Keyboard Shortcuts
Likes
Search
Main app in new process vs new thread (Python)
All Python examples on the Internet show to run main app (class extending?EClient and?EWrapper) in new thread, something like this:
app = TestApp()
app.connect("127.0.0.1", 7497, 1)
time.sleep(1)
api_thread = threading.Thread(target=app.run)
api_thread.start()
?But I wonder, if can run it in new process. Here is what I tried: from ibapi.client import EClient
from ibapi.wrapper import EWrapper
from ibapi.contract import Contract
from ibapi.ticktype import TickTypeEnum
import time
import datetime
import threading
import psutil
from multiprocessing import Process
?
?
class TestApp(EWrapper, EClient):
? ? def __init__(self):
? ? ? ? EClient.__init__(self, self)
? ? ? ? self.nextValidOrderId = None
? ? ? ? self.price_list = []
? ? ? ??
? ? def error(self, reqId, errorCode, errorString, advancedOrderRejectJson):
? ? ? ? print("Error: ", reqId, " ", errorCode, " ", errorString, "", advancedOrderRejectJson)
? ??
? ? def tickPrice(self, reqId, tickType, price, attrib):
? ? ? ? super().tickPrice(reqId, tickType, price, attrib)
? ? ? ? self.price_list.append(price)
? ? ? ?
contract = Contract()
contract.symbol = "SPY"
contract.secType = "STK"
contract.exchange = "SMART"
contract.currency = "USD"
contract.primaryExchange = "ARCA"
app = TestApp()
app.connect("127.0.0.1", 7497, 1)
time.sleep(1)
?
p = Process(target = app.run)
p.start()
p.join()
time.sleep(2)
?
app.reqMktData(101, contract , '', False, False, [])
?
time.sleep(2)
app.disconnect()
Unfortunately, it just hangs and does not terminate. I wonder what might be the reason? Does it matter that I run it in Jupyter Notebook? ? |
What is it that you are trying to achieve? Why is the "standard" way of using TWS API insufficient? TWS API runs a reader thread that receives all communication data packets from TWS/IBGW, decodes them, and calls the appropriate callback function to alert you about the response. That works well when all threads run within the same process and have access to the same memory and global variables. But that will not work when the TWS API reader runs in a separate process. The nature of processes is that they cannot simply access each other's memory or directly call some method in a separate process. So you would need to implement your own API for your "client process" to communicate with your "TWS API process". That will be a lot of work even if you start with a library or module that provides a basic infrastructure. And the communication with TWS/IBGW will be slower since requests have to get from your client process first to your API process before they are sent to TWS/IBG and the responses will have to take the same way back. ´³¨¹°ù²µ±ð²Ô PS. Before I forget.? Do yourselves a favor and lose all the? time.sleep() calls. TWS API is an asynchronous protocol where it is undetermined how long it will take for responses to your requests to arrive. If your client runs out of things to do between the request and the arrival of the response(s), the thread has to wait but never sleep. The only acceptable use of sleep in relation with TWS API is when you need to? pace requests (for example to observe the 50 requests/second limit for older TWS/IBGW or the historical data download maximum requests rates). But in these cases, you delay the next request (the sleep is before the request) instead of sleeping after the request in the hope that some response came back while your thread slept. Every language has some form of support for these asynchronous communications and for Python you may want to look at asyncio or take a peek at ib_insync for an excellent implementation. And yes, even IBKR samples and those best selling "Get rich? fast with Python" books are just full of them. It's still wrong, dangerous, and borderline malpractice. On Thu, Jan 5, 2023 at 11:21 AM, GreenGreen wrote:All Python examples on the Internet show to run main app (class extending?EClient and?EWrapper) in new thread, something like this: |
Thank you, ´³¨¹°ù²µ±ð²Ô!
In python most TWS API example show how to create new thread and run TWS API reader in there. As far as I know in Python threads cannot run in parallel due to GIL (general interpreter lock).? I would like to have two separate processes (running in parallel) 1) One is for TWS API reader 2) One for processing data (I want to make sure that processing data does not hold back data stream) Thank you again for clarification. It looks like I will have to use shared variables to communicate between processes. Even after reading buddy response, I am still puzzled why my code did not work. Child process (TWS API reader) was supposed to terminate after parent process finished.? I simply copied time.sleep() from examples that I have seen. Dropping it now... |
You can use a class for each process:
And in main: Que'ing in and out of a class/process makes it easy to add more processes when you need more expediant data processing. You can push a fairly large np array onto a que if you have alot of data to pass around. |
On Thu, Jan 5, 2023 at 08:17 PM, GreenGreen wrote:
If you read the man page for fork carefully you'll see it says... The child process and the parent process run in separate memory spaces. And, ´³¨¹°ù²µ±ð²Ô kindly pointed out that the API works well when all threads run within the same process and have access to the same memory and global variables. So... when you call:
and then, subsequently:
You just might not be calling the functions in the process you think you are. In other words, you need to make sure you're referring to the same "app", because now there are two (one in the child and one in the parent). So, the problem just might be that you're having the parent operate on things that are actually inside the child (or vice versa) and without doing some special IPC like shared memory etc this won't work. And even then, be careful because the API may not be designed to operate that way necessarily. If you want the parent to "control" the lives of the children (count them, spawn a few, etc) that's one thing. But you can't just assume they share everything off the bat.. they share very little. A fork basically just starts a copy of the parent process in separate address space. Think of it like running the same program twice from the command-line. If you did that you'd need a different client ID for each and two different tcp/ip sockets and probably call reqMktData on different contracts, right? This is why I suggested you get more familiar with what fork does, since under the hood that's likely what a new Python Anyway, at the end of the day when ´³¨¹°ù²µ±ð²Ô asks What is it that you are trying to achieve? he's being practical. However, when I suggested you read the man page for fork it was for you to learn something in an academic sense. For the life of me IDK why you'd actually want to complicate things in the way you're trying to, lol. There's nothing wrong with using multiprocessing, I actually prefer it. But it's always easier to first use the facilities built into the OS before rewriting things in Python... you know, top, kill, pgrep, Unix pipes, the shell's ampersand operator etc. Then, if you're doing the same thing over and over whip something up in Python. Just my 2cents. Good luck! |
Thank you, buddy!
So this piece (TWS API reader) runs in child process and does effectively nothing since it does not see request for data p = Process(target = app.run)
p.start()
p.join() while this piece runs in parent process? app.reqMktData(101, contract , '', False, False, []) ?
time.sleep(2)
app.disconnect()
I expect parent process to terminate without any problem and this should kill child process as well, but it does not happen. It just hangs.? Good point about OS resources, but my knowledge of OS are so limited that it looks like Python path is a bit easier. Thank you again for clarifications!? |
On Thu, Jan 5, 2023 at 06:59 PM, ´³¨¹°ù²µ±ð²Ô Reinold wrote:
This is the best advice IMHO. Specifically, I'd take a close look at this code:
It's the and basically looks like what you want. Otherwise, unless you know exactly what to look for, sifting through random examples on the internet can be a wild goose chase. |
On Thu, Jan 5, 2023 at 11:30 PM, GreenGreen wrote:
Yeah, well... give it time and that stuff will become obvious. I was just taking an educated guess but if you're really curious you can actually read through the Python code for multiprocessing . Look inside process.py and subsequently context.py to get the final word on exactly what's happening "under the hood". Have fun :-) |