What would be the best way to do on-the-fly computations using Numpy, Pandas for trading using IB TWS API, writing to/from mySQL, and in the meanwhile receive and send data and orders?
From (and ) I conclude that CPU-bound programs, those that spend most of the their time processing data (like on the-fly computations using Pandas which is synchronous single threaded), solutions like threading (pre-emptive multitasking) and asyncio are inferior (and may disrupt the sending/receipt of data or introduce some lag or data drops) since there is only one Python interpreter on a single core/processor (asyncio aligns I/O processes more efficiently) that takes care of both the CPU computations and checking for socket data.
I know there is? a ib_insync library that implements cooperative multitasking (asyncio) in Python, but will the overall program be fast enough if you do computations using large dataframes? I read that on a socket level there might not be any limitations (¡°socket itself is actually thread-safe¡±), as suggested by . I do not assume that Ptyhon has it implemented this way since the Global Interpreter Lock is in place (single thread, synchronous). From :
IB.sleep(0); may be an alternative, but I do not think a Pandas groupby() can be interrupted once called.
Should I use the Multiprocessing library and do every task on its own processor (thus dataframe calculations, mySQL connection read/write, sending and receiving orders and data)?
Or should I combine some things using (I think some next level threading which can be combined with the normal way of interpretation of code by Python) or the (has to do with built-in async for Scikit-Learn, Pandas, etc. functions, if I understand correctly) library? I think this is next level, and may be an overkill.
Or should stick to Java (I do not know if there is a Pandas like alternative which allows easy manipulation of data and performing machine learning for example, but threading may be organized better)?
Any suggestions or tutorials?
(I read something about was well, but that¡¯s a whole different story since it is not directly applicable to IB TWS API, and may be prone to delays resulting from single core processing/interpreting by Python as well, if I understand correctly)
