开云体育

What would be the best way to do on-the-fly computations using Numpy, Pandas for trading using IB TWS API, writing to/from mySQL, and in the meanwhile receive and send data and orders?

From (and ) I conclude that CPU-bound programs, those that spend most of the their time processing data (like on the-fly computations using Pandas which is synchronous single threaded), solutions like threading (pre-emptive multitasking) and asyncio are inferior (and may disrupt the sending/receipt of data or introduce some lag or data drops) since there is only one Python interpreter on a single core/processor (asyncio aligns I/O processes more efficiently) that takes care of both the CPU computations and checking for socket data.

I know there is? a ib_insync library that implements cooperative multitasking (asyncio) in Python, but will the overall program be fast enough if you do computations using large dataframes? I read that on a socket level there might not be any limitations (“socket itself is actually thread-safe”), as suggested by . I do not assume that Ptyhon has it implemented this way since the Global Interpreter Lock is in place (single thread, synchronous). From :

IB.sleep(0); may be an alternative, but I do not think a Pandas groupby() can be interrupted once called.

Should I use the Multiprocessing library and do every task on its own processor (thus dataframe calculations, mySQL connection read/write, sending and receiving orders and data)?
Or should I combine some things using (I think some next level threading which can be combined with the normal way of interpretation of code by Python) or the (has to do with built-in async for Scikit-Learn, Pandas, etc. functions, if I understand correctly) library? I think this is next level, and may be an overkill.

Or should stick to Java (I do not know if there is a Pandas like alternative which allows easy manipulation of data and performing machine learning for example, but threading may be organized better)?

Any suggestions or tutorials?

(I read something about was well, but that’s a whole different story since it is not directly applicable to IB TWS API, and may be prone to delays resulting from single core processing/interpreting by Python as well, if I understand correctly)

I love Python but it has its own place and it's not good for real-time-ish parallel compute. You can hack around things but in the end it won't be crazy.

You can consider Go and node.js from mainstream languages (apart from C++).

I personally always wanted to try Elixir but never got around - perfect for highly concurrent stuff.?

On Sat, Mar 13, 2021 at 2:52 PM matt <rmdeboer82@...> wrote:

What would be the best way to do on-the-fly computations using Numpy, Pandas for trading using IB TWS API, writing to/from mySQL, and in the meanwhile receive and send data and orders?

From (and ) I conclude that CPU-bound programs, those that spend most of the their time processing data (like on the-fly computations using Pandas which is synchronous single threaded), solutions like threading (pre-emptive multitasking) and asyncio are inferior (and may disrupt the sending/receipt of data or introduce some lag or data drops) since there is only one Python interpreter on a single core/processor (asyncio aligns I/O processes more efficiently) that takes care of both the CPU computations and checking for socket data.

I know there is? a ib_insync library that implements cooperative multitasking (asyncio) in Python, but will the overall program be fast enough if you do computations using large dataframes? I read that on a socket level there might not be any limitations (“socket itself is actually thread-safe”), as suggested by . I do not assume that Ptyhon has it implemented this way since the Global Interpreter Lock is in place (single thread, synchronous). From :

IB.sleep(0); may be an alternative, but I do not think a Pandas groupby() can be interrupted once called.

Should I use the Multiprocessing library and do every task on its own processor (thus dataframe calculations, mySQL connection read/write, sending and receiving orders and data)?
Or should I combine some things using (I think some next level threading which can be combined with the normal way of interpretation of code by Python) or the (has to do with built-in async for Scikit-Learn, Pandas, etc. functions, if I understand correctly) library? I think this is next level, and may be an overkill.

Or should stick to Java (I do not know if there is a Pandas like alternative which allows easy manipulation of data and performing machine learning for example, but threading may be organized better)?

Any suggestions or tutorials?

(I read something about was well, but that’s a whole different story since it is not directly applicable to IB TWS API, and may be prone to delays resulting from single core processing/interpreting by Python as well, if I understand correctly)