Hello all,

Is it possible to run multiple strategies at once, provided that each instance of python has a different client id and requests the next valid order ID before each order?

Thanks for the help!

The short answer is yes. Up to 32 of them as described in the TWS API Reference Guide section The different clients can trade in the same account or, in case you have? FA or FF accounts, in the various linked accounts.

But is that really what you want to do? You'll have to implement a lot of communication amongst the various independently running strategies so that they do not step on each others toes or create undesirable situations such as (but not limited to):

• One strategy sells position that the other just bought
• Order placements fail for one or more strategies since the account's buying power or margin cushion have been exhausted by the other strategies
• Depending on the account type, uncoordinated strategies can inadvertently cause problems such as triggering the "pattern day trading" rule
• If several strategies act upon the same instrument, duplicate market data feeds have to be sent to multiple clients

It may be simpler to implement and better to control if you have a single client that monitors and executes several strategies with a shared account management and trading module. But this has really nothing to do with the TWS API since these are application level architecture and design decisions.

闯ü谤驳别苍

On Mon, Mar 27, 2023 at 05:59 PM, 闯ü谤驳别苍 Reinold wrote:

The short answer is yes. Up to 32 of them

Theoretically this is true but beware there's a somewhat gross since 10.x that probably hasn't been fixed yet. It could cause problems.

开云体育

Lol, thanks Richard. IIRC trying to sidestep the problem by canceling the mkt data prior to exit actually didn't work (see here).

It doesn't matter to me though. It's a known problem. The new 10.x version of software simply does not honor the contract as documented. Unless of course they fixed the bug... but I haven't checked in a long time.

I know I'm being pedantic but sometimes being pedantic is useful. I'll let others judge for themselves.

-Very best

开云体育

On Tue, Mar 28, 2023 at 10:33 PM, Richard L King wrote:

can you give me a detailed description of the steps you go through to demonstrate that this workaround doesn’t work, so I can verify or refute your claim

Yes, I love your spirit! The exact instructions are located . Read and follow them precisely. As I understand your workaround, it is to add the line this.cancelMktData(os.getpid()) right before the line self.disconnect()... that doesn't help. IOW:

--- example.py.~1~ 2023-03-28 19:04:06.322531726 -0400
+++ example.py     2023-03-28 19:07:29.589006097 -0400
@@ -31,6 +31,7 @@
             attrib: TickAttrib,
     ):
         print( "Got a price tick, exiting." )
+        this.cancelMktData(os.getpid()) 
         self.disconnect()
     def error(self, reqId: TickerId, errorCode: int,
               errorString: str, advancedOrderRejectJson = ""):

I've tried this again today with the latest IBGW and the bug still exists and this proposed workaround does not help.

I know you're a Windows person Richard and you're free to use Windows if you want, although I'd prefer you didn't in order to make sure test environments match as closely as possible. However, I have reproduced the problem in a Windows VM as well, but some time ago. You can install python3 for Windows from .

Just please remember, follow the gist exactly. IIRC last time you tried to summarize and port the example to VB or C# or something and I'm guessing in that process may have messed something up. Please be as thoughtful and deliberate as possible. Patches are appreciated.

Make sure to read the entire gist very thoroughly, it is a subtle bug and very easy to misunderstand or see it work once and then, prematurely conclude that you've found a workaround or it's a heisenbug. If you perform the test properly you will see it is clearly a bohrbug.

If the 10.x version of TWS/IBGW fully supported multiple clients (32 no less) you should be able to run the example program over and over and over and over using the pid as the clientID. It's not expected to run the example simultaneously... just sequentially. There doesn't ever need to be more than one client connected. The IB support rep who was assigned to work with me on this has acknowledged their developers know it's a problem but advised I don't hold my breath for a fix.

Lol, I don't actually care that much... my software is printing plenty of money without the functionality. I just thought it was interesting because I stumbled over it while using my tool-set in an ad-hoc fashion (not trading).

It is also important that you test with real-time data using a live account during RTH. The bug can be reproduced intermittently w/ a test account, delayed/frozen data, or outside RTH... but that should not be the final test. The problem only happens with absolute certainty in a "production" environment.

The example code is very short and apparently correct (according to documentation); it was actually written by IB support. And, again, the program should be able to run over and over and over, never hanging on ticks... certainly for something as liquid as VZ.

Because of this, at the end of the day, I agree with Jurgen 100%:

On Mon, Mar 27, 2023 at 05:59 PM, 闯ü谤驳别苍 Reinold wrote:

have a single client that monitors and executes

It's the only real "workaround"; which is to say there's actually no way to workaround. If someone wants to "run multiple algorithms in parallel" my suggestion would be they look at using ; it's close enough.

Good luck and enjoy!

On Wed, Mar 29, 2023 at 02:03 AM, buddy wrote:

   this.cancelMktData(os.getpid()) 
   self.disconnect()

To rule out any socket-closing-vagaries, I would suggest placing time.sleep(1) between these two lines to give the cancel request some time to be really sent over to the other side.

-- Ewald

On Wed, Mar 29, 2023 at 05:33 PM, Ewald de Wit wrote:

I would suggest placing time.sleep(1) between these two lines

I just tried this and it does seem to mitigate the problem a bit. I cannot say it's a proper fix(tm) by any means. But, the inevitable hanging is prolonged. I guess that's worth something? Anyway, this makes me think the bug they introduced with 10.x has a race condition nature. They're certainly tough ones to track and fix.

Thanks for all the consideration and suggestions.

开云体育

On Fri, Mar 31, 2023 at 06:26 PM, Richard L King wrote:

took the trouble to understand what I was doing

I do understand. But I'm not going to argue about this with you any more. Choose to believe what you want.

On Fri, Mar 31, 2023 at 06:26 PM, Richard L King wrote:

With that correction, it seems to work fine (though you seem reluctant to admit it – so can you provide any examples of where it doesn’t help?).

It does help, yes... but it does not solve the problem. It's pretty clearly a race condition and adding a delay ends up only masking the problem, not solving it. If you run the example enough times it will still hang eventually. You should try it and see for yourself to see my point. You can add a longer and longer delay and then the problem may only happen 1 in a 1000 times but nonetheless it is not a fix or even a good workaround IMHO.

This is the nature of race conditions... you can mask them if you want. The challenge is to reliably reproduce them so they can be fixed properly instead; at least from a development perspective.

I think people should be aware of this. If someone is happy with a periodic failure that's up to them.

I hope your respiratory situation is improving rapidly, Richard, and that the nasty COVID was not involved.

We are actually in complete agreement on market data for multiple clients. In fact we run multiple clients that hold data subscriptions simultaneously all the time (same or different instruments) and we have never seen an issue with that. My "single client" comment was intended to relate to order and account management exclusively and not to market data sharing by independent clients:

It may be simpler to implement and better to control if you have a single client that monitors and executes several strategies with a shared account management and trading module.

Rereading my reply now makes me question what exactly I was thinking about when I added the fourth bullet on market data feeds. The first three bullets adequately address the kinds of issues multiple parallel and independent trading strategies better communicate about but data feeds to multiple clients is not an issue.

The post took a bit of a "left turn" and became an entirely different post when Buddy pointed to a bug he thinks he discovered in V10 TWS/IBGW versions. I don't dispute that some kind of race condition may exist, but we have not experienced that issue ourselves. And from the description, the issue would not be an impediment to running multiple trading strategies for the same instrument in independent Python clients.

And I can believe your finding that the issue can be avoided entirely by simply cancelling the market data subscription before the client exits.

闯ü谤驳别苍

..

I completely disagree with you here (and with 闯ü谤驳别苍 for that matter, and I’ll get back to him on that when I can find the time). As I’ve pointed out before, I run multiple clients sharing the same or overlapping data streams all day every day, and have been doing so since 2003, and I simply have no problems.

?

As an illustration, attached is a screenshot of three DataCollector programs (two of them write to different SQL Server database, and the third writes to text files (all are connected to the live TWS). As far as concurrent programs that are actually trading, as opposed to consuming data, are concerned, there are things that need consideration, but sharing data isn’t one of them.

?

Richard

开云体育

On Tue, Apr 4, 2023 at 05:38 PM, Richard L King wrote:

without first cancelling the market data stream

Richard, I hate to harp on this but I think your understanding of the bug is coming along and so I would like to point out, again, that cancelling the market data stream does not prevent the eventual deadlock.

In fact it is very easy to refute that it does... just apply the proposed fix to the example and then run the example in a loop.

In other words, apply this:

--- example.py.~1~      2023-04-05 12:26:11.886098467 -0400
+++ example.py  2023-04-05 12:40:22.023094581 -0400
@@ -1,4 +1,4 @@
-import os
+import os, time
 from ibapi.client import *
 from ibapi.wrapper import *
 
@@ -31,6 +31,8 @@
             attrib: TickAttrib,
     ):
         print( "Got a price tick, exiting." )
+        self.cancelMktData(os.getpid())
+        time.sleep(1)
         self.disconnect()
     def error(self, reqId: TickerId, errorCode: int,
               errorString: str, advancedOrderRejectJson = ""):

to the example and then run in a forever loop, e.g:

#!/usr/bin/env bash

ii=1
while [[ $ii -gt 0 ]]; do
    echo -n ">>>>>>>>>>  $ii "; date
    ((ii+=1))
    python3 ./example.py
done

The deadlock may not occur immediately but will happen in somewhat short order. When I tried this the loop would hang by the 12th iteration with a very high likelihood; usually earlier. This behavior is certainly indicative of a race condition; be it in a destructor or reference counter as you suggest.

This is why I was encouraging you to forego "issuing API commands discretely" and test the example programmatically and thoroughly. Race conditions often disappear in a debugger or when you "step" through code because the code timing is affected. Additionally, in a real-world application, attempting to "fix" the race condition by introducing an artificial delay wouldn't help since the program would be running in parallel at random overlapping times and therefore any artificial delay would be canceled out by the start times.

Anyway, I'm glad you're feeling better.

On Sat, Apr 1, 2023 at 10:14 PM, 闯ü谤驳别苍 Reinold wrote:

The post took a bit of a "left turn" and became an entirely different post

闯ü谤驳别苍, I'll be the first to admit that the bug I've been droning on about is a corner case. However... it is a very good example of why serious consideration should be given to the decision of implementing a multi-threading/processing design. Multi-threading/processing can sometimes be difficult to implement and often creates very tricky problems that even experienced folks have problems troubleshooting and understanding. And, even if the application design is solid, sometimes the cause is poor vendor support which can be out of the developers control.

开云体育

On Wed, Apr 5, 2023 at 10:56 PM, Richard L King wrote:

By the way, it’s not necessary to treat me as if I’m a junior programmer that knows nothing about debugging, race conditions, etc. Look up the expression ‘teach your grandmother to suck eggs’ – I’m your grandmother!…

I'm just trying to help, and get help, working through a tricky problem. Sorry to have made you feel slighted.