开云体育

On Tue, Dec 24, 2024 at 03:40 AM, buddy wrote:

"this is a bug (full stop)"

Of course, what's also interesting, is this could actually be a bug w/ 10.19 instead. That is, 10.19 might be providing bad data in a timely manner instead of timing out.

Unfortunately, it's difficult to be certain without digging much much deeper into the IBKR infra and the TWS/IBGW code.

• when the data never arrives in 10.30+ or
• when it does arrive but not within the time @ajn is willing to wait for. Even if data always arrives within that time with IBGW 10.19.

The first case smells like a bug (as long as data should be available), while the second case is simply a change in systemic behavior. Even if that behavior change, in the eyes of the user, is a degradation and causes previously working timeouts to now fail. That is the crux with timeouts - they are always too long until they are suddenly not long enough.

During my due diligence before switching from V9 TWS/IBGW to the first stable V10 (probably around TWS 10.10) I found that timing for certain operations had changed. Some for the better and some for the worse. So it is not unreasonable to assume that something similar happened between 10.19 and 10.30+.

To quickly check that hypothesis I went back to my records and compared all reqMkt() subscriptions for SPY ETF in December 2023 (when I used stable TWS/IBGW 10.19) with those from MTD December 2024 (with stable TWS/IBGW 10.30). And guess what:

• The average time from reqMktData() to the first arrival of a data tick was 0.3 seconds (300ms) with TWS 10.19 last year?
• but it is now 3.2 seconds (3,200ms) with TWS 10.30 this month

Small things other than the TWS version have changed on my side since December 2023, but I cannot imagine that any of those changes could cause the longer time to first tick. But I will run tests with 10.19 to confirm when the markets are closed later in the week. Attached a quick xlsx with the raw results.

I still would not call this a bug (though this change of behavior may have not been intended by the developers)

闯ü谤驳别苍

On Tue, Dec 24, 2024 at 06:53 AM, 闯ü谤驳别苍 Reinold wrote:

I still would not call this a bug

Well, for practical purposes, whether it's a "bug" or not is a moot point. But, just as you wouldn't call it a bug, I wouldn't necessarily call it a great improvement either ;-)

That said, I understand your viewpoint, and would simply remind future readers that it's wise to distinguish between so-called "hard" vs "soft" real-time requirements when building these types of programs.

On Mon, Dec 23, 2024 at 10:31 PM, 闯ü谤驳别苍 Reinold wrote:

Just curious whether the data eventually arrives if you give it enough time.

Twenty seconds seems to be a pretty long time (see attached).

How long do you think we should wait before crossing over from thinking this is a "feature" to "bug"?

Ah... good old Sorites paradox, never fails to amuse :-)

I still believe there is no bug or anything wrong with reqMktData real time feeds. And snapshots are probably what you are looking for in the first place.

TWS API documentation says about reqMktData snapshots that: ... it is possible to request a snapshot of the current state of the market once instead of requesting a stream of updates continuously as market values change.

What you experience is likely that, at times, not all of the market values you are interested in change during the 20 seconds you are looking at the market. You could easily verify that yourselves by downloading TickByTick historical data for the time periods that fail your test.

I took a quick look at the "missing" BID price for MTDR on 20241217:

• You request real-time data for MTDR@NYSE only, instead of MTDR@SMART. On that day, NYSE handled only 22% of all trades. They handled 51% of the total volume, but I bet most of it was related to the MOO and MOC auctions.
• And then there were only 563 trades during the entire trading session (RTH and outside of RTH) on NYSE. That means there were definitely long periods of time that day (way longer than 20 seconds) where the BID price (or other "market values") just did not change and, therefore, no reports would have taken place in the real-time stream. But that would be no bug, just "nothing to report".
• Requesting data for SMART instead of only for NYSE may increase your chance of seeing "market values change" while you look. But even then, MTDR had only 2,563 trades (compared with 177,164 for APPL) and long quiet periods would have to be expected.

I don't have enough data about your 10.19 vs 10.30+ comparisons to explain the differences your are seeing. It may very well be that IBGW/TWS 10.19, upon subscription, send some "recent historical values" and 10.30+ are now cleaner and send "only market values that changed". But that is just a guess and there may be other explanations.

闯ü谤驳别苍

?

?

On Thu, Dec 26, 2024 at 05:36 AM, 闯ü谤驳别苍 Reinold wrote:

It may very well be that IBGW/TWS 10.19, upon subscription, send some "recent historical values" and 10.30+ are now cleaner and send "only market values that changed".

The above is an interesting thought. But, I feel obligated to note it would be a breaking change to the interface. So basically still a pretty big "bug/problem"... just not in the implementation.

And even if it's "cleaner" over the wire, changing an interface without also changing its signature is terrible practice because it silently breaks end user applications (as @ajn may have found out).

With all due respect 闯ü谤驳别苍, dismissing this or trying to spin it as a good thing(tm) strikes me as extremely forgiving or just naive.

Finally, if this speculation is correct, it makes me think that extra caution should be exercised prior to using 10.30+ in production. Since, I wouldn't want to have any code running the relied on defaults being set to "recent historical values" and didn't immediately fail loudly when such an assumption quietly disappeared; which seems like a possibility.

On Thu, Dec 26, 2024 at 05:36 AM, 闯ü谤驳别苍 Reinold wrote:

Requesting data for SMART instead

I thought this was an interesting observation as well. So, I changed the input list to just 800 of the most liquid stocks (see attached) and specified SMART as the exchange instead. Additionally, I changed the timeout to 45 seconds. Failure occurred in the same manner.

Of course, your theory that the semantics w.r.t. reqMktData have mysteriously changed still holds water. And, even though the professional way to distribute such a change would have been to implement reqMktData2 and optionally deprecate reqMktData (so that there were no unpleasant surprises), at least there appears to be some form of sour logic to what's going on.

However, without confirmation from IBKR there's no way to know this with absolute certainty, and that's not great. Regardless... it seems the best advice, at the present time, might be as previously suggested, to first initialize things with a snapshot.

If IBKR takes responsibility for any of this I'd absolutely love to know about it. Keep us posted @ajn.