开云体育

On Thu, Dec 26, 2024 at 05:36 AM, 闯ü谤驳别苍 Reinold wrote:

Requesting data for SMART instead

I thought this was an interesting observation as well. So, I changed the input list to just 800 of the most liquid stocks (see attached) and specified SMART as the exchange instead. Additionally, I changed the timeout to 45 seconds. Failure occurred in the same manner.

Of course, your theory that the semantics w.r.t. reqMktData have mysteriously changed still holds water. And, even though the professional way to distribute such a change would have been to implement reqMktData2 and optionally deprecate reqMktData (so that there were no unpleasant surprises), at least there appears to be some form of sour logic to what's going on.

However, without confirmation from IBKR there's no way to know this with absolute certainty, and that's not great. Regardless... it seems the best advice, at the present time, might be as previously suggested, to first initialize things with a snapshot.

If IBKR takes responsibility for any of this I'd absolutely love to know about it. Keep us posted @ajn.

On Thu, Dec 26, 2024 at 05:36 AM, 闯ü谤驳别苍 Reinold wrote:

It may very well be that IBGW/TWS 10.19, upon subscription, send some "recent historical values" and 10.30+ are now cleaner and send "only market values that changed".

The above is an interesting thought. But, I feel obligated to note it would be a breaking change to the interface. So basically still a pretty big "bug/problem"... just not in the implementation.

And even if it's "cleaner" over the wire, changing an interface without also changing its signature is terrible practice because it silently breaks end user applications (as @ajn may have found out).

With all due respect 闯ü谤驳别苍, dismissing this or trying to spin it as a good thing(tm) strikes me as extremely forgiving or just naive.

Finally, if this speculation is correct, it makes me think that extra caution should be exercised prior to using 10.30+ in production. Since, I wouldn't want to have any code running the relied on defaults being set to "recent historical values" and didn't immediately fail loudly when such an assumption quietly disappeared; which seems like a possibility.

I still believe there is no bug or anything wrong with reqMktData real time feeds. And snapshots are probably what you are looking for in the first place.

TWS API documentation says about reqMktData snapshots that: ... it is possible to request a snapshot of the current state of the market once instead of requesting a stream of updates continuously as market values change.

What you experience is likely that, at times, not all of the market values you are interested in change during the 20 seconds you are looking at the market. You could easily verify that yourselves by downloading TickByTick historical data for the time periods that fail your test.

I took a quick look at the "missing" BID price for MTDR on 20241217:

• You request real-time data for MTDR@NYSE only, instead of MTDR@SMART. On that day, NYSE handled only 22% of all trades. They handled 51% of the total volume, but I bet most of it was related to the MOO and MOC auctions.
• And then there were only 563 trades during the entire trading session (RTH and outside of RTH) on NYSE. That means there were definitely long periods of time that day (way longer than 20 seconds) where the BID price (or other "market values") just did not change and, therefore, no reports would have taken place in the real-time stream. But that would be no bug, just "nothing to report".
• Requesting data for SMART instead of only for NYSE may increase your chance of seeing "market values change" while you look. But even then, MTDR had only 2,563 trades (compared with 177,164 for APPL) and long quiet periods would have to be expected.

I don't have enough data about your 10.19 vs 10.30+ comparisons to explain the differences your are seeing. It may very well be that IBGW/TWS 10.19, upon subscription, send some "recent historical values" and 10.30+ are now cleaner and send "only market values that changed". But that is just a guess and there may be other explanations.

闯ü谤驳别苍

?

?

On Mon, Dec 23, 2024 at 10:31 PM, 闯ü谤驳别苍 Reinold wrote:

Just curious whether the data eventually arrives if you give it enough time.

Twenty seconds seems to be a pretty long time (see attached).

How long do you think we should wait before crossing over from thinking this is a "feature" to "bug"?

Ah... good old Sorites paradox, never fails to amuse :-)

On Tue, Dec 24, 2024 at 06:53 AM, 闯ü谤驳别苍 Reinold wrote:

I still would not call this a bug

Well, for practical purposes, whether it's a "bug" or not is a moot point. But, just as you wouldn't call it a bug, I wouldn't necessarily call it a great improvement either ;-)

That said, I understand your viewpoint, and would simply remind future readers that it's wise to distinguish between so-called "hard" vs "soft" real-time requirements when building these types of programs.

• when the data never arrives in 10.30+ or
• when it does arrive but not within the time @ajn is willing to wait for. Even if data always arrives within that time with IBGW 10.19.

The first case smells like a bug (as long as data should be available), while the second case is simply a change in systemic behavior. Even if that behavior change, in the eyes of the user, is a degradation and causes previously working timeouts to now fail. That is the crux with timeouts - they are always too long until they are suddenly not long enough.

During my due diligence before switching from V9 TWS/IBGW to the first stable V10 (probably around TWS 10.10) I found that timing for certain operations had changed. Some for the better and some for the worse. So it is not unreasonable to assume that something similar happened between 10.19 and 10.30+.

To quickly check that hypothesis I went back to my records and compared all reqMkt() subscriptions for SPY ETF in December 2023 (when I used stable TWS/IBGW 10.19) with those from MTD December 2024 (with stable TWS/IBGW 10.30). And guess what:

• The average time from reqMktData() to the first arrival of a data tick was 0.3 seconds (300ms) with TWS 10.19 last year?
• but it is now 3.2 seconds (3,200ms) with TWS 10.30 this month

Small things other than the TWS version have changed on my side since December 2023, but I cannot imagine that any of those changes could cause the longer time to first tick. But I will run tests with 10.19 to confirm when the markets are closed later in the week. Attached a quick xlsx with the raw results.

I still would not call this a bug (though this change of behavior may have not been intended by the developers)

闯ü谤驳别苍

On Tue, Dec 24, 2024 at 03:40 AM, buddy wrote:

"this is a bug (full stop)"

Of course, what's also interesting, is this could actually be a bug w/ 10.19 instead. That is, 10.19 might be providing bad data in a timely manner instead of timing out.

Unfortunately, it's difficult to be certain without digging much much deeper into the IBKR infra and the TWS/IBGW code.

On Mon, Dec 23, 2024 at 10:31 PM, 闯ü谤驳别苍 Reinold wrote:

Just curious whether the data eventually arrives if you give it enough time.

Even if it did, that wouldn't explain why it arrives in time when using 10.19. If you run the test case as prescribed, and compare the results of 10.19 vs 10.30+, you'll probably see what I mean.

I suppose it could be classified as "not a bug" if IBKR wants to intentionally degrade the service they once provided. And, as a skeptic, I wouldn't put that past them. Nevertheless, from a technical user's perspective, this is at best a [massive] shortcoming IMHO.

After all, without any SLA they could just hold data back completely, or provide the same QOS for DELAYED and REALTIME. It's an absurd proposition so I'm leaning toward "this is a bug (full stop)".

Playing that much faster and looser w/ their specifications would leave a bad taste in an engineer's mouth... although a lawyer might love it.

:-/

Rarely is lack of "last hop" bandwidth the root cause for network issues. In my experience it is pretty much always latency. That is latency from processing, end-to-end "flight time", congestion along the route, traffic shaping games played by the ISP, or packet loss and retransmission. But since you can reproduce what you are seeing with a t3.2xlarge AWS instance, I am pretty sure that network issues are not the root cause of what you are seeing. Having said that, latency (or changes of latency) could very well make your experience better or worse.

I am still not convinced that there is a "bug" here and I was wondering, whether you have tried (or could try) some runs without timeouts. At least the Python code currently gives up and cancels the subscription after 20 seconds, right? Maybe it just takes that long occasionally when you send massive numbers of subscriptions and cancellations.

• There is no interface contract or service level agreement that specifies how long it will take for the first market data ticks to arrive after a successful subscription. It may just be that it sometimes takes a while. Especially since in many cases a set of historical tick (last close, open, ...) is sent before the real-time data starts.
• The API documentation for market data snapshot says, that the tickSnapshotEnd() callback "... will indicate the snapshot reception is finished. Expected to occur 11 seconds after beginning of request." It does not guarantee that any data has been sent before the end of the 11 second period.
• I checked my records for the SPY ETF how long it takes from the market data subscription to the first data arriving. This comparison is not perfect, since my client subscribes between 16:00 and 17:00 US/Central, keeps the subscriptions active for 24 hours, and only makes few subscriptions compared to your 1,000++ runs. For about 2,500 subscriptions over the last few years, max time for the arrival of the first data tick was 6.3s with an average of 510ms and stdev of 794ms. That does not include quite a few cases where the first tick did not arrive for more than a minute and was a "LastTimeStamp" callback and not a data tick.

Just curious whether the data eventually arrives if you give it enough time.

闯ü谤驳别苍

PS. Just as an FYI attached recent latency and packet loss measurements for the IBKR entry points as experienced by a server outside of Chicago.

On Thu, Dec 19, 2024 at 10:23 PM, ajn wrote:

It is often happening on the second loop or later.

Ok, I managed to reproduce the problem.

On Thu, Dec 19, 2024 at 10:23 PM, ajn wrote:

However I did notice 2 things

I just ran your sample program again, after hours w/ realtime, data_lines=75, loops=5 and everything worked OK using IBGW 10.30.1s. IDK what to say... I may try it again tomorrow since you indicate there's more likelihood of a problem during RTH.

However, I'm also wondering if your network connection may have slightly degraded in the recent past? Is someone streaming HDTV while you test, lol? Are you sure you can't reproduce the problem w/ 10.19? I suppose it's possible that 10.30 has just enough extra network demand than 10.19 to make a difference but...

I mention this mostly because my server has an exceptionally good connection (and I'm located close to the USA NE data center). When I look at your email headers I notice you're in Sweden, so if you're using a local computer... you might have more variation in connection quality. Just a guess since the code itself looks OK and the test results look OK.

1. I have run the test after hours and the frequency of errors is significantly less. I get 1 failure per 5 loops. whereas during trading hours I was getting maybe 5-20 per same test. So maybe it is related somehow on how IB infra is loaded? i.e. the infra is probably less loaded now (just a guess)
2. I can repeat the issue for REALTIME and DELAYED options on paper trading account which HAS market data subscriptions. However, at least now, after hours trying it on the account which DOESN'T have subscription doesn't generate issues with DELAYED. Maybe it has to do with after hours or maybe DELAYED means different things for account with subscription and without. I don't know.? So I can confirm the issue for account with subscription, yet after hours it is less rare one really need to run 5 loops to be sure.?

On Wed, Dec 18, 2024 at 12:08 AM, ajn wrote:

Actually, running against a paper account is perfectly fine (this is how I run it all day). I believe running it against a paper account with no subscription will also work (need to set DELAYED data).

I presume you mean the problem will still exhibit itself. Can you confirm? Because I just ran your last sample program (w/ inconsequential changes for convenience; see attached) against a demo acct using DELAYED and it finished successfully.

• TWGW ver: 10.30.1s
• IBAPI ver: 1030.01
• Platform: Linux 4.4.0-260-generic #294-Ubuntu SMP Fri Sep 27 16:14:57 UTC 2024 x86_64 GNU/Linux

Output:

Using args Namespace(data_lines=65, file='symbols.csv', global_cancel=False, loops=1, market_data_type='DELAYED', port=4002)                                                   
serverVersion:187 connectionTime:b'20241219 15:00:41 EST'
NextValidId received: 1
NextValidId set     : 1000009
Executing requests
LOOP 0 START
1000074 Waiting... Symbols in work : 65, done symbols: 0
1000209 Waiting... Symbols in work : 65, done symbols: 135
1000334 Waiting... Symbols in work : 65, done symbols: 260
1000459 Waiting... Symbols in work : 65, done symbols: 385
1000584 Waiting... Symbols in work : 65, done symbols: 510
1000709 Waiting... Symbols in work : 65, done symbols: 635
1000834 Waiting... Symbols in work : 65, done symbols: 760
1000959 Waiting... Symbols in work : 65, done symbols: 885
1001080 Waiting... Symbols in work : 65, done symbols: 1006
1001207 Waiting... Symbols in work : 65, done symbols: 1133
1001292 Waiting... Symbols in work : 24, done symbols: 1259
             Number of not done symbols: 0
reqId     symbol done  bid     ask     last    volume  high    low     last_timestamp   
                                                                                       
DONE reqMktData_test 51.7 seconds
   ALL Tests DONE 

E.g. in C#: client.reqMktData(reqId, contractObject, string.Empty, true, false, null);