Dear Twsapi experts,
Since 10.30 gateway/tws looks like reqMktData cannot be relied upon and I am often not getting the data I am asking for. And I am talking about basic bid/ask/last etc data for popular contracts with high volume of trading on Nasdaq/NYSE (like MSFT, GOOG etc) nothing exotic. It happens so often that it makes it impossible to use reqMktData in practice (up to 10% call failure rate). Yet I've been using reqMktData for 5+ years both in test and live trading, it was pretty reliable for me, no complains so far (it was slow, sure, but reliable :) ). Moreover for gateway/tws 10.19 it works as expected still today just 10.30+ drops data. Making me believe that something got broken in 10.30 and newer software.
The issue, as I will describe below seems to be related to resource usage/management of 10.30+ software, which makes me wonder what else tws is dropping once it gets somehow pushed. I could imagine that IB is loosing some data here which used to be perfect/fine it might as well drop other api call/callback (like drooping sending an order or dropping some order status data would be really bad). Thus making it more urgent to fix it.?
?
I of course reported the issue to IB with logs clearly showing that the gateway didn't provide the data it was asked. Moreover, I shipped them this test app which I will share below which (for me) 100% repeats the issue. And IB went silent for 10+ days. I have no idea atm if they confirm the issue and busy fixing or it just ended up at the bottom of the backlog. Thus really appreciate the help of the community here. Maybe I am not alone experiencing this issue.
?
My ask for community is to run the test app provided (I did one in python and one in java. both languages experience the issue more strongly pointing towards IB software issue) if more can repeat the issue (and there is no issues in the my test code :) ) maybe it will help IB to rise the urgency for fixing the issue.?
I am using IBAPI 1030.01. It works for connecting to any gateway/TWS versions. I have not tested it with other IBAPI SDK version.?
in this api there is folder \samples\Python\Testbed extract provided python.zip in that folder (details for Java are later). Then, run
Program_get_bid.py -f symbols.csv -m REALTIME --data-lines 75 --loops 5 --port 4001
this will read symbols (about 1200 stocks from big, liquid companies on Nasdaq/NYSE), connect to your gateway (maybe you use a different port). will ask for REALTIME data (if you don't have subscription to US stocks + realtime addon, FROZEN, DELAYED seem to be experience the same issue). Test also will make sure that we don't ask for more that 75 contract simultaneously and run this operation 5 loops?
The trick with this bug is that it is doesn't happen for a specific contract or specific order of API calls. In my experience it happens "randomly" after the gateway was used for some time. So this test app asks for data using self.reqMktData(self.nextValidOrderId, contract, "", False, False, []) and monitors callbacks if it receives the following data ["bid", "ask", "last", "volume", "last_timestamp", "high", "low"]. it waits for 20 seconds for this data to appear and if doesn't appear (this is the bug I am talking about) it declares this call as failed and prints out as TIMEOUT. Some runs will experience more issues than others. Some will have zero issues. The important bit that in my setup 5 loops of 1200 contracts always causes some failures if connected 10.30+ gateway/TWS and zero issues on 10.19. It is true for both Linux and Windows.?
A fragment of output?
?
All failed symbols are printed during the run and summarized with info received at the end. Also reqId is printed, so one can go into gateway log and search for this ID. All the searches I did in the gateway's log it was clear that gateway really didn't send the info the app is missing (i.e. it is not the test app which missed it somehow but the gateway didn't even send it)
Notes
- Program_get_bid.py is based on Program.py. One can easily compare the differences to the "official" sample. It is really minimal difference to make sure we can ship requests without overloading API. And again this code works just fine with 10.19 gateway and fails 100% on 10.30.?
- Of course it could be situation there bid/ask is missing, but those are rare for such common stocks (I have filtered stocks by market cap and I think that list has only stocks with 10+B in market cap and some trading volume, precisely so we don't run into such issues in this test). And if that is the case, there is no excuse for missing last/volume/high/low/last_timestamp anyway. It is not present on the screenshot, but happens often
- java code is should be extracted to samples\Java\samples\testbed. However make a copy of that folder so you can compare Tesbed.java and EWrapperImpl.java to original files.?
- I call it with following parameter (or change directly in the code) -p 4001 -m Real-Time -d 75 -l 5 -f symbols.csv
- again if you don't have subscription, -m Delayed seems to be causing same issues. but again, Real-Time is the most interesting.
- This is not the code I run in production. I just wanted something which repeats the issue 100% and I can share it with IB and community here. So it is fast ad-hoc code which I was trying to keep as close as possible to the official samples (java was AI driven conversion from python, but I was trying to figure out if it is api language related or not). Fingers crossed I didn't do mistakes here. Feel free to comment?
- In my case I see that test is failing about 3-50 symbols out of 6000 calls (5 loops of 1200 contracts). In my production where much more complex loading of the gateway is happening (read account, do trades, monitor trades etc) I see about 10% of calls failing. Which makes me believe that it has maybe something to do with resource usage/API loading of the gateway.?
- dropping -m 20 or lower symbols simultaneously slightly reduces chances of dropping data, again pointing to resource issue, maybe
- If I run the test connecting to a TWS instead of the gateway the symbols which I have in the watchlist are somewhat overrepresented in the list of "failed" contracts in the test.?
Regards,
AJ
