reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS

Hi AJ,?

I connect via TWS and my TWS version is 10.30.1s, whereas the version of my CSharpAPI.dll is?10.15.2.0.?

"I didn't receive all of ["bid", "ask", "last", "volume", "last_timestamp", "high", "low"]"

?I think this might be the difference. I only request snapshots every few seconds.?

E.g. in C#: client.reqMktData(reqId, contractObject, string.Empty, true, false, null);

As said, 50.000+ requests per day and working fine. Of?course, I only get 'the last Last' snapshot, for my app that's fine.

Hope it helps,

Daniel.

On Thu, Dec 19, 2024 at 5:53?PM ajn via <andrei.jefremov=[email protected]> wrote:

In the attached zip I have the app. It is now only 250 lines including all the IB connection and argument parsing. The working function are all the same as before (described earlier) I just deleted everything else. I do agree, as a standalone app it gets much more readable

?

Program_get_bid.py -f symbols.csv -m REALTIME --data-lines 75 --loops 5 --port 4001

?

As an added bonus this app can be run from anywhere, not just from python samples folder. so long your python environment has 10.30 installed

?

AJ

--

Daniel

#53676

In the attached zip I have the app. It is now only 250 lines including all the IB connection and argument parsing. The working function are all the same as before (described earlier) I just deleted everything else. I do agree, as a standalone app it gets much more readable

?

Program_get_bid.py -f symbols.csv -m REALTIME --data-lines 75 --loops 5 --port 4001

?

As an added bonus this app can be run from anywhere, not just from python samples folder. so long your python environment has 10.30 installed

?

AJ

Program_get_bid.zip

Program_get_bid.py

#53675

Thanks Daniel,

?

Just to be clear the problem is not that I am not receiving the callbacks at all. The problem is that I am receiving not entire set of? the data I expect for some of the contracts (always random which contracts will be missing some data). Sometime it is bid which is missing, sometimes ask, sometimes last etc. bit random misses. And everytime I check gateway log for the request I had issues with It is clear that that piece of data is not there (say I see that for req 123456 I get bid, ask, volume but not a last value etc.).?

?

First off: note that `Contract.primaryExch` is not the same as `Contract.Exchange`. You might be constructing invalid Contract objects.?

I am 99.99% it is not the contract issue, as I am getting most of data,?

?

Moreover, today I was experimenting with unsubscribing/subscribing for data for a contract which at this point was missing data (my definition of "missing" is that after subscription I didn't receive all of ["bid", "ask", "last", "volume", "last_timestamp", "high", "low"] within 20 seconds at least once, typically one or two would be missing), and 100% of time I was receiving the data after a second subscription. So contract part is very likely fine.?

?

Possible issues:

1: make sure you comply with the max 50 requests per second pacing violation. I don't know if you'll receive an error message if that's your case.

2: error code 10197 "No market data during competing session": if you're pulling reqMktData and at the same time visualizing the data in TWS, you might get this error and hence not receive an eWrapper callback.?

Yes, I monitor error codes. no such errors. As for pacing, it used to be needed but SDK I believe build it in and there is an option ?setConnectOptions("+PACEAPI"), which I use for several years now on top of own pacer anyway (just ot be safe, remember first versions were glitchy). And typically pacing errors result that you don't get any data at all. which is not current case.?

?

About reqMktData: just for fun I checked my code and out and yesterday I made 54.000 calls to it, all received a callback except few

Great stats, :) it used to be like this for me up and including gateway 10.19.?

what is your definition of "received"? are you looking for say "last" only or a bit wider scope like I mentioned above?
What version of gateway/TWS are you using (not API) ?? Gateway 10.19 and older work fine for me (and have been for 5 years), yet once I switch gateway to 10.30 or 10.31 I immediately start seeing issues.?

AJ

#53670

Hi AJN.?

First off: note that `Contract.primaryExch` is not the same as `Contract.Exchange`. You might be constructing invalid Contract objects.?

About reqMktData: just for fun I checked my code and out and yesterday I made 54.000 calls to it, all received a callback except few, like ~50, because of the reason #2 mentioned below.?

Possible issues:

1: make sure you comply with the max 50 requests per second pacing violation. I don't know if you'll receive an error message if that's your case.

2: error code 10197 "No market data during competing session": if you're pulling reqMktData and at the same time visualizing the data in TWS, you might get this error and hence not receive an eWrapper callback.?

Note I'm in C# API 10.15.2.0 - no capacity to test on another version at?the moment.

Hope it helps.?

Show quoted text

On Wed, Dec 18, 2024 at 9:09?PM ajn via <andrei.jefremov=[email protected]> wrote:

I created a second version where I removed all code which is not absolutely necessary It is roughly 150 line of code which one easily sees comparing Program.py from samples to Program_get_bid.py. As per first post: The file is part of \samples\Python\Testbed?extract provided python.zip in that folder Then, run (or even better run from debugger and step)
Program_get_bid.py -f symbols.csv -m REALTIME --data-lines 75 --loops 5 --port 4001

?

There is 3 new functions left. Main entry , runs the test in the loop

?

Actual code subscribing to data. making sure the datalines limit is not exceeded. And knowing that Subscription is expensive I typically run 65% of line limit. I.e. for test accounts where I have only 100 lines I would run 65 max. So we never run into "max data lines limit reached" error.?

?

monitor_not_done is called after each subscription and stops new subscribtions once we reach the limit or (finalize = True) simply waits until all subscriptions returned the data or Timed out.

?

The wrappers are easier described by comparison to Program.py the original test app. Really all wrappers do is saving incoming data

?

and there is unsubscribing bit which is driven by wrappers (here on example of error), if all data is present the cancel is called and sets "sentCancelMktData" which gives signal to test loop that one received all the data

?

That is pretty much it.?

?

--

Daniel

#53669

On Wed, Dec 18, 2024 at 08:09 PM, ajn wrote:

where I removed all code which is not absolutely necessary It is roughly 150 line of code which one easily sees comparing Program.py from samples to Program_get_bid.py.

Pull it out and make it stand alone. Get rid of ALL SUPERFLUOUS things like colorization constants... or no one will want to help you.

#53664

I created a second version where I removed all code which is not absolutely necessary It is roughly 150 line of code which one easily sees comparing Program.py from samples to Program_get_bid.py. As per first post: The file is part of \samples\Python\Testbed?extract provided python.zip in that folder Then, run (or even better run from debugger and step)
Program_get_bid.py -f symbols.csv -m REALTIME --data-lines 75 --loops 5 --port 4001

?

There is 3 new functions left. Main entry , runs the test in the loop

?

Actual code subscribing to data. making sure the datalines limit is not exceeded. And knowing that Subscription is expensive I typically run 65% of line limit. I.e. for test accounts where I have only 100 lines I would run 65 max. So we never run into "max data lines limit reached" error.?

?

monitor_not_done is called after each subscription and stops new subscribtions once we reach the limit or (finalize = True) simply waits until all subscriptions returned the data or Timed out.

?

The wrappers are easier described by comparison to Program.py the original test app. Really all wrappers do is saving incoming data

?

and there is unsubscribing bit which is driven by wrappers (here on example of error), if all data is present the cancel is called and sets "sentCancelMktData" which gives signal to test loop that one received all the data

?

That is pretty much it.?

?

Testbed_python.zip

#53663

I have implemented reqContractDetails and start constructing contract using conId I received.
so instead of this

Like this

?

It made no difference, unfortunately. still not receiving some data

#53662

On Wed, Dec 18, 2024 at 12:08 AM, ajn wrote:

I didn't delete that IB boiler code to make the comparison obvious, if I do the app would be tiny. I can do that easily = keep only code which gets executed if that will make everyone more comfortable)

This would be greatly appreciated. If you can make the test case as small and simple as humanly possible... this will help people (esp. IBKR internal developers, who need extra help because they're mostly inept) to id and possibly fix the problem.

#53653

Thanks 闯ü谤驳别苍,

very valid points. I was really afraid that complexity of this bug will cause problems. So I didn't write code from scratch but did least possible modifications the the official sample app IB provides as a common ground (Program.py in case of python) precisely hoping that maybe it will be bit more familiar to people and will save time and effort for everyone (at least to IB developers as I sent it first to them) and the way to look at that code is to run a comparison to the original IB code to really see the difference = the actual new code which is executed. And the code I have added there is less than 200 lines, the rest is the "full" test app, which can be ignored (or at least I was hoping people who would look at it would have more trust to IB's code :) and everyone has the original app to compare. I didn't delete that IB boiler code to make the comparison obvious, if I do the app would be tiny. I can do that easily = keep only code which gets executed if that will make everyone more comfortable)
In the nutshell
reqMktData_test (set a break point here and easy walk)
? ?calls reqMktData_test_work in a loop
? ? ? all contracts are sent to reqMktData
? ? ? the process is slowed down and finished by monitor_not_done so we don't have more than x data-lines used at the same time, that function monitors that all data is received and we can cancel the subscription?
? ? ? the results are printed using process_and_export_data

the following wrapper callbacks take the incoming data (unfortunately reqMktData triggers all those)

tickPrice
tickSize
tickGeneric
tickString
error

?

That will generally require us to read and understand the code before?

Actually, running against a paper account is perfectly fine (this is how I run it all day). I believe running it against a paper account with no subscription will also work (need to set DELAYED data).?

?Calling reqMktData creates a market data subscription for a real-time data feed that is intended to stay up for some time

Totally. In my live application I have several hundreds of data lines to work with and slightly less number of contracts I am interested in trading "now". and about 1500 - 2500 "full universe". So I go through such cycle to get latest on the market and maintain the subscription for those several hundreds I am interested in for a while (and there I run into issues that some data is not showing up for 20+ seconds (which I consider never)). Of course as I wanted the "simplest" app showing the issues. As you rightly say, it is a large chunk of code already :) I didn't want to make it larger by having another 200 lines "ok here we subscribed and waiting, and 3 min latter suddenly some of those lines lack data"?

Your code is peppered with sleep statements,

Thanks, I double checked (happy to double check). The code which I added is only to not have the busy waiting while waiting for data delivered by a different thread. There is one place I added a slowdown (but it default to zero) in tickPrice as I was experimenting with hypothesis "what if the problems are caused by me processing the data too long", I didn't find correlation. I should have removed that before posting the code to reduce mental strain of readers. Sorry about that (yet again, that line does nothing). But indeed there are several cases in IB original code (which is not used in this test) which relies on sleep before cancelling. The test code data path doesn't rely on IB sending data within specific time as far as I can see.??

?

My environment makes many of these subscriptions just before the start of the trading session

Another interesting bit. Thanks for sharing. As in this test we subscribe in the middle of the trading AND in case of TWS contracts from the watchlist definitely overrepresented in the list of failed symbols. Maybe something got broken in IB here. That is very tough to debug, as you get one shot per day :)?

you are using market data requests well outside of the "design envelope" and, while that worked in the past, it does not any longer.

The worrying bit that even if I do it for 5-20 data lines at a time (as a result slowly crawling). I still see the data missing. I see that it less to do with frequency of the requests?

Try a loop of reqContractDetails for all contracts

Interesting, I didn't know about this possible issue. Will try. Although since we run several loops on the same "home brew" contracts, I would expected that after the first loop all will be good. But I will try. At this point I would try anything which has any chance of solving the issue before IB retires the 10.19 gateway and would force move to 10.30

?

#53651

This is a pretty tall ask, ajn. You send 2,000++ lines of source code that you ask us to run. That will generally require us to read and understand the code before we can even consider running against our IBGW/TWS instances. I cannot invest that kind of effort.

If I understand your test beds correctly (and I took only a very quick peak), your code subscribes to and then quickly cancels market data subscriptions for more than 1,000 symbols in a rapid sequence. I assume that is to work around the default 100 simultaneous subscriptions limit. And I am sure the difference in behavior between IBGW/TWS 10.19 and 10.30 is real (you seem to have done a lot of work on this), but I am not convinced yet, that this is a bug or issue with IBGW/TWS. It appears as if you are using market data requests well outside of the "design envelope" and, while that worked in the past, it does not any longer. Your code is peppered with sleep statements, which is always a big red flag for me since it makes code vulnerable the changes in timing behavior between versions of IBGW/TWS.

Calling reqMktData creates a market data subscription for a real-time data feed that is intended to stay up for some time and is a pretty expensive operation (from IBGW/TWS and IBKR infrastructure point of view). Canceling the? feed, while possibly less expensive, is also not trivial. My environment makes many of these subscriptions just before the start of the trading session, keeps them open and receives the data feeds for 23 hours, and disconnects them once the session is over. I receive between 300 and 1000 callbacks in average every second for the entire 23 hours with peak periods of 10,000 callbacks per second and more. But as far as I can tell, I do not experience any loss of data with TWS/IBGW 10.30. But I will take a look another look at my data to make sure.

Not really sure what I can tell you other than this. Try a loop of reqContractDetails for all contracts you need and use the complete contracts returned by IBKR instead of your less complete "home made" contract objects during your reqMktData loops. That will increase the chance that IBGW/TWS caches the information it needs and reduces the likelihood that additional costly HMDS queries need to be made for each subsequent reqMktData call.

闯ü谤驳别苍

#53650

Dear Twsapi experts,

Since 10.30 gateway/tws looks like reqMktData cannot be relied upon and I am often not getting the data I am asking for. And I am talking about basic bid/ask/last etc data for popular contracts with high volume of trading on Nasdaq/NYSE (like MSFT, GOOG etc) nothing exotic. It happens so often that it makes it impossible to use reqMktData in practice (up to 10% call failure rate). Yet I've been using reqMktData for 5+ years both in test and live trading, it was pretty reliable for me, no complains so far (it was slow, sure, but reliable :) ). Moreover for gateway/tws 10.19 it works as expected still today just 10.30+ drops data. Making me believe that something got broken in 10.30 and newer software.

The issue, as I will describe below seems to be related to resource usage/management of 10.30+ software, which makes me wonder what else tws is dropping once it gets somehow pushed. I could imagine that IB is loosing some data here which used to be perfect/fine it might as well drop other api call/callback (like drooping sending an order or dropping some order status data would be really bad). Thus making it more urgent to fix it.?

?

I of course reported the issue to IB with logs clearly showing that the gateway didn't provide the data it was asked. Moreover, I shipped them this test app which I will share below which (for me) 100% repeats the issue. And IB went silent for 10+ days. I have no idea atm if they confirm the issue and busy fixing or it just ended up at the bottom of the backlog. Thus really appreciate the help of the community here. Maybe I am not alone experiencing this issue.

?

My ask for community is to run the test app provided (I did one in python and one in java. both languages experience the issue more strongly pointing towards IB software issue) if more can repeat the issue (and there is no issues in the my test code :) ) maybe it will help IB to rise the urgency for fixing the issue.?

I am using IBAPI 1030.01. It works for connecting to any gateway/TWS versions. I have not tested it with other IBAPI SDK version.?

in this api there is folder \samples\Python\Testbed extract provided python.zip in that folder (details for Java are later). Then, run
Program_get_bid.py -f symbols.csv -m REALTIME --data-lines 75 --loops 5 --port 4001
this will read symbols (about 1200 stocks from big, liquid companies on Nasdaq/NYSE), connect to your gateway (maybe you use a different port). will ask for REALTIME data (if you don't have subscription to US stocks + realtime addon, FROZEN, DELAYED seem to be experience the same issue). Test also will make sure that we don't ask for more that 75 contract simultaneously and run this operation 5 loops?

The trick with this bug is that it is doesn't happen for a specific contract or specific order of API calls. In my experience it happens "randomly" after the gateway was used for some time. So this test app asks for data using self.reqMktData(self.nextValidOrderId, contract, "", False, False, []) and monitors callbacks if it receives the following data ["bid", "ask", "last", "volume", "last_timestamp", "high", "low"]. it waits for 20 seconds for this data to appear and if doesn't appear (this is the bug I am talking about) it declares this call as failed and prints out as TIMEOUT. Some runs will experience more issues than others. Some will have zero issues. The important bit that in my setup 5 loops of 1200 contracts always causes some failures if connected 10.30+ gateway/TWS and zero issues on 10.19. It is true for both Linux and Windows.?

A fragment of output?

?

All failed symbols are printed during the run and summarized with info received at the end. Also reqId is printed, so one can go into gateway log and search for this ID. All the searches I did in the gateway's log it was clear that gateway really didn't send the info the app is missing (i.e. it is not the test app which missed it somehow but the gateway didn't even send it)

Notes

Program_get_bid.py is based on Program.py. One can easily compare the differences to the "official" sample. It is really minimal difference to make sure we can ship requests without overloading API. And again this code works just fine with 10.19 gateway and fails 100% on 10.30.?
Of course it could be situation there bid/ask is missing, but those are rare for such common stocks (I have filtered stocks by market cap and I think that list has only stocks with 10+B in market cap and some trading volume, precisely so we don't run into such issues in this test). And if that is the case, there is no excuse for missing last/volume/high/low/last_timestamp anyway. It is not present on the screenshot, but happens often
java code is should be extracted to samples\Java\samples\testbed. However make a copy of that folder so you can compare Tesbed.java and EWrapperImpl.java to original files.?
1. I call it with following parameter (or change directly in the code) -p 4001 -m Real-Time -d 75 -l 5 -f symbols.csv
2. again if you don't have subscription, -m Delayed seems to be causing same issues. but again, Real-Time is the most interesting.
This is not the code I run in production. I just wanted something which repeats the issue 100% and I can share it with IB and community here. So it is fast ad-hoc code which I was trying to keep as close as possible to the official samples (java was AI driven conversion from python, but I was trying to figure out if it is api language related or not). Fingers crossed I didn't do mistakes here. Feel free to comment?
1. In my case I see that test is failing about 3-50 symbols out of 6000 calls (5 loops of 1200 contracts). In my production where much more complex loading of the gateway is happening (read account, do trades, monitor trades etc) I see about 10% of calls failing. Which makes me believe that it has maybe something to do with resource usage/API loading of the gateway.?
2. dropping -m 20 or lower symbols simultaneously slightly reduces chances of dropping data, again pointing to resource issue, maybe
3. If I run the test connecting to a TWS instead of the gateway the symbols which I have in the watchlist are somewhat overrepresented in the list of "failed" contracts in the test.?

Regards,

AJ

Testbed_Python.zip

testbed_java.zip

开云体育

reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS