I am trying to troubleshoot a problem I’m having with reqMktData. After starting TWS, my first run of reqMktData for several hundred contracts successfully gets data for them all. However, on subsequent runs for the same batch of contracts, market data is missing for about 30% of them. That missing 30% is always the same contracts.

Restarting TWS always fixes it for the first run again, then the same problem repeats on subsequent runs until restarting TWS again. Just reconnecting to the API does not fix it, it has to be a full TWS restart.? I use cancelMktData after each batch, so it is not an issue of exceeding the ticker limit.

Anyone encountered something similar or have ideas on what I could try?

开云体育

Sure. Here is some more detail on the steps I am taking:

?

1. Start collecting tickPrice messages.

2. Loop through about 200 contracts calling reqMktData, with a delay of 30ms between each iteration to prevent going over the 50 messages per second limit.

3. Wait an additional 10 seconds to give the ticker data extra time to stream in.

4. Stop collecting tickPrice messages.

5. Loop through all 200 contracts again calling cancelMktData, with a delay of 30ms between each iteration.

?

I am specifically after the Last Price or Close Price ticks for those contracts. On the first run after a fresh TWS restart, it finds those ticks for all contracts. Any subsequent runs is when it misses getting those ticks for about 30% of the contracts.

?

Additional information:

?

• I am not using snapshots.

• I do use the same request IDs for each batch. I assume this is safe to do after using cancelMktData for the previous batch’s request IDs. I tried changing it to use unique request IDs for each batch and it did not make any difference.
• No error callbacks at all for the batches that are missing data. I do still receive several tickPrice messages for the problem contracts on the subsequent batches, but not the Last Price or Close Price ticks that I am after. The Last Price and Close Price ticks for those contracts are only returned on the first run after a TWS restart.
• I have tried keeping the ticker data streaming from 10 seconds all the way up to 5 minutes, and it did not make a difference. The Last Price and Close Price ticks for certain contracts still never came in on the subsequent runs.
• I have over the default 100 market data lines. I know that hitting the limit is not the issue here, because in the past I have hit that limit and received an error message both in TWS and API error callbacks. I am not getting those errors now. Even so, I have tried doing contract batch sizes smaller than 100, and it does not fix the issue.?

?

Thanks.

You say you don't use snapshots. But do you have market data subscriptions for all instruments you request data for? Or do you request delayed data?

The two items that jump at me are "200 instruments" and "wait an additional 10 seconds".

Let's start with the 200 instruments.

From your pseudo code it sounds like your pacing of 30ms between requests results in 200 reqMktData over a 6 second period. You then wait for 10 seconds and cancel the requests. That means you will have 200 simultaneous subscriptions at the end of the request loop and for the 10 second wait, right? Or do you cancel any of the subscriptions during the 10 second wait period (say once you got the data you want)?

There is a unless you purchase "quote boosters", spend more that $1,600 in commission per month, or have an account value of at least $2,000,000. The limit is across all clients and TWS so the limit for your API client might actually be lower if, for example, your TWS GUI holds subscriptions that are different from the ones your client requests.

Assuming you are subject to the 100 subscriptions consider changing your loop such that:

• You keep an "active subscriptions" counter that tells you how many simultaneous reqMktData you have so that you never exceed the limit.
• You cancel individual subscriptions (and adjust the counter) as soon as data you are interested in has arrived or you receive an error for a reqMktData

Depending on your programming language you can use a counter that also synchronizes your threads, such as the . You'd initialize the latch with 100 and decrement it by one before you call reqMktData. The thread will be blocked once you hit and while you stay at 0. Your data and error handlers? would cancel subscriptions and increment the latch.

In fact you could probably eliminate the 30ms pacing if you set the latch counter to 50 without impeding performance.

Wait an additional 10 seconds

You need to make sure you perform this wait such that the message reader thread is not blocked. So make it a "wait with timeout" and not a "sleep for 10 seconds)" As a quick check, do you receive data callbacks during the 10 second wait or do you get all callbacks after that wait period?

Other

You say that reqId recycling works for you, but it is not a "best practice". Once your clients et a little more complex you'd be better off with:

• a monotonously increasing reqId for each session that does not reuse reqid while the client holds a connection to TWS/IBGW
  
• An architecture where reqId are handled internally and automatically and hide the concept of reqId from your decision and trading logic layer

闯ü谤驳别苍

开云体育

I appreciate the feedback.

Yes I have been opening 200 simultaneous market data subscriptions, then pausing for 10 seconds, then canceling all 200. This has not given me any TWS or API error messages, so I assume the IB account has “quote boosters” to allow more than 100 at a time. It is a relative’s IB account I am doing development for, so I’m not sure at this time what extra market data perks they have purchased. Regardless, my issue has still been occurring when I limit it to 100, which leads me to believe this is not an issue of exceeding subscription limits.

I am using the C# API. I use separate threads for making requests and consuming messages, so I do not believe any thread blocking could be causing the issue. I receive and process the stream of tickPrice messages throughout the whole timeline of calling reqMktData, waiting 10 seconds, and calling cancelMktData.

I will tinker around with lowering the batch sizes even more and isolating specific contracts that do not give me the ticks I want on the 2nd run. I’ll then try making repeated requests for a single problem contract and see what happens. I may give snapshots a try too, although I recall reading some complaints about their accuracy on this forum.

I’ll post again here if I glean any interesting information or conclusions.

Would definitely be interesting to hear what you learn.

The only other suggestion I have is to run a test with TWS API logging enabled. Under "File -> Global Configuration -> API -> Setting" enable both "Create API message log file" and "Include Market Data in API log file", That way you can see whether TWS does send the data (and the issue is within our code) or whether TWS actually sends no data when you experience missing data.

闯ü谤驳别苍