开云体育

Dear Community,
i have noticed that some historical Data requests seem to go "unanswered" somehow. I compared the tws log and the api log which are set to full detail. In the tws log there are the requested bars that were requested, but in the api logs they are missing. Does someone have any information about this ?
regards
Simon

A couple clarification questions, Simon.

When you say "tws log" and "api log", do you refer to the logs you access through Help -> Troubleshooting -> Diagnostics in TWS? Or is the "api log" from within your client application?

Assuming the logs are both from TWS, have you enabled the option "Include Market Data in API log file" in the API Global Configuration section? I believe that is disabled by default.

Does your application get any error codes (such as no data available or pacing violation complaints)?

And finally, what bar size and duration do you use for your requests?

闯ü谤驳别苍

On Wed, Jul 3, 2024 at 05:17 PM, <simon.meier1987@...> wrote:

Hello 闯ü谤驳别苍,

exactly, the tws log and api log are extracted from the tws, through Help -> Troubleshooting -> Diagnostics. The "Include Market Data in API log file" is enabled, otherwise i wouldn't have the first request in the api log.

There are errors but these are the same in both cases. I request? "1 min"? candles for the past 3 weeks, for two contracts. The basic approach is as follows:
I make use of reqHistoricalData(...) request function. First i call m_pClient->reqHistoricalData(...) for the desired contract then i wait for 4 seconds and then call m_pClient->cancelHistoricalData(...), because this was done in the examples for the tws api.?

This is the first step to acquire market data for the past 3 weeks at startup of my trading roboter / client to avoid gaps in the data (if paused for some days). After this initial update is done, i turn this functionality off and only request the tickbytick data (bid / ask) and construct the candlesticks/bars myself, as i found out, that due to data limitations the TWS doesnt deliver the data consistently by using only reqHistoricalData(...).?

The data is delivered by:

TestCppClient::historicalData(...) + TestCppClient::historicalDataEnd(...). ?But in this case, the callback delivered the first request but not the second one (I restarted the client shortly afterwards, therefore the second request). In tws-exported-logs.txt it shows
both data requests (and the data) but not in api-exported-logs.txt. api-exported-logs.txt only shows data for the first request.

I hope this clarifies my question.

regards,
Simon

I am not sure which example the subscribe/wait/unsubscribe came for, but you just need to ditch that. It makes no sense, especially if the "wait" is actually done with a call to "sleep". Just subscribe, make the 3 weeks of 1min" call, pick up the bars through the historicalData callback and wrap it up once you get the historicalDataEnd callback.

I am not doing that much historical data download and my small utility just does that (no attempts to be clever about it) and I just an it for three US stocks and one from Germany (I just do;t have a broad subscription to European stocks:

• With RTHonly set to true, downloading 3 weeks of 1 min bars takes less than 1 second for the US stocks and less than 2 seconds for the EUR stock. There are more bars for the European stock (1/2hr longer RTH session) and the packet "flight time" between the IIBKR data server in Switzerland to my server is 10times of that to the US server.
• With RTHonly set to false, the same request takes up to 7 sec for the US stock and 13sec for the one in Europe.

I can't say for sure why TWS does not like your current approach, but I am confident that everything will work just fine for you when you take out the wait/sleep and subscription cancellation.

Here the detail results for the eight runs:

闯ü谤驳别苍

Ok, i will consider your advice. If i change the code accordingly, does TWS send me 1 minute candles afterwards, after the request is handled, or does it consider this request a "one time request" ? (=So jobs done after sending 3-week data)

Or in other words: i change it the way your recommended, then i receive the 3-week data , turn the functionality off and afterwards construct the candles with the bid/ask as previously ?

If you set keepUpToDate during the reqHistoricalData call to false, the historical data download is a one time request and nothing has to be done after the historicalDataEnd callback.

If you set keepUpToDate during the reqHistoricalData call to true, you basically have an open ended subscription and 1min data bars keep on coming after the historicalDataEnd callback and until you call cancelHistoricalData.

While that works, I'd suggest you stay with one of the real-time subscriptions instead. TickByTick is fine, but if your logic works off of 1min bars, consider 5 sec realtime bars.

The keepUpToDate subscription comes with a bunch of downsides:

• Data is served from the HMDS servers (Historical Market Data Server) and not the real-time data farms for the instrument.
• At least for the instruments I tried that for, I would get multiple "corrected" bars for the same tome period and I could only be sure that I had stable data when the bar for the next period arrived.
• And since data comes from the HMDS servers, it looks like TWS/IBGW silently drops the subscription during the nightly reset/maintenance window (and possibly during other reset events). All realtime subscriptions survive these resets.

闯ü谤驳别苍

Thank you for the tips . I will reply here what was the solution for me after trying the variants.

Basically all i want is to get a stable (100% stable) stream of 1 minute candles for the contracts that are stored in some list. For my approach there is no need for a smaller timeframe.

regards,
Simon

Hi 闯ü谤驳别苍,
the problem with the historical data hasn't occured since i changed the code according to your suggestions. Thanks alot for the tip! Concerning the variants you gave me, i need more time for try them out.

regards,
Simon

While deleting I may make your code work (better), it is clearly not the solution to the problem. The correct solution would be to actually handle I.

The API documentation says:

The historical data will be delivered via the method in the form of candlesticks. ... If reqHistoricalData was invoked with keepUpToDate = false, once all candlesticks have been received the marker will be sent. Otherwise updates of the most recent partial five-second bar will continue to be returned in real time to . ?The keepUpToDate functionality can only be used with bar sizes 5 seconds or greater ...

The historical bars from your request get returned to you by ? and you will receive one callback (bar) per time period.

Once all historical bars have been returned to you, real-time bars arrive through I so that your application is aware of the fact that bars are now real-time. Internally, your client may use similar or the same code for handling them, but real-time bars are different from the historical ones and generally, you may need to do some per-processing (hence the different callback).

The important word in the documentation is "partial" five-second bar. While you receive no more than one historical bar per period, you can get several "corrected" real-time bars for each period and only the last one you receive correctly describes the period. In I you would have some code that filters out bars that are "too partial" for your purpose, something you would not do for historical bars. The filtering may not be trivial (unless your client can wait until the first bar for the next period arrives).

For the instruments I monitor, historical bar data with keepUpToDate = true had another major drawback that make them unusable for me: subscriptions do not survive the nightly reset window (and probably other HMDS farm resets or TWS/IBGW disconnect/reconnect cycles throughout the day). TWS/IBGW are very good at maintaining realTime data feed subscriptions (such as the ) but historical data requests or subscriptions just stop.

IMHO the 5 second realTime bar is a much better solution than keeping historical data up to date-

Maybe, but why do you waste your time with this and make this so complicated? You want to formally verify correctness of your code by an analytical proof (nothing wrong there if you have the tools to do so) but that perfect code will make trading decisions for your real money based on non-deterministic data.

The best you can hope for from historical bar data with keepUpToDate == true is what the documentations says: "a subscription is made to return updates of unfinished real time bars as they are available". Historical data feeds are come with limitations, may be subject to throttling by IBKR, and are good enough for a updating chart such those in TWS, but I would not trade real money based on it.

Let me try my advice one last time. Go with 5 second real-time bars:

• TWS/IBGW maintain the subscriptions for you even through multiple disconnect/connect cycles and you get a solid feed
• For many instruments you get a bar even for periods with no activity (volume/count is 0). That can be used as a heartbeat indicator that gives your application the assurance, that the data feed from TWS is working
• If your algorithm works on 1min bars, assembling 1min bars from 5 second bars is trivial and deterministic. You can also observe the building if the 1min bar with updates every 5 seconds, which you can use as an early warning "earthquake" detector in case the instrument you are trading has a sudden large change in price.
• I ran a quick report on some of my collected data. For about 400mio real-time bars, the average arrival delay (past the end of the period) is 968ms (median 1010ms). So you will have a solid 1min bar about 1 second after each 1min period.