¿ªÔÆÌåÓý

ctrl + shift + ? for shortcuts
© 2025 Groups.io

Re: Source of the ticks timestamps


 

¿ªÔÆÌåÓý

Realtime data returned by TWS from reqMktData() does not contain a timestamp. ib_insync itself adds the timestamp using your computer clock. So indeed there is no way at all to know when the tick was 'really' produced.

?

Data returned from reqTickByTick() DOES include a timestamp, but it's limited to a resolution of 1 second. I don't know offhand if it's added by the IB servers or by TWS, but I think this has been discussed before so try searching.

?

IBKR does not aggregate data returned from reqMktData: it samples it. If you compare the data returned from reqMktData with a source that definitely includes every tick, you find that every tick from reqMktData is in the full data, but not vice versa.

?

The sampling mechanism is very simple. Time is divided into fixed length intervals of around 300 millisecs for stocks and futures and 100 millisecs for forex. For any given instrument, the IBKR market data server records each tick as it arrives. At the end of each period it sends the current value for each tick type to TWS, but only if it is different from the value set to the user at the end of the previous period. To enable this to happen, it must also keep a record of what it sent last time.

?

This has implications: in particular, if the previous value for 'last price' was, say, 1500, and during the next period the exchange sends one or more trade reports with different values, then at the end of the period only the latest 'last price' is sent to the API, and if that happens to be the same as at the start of the period then no 'last price' is sent at all (but a 'last size' might be sent if that is different, and a 'volume' will be sent because that has increased). This mechanism explains how the reqMktData stream sometimes results in highs and lows of bars being incorrect by a tick or two: they just weren't sent because no change from the start of the 300ms period had been detexcted.

?

Given that there are many tens of thousands of TWS users at any one time, many of whom will have large numbers of tickers running, this might sound like a massive amount to keep track of, but it's actually quite simple (though this is speculation ¨C I don't know for certain exactly how IBKR manage this). If the time period length is 300 millseconds, you could imagine that the IBKR data farm has 3 servers for a particular stock or future. When a user requests data for that instrument, he is allocated to one of these 3 servers, and all his subsequent data (for that instrument) come from that server. Thus that server only needs to record one set of previously sent ticks, not one for each user. And that server now has 300 milliseconds ?to send out each tick to all the users it services, so the load is fairly constant and predictable. Obviously one server would serve data from many different instruments, and conversely each instrument's data would be disseminated to by multiple servers, but the whole thing can be sliced and diced and load-balanced in such a way that the tick data stream is never more than 300 ms behind the market.

?

When I first starting investigating this mechanism back in 2003, every market data line in TWS and every market data request via the API counted against your allocation of 100 tickers, even if you requested the same ticker more than once. This was presumably because the request was allocated to a different server, and in fact the data streams received were different.. But at some stage in the late noughties (I think) this was refined so that the same request from the same user was only counted once: so now, if I request data for a given instrument from several different API client programs, that is only counted as one market data line, and they all receive exactly the same data (and that even applies whether the data is requested via the live or the paper-trading account).

?

Sorry for the lengthy reply, but I think it helps to have a clear picture of what is going on here.

?

?

?

From: [email protected] <[email protected]> On Behalf Of Alex Gorbachev
Sent: 16 February 2021 04:45
To: [email protected]
Subject: [TWS API] Source of the ticks timestamps

?

Hi all.

?

This must be a very noob question...

I was trying to track the delay of real time data I get. I use the ib_insync Python module which makes things much easier in Python.

?

First of all, ping to? takes about 27ms.

?

For real-time bars I take my current system timestamp and then subtract?the bar time minus 5 seconds (bar size). That gives me 500-1000ms.

?

For market data ticks, I subscribe using?reqMktData and the difference between time in the ticker (including all ticks in it - they have the same timestamp) and system time when I get them is only about?0.2ms (that's it - 200 microseconds). Now, that makes me think that ticker timestamp produced through?reqMktData subscription is actually generated in TWS (or IB Gateway which is what I use) and not on IBRK servers. Right? Thus, there is no way to say when those ticks were really produced. Is there?

?

I haven't looked at TickByTick yet but I see that it has a very low limit on simultaneous subscriptions (3) so won't be useful for as many contracts as I need.

?

Also, could someone point me on how exactly IBRK samples or aggregates ticks for?reqMktData? Do I every time just get aggregated ticks by type and price since the last update?

?

Thanks,

Alex

?

Join [email protected] to automatically receive all group messages.