¿ªÔÆÌåÓý

ctrl + shift + ? for shortcuts
© 2025 ¿ªÔÆÌåÓý
  1. Twsapi
  2. Topics

ReqHistoricalTicks() & Futures Data Limitations, Rolling Expiring Contracts

 

Hi all,

The TWS API docs say that 2 years of expired futures contracts is available for historical data. Is that true for tick data? I am wondering if I could actually get multiple years, even 2, of tick data on NQ using this. I assume I would have to write a script to handle contract rolling. I haven't wrote a script yet to get around the max 1,000 data points per request limit, but I saw threads on here discussing how to do just that.

For people who commented on that thread, or anyone with information on acquiring large amounts of affordable tick data, is it even possible to get 2 years worth of tick data on futures contracts like NQ & ES using IB? Assuming I could handle the contract rolling and other necessary steps to format the data. Or should I look to get it another way? Is there any cheap tick data banks even out there? I am only willing to consider that if this option is completely exhausted/impossible.

Thanks,
Brendan
 

You can indeed get approx 2 years of high resolution tick data for expired futures with Just grabbed trade ticks for VXV1 on 2021-09-21.

But you have to be patient since requesting that amount of data will take some time and you want to have some kind or restart logic to work around TWS/IBGW daily restarts and other disruptions. For very active futures, be prepared to get more than 1,000 ticks per call occasionally since returned data is always for full seconds.

Contract rolling can be implemented by requesting lists of active and expired contracts via . Those contracts contain each future's expiration details so you can construct a "continuous future" from that list.

There are various data services out there that sell or rent historical futures ticks data but I have no experience to share and some of them can expensive. They may be interesting, though, if you need data for many futures or if you need event times with a resolution of higher than one second.

´³¨¹°ù²µ±ð²Ô
 

Jurgen,

Awesome news. How long did it take you request and retrieve that amount of data? I assume you went the route the thread suggests, where you store the most recent timestamp and make a request with said timestamp as your startDateTime until you hit current day?

Thanks,
Brendan
 

Also, right now this strategy will only require having useRTH set to true. With that being said, I will probably just get it over with and download all hours. If I didn't though, what would you estimate the difference in time would be for retrieving NQ e-minis tick data for 2 years?
 

This is a numbers game and you'll have to just try it out yourselves. It all depends on the number of ticks (trades, BidAsk, Midpoint) the future produces, whether you want data just for the front month or whether you want to download all ticks for all tradable futures for each day.

ES easily gets to 800K or1Mio trades a day, NQ is probably 1/2 that, ZN 1/5 and VIX 1/10. BidAsk or Midpoint ticks are 5x to 10x those of trades. So you'll be looking at several 100Mio to 1B ticks for two years worth of data. I'll take a long time but then you'll only do that once.

IBKR paces the speed at which they provide the data and speeds may be higher on weekends and during the night. Speeds also tend to be higher at the start and slow down if your process keeps on requesting data for extended periods of time. When I did a quick run yesterday to see whether our tool still works (we never really download historical data any longer) it took 200 seconds to get 71,000 ES trades in 71 requests. That's an initial rate of 1.25 Mio trades per hour but will very likely get slower if you keep on doing that for several hours.

´³¨¹°ù²µ±ð²Ô


On Wed, Sep 20, 2023 at 03:33 PM, <blydon12@...> wrote:
Also, right now this strategy will only require having useRTH set to true. With that being said, I will probably just get it over with and download all hours. If I didn't though, what would you estimate the difference in time would be for retrieving NQ e-minis tick data for 2 years?
 

hello,
I've tried reqContractDetails with NQ future, and here is below the contract list returned in contractDetails. As you can see, there are?only non-expired contracts. So is there anything to do to get expired Contracts ?

contract = Contract()
contract.symbol = "NQ"
contract.secType = "FUT"
self.reqContractDetails(self.reqID, contract) ?

def contractDetails(self, reqId: int, contractDetails: ContractDetails):
? ? super().contractDetails(reqId, contractDetails)
? ? printinstance(contractDetails)
? ? # ! [contractdetails]
 

Set the IncludeExpired field in the Contract object to true, as described in the .
 

Running a script right now to get 2 years of tick data for NQ. Seems to be restricting my request limits to every 6 seconds. Are there times when this could improve? It is Sunday @ 2:30 p.m. where I am right now for reference.
 

Just after 15:10 US/Central this afternoon, I request Historical TickByTickLast data for Friday's NQZ3 session (20230928 17:00 through 20230929 16:00 US/Central):
  • I received 451,009 TickByTickLast objects for NQZ3
  • It took 445 requests/responses and 1,045 seconds to receive all data
  • that is an average of 2.347s per request (elapsed time)
  • but the median was sub-second at 0.659s.
We don't download historical data a lot so we did not put a lot of thought into the little tool:
  • it is a single threaded event processor (no sleeps, delays, or built-in pacing)
  • requests data in 1,000 tick chunks in reverse time order
  • converts each returned TickByTickLast into a relatively expensive immutable Java object
  • accumulates the objects in a list that is serialized into streamable Json objects and written to file each time more than 10,000 ticks have been accumulated

That means the tool makes about 9 out of 10 requests immediately (a few micro seconds) after the callback for the previous request. There is a short processing delay before every tenth request (5ms to 40ms) for the data serialization and file storage.

Now, the interesting finding is the discrepancy between average and median response times. IBKR paced the responses within chunks of ~60 seconds, each time with roughly the following rhythm:

  • Ten requests with response times around 600ms each
  • One request with 3.5 seconds
  • Three requests with 600ms each
  • One request with 4.5 seconds
  • One request with 600ms
  • One request with 5.5 seconds
  • Three requests with 6 seconds

  • One request with 12 seconds

Attached a couple charts of what that looked like. I do not have data on how long they keep that 60 second chunk rhythm up in case you download a couple years worth of data. My gut tells me that you will not be able to keep up the less-than 3 second average for very long runs.

´³¨¹°ù²µ±ð²Ô




On Sun, Oct 1, 2023 at 01:39 PM, <blydon12@...> wrote:
Running a script right now to get 2 years of tick data for NQ. Seems to be restricting my request limits to every 6 seconds. Are there times when this could improve? It is Sunday @ 2:30 p.m. where I am right now for reference.

Charts.pdf
Charts.pdf

 

I am doing this from my sim account. I should probably switch to my live account to hopefully get better times?

On Sun, Oct 1, 2023 at 7:04 PM ´³¨¹°ù²µ±ð²Ô Reinold via <TwsApiOnGroupsIo=[email protected]> wrote:
Just after 15:10 US/Central this afternoon, I request Historical TickByTickLast data for Friday's NQZ3 session (20230928 17:00 through 20230929 16:00 US/Central):
  • I received 451,009 TickByTickLast objects for NQZ3
  • It took 445 requests/responses and 1,045 seconds to receive all data
  • that is an average of 2.347s per request (elapsed time)
  • but the median was sub-second at 0.659s.
We don't download historical data a lot so we did not put a lot of thought into the little tool:
  • it is a single threaded event processor (no sleeps, delays, or built-in pacing)
  • requests data in 1,000 tick chunks in reverse time order
  • converts each returned TickByTickLast into a relatively expensive immutable Java object
  • accumulates the objects in a list that is serialized into streamable Json objects and written to file each time more than 10,000 ticks have been accumulated

That means the tool makes about 9 out of 10 requests immediately (a few micro seconds) after the callback for the previous request. There is a short processing delay before every tenth request (5ms to 40ms) for the data serialization and file storage.

Now, the interesting finding is the discrepancy between average and median response times. IBKR paced the responses within chunks of ~60 seconds, each time with roughly the following rhythm:

  • Ten requests with response times around 600ms each
  • One request with 3.5 seconds
  • Three requests with 600ms each
  • One request with 4.5 seconds
  • One request with 600ms
  • One request with 5.5 seconds
  • Three requests with 6 seconds

  • One request with 12 seconds

Attached a couple charts of what that looked like. I do not have data on how long they keep that 60 second chunk rhythm up in case you download a couple years worth of data. My gut tells me that you will not be able to keep up the less-than 3 second average for very long runs.

´³¨¹°ù²µ±ð²Ô




On Sun, Oct 1, 2023 at 01:39 PM, <blydon12@...> wrote:
Running a script right now to get 2 years of tick data for NQ. Seems to be restricting my request limits to every 6 seconds. Are there times when this could improve? It is Sunday @ 2:30 p.m. where I am right now for reference.
 

My tool is extremely similar to yours, so no bottlenecks in code either.

On Sun, Oct 1, 2023 at 8:07 PM Brendan Lydon <blydon12@...> wrote:
I am doing this from my sim account. I should probably switch to my live account to hopefully get better times?

On Sun, Oct 1, 2023 at 7:04 PM ´³¨¹°ù²µ±ð²Ô Reinold via <TwsApiOnGroupsIo=[email protected]> wrote:
Just after 15:10 US/Central this afternoon, I request Historical TickByTickLast data for Friday's NQZ3 session (20230928 17:00 through 20230929 16:00 US/Central):
  • I received 451,009 TickByTickLast objects for NQZ3
  • It took 445 requests/responses and 1,045 seconds to receive all data
  • that is an average of 2.347s per request (elapsed time)
  • but the median was sub-second at 0.659s.
We don't download historical data a lot so we did not put a lot of thought into the little tool:
  • it is a single threaded event processor (no sleeps, delays, or built-in pacing)
  • requests data in 1,000 tick chunks in reverse time order
  • converts each returned TickByTickLast into a relatively expensive immutable Java object
  • accumulates the objects in a list that is serialized into streamable Json objects and written to file each time more than 10,000 ticks have been accumulated

That means the tool makes about 9 out of 10 requests immediately (a few micro seconds) after the callback for the previous request. There is a short processing delay before every tenth request (5ms to 40ms) for the data serialization and file storage.

Now, the interesting finding is the discrepancy between average and median response times. IBKR paced the responses within chunks of ~60 seconds, each time with roughly the following rhythm:

  • Ten requests with response times around 600ms each
  • One request with 3.5 seconds
  • Three requests with 600ms each
  • One request with 4.5 seconds
  • One request with 600ms
  • One request with 5.5 seconds
  • Three requests with 6 seconds

  • One request with 12 seconds

Attached a couple charts of what that looked like. I do not have data on how long they keep that 60 second chunk rhythm up in case you download a couple years worth of data. My gut tells me that you will not be able to keep up the less-than 3 second average for very long runs.

´³¨¹°ù²µ±ð²Ô




On Sun, Oct 1, 2023 at 01:39 PM, <blydon12@...> wrote:
Running a script right now to get 2 years of tick data for NQ. Seems to be restricting my request limits to every 6 seconds. Are there times when this could improve? It is Sunday @ 2:30 p.m. where I am right now for reference.

1 - 11 of 11