|
I can't believe that the date/time format has any impact. I went with the "Date/Time in the instrument's time zone" approach a long time ago. The only major difference between our approaches is the direction of time for a sequence of requests.
I don't use historical data download that much, but I don't recall a case where a request would hang forever (in other words no historicalDataEnd() or error() callback). My client is entirely event driven and basically sends the next request upon the receipt of the previous request's historicalDataEnd() callback. Though the framework between the client and TWS API handles collecting bars and returning all bars for a request as a list of bars.
´³¨¹°ù²µ±ð²Ô
?
?
On Wed, Feb 5, 2025 at 11:03 AM, David Armour wrote:
Hi Jurgen,
?
How does that work when you get no data and no historicalDataEnd() callback? If you have no timeout then your code would just hang wouldn't it?
?
I used to download backwards, but it gave me other problems so I reversed it. I always check once I have received the historicalDataEnd() that I received the expected number of bars. So far, I have never had an error for missing bars. (I have a function to manage holidays and short trading days.)
?
I must say your delays seem long to me, but I have never checked mine other than they are all less than 15secs (based on my start latch timeout). I only ever got a timeout after 15secs with problems like the one I am facing now with QQQ. I have downloaded hundreds of symbols with long histories of 1min and 5min bars so this is not stated without some experience.
?
This is just one of those problems that IB throws at us now and again with data.
?
Going to sleep on it. May try using a different timestamp format tomorrow and see if that has any impact. Cannot really think of anything else to try.
?
?
|
Hi Jurgen,
?
How does that work when you get no data and no historicalDataEnd() callback? If you have no timeout then your code would just hang wouldn't it?
?
I used to download backwards, but it gave me other problems so I reversed it. I always check once I have received the historicalDataEnd() that I received the expected number of bars. So far, I have never had an error for missing bars. (I have a function to manage holidays and short trading days.)
?
I must say your delays seem long to me, but I have never checked mine other than they are all less than 15secs (based on my start latch timeout). I only ever got a timeout after 15secs with problems like the one I am facing now with QQQ. I have downloaded hundreds of symbols with long histories of 1min and 5min bars so this is not stated without some experience.
?
This is just one of those problems that IB throws at us now and again with data.
?
Going to sleep on it. May try using a different timestamp format tomorrow and see if that has any impact. Cannot really think of anything else to try.
?
?
|
Hi Dale,
?
Perhaps you missed it in the long discussion below, but that is exactly what was done. I am unable to get data before 5th June 2018.
?
Somehow, Jurgen is able to though.
?
I suspect there is some difference in the call to reqHistoricalData() somewhere. For example, I am using the UTC format YYYYMMDD-HH:mm:ss but I believe that Jurgen is using the format like "YYYYMMDD HH:mm:ss US/EASTERN" as he mentioned it below. Not sure why that would make a difference.
?
?
|
Hmm. I use no set timeouts and I do not cancel nor restart requests.
But I perform historical data downloads in reverse time order (and I know several other members do that, too). When you make a request you specify the "end date and time" plus a duration, but you may or may not get data for that entire duration. So once you receive data, simply start the next request based upon the earliest time stamp you received. I am wondering whether your delays are caused because you make requests in ascending time order. That might go against the IBKRs ability of prefetching and pipelining the data for you.
For a download of 1min data with 1day duration that started at 2018-06-03 and ended at 2018-05-01 (in reverse time order) I get the following delays between request and data arrival (in seconds): 1, 1, 6, 1, 7, 6, 7, 18, 19, 1, 1, 6, 6, 6, 7, 12, 24, 1, 1, 6, 6, 1, 6
That is the kind of pattern I see for longer downloads, too.
´³¨¹°ù²µ±ð²Ô
?
On Wed, Feb 5, 2025 at 10:16 AM, David Armour wrote:
I get no data. Right now I am running a job starting from 2015-01-01 and working my way forward day by day. Up to now I get nothing returned and I am at 2015-06-01.
My code waits 15secs for the data download to start so it is a little slow when no data is returned. It is a latch using a conditional_variable.
?
?
|
maybe get 1 day of data, 3 years ago, then 4 years ago...etc. to see where and how the timing starts to change?
On Wed, Feb 5, 2025 at 11:26?AM David Armour via <dave.armour= [email protected]> wrote: I tried setting the wait up to 10mins when you suggested it before. It just timed out after 10mins. No data is being sent and frankly waiting 10mins is ridiculous. The first set of data is always sent in less than 1 second in my experience.
?
Also, I am following the historical data downloading pacing guidelines and 1 day is normally what is acceptable for 1 minute data. In the past I have asked for more only to see my request get slower and slower until it ground to a halt. This was due to soft pacing being enforced by IB. (It may not be the case now, but it certainly was in the past and I found 1 day at? a time to be a sweetspot.)
?
I get all other stock data just fine the way I do it. It is only QQQ that is having an issue.
?
?
|
I tried setting the wait up to 10mins when you suggested it before. It just timed out after 10mins. No data is being sent and frankly waiting 10mins is ridiculous. The first set of data is always sent in less than 1 second in my experience.
?
Also, I am following the historical data downloading pacing guidelines and 1 day is normally what is acceptable for 1 minute data. In the past I have asked for more only to see my request get slower and slower until it ground to a halt. This was due to soft pacing being enforced by IB. (It may not be the case now, but it certainly was in the past and I found 1 day at? a time to be a sweetspot.)
?
I get all other stock data just fine the way I do it. It is only QQQ that is having an issue.
?
?
|
15 seconds isn't near long enough to wait for a reach that far back.
for 2019 data, i believe it'd take several minutes (as in > 4) to start getting any reply messages. I'd also suggest you request more than 1 day at a time and storing the response on? your side since it takes so long?
On Wed, Feb 5, 2025 at 11:16?AM David Armour via <dave.armour= [email protected]> wrote: I get no data. Right now I am running a job starting from 2015-01-01 and working my way forward day by day. Up to now I get nothing returned and I am at 2015-06-01.
My code waits 15secs for the data download to start so it is a little slow when no data is returned. It is a latch using a conditional_variable.
?
Here is a snapshot of the log:
?
?
You can see it making the request, then 15secs later cancelling it and issuing a new request for the following day.
|
I get no data. Right now I am running a job starting from 2015-01-01 and working my way forward day by day. Up to now I get nothing returned and I am at 2015-06-01.
My code waits 15secs for the data download to start so it is a little slow when no data is returned. It is a latch using a conditional_variable.
?
Here is a snapshot of the log:
?
?
You can see it making the request, then 15secs later cancelling it and issuing a new request for the following day.
|
What happens when you start the 1min data download with an end-date of 3rd June 2018 or 2nd June 2018?
?
On Wed, Feb 5, 2025 at 10:02 AM, David Armour wrote:
After getting the X11 running with XMing on the WSL setup, I checked the chart again and it is the same as Windows TWS, so the progress I thought I had made was not real.
?
Tried to load the data from the TWS Linux and have the same failure for QQQ at 5th June 2018. However, daily data is no problem for me all the way back to 1999.
?
Ready to stop, but would still like to hear if anyone sees the same problem on the 1 minute chart at that date as per my screenshots in the post below. Just want to be sure it is not a "me" problem.
|
After getting the X11 running with XMing on the WSL setup, I checked the chart again and it is the same as Windows TWS, so the progress I thought I had made was not real.
?
Tried to load the data from the TWS Linux and have the same failure for QQQ at 5th June 2018. However, daily data is no problem for me all the way back to 1999.
?
Ready to stop, but would still like to hear if anyone sees the same problem on the 1 minute chart at that date as per my screenshots in the post below. Just want to be sure it is not a "me" problem.
|
Hi. I ran TWS Linux in the Windows WSL and managed to get the charts to work properly at and before 5th June 2018. This is a step forward.
?
I am now going to run my program connecting to that TWS instead of the Windows one and will see if I can get 10y+ of data.
?
Making progress...
|
As I said before, I can get "1 min" data down to 2004-12-01 and "1 day" data down to the funds inception date on "1999-03-10".
When I use your parameters (duration 1 day, bar 1 min, rthonly true), the download stops with a "no data" error after the data for Monday 2018-06-04 has been received completely.
Restarting with an end date of "20180604 00:00:00 US/Eastern" immediately fails with a "no data" error.
Restarting with an end date of "20180603 00:00:00 US/Eastern" works fine and data keeps on coming properly for every day from Friday 2018-06-01 and below.
So it looks like I can get all 1min data, just have to restart after 2018-06-04.
´³¨¹°ù²µ±ð²Ô
?
On Wed, Feb 5, 2025 at 01:04 AM, David Armour wrote:
I tried IBGateway (latest) and I did not get the Unknown Contract error but it would not load any data before 5th June 2018.
?
I completely deleted my TWS and re-installed it, taking care to switch off download layout from server so I had a fresh install. Same situation i.e. no Unknown Contract error, but also no data loading before 5th June 2018. The charts also have the same "break" and missing data before 5th June 2018.
?
My reqHistoricalData() is loading 1 day on 1 minute bars. Asking for TRADES and RTH = true and keepUpToDate = FALSE. These parameters work on SPY, DIA and IWM with no problems.
?
?
@Jurgen, what parameters are you using for your reqHistoricalData() call?? Also, do you have any trouble with the QQQ chart before June '18?
?
The only other difference I can see is the Windows vs Linux. I can try to run from my Linux environment and see what happens.
|
I tried IBGateway (latest) and I did not get the Unknown Contract error but it would not load any data before 5th June 2018.
?
I completely deleted my TWS and re-installed it, taking care to switch off download layout from server so I had a fresh install. Same situation i.e. no Unknown Contract error, but also no data loading before 5th June 2018. The charts also have the same "break" and missing data before 5th June 2018.
?
My reqHistoricalData() is loading 1 day on 1 minute bars. Asking for TRADES and RTH = true and keepUpToDate = FALSE. These parameters work on SPY, DIA and IWM with no problems.
?
?
@Jurgen, what parameters are you using for your reqHistoricalData() call?? Also, do you have any trouble with the QQQ chart before June '18?
?
The only other difference I can see is the Windows vs Linux. I can try to run from my Linux environment and see what happens.
|
I am running TWS and IBGW Build 10.30.1t, Jan 7, 2025 3:54:41 PM on Linux. Results are identical.
Since you got an error on 2018-06-05, I ran a couple downloads (one with TWS and one with IBGW) starting at 2019-02-01 going backwards 1 month at a time. Both went through 2018-06-05 without a problem and I stopped them when they were well into 2017. But there was missing data around early June 2018.
´³¨¹°ù²µ±ð²Ô
?
?
On Tue, Feb 4, 2025 at 08:23 PM, David Armour wrote:
First, thanks for all your suggestions. Let me answer those suggestions:
?
Orionn:
Have you tried "SMART/NASDAQ" or "SMART/ISLAND" in the exchange field ??
Are you setting only the security type, exchange, symbol and currency fields ??
Are you setting the symbol or local symbol field ?
?
Yes, both NASDAQ and ISLAND, switching on and off compatible mode. It did not work.
Normally I use reqContractDetails() to get the contract like Jurgen suggested but I was running out of ideas so started setting the contract manually in case the ConID had changed at some point. Nothing worked.
?
Are you able to open a 1 minute chart of QQQ in TWS going back to more than 6 years from now ?
?
This is where it gets weird. I tried to open a 1 minute chart and the data stops at 5th June 2018. However, there was data showing before it at what ever date/time I moved the chart back to. If you moved the mouse over the break it jumps from the start date to 5th June 2018. I am adding a pic so you can see what I mean. See on bottom left of screenshot the date is 4th Jan 2010 as that is what I set the chart date at. You can see the break and the next date being 5th June (it is actually 2018, take my word for it).
?
I stepped the 1 minute chart backwards from around that time in June and sure enough, the chart threw the error UNKNOWN CONTRACT. See below (look at the top left for the error).
?
?
?
Jurgen
?
As mentioned above, I usually use the contract returned by reqContractDetails(). The manual attempt was frustration and the last resort of a 'drowning man'!
?
I have tried Demo Account and Live Account and both are the same. I have yet to try the IBGateway. @Jurgen, were you using your code connected to TWS or IBGateway?
?
Is there some reason my version of TWS would be acting this way? As far as I know the data is not stored locally on the PC so not sure why it might impact.
?
Could someone else try to get a 1 minute chart of QQQ at this timeframe and see if they get this error?
?
My TWS version is:
Build 10.33.1f, Jan 27, 2025 3:58:40 PM
Jolt Build 1.18.12, Sep 5, 2024 06:57:46 PM
Nia Build 2.25.8, Jan 3, 2025 05:52:02 PM
ModelNav Build 1.13.2, Jan 24, 2022 04:59:21 PM
Riskfeed Build 2.46.0, Dec 3, 2024 05:33:36 PM ?
Java Version: 1.8.0_202,
OS: Windows 10 (amd64, 10.0),
Memory: heap max 4096Mb, current 294Mb?
?
?
|
Some more comments came in while I was writing that:
?
Dale
?
It is not a matter of waiting unfortunately. TWS immediately triggers the error "Unknown Contract" when I submit the reqHistoricalData().
?
?
J G
?
I will have a look at that ConId, but it seems Jurgen was able to get data with the newer ConId if we refer to his post.
?
?
In my view, if Jurgen can get data and I cannot because my TWS throws errors, then something is weird and more likely IBKR related. I suspect different versions of TWS being used. I am going to try IBGateway now and will report the findings.
|
First, thanks for all your suggestions. Let me answer those suggestions:
?
Orionn:
Have you tried "SMART/NASDAQ" or "SMART/ISLAND" in the exchange field ??
Are you setting only the security type, exchange, symbol and currency fields ??
Are you setting the symbol or local symbol field ?
?
Yes, both NASDAQ and ISLAND, switching on and off compatible mode. It did not work.
Normally I use reqContractDetails() to get the contract like Jurgen suggested but I was running out of ideas so started setting the contract manually in case the ConID had changed at some point. Nothing worked.
?
Are you able to open a 1 minute chart of QQQ in TWS going back to more than 6 years from now ?
?
This is where it gets weird. I tried to open a 1 minute chart and the data stops at 5th June 2018. However, there was data showing before it at what ever date/time I moved the chart back to. If you moved the mouse over the break it jumps from the start date to 5th June 2018. I am adding a pic so you can see what I mean. See on bottom left of screenshot the date is 4th Jan 2010 as that is what I set the chart date at. You can see the break and the next date being 5th June (it is actually 2018, take my word for it).
?
I stepped the 1 minute chart backwards from around that time in June and sure enough, the chart threw the error UNKNOWN CONTRACT. See below (look at the top left for the error).
?
?
?
Jurgen
?
As mentioned above, I usually use the contract returned by reqContractDetails(). The manual attempt was frustration and the last resort of a 'drowning man'!
?
I have tried Demo Account and Live Account and both are the same. I have yet to try the IBGateway. @Jurgen, were you using your code connected to TWS or IBGateway?
?
Is there some reason my version of TWS would be acting this way? As far as I know the data is not stored locally on the PC so not sure why it might impact.
?
Could someone else try to get a 1 minute chart of QQQ at this timeframe and see if they get this error?
?
My TWS version is:
Build 10.33.1f, Jan 27, 2025 3:58:40 PM
Jolt Build 1.18.12, Sep 5, 2024 06:57:46 PM
Nia Build 2.25.8, Jan 3, 2025 05:52:02 PM
ModelNav Build 1.13.2, Jan 24, 2022 04:59:21 PM
Riskfeed Build 2.46.0, Dec 3, 2024 05:33:36 PM ?
Java Version: 1.8.0_202,
OS: Windows 10 (amd64, 10.0),
Memory: heap max 4096Mb, current 294Mb?
?
?
|
Currently QQQ's conid is 320227571. Previously it has had a conid of 43661924. Maybe you can get older historical data using this conid.
|
As a matter of practice I never make requests with "home made" contracts, but rather use contract objects received from reqContractDetails() queries. That way you don't have to keep track of which fields each requests really wants and when additional fields are added over time.
With one of those contracts, I just grabbed data for QQQ from 20++ years ago:
- With an initial end date of 2005-02-01, my downloader grabbed 1 week periods of 1min data (RTHonly=false)
- Each 1-week chunk took between 6 and 30 seconds (average 15 seconds)
- The process stopped with an error 162 "HMDS query returned no data" once the downloader tried to grab data from before 2004-12-01 at 07:00 US/Central
For completeness, below the non-null/initialized fields of the contract.
´³¨¹°ù²µ±ð²Ô
?
?
{
? "m_conid" : 320227571,
? "m_symbol" : "QQQ",
? "m_secType" : "STK",
? "m_exchange" : "SMART",
? "m_primaryExch" : "NASDAQ",
? "m_currency" : "USD",
? "m_localSymbol" : "QQQ",
? "m_tradingClass" : "NMS",
? "m_includeExpired" : false
}
?
On Tue, Feb 4, 2025 at 07:50 AM, David Armour wrote:
Recently I have been trying to backtest a new strategy using 1 minute bars. I was able to get 10 years worth of 1 minute data for SPY, IWM and DIA but not for QQQ. The longest I can get for QQQ is ~6 years.
?
For older dates the reqHistoricalData() call comes back with a strange error :? "Unrecognised Contract".
Does anyone know why this is?
?
Note I tried, SMART, ISLAND, NASDAQ as the exchanges and all came back with the same issue.
?
Wondering if it is a temporary glitch or a genuine problem and I will not be able to get data that far back for QQQ.
?
Any input would be helpful.
|
How long does your app wait for the response?
When I go for futures data 4 or 5 years ago, it's a long wait before it starts sending data.
Have you tried "SMART/NASDAQ" or "SMART/ISLAND" in the exchange field ?
Are you setting only the security type, exchange, symbol and currency fields ?
Are you setting the symbol or local symbol field ?
Are you able to open a 1 minute chart of QQQ in TWS going back to more than 6 years from now ?
--
|
Have you tried "SMART/NASDAQ" or "SMART/ISLAND" in the exchange field ?
Are you setting only the security type, exchange, symbol and currency fields ?
Are you setting the symbol or local symbol field ?
Are you able to open a 1 minute chart of QQQ in TWS going back to more than 6 years from now ?
--
|
