|
Recently I have been trying to backtest a new strategy using 1 minute bars. I was able to get 10 years worth of 1 minute data for SPY, IWM and DIA but not for QQQ. The longest I can get for QQQ is ~6 years.
?
For older dates the reqHistoricalData() call comes back with a strange error :? "Unrecognised Contract".
Does anyone know why this is?
?
Note I tried, SMART, ISLAND, NASDAQ as the exchanges and all came back with the same issue.
?
Wondering if it is a temporary glitch or a genuine problem and I will not be able to get data that far back for QQQ.
?
Any input would be helpful.
|
You can look at flat files on ?
Recently I have been trying to backtest a new strategy using 1 minute bars. I was able to get 10 years worth of 1 minute data for SPY, IWM and DIA but not for QQQ. The longest I can get for QQQ is ~6 years.
?
For older dates the reqHistoricalData() call comes back with a strange error :? "Unrecognised Contract".
Does anyone know why this is?
?
Note I tried, SMART, ISLAND, NASDAQ as the exchanges and all came back with the same issue.
?
Wondering if it is a temporary glitch or a genuine problem and I will not be able to get data that far back for QQQ.
?
Any input would be helpful.
|
There was a period where QQQ was QQQQ.? Maybe that is what you are
encountering...
On 2/4/2025 8:50 AM, David Armour via
groups.io wrote:
toggle quoted message
Show quoted text
Recently I have been trying to backtest a new strategy using
1 minute bars. I was able to get 10 years worth of 1 minute data
for SPY, IWM and DIA but not for QQQ. The longest I can get for
QQQ is ~6 years.
?
For older dates the reqHistoricalData() call comes back with
a strange error :? "Unrecognised Contract".
Does anyone know why this is?
?
Note I tried, SMART, ISLAND, NASDAQ as the exchanges and all
came back with the same issue.
?
Wondering if it is a temporary glitch or a genuine problem
and I will not be able to get data that far back for QQQ.
?
Any input would be helpful.
|
Thanks for your idea. I did check that before I posted and confirmed that QQQQ->QQQ in 2011 so should not affect my request.
?
My thought was that the ConID may have changed so I did a query just with symbol name and SMART and USD and nothing else but it came back with the same error so it seems impossible to pull 1 minute data from that time.
|
Have you tried "SMART/NASDAQ" or "SMART/ISLAND" in the exchange field ?
Are you setting only the security type, exchange, symbol and currency fields ?
Are you setting the symbol or local symbol field ?
Are you able to open a 1 minute chart of QQQ in TWS going back to more than 6 years from now ?
--
|
How long does your app wait for the response?
When I go for futures data 4 or 5 years ago, it's a long wait before it starts sending data.
Have you tried "SMART/NASDAQ" or "SMART/ISLAND" in the exchange field ?
Are you setting only the security type, exchange, symbol and currency fields ?
Are you setting the symbol or local symbol field ?
Are you able to open a 1 minute chart of QQQ in TWS going back to more than 6 years from now ?
--
|
As a matter of practice I never make requests with "home made" contracts, but rather use contract objects received from reqContractDetails() queries. That way you don't have to keep track of which fields each requests really wants and when additional fields are added over time.
With one of those contracts, I just grabbed data for QQQ from 20++ years ago:
- With an initial end date of 2005-02-01, my downloader grabbed 1 week periods of 1min data (RTHonly=false)
- Each 1-week chunk took between 6 and 30 seconds (average 15 seconds)
- The process stopped with an error 162 "HMDS query returned no data" once the downloader tried to grab data from before 2004-12-01 at 07:00 US/Central
For completeness, below the non-null/initialized fields of the contract.
闯ü谤驳别苍
?
?
{
? "m_conid" : 320227571,
? "m_symbol" : "QQQ",
? "m_secType" : "STK",
? "m_exchange" : "SMART",
? "m_primaryExch" : "NASDAQ",
? "m_currency" : "USD",
? "m_localSymbol" : "QQQ",
? "m_tradingClass" : "NMS",
? "m_includeExpired" : false
}
?
On Tue, Feb 4, 2025 at 07:50 AM, David Armour wrote:
Recently I have been trying to backtest a new strategy using 1 minute bars. I was able to get 10 years worth of 1 minute data for SPY, IWM and DIA but not for QQQ. The longest I can get for QQQ is ~6 years.
?
For older dates the reqHistoricalData() call comes back with a strange error :? "Unrecognised Contract".
Does anyone know why this is?
?
Note I tried, SMART, ISLAND, NASDAQ as the exchanges and all came back with the same issue.
?
Wondering if it is a temporary glitch or a genuine problem and I will not be able to get data that far back for QQQ.
?
Any input would be helpful.
|
Currently QQQ's conid is 320227571. Previously it has had a conid of 43661924. Maybe you can get older historical data using this conid.
|
First, thanks for all your suggestions. Let me answer those suggestions:
?
Orionn:
Have you tried "SMART/NASDAQ" or "SMART/ISLAND" in the exchange field ??
Are you setting only the security type, exchange, symbol and currency fields ??
Are you setting the symbol or local symbol field ?
?
Yes, both NASDAQ and ISLAND, switching on and off compatible mode. It did not work.
Normally I use reqContractDetails() to get the contract like Jurgen suggested but I was running out of ideas so started setting the contract manually in case the ConID had changed at some point. Nothing worked.
?
Are you able to open a 1 minute chart of QQQ in TWS going back to more than 6 years from now ?
?
This is where it gets weird. I tried to open a 1 minute chart and the data stops at 5th June 2018. However, there was data showing before it at what ever date/time I moved the chart back to. If you moved the mouse over the break it jumps from the start date to 5th June 2018. I am adding a pic so you can see what I mean. See on bottom left of screenshot the date is 4th Jan 2010 as that is what I set the chart date at. You can see the break and the next date being 5th June (it is actually 2018, take my word for it).
?
I stepped the 1 minute chart backwards from around that time in June and sure enough, the chart threw the error UNKNOWN CONTRACT. See below (look at the top left for the error).
?
?
?
Jurgen
?
As mentioned above, I usually use the contract returned by reqContractDetails(). The manual attempt was frustration and the last resort of a 'drowning man'!
?
I have tried Demo Account and Live Account and both are the same. I have yet to try the IBGateway. @Jurgen, were you using your code connected to TWS or IBGateway?
?
Is there some reason my version of TWS would be acting this way? As far as I know the data is not stored locally on the PC so not sure why it might impact.
?
Could someone else try to get a 1 minute chart of QQQ at this timeframe and see if they get this error?
?
My TWS version is:
Build 10.33.1f, Jan 27, 2025 3:58:40 PM
Jolt Build 1.18.12, Sep 5, 2024 06:57:46 PM
Nia Build 2.25.8, Jan 3, 2025 05:52:02 PM
ModelNav Build 1.13.2, Jan 24, 2022 04:59:21 PM
Riskfeed Build 2.46.0, Dec 3, 2024 05:33:36 PM ?
Java Version: 1.8.0_202,
OS: Windows 10 (amd64, 10.0),
Memory: heap max 4096Mb, current 294Mb?
?
?
|
Some more comments came in while I was writing that:
?
Dale
?
It is not a matter of waiting unfortunately. TWS immediately triggers the error "Unknown Contract" when I submit the reqHistoricalData().
?
?
J G
?
I will have a look at that ConId, but it seems Jurgen was able to get data with the newer ConId if we refer to his post.
?
?
In my view, if Jurgen can get data and I cannot because my TWS throws errors, then something is weird and more likely IBKR related. I suspect different versions of TWS being used. I am going to try IBGateway now and will report the findings.
|
I am running TWS and IBGW Build 10.30.1t, Jan 7, 2025 3:54:41 PM on Linux. Results are identical.
Since you got an error on 2018-06-05, I ran a couple downloads (one with TWS and one with IBGW) starting at 2019-02-01 going backwards 1 month at a time. Both went through 2018-06-05 without a problem and I stopped them when they were well into 2017. But there was missing data around early June 2018.
闯ü谤驳别苍
?
?
On Tue, Feb 4, 2025 at 08:23 PM, David Armour wrote:
First, thanks for all your suggestions. Let me answer those suggestions:
?
Orionn:
Have you tried "SMART/NASDAQ" or "SMART/ISLAND" in the exchange field ??
Are you setting only the security type, exchange, symbol and currency fields ??
Are you setting the symbol or local symbol field ?
?
Yes, both NASDAQ and ISLAND, switching on and off compatible mode. It did not work.
Normally I use reqContractDetails() to get the contract like Jurgen suggested but I was running out of ideas so started setting the contract manually in case the ConID had changed at some point. Nothing worked.
?
Are you able to open a 1 minute chart of QQQ in TWS going back to more than 6 years from now ?
?
This is where it gets weird. I tried to open a 1 minute chart and the data stops at 5th June 2018. However, there was data showing before it at what ever date/time I moved the chart back to. If you moved the mouse over the break it jumps from the start date to 5th June 2018. I am adding a pic so you can see what I mean. See on bottom left of screenshot the date is 4th Jan 2010 as that is what I set the chart date at. You can see the break and the next date being 5th June (it is actually 2018, take my word for it).
?
I stepped the 1 minute chart backwards from around that time in June and sure enough, the chart threw the error UNKNOWN CONTRACT. See below (look at the top left for the error).
?
?
?
Jurgen
?
As mentioned above, I usually use the contract returned by reqContractDetails(). The manual attempt was frustration and the last resort of a 'drowning man'!
?
I have tried Demo Account and Live Account and both are the same. I have yet to try the IBGateway. @Jurgen, were you using your code connected to TWS or IBGateway?
?
Is there some reason my version of TWS would be acting this way? As far as I know the data is not stored locally on the PC so not sure why it might impact.
?
Could someone else try to get a 1 minute chart of QQQ at this timeframe and see if they get this error?
?
My TWS version is:
Build 10.33.1f, Jan 27, 2025 3:58:40 PM
Jolt Build 1.18.12, Sep 5, 2024 06:57:46 PM
Nia Build 2.25.8, Jan 3, 2025 05:52:02 PM
ModelNav Build 1.13.2, Jan 24, 2022 04:59:21 PM
Riskfeed Build 2.46.0, Dec 3, 2024 05:33:36 PM ?
Java Version: 1.8.0_202,
OS: Windows 10 (amd64, 10.0),
Memory: heap max 4096Mb, current 294Mb?
?
?
|
I tried IBGateway (latest) and I did not get the Unknown Contract error but it would not load any data before 5th June 2018.
?
I completely deleted my TWS and re-installed it, taking care to switch off download layout from server so I had a fresh install. Same situation i.e. no Unknown Contract error, but also no data loading before 5th June 2018. The charts also have the same "break" and missing data before 5th June 2018.
?
My reqHistoricalData() is loading 1 day on 1 minute bars. Asking for TRADES and RTH = true and keepUpToDate = FALSE. These parameters work on SPY, DIA and IWM with no problems.
?
?
@Jurgen, what parameters are you using for your reqHistoricalData() call?? Also, do you have any trouble with the QQQ chart before June '18?
?
The only other difference I can see is the Windows vs Linux. I can try to run from my Linux environment and see what happens.
|
As I said before, I can get "1 min" data down to 2004-12-01 and "1 day" data down to the funds inception date on "1999-03-10".
When I use your parameters (duration 1 day, bar 1 min, rthonly true), the download stops with a "no data" error after the data for Monday 2018-06-04 has been received completely.
Restarting with an end date of "20180604 00:00:00 US/Eastern" immediately fails with a "no data" error.
Restarting with an end date of "20180603 00:00:00 US/Eastern" works fine and data keeps on coming properly for every day from Friday 2018-06-01 and below.
So it looks like I can get all 1min data, just have to restart after 2018-06-04.
闯ü谤驳别苍
?
On Wed, Feb 5, 2025 at 01:04 AM, David Armour wrote:
I tried IBGateway (latest) and I did not get the Unknown Contract error but it would not load any data before 5th June 2018.
?
I completely deleted my TWS and re-installed it, taking care to switch off download layout from server so I had a fresh install. Same situation i.e. no Unknown Contract error, but also no data loading before 5th June 2018. The charts also have the same "break" and missing data before 5th June 2018.
?
My reqHistoricalData() is loading 1 day on 1 minute bars. Asking for TRADES and RTH = true and keepUpToDate = FALSE. These parameters work on SPY, DIA and IWM with no problems.
?
?
@Jurgen, what parameters are you using for your reqHistoricalData() call?? Also, do you have any trouble with the QQQ chart before June '18?
?
The only other difference I can see is the Windows vs Linux. I can try to run from my Linux environment and see what happens.
|
Hi. I ran TWS Linux in the Windows WSL and managed to get the charts to work properly at and before 5th June 2018. This is a step forward.
?
I am now going to run my program connecting to that TWS instead of the Windows one and will see if I can get 10y+ of data.
?
Making progress...
|
After getting the X11 running with XMing on the WSL setup, I checked the chart again and it is the same as Windows TWS, so the progress I thought I had made was not real.
?
Tried to load the data from the TWS Linux and have the same failure for QQQ at 5th June 2018. However, daily data is no problem for me all the way back to 1999.
?
Ready to stop, but would still like to hear if anyone sees the same problem on the 1 minute chart at that date as per my screenshots in the post below. Just want to be sure it is not a "me" problem.
|
What happens when you start the 1min data download with an end-date of 3rd June 2018 or 2nd June 2018?
?
On Wed, Feb 5, 2025 at 10:02 AM, David Armour wrote:
After getting the X11 running with XMing on the WSL setup, I checked the chart again and it is the same as Windows TWS, so the progress I thought I had made was not real.
?
Tried to load the data from the TWS Linux and have the same failure for QQQ at 5th June 2018. However, daily data is no problem for me all the way back to 1999.
?
Ready to stop, but would still like to hear if anyone sees the same problem on the 1 minute chart at that date as per my screenshots in the post below. Just want to be sure it is not a "me" problem.
|
I get no data. Right now I am running a job starting from 2015-01-01 and working my way forward day by day. Up to now I get nothing returned and I am at 2015-06-01.
My code waits 15secs for the data download to start so it is a little slow when no data is returned. It is a latch using a conditional_variable.
?
Here is a snapshot of the log:
?
?
You can see it making the request, then 15secs later cancelling it and issuing a new request for the following day.
|
15 seconds isn't near long enough to wait for a reach that far back.
for 2019 data, i believe it'd take several minutes (as in > 4) to start getting any reply messages. I'd also suggest you request more than 1 day at a time and storing the response on? your side since it takes so long?
On Wed, Feb 5, 2025 at 11:16?AM David Armour via <dave.armour= [email protected]> wrote: I get no data. Right now I am running a job starting from 2015-01-01 and working my way forward day by day. Up to now I get nothing returned and I am at 2015-06-01.
My code waits 15secs for the data download to start so it is a little slow when no data is returned. It is a latch using a conditional_variable.
?
Here is a snapshot of the log:
?
?
You can see it making the request, then 15secs later cancelling it and issuing a new request for the following day.
|
I tried setting the wait up to 10mins when you suggested it before. It just timed out after 10mins. No data is being sent and frankly waiting 10mins is ridiculous. The first set of data is always sent in less than 1 second in my experience.
?
Also, I am following the historical data downloading pacing guidelines and 1 day is normally what is acceptable for 1 minute data. In the past I have asked for more only to see my request get slower and slower until it ground to a halt. This was due to soft pacing being enforced by IB. (It may not be the case now, but it certainly was in the past and I found 1 day at? a time to be a sweetspot.)
?
I get all other stock data just fine the way I do it. It is only QQQ that is having an issue.
?
?
|
maybe get 1 day of data, 3 years ago, then 4 years ago...etc. to see where and how the timing starts to change?
On Wed, Feb 5, 2025 at 11:26?AM David Armour via <dave.armour= [email protected]> wrote: I tried setting the wait up to 10mins when you suggested it before. It just timed out after 10mins. No data is being sent and frankly waiting 10mins is ridiculous. The first set of data is always sent in less than 1 second in my experience.
?
Also, I am following the historical data downloading pacing guidelines and 1 day is normally what is acceptable for 1 minute data. In the past I have asked for more only to see my request get slower and slower until it ground to a halt. This was due to soft pacing being enforced by IB. (It may not be the case now, but it certainly was in the past and I found 1 day at? a time to be a sweetspot.)
?
I get all other stock data just fine the way I do it. It is only QQQ that is having an issue.
?
?
|
