¿ªÔÆÌåÓý

ctrl + shift + ? for shortcuts
© 2025 Groups.io

Error when requesting 20 years of price data for specific ticker [CHTR]


 

I have a strange issue when requesting data for a specific ticker: CHTR (for?Charter Communications Inc. - Class A). Below you can see a printout of my console, showing that price data for other test tickers is correctly retrieved. However, when reaching CHTR, it stops and says I don't have access to a database that can provide the data:



However, I do have access to multiple databases. I'm assuming that the most relevant one is the NASDAQ one - highlighted in green below. And I know the database is working because my code retrieves data for CHRW and CSX, both Nasdaq listed.



But there is a workaround - if I open CHTR in a chart with 20 years of weekly data, I can now retrieve the data for CHTR. See below the chart and the retrieved data for CHTR on my console:





Also, if I ask for a couple of weeks data for CHTR, even without opening a chart with the ticker in it, the data retrieval works fine.

Below is the test code I use. It works for multiple tickers, for multiple exchanges etc. I only have an issue with this one security (that I'm aware so far).

IBKR help desk will send me the typical answer "we are not a data provider" etc. Sure, but ironically IBKR's API is faster than the other three backup databases I have.

Any insight on this issue would be greatly appreciated,

Danilo
___ ___ ___ ___ ___?

CODE:

#%% DATA REQUEST EXAMPLES
?
# VARIOUS EXAMPLES OF DATA REQUEST FROM INTERACTIVE BROKERS
?
from ibapi.client import EClient
from ibapi.wrapper import EWrapper
from ibapi.contract import Contract
from ibapi.common import TickerId
?
from threading import Thread
import time
?
import pandas
import sys
?
class IBapi(EWrapper, EClient):
?
? ? def __init__(self):
? ? ? ? EClient.__init__(self, self)
? ? ? ? self.data = []
? ? ? ? self.data_sent = -1 # used to check if full data requested returned
?
? ? def error(self, reqId:TickerId, errorCode:int, errorString:str, advancedOrderRejectJson = ""):
? ? ? ? ?if advancedOrderRejectJson:
? ? ? ? ? ? ?print("\nError. Id:", reqId, "\nCode:", errorCode, "\nMsg:", errorString, "\nAdvancedOrderRejectJson:", advancedOrderRejectJson)
? ? ? ? ?else:
? ? ? ? ? ? ?print("\nError. Id:", reqId, "\nCode:", errorCode, "\nMsg:", errorString)
?
? ? def historicalData(self, reqId, bar):
? ? ? ? self.data.append([bar.date, bar.close])
?
? ? def historicalDataEnd(self, reqId, start, end):
? ? ? ? self.data_sent = reqId
?
def run_loop():
? ? app.run()
?
app = IBapi()
app.connect('127.0.0.1', 7496, 0)
?
#Start the socket in a thread
api_thread = Thread(target=run_loop, daemon=True)
api_thread.start()
time.sleep(0.5) #Sleep interval to allow time for connection to server
?
?
# HISTORICAL DATA TEST OF DATABASES vs. API
?
# AAP (NYSE) and CHTR (Nasdaq):
# Had to sign-up for "NYSE (Network A/CTA) Billed by Broker" @ $45/month and "NASDAQ (Network C/UTP)" @ $25/month
# Didn't work with "NYSE American, BATS, ARCA, IEX, and Regional Exchanges (Network B)" @ $25/month
# "Worked" with "Cboe One" @ $5/month but data is inconsistent (e.g. multiple days with same price)
# The "NYSE American, BATS, ARCA, IEX, and Regional Exchanges (Network B)" was better to show streaming prices on TWS (with the Cboe One, live data disapears when using a Basket trade)
?
# Volvo worked with "Nordic Equity" @ 36 EUR/month
# Didn't work with "Turquoise ECNs and Gettex Europe" @ 9.5 GBP/month
# But didn't work with "European (BATS/Chi-X)" @ 31.5 GBP/month
?
# Nestle worked with "European (BATS/Chi-X) Equities" @ GBP 31.50/Month;?
# "SIX Swill Exchange" @ 32.5 CHF/Month partialy worked (would cause an error for 20 years of data request)
# "Tuquoise ECNs and Gettex" didn't work (same for Volvo - see above)
# Noticed that when using "SMART" for exchange getting different figures for Neste (vs. EOD and Yahoo Finance)
# Tried specific exchanges: AQXECH - no permission; BATECH: matched other databases; CHIXCH: permission but a few figures still wrong
# EBS - no permission; TRQXCH: no permission
?
contracts = {'NYSE listed 1':? ?['AAP',? ? 'SMART',? 'USD'],
? ? ? ? ? ? ?'NYSE listed 2':? ?['AGCO',? ?'SMART',? 'USD'],
? ? ? ? ? ? ?'Nasdaq listed 1': ['CHRW',? ?'SMART',? 'USD'],
? ? ? ? ? ? ?'Nasdaq listed 2': ['CSX',? ? 'SMART',? 'USD'],
? ? ? ? ? ? ?'Nasdaq listed 3': ['CHTR',? ?'SMART',? 'USD'],
? ? ? ? ? ? ?'Index':? ? ? ? ? ?['SPY',? ? 'SMART',? 'USD'],
? ? ? ? ? ? ?'Nestle':? ? ? ? ? ['NESN',? ?'BATECH', 'CHF'],
? ? ? ? ? ? ?'Volvo':? ? ? ? ? ?['VOLV.B', 'SFB',? ? 'SEK']}
?
#Create contract object
con = Contract()
con.secType = 'STK'
?
# short-term data
for c in contracts:
? ? con.symbol = contracts[c][0]
? ? con.exchange = contracts[c][1]
? ? con.currency = contracts[c][2]
?
? ? #Request Market Data
? ? reqId = 100
? ? app.data = []
? ? app.data_sent = -1
? ? app.reqHistoricalData(reqId, con, '', '2 W', '1 day', 'ADJUSTED_LAST', 1, 1, False, [])
? ? while app.data_sent != reqId:
? ? ? ? try:
? ? ? ? ? ? time.sleep(0.1)
? ? ? ? except:
? ? ? ? ? ? print("\nError while reading security "+contracts[c][0]+"\n")
? ? ? ? ? ? app.disconnect()
? ? ? ? ? ? sys.exit(1) # will exit code either due to an true error or if CTRL+C pressed
?
? ? # Work with "data" from IB API
? ? df = pandas.DataFrame(app.data, columns=['DateTime', 'Adj. Close'])
? ? df['DateTime'] = pandas.to_datetime(df['DateTime'], format='%Y%m%d')
? ? print('\n2 Weeks of price data for {}:\n'.format(con.symbol))
? ? print(df)
?
?
# long-term data
for c in contracts:
? ? con.symbol = contracts[c][0]
? ? con.exchange = contracts[c][1]
? ? con.currency = contracts[c][2]
?
? ? #Request Market Data
? ? reqId = 100
? ? app.data = []
? ? app.data_sent = -1
? ? app.reqHistoricalData(reqId, con, '', '20 Y', '1 day', 'ADJUSTED_LAST', 1, 1, False, [])
? ? while app.data_sent != reqId:
? ? ? ? try:
? ? ? ? ? ? time.sleep(0.1)
? ? ? ? except:
? ? ? ? ? ? print("\nError while reading security "+contracts[c][0]+"\n")
? ? ? ? ? ? app.disconnect()
? ? ? ? ? ? sys.exit(1) # will exit code either due to an true error or if CTRL+C pressed
?
? ? # Work with "data" from IB API
? ? df = pandas.DataFrame(app.data, columns=['DateTime', 'Adj. Close'])
? ? df['DateTime'] = pandas.to_datetime(df['DateTime'], format='%Y%m%d')
? ? print('\n20 years of price data for {}:\n'.format(con.symbol))
? ? print(df)
?
app.disconnect()


 

Just suggestions: try CHTR with ISLAND instead of SMART. if failed then try with 'TRADES' instead of 'ADJUSTED_LAST'


 

The TWS API Guide suggests for retrieval:
  • In general, a smart-routed historical data requests will require subscriptions to all exchanges on which a instrument trades.
  • For instance, a historical data request for a pink sheet (OTC) stock which trades on ARCAEDGE will require the subscription "OTC Global Equities" or "Global OTC Equities and OTC Markets" for ARCAEDGE in addition to the regular subscription (e.g. "OTC Markets").
That is the reason for Gordon to suggest you use ISLAND (direct routing) instead of SMART (smart-routed). Check the ContractDetails object for CHTR. They are listed on more than 25 exchanges (real and virtual ones) and there is probably one that you do not have subscribed to. The consequence, however, is that the returned data is based solely on trades that took place on ISLAND. If your approach needs all data from all exchanges, you'd have to get additional subscriptions.

For licensing reasons, IBKR has to be much more stringent when they provide you market data in machine readable form via the API than when they display charts within TWS. So I am not surprised that, for charting, TWS may receive more data than what you have subscribed to for and that you can retrieve unsubscribed but locally cached data after pulling up the chart in TWS.

Hope this helps,
´³¨¹°ù²µ±ð²Ô

PS. Going forward, please do not post large chunks of code. TWS API is language independent and discussions are more meaningful for the entire group when we focus on API behavior, not the implementation in one language. Chances are you get more and better responses that way, too.



On Wed, Aug 9, 2023 at 01:46 PM, Rational-IM wrote:

I have a strange issue when requesting data for a specific ticker: CHTR (for?Charter Communications Inc. - Class A). Below you can see a printout of my console, showing that price data for other test tickers is correctly retrieved. However, when reaching CHTR, it stops and says I don't have access to a database that can provide the data:

However, I do have access to multiple databases. I'm assuming that the most relevant one is the NASDAQ one - highlighted in green below. And I know the database is working because my code retrieves data for CHRW and CSX, both Nasdaq listed.

But there is a workaround - if I open CHTR in a chart with 20 years of weekly data, I can now retrieve the data for CHTR. See below the chart and the retrieved data for CHTR on my console:

Also, if I ask for a couple of weeks data for CHTR, even without opening a chart with the ticker in it, the data retrieval works fine.

Below is the test code I use. It works for multiple tickers, for multiple exchanges etc. I only have an issue with this one security (that I'm aware so far).

IBKR help desk will send me the typical answer "we are not a data provider" etc. Sure, but ironically IBKR's API is faster than the other three backup databases I have.

Any insight on this issue would be greatly appreciated,

Danilo


 

Thank you for the tips - I will try tomorrow morning with the ISLAND exchange settings (before opening the chart I mentioned above) to check if it acquires the data without a glitch. I can't use "TRADES"? because it brings the price?unadjusted for dividends. See the detta for PCAR (a company that pays dividends):



What is quite scary is how different the prices are from another source (FactSet) - see below. But I will address this later:



The price for ISLAND and SMART (using "ADJUSTED_LAST") is similar (a little scary that it isn't the same, but close enough) for CHTR:



I will post back here re: data acquisition before trying the workaround (once you do it, the code works during the whole day).

Thank you again,

Danilo


 

Interesting!
You probably noticed on your example that recent days are way more similar (identical)
The reason a close differ to an adjusted close are generally fix values (div/split etc ...) I mean not related to the place of exchange itself.
I won't be surprised that SMART Adjusted Close and NASDAQ Adjusted Close differ in 2002 because the Close itself did differ more from one exchange to another in theses epoch than the 'current epoch' post "Flash boys"


 

May I guess that the reason for the difference in CHTR prices?between Island and Smart in the last pair of screenshots must be that the prices quoted are for very different dates - 2010 vs 2009...
--
Best,
DS


 

Great catch from @ds-avatar: the dates are indeed different (so the difference in the last pair of screenshots are not meaningfull.

So this morning, I run my code but using ISLAND for CHTR. And it worked on the first trial (without my workaround of opening a chart with share prices for CHTR) - see below.

Here is a puzzling part: @´³¨¹°ù²µ±ð²Ô Reinold allerted me to the following: "In general, a smart-routed historical data requests will require subscriptions to all exchanges on which a instrument trades". So I got the "validExchanges" info for the contracts I was using to ask for price data. See below that after each set of price data, there is now a "CONTRACT DETAILS" section. Note that for CHRW and CSX (two other Nasdaq-listed stocks) the "Valid Exchanges" are exactly the same as for CHTR. So not sure why the issue happens only with CHTR (and not the other Nasdaq names in my test list).

Tomorrow I will run the code with "SMART"? for CHTR and see if the error happens again and report back.

Thank you all for the help so far.


 

If I rephrase your concern I understand it as: why different tickers with exactly same exchanges behave differently on historical request ?

I paraphrase here what Jurgen already said:
I agree that CHTR is not an exotic instrument, and I am not familiar enough with constraints that can arise from some 'exchanges' like FOXRIVER or CSFBALGO? or IBEOS etc ...
however the principle in order to fulfill your request is that IB try to fetch data, but doing so IB need access to some 'exchanges' that you are not allowed to get data from. (And maybe you will never be able to access, or finding which exchange is part of the SMART aggregate and induce the failure is too much time consuming)
Worth a try submitting this to IB support to know which exchange was missing in your subscription (if it's a real miss) they may have tools to diagnose that faster than trough API with trial/error.

Somewhere SMART create catch 22 situations for IB, because IB know that there are accurate data available but they can't deliver it for copyright reason, so it is somewhere safer that IB complain about your subscription rather than answering wrong info.
It became your decision to fall back on an exchange part of your subscriptions.
You can reverse the issue and understand it as on CSX or CHRW, IB doesn't need access to specialized exchange to deliver you the data.

Personal opinion
I never use SMART when fetching historical data, because when IB return data:
1- I don't know what I am looking at.

2- if a bar or a price is missing from primary exchange, I better know it in order to anticipate an execution issue, like hiding a lack of liquidity, or ...
I may miss the beauty of SMART and I don't know what TWS is doing for chart, but so far so good.

3- As most of this algo business is working by doing comparisons, it could be better to decide a constant sound reference, related to your algo, even if wrong towards the rest of the world, could be right within the mindset of your algos.
Or quit the opposite and knowing precisely any differences that may exist between exchanges.