Keyboard Shortcuts
Likes
- Twsapi
- Messages
Search
|
Re: ib market data. is it clean and any good alterantive sources?
toggle quoted message
Show quoted text
|
|
Re: requesting midpoint data
开云体育Why don’t you just ask IBKR? If anybody is likely to be able to provide you with chapter and verse, it’s them. ? Richard ? ? From: [email protected] <[email protected]> On Behalf Of tbrown122387 via groups.io
Sent: 07 February 2025 22:52 To: [email protected] Subject: [TWS API] requesting midpoint data ? I'm trying to request midpoint data (for symbols such as SPY, QQQ, etc.) via reqTickByTickData() and says "For some securities, getting tick-by-tick data requires Level 2 data bundles." ? 1/3: Did this change? If so, was that recent? I thought I remembered being able to do this while only subscribing to level 1. ? 2/3: Does that mean I need: ?US Equity and Options Add-On Streaming Bundle (P)Includes streaming real-time quotes for NYSE (CTA/Network A), AMEX (CTA/Network B), NASDAQ (UTP/Network C), and OPRA (US Options). In order to subscribe to US Equity and Options Add-On Streaming Bundle (P), the user must already be subscribed to US Securities Snapshot Bundle (P). ? Can I get rid of what I bought earlier this week? ? NYSE American, BATS, ARCA, IEX, and Regional Exchanges (Network B)(P,L1)Provides real-time quotes for stocks listed outside of NYSE and NASDAQ. Exchange listings include (but not limited to) those on NYSE American (formerly known as AMEX), ARCA, BATS, and IEX. Examples include IBKR, SPY, and VXX. ? 3/3: If I end up having to subscribe to the fancier one, I assume I'm also allowed to use reqMarketDepth() to subscribe to the data in an alternative format. Is that right? |
|
Re: ib market data. is it clean and any good alterantive sources?
how long have you been using it? appreciate if you can share real experience, like telling me oh i have been extracting them daily, hourly for some years and found the data quality high and without issue when extracting them.
?
when in production, i want something consistent (without format change, data change or delay).?
?
don't know how reliable polygon.io is but i would prefer something with long history |
|
Re: ib market data. is it clean and any good alterantive sources?
What would be timely for you? I found to be a good source, especially as they provide flat file data downloads on a daily basis:? On Sat, 8 Feb 2025 at 07:54, alan chau via <curiosityhk2019=[email protected]> wrote:
|
|
ib market data. is it clean and any good alterantive sources?
i have been using IB API for pacing order only. can anyone tell if the US equities/ETF data from IB clean and up-to-date?
?
Recently i am looking for data source which provide data timely and clean. my focus is US equities/ETF data, can anyone suggest some free (or low cost) alternative for bloomberg terminal? better have python API to extract them.
?
especially for production purpose, they must update and provide the data timely with less frequent format/changes in their raw sources.
?
understand that this question may not be that related to IB API here. but i do think the people here can address my question efficiently. appreciate if anyone can share the experience to me here.
?
thanks! |
|
requesting midpoint data
I'm trying to request midpoint data (for symbols such as SPY, QQQ, etc.) via reqTickByTickData() and says "For some securities, getting tick-by-tick data requires Level 2 data bundles."
?
1/3: Did this change? If so, was that recent? I thought I remembered being able to do this while only subscribing to level 1.
?
2/3: Does that mean I need:
?US Equity and Options Add-On Streaming Bundle (P)Includes streaming real-time quotes for NYSE (CTA/Network A), AMEX (CTA/Network B), NASDAQ (UTP/Network C), and OPRA (US Options). In order to subscribe to US Equity and Options Add-On Streaming Bundle (P), the user must already be subscribed to US Securities Snapshot Bundle (P). ? Can I get rid of what I bought earlier this week? ? NYSE American, BATS, ARCA, IEX, and Regional Exchanges (Network B)(P,L1)Provides real-time quotes for stocks listed outside of NYSE and NASDAQ. Exchange listings include (but not limited to) those on NYSE American (formerly known as AMEX), ARCA, BATS, and IEX. Examples include IBKR, SPY, and VXX. ? 3/3: If I end up having to subscribe to the fancier one, I assume I'm also allowed to use reqMarketDepth() to subscribe to the data in an alternative format. Is that right? |
|
Re: Decimal remaining value retrival
The short answer is no. When you look at the data model, one Order is related to (or filled by) one or more Trades (executions). "filled" and "remaining" are attributes of Orders and not attributes of Trades. As such they are reported ithough orderStatus() callbacks, where they belong. When you manage orders and trades take place, you will receive orderStatus() callbacks right before or after execDetails() callbacks so your client will have that information. 闯ü谤驳别苍 ?
?
On Thu, Feb 6, 2025 at 09:13 AM, <tarun.joshi@...> wrote:
|
|
Decimal remaining value retrival
Hello Folks,
Is is possible to get Decimal remaining value? in the below callback //! [execdetails] void execDetails(int reqId, const Contract& contract, const Execution& execution) { ? printf("ExecDetails. ReqId: %d - %s, %s, %s - %s, %s, %s, %s, %s\n", reqId, contract.symbol.c_str(), ? ? ? ? ?contract.secType.c_str(), contract.currency.c_str(), execution.execId.c_str(), ? ? ? ? ?longMaxString(execution.orderId).c_str(), decimalStringToDisplay(execution.shares).c_str(), ? ? ? ? ?decimalStringToDisplay(execution.cumQty).c_str(), intMaxString(execution.lastLiquidity).c_str()); like we receive in Order Status callback void orderStatus(OrderId orderId, const std::string& status, Decimal filled, Decimal remaining, ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?double avgFillPrice, int permId, int parentId, double lastFillPrice, int clientId, ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?const std::string& whyHeld, double mktCapPrice) { ? printf( ? ? ? "OrderStatus. Id: %ld, Status: %s, Filled: %s, Remaining: %s, AvgFillPrice: %s, PermId: %s, LastFillPrice: %s, " ? ? ? "ClientId: %s, WhyHeld: %s, MktCapPrice: %s\n", ? ? ? orderId, status.c_str(), decimalStringToDisplay(filled).c_str(), decimalStringToDisplay(remaining).c_str(), ? ? ? doubleMaxString(avgFillPrice).c_str(), intMaxString(permId).c_str(), doubleMaxString(lastFillPrice).c_str(), ? ? ? intMaxString(clientId).c_str(), whyHeld.c_str(), doubleMaxString(mktCapPrice).c_str()); I wan't to get? decimalStringToDisplay(remaining).c_str() in ?execdetails callback. Thanks. |
|
Re: QQQ 1 Minute Data >6years?
I can't believe that the date/time format has any impact. I went with the "Date/Time in the instrument's time zone" approach a long time ago. The only major difference between our approaches is the direction of time for a sequence of requests. I don't use historical data download that much, but I don't recall a case where a request would hang forever (in other words no historicalDataEnd() or error() callback). My client is entirely event driven and basically sends the next request upon the receipt of the previous request's historicalDataEnd() callback. Though the framework between the client and TWS API handles collecting bars and returning all bars for a request as a list of bars. 闯ü谤驳别苍 ? ? On Wed, Feb 5, 2025 at 11:03 AM, David Armour wrote:
|
|
Re: QQQ 1 Minute Data >6years?
Hi Jurgen,
?
How does that work when you get no data and no historicalDataEnd() callback? If you have no timeout then your code would just hang wouldn't it?
?
I used to download backwards, but it gave me other problems so I reversed it. I always check once I have received the historicalDataEnd() that I received the expected number of bars. So far, I have never had an error for missing bars. (I have a function to manage holidays and short trading days.)
?
I must say your delays seem long to me, but I have never checked mine other than they are all less than 15secs (based on my start latch timeout). I only ever got a timeout after 15secs with problems like the one I am facing now with QQQ. I have downloaded hundreds of symbols with long histories of 1min and 5min bars so this is not stated without some experience.
?
This is just one of those problems that IB throws at us now and again with data.
?
Going to sleep on it. May try using a different timestamp format tomorrow and see if that has any impact. Cannot really think of anything else to try.
?
? |
|
Re: QQQ 1 Minute Data >6years?
Hi Dale,
?
Perhaps you missed it in the long discussion below, but that is exactly what was done. I am unable to get data before 5th June 2018.
?
Somehow, Jurgen is able to though.
?
I suspect there is some difference in the call to reqHistoricalData() somewhere. For example, I am using the UTC format YYYYMMDD-HH:mm:ss but I believe that Jurgen is using the format like "YYYYMMDD HH:mm:ss US/EASTERN" as he mentioned it below. Not sure why that would make a difference.
?
? |
|
Re: QQQ 1 Minute Data >6years?
Hmm. I use no set timeouts and I do not cancel nor restart requests. But I perform historical data downloads in reverse time order (and I know several other members do that, too). When you make a request you specify the "end date and time" plus a duration, but you may or may not get data for that entire duration. So once you receive data, simply start the next request based upon the earliest time stamp you received. I am wondering whether your delays are caused because you make requests in ascending time order. That might go against the IBKRs ability of prefetching and pipelining the data for you. For a download of 1min data with 1day duration that started at 2018-06-03 and ended at 2018-05-01 (in reverse time order) I get the following delays between request and data arrival (in seconds): 1, 1, 6, 1, 7, 6, 7, 18, 19, 1, 1, 6, 6, 6, 7, 12, 24, 1, 1, 6, 6, 1, 6 That is the kind of pattern I see for longer downloads, too. 闯ü谤驳别苍 ? On Wed, Feb 5, 2025 at 10:16 AM, David Armour wrote:
|
|
Re: QQQ 1 Minute Data >6years?
maybe get 1 day of data, 3 years ago, then 4 years ago...etc. to see where and how the timing starts to change? On Wed, Feb 5, 2025 at 11:26?AM David Armour via <dave.armour=[email protected]> wrote:
|
|
Re: QQQ 1 Minute Data >6years?
I tried setting the wait up to 10mins when you suggested it before. It just timed out after 10mins. No data is being sent and frankly waiting 10mins is ridiculous. The first set of data is always sent in less than 1 second in my experience.
?
Also, I am following the historical data downloading pacing guidelines and 1 day is normally what is acceptable for 1 minute data. In the past I have asked for more only to see my request get slower and slower until it ground to a halt. This was due to soft pacing being enforced by IB. (It may not be the case now, but it certainly was in the past and I found 1 day at? a time to be a sweetspot.)
?
I get all other stock data just fine the way I do it. It is only QQQ that is having an issue.
?
? |
|
Re: QQQ 1 Minute Data >6years?
15 seconds isn't near long enough to wait for a reach that far back. for 2019 data, i believe it'd take several minutes (as in > 4) to start getting any reply messages. I'd also suggest you request more than 1 day at a time and storing the response on? your side since it takes so long? On Wed, Feb 5, 2025 at 11:16?AM David Armour via <dave.armour=[email protected]> wrote:
|
|
Re: QQQ 1 Minute Data >6years?
I get no data. Right now I am running a job starting from 2015-01-01 and working my way forward day by day. Up to now I get nothing returned and I am at 2015-06-01.
My code waits 15secs for the data download to start so it is a little slow when no data is returned. It is a latch using a conditional_variable. ?
Here is a snapshot of the log:
?
?
You can see it making the request, then 15secs later cancelling it and issuing a new request for the following day. |
|
Re: QQQ 1 Minute Data >6years?
What happens when you start the 1min data download with an end-date of 3rd June 2018 or 2nd June 2018?
?
On Wed, Feb 5, 2025 at 10:02 AM, David Armour wrote:
|
|
Re: QQQ 1 Minute Data >6years?
After getting the X11 running with XMing on the WSL setup, I checked the chart again and it is the same as Windows TWS, so the progress I thought I had made was not real.
?
Tried to load the data from the TWS Linux and have the same failure for QQQ at 5th June 2018. However, daily data is no problem for me all the way back to 1999.
?
Ready to stop, but would still like to hear if anyone sees the same problem on the 1 minute chart at that date as per my screenshots in the post below. Just want to be sure it is not a "me" problem. |
|
Re: QQQ 1 Minute Data >6years?
Hi. I ran TWS Linux in the Windows WSL and managed to get the charts to work properly at and before 5th June 2018. This is a step forward.
?
I am now going to run my program connecting to that TWS instead of the Windows one and will see if I can get 10y+ of data.
?
Making progress... |