¿ªÔÆÌåÓý

ctrl + shift + ? for shortcuts
© 2025 Groups.io
Date

C++ reqHistoricalData() Inconsistent Responses

 

Hi,

I receive pretty inconsistent responses from IB using this call: m_pClient->reqHistoricalData(1001, m_contract, "", "1 D", "1 min", "TRADES", 1, 1, true, TagValueListSPtr());

There seem to be no issues w/ the m_contract setup, but nonetheless, it's set up is the following: symbol->'NQ', localSymbol->'NQZ3', exchange -> "CME", secType -> "FUT", lastTradeDateOrContractMonth -> "202312", currency "USD"

I am building responses and storing them in an std::unordered_map<std::string, std::variant<double, float, int, std::string>>, does not matter really, but the raw responses often come back with only data for the candle that just closed, and the current candle. 'time', 'open', 'high', 'low', 'close', 'volume' are the only columns I am interested in. (I am thinking of my unordered_map as a dataframe, hence the columns lingo, thanks pandas...)

I am removing the current candle so I am left with just the most recent closed candle, leaving me with 1 instead of 2 candles. Other times I get all candles of the current day. If it is 10:14:00 A.M. I get 44 candles once I sort by timestamp and remove duplicates. The useRTH, as seen above, is set to 1 so I only get from 9:30 onwards, no premarket etc. I do not know why this behavior occurs, has anyone else seen this or see why I may be getting inconsistent responses? I request in a loop every 60 seconds for the latest candle/information.

Thanks


Re: TWS queue buildup

 

There is no TWS API function to monitor the amount of unprocessed data in the socket that your client uses to connect with TWS API and I am not sure that would be a good approach in the first place. You just need to structure your client such that it is event driven and always ready to process TWS API callback function calls. Your callbacks:
  • should not directly perform very time consuming processing tasks
  • should not directly perform tasks such as file or database operations that might block the callback for extended or unbounded periods of time
  • and your client shall never call a sleep function.

Callbacks alert you about the arrival of new data and trigger processing (possibly at a later time). Simple and fast tasks can be performed right away, but anything more serious needs to be handled by a thread other than the callback thread. Every programming language has built-in or library add-ons that provide the tools you need for an event-driven client structure.

In this setup, your client can manage the data flow actively and by importance of the message:
  • it could drop certain messages by performing no task in the callback function (say you drop Bid/Ask or account value updates at peak times)
  • or it could buffer lower priority data in a queue for later processing
  • but it would always be ready to process important messages such as order updates, execDetails, and error messages or the most critical data your trading logic needs to be aware of immediately

We have never seen any backup issues and process in excess of 20,000 messages per second in peak situations.

´³¨¹°ù²µ±ð²Ô


Re: How to contact IBKR on weekend to unlock test userid

 

And this morning it mysteriously unlocked itself .. DOH.


TWS queue buildup

 

Hi:
as I add more symbols to my monitoring with real-time mkt data subscription, I start noticing that there's a queue buildup of tickPrice messages (and perhaps all other as well).
It shows itself as a delay in refreshing quotes in my application. When I stop my application, I can see a final "avalanche" of queued messages in the console output.
I guess I have a race condition somewhere in the code, I am going to look into it.
Meanwhile, do you know any way to monitor the ibapi incoming queue, so that I can detect a buildup and delay early?


How to contact IBKR on weekend to unlock test userid

 

Hi All,

IBKR locked me out of my TEST id for too many login attempts and I can't re-set without calling them.? This happened on a Sunday morning (I noticed it about 10 am Sunday NYT).? The only choice I seemed to have was to bring down TWS.? TWS logs you out and then keeps trying to re-log you in and eats up the valid attempts.? Really clever ...? I use auto re-start.? BTW, this doesn't happen all the time - I guess only when IB feels like it.? I can re-set the password (which is fine) but can't un-lock it.

I tried calling (in the US) 877-442-2757 and was told to call back during normal business hours.

Anyone else have a different number?

BTW, the PRODUCTION login also was locked out for too many attempts and I was able to re-set it on-line? - DOH?

What am I missing?

Thanks, as always,
Lou Dudka


Re: Basic question API for Excel : just get last price for a list of stocks

 

I managed to get something using RTD formula.?


Basic question API for Excel : just get last price for a list of stocks

 

Sorry, I am new to the API.

I have a serie of stocks in column, then a column with my purchase price and I just want the API to collect the last price to be able to compute variation since purchase. If possible I would like the stocks to be ordered in %variation descending order.

Can you give me guidance with how to do this ?

Thanks in advance.


Re: conflicting orderStatus messages

 

Indeed place and cancel is more expensive in terms of time, as it takes more to verify the order's been actually cancelled. And if skip the wait, I have seen multiple orders executed instead of one, because the cancellation never went through, but ended up with execution. On the other hand, modification is cheap, as the worst case scenario is modification rejection which is easy to handle and doesn't hurt me. Also considering the message flow cap, modification is a preferred choice. So far I mostly get errors 104 and 10148, and occasionally 103 too.


Re: limit orders can have status "Submitted" but not show up in Level2 or NBBO ?

 

Will do. ?I turned opted out of internalization Thursday morning. ?I believe that I was still seeing the issue after that, during the day Thursday, but I don't think I saw it this afternoon. ? I will watch for it next week and report back mid-week.


Re: limit orders can have status "Submitted" but not show up in Level2 or NBBO ?

 

Great, tell us please whatever you find out


El jue, 9 nov 2023 a las 16:04, <youngneal@...> escribi¨®:
Thanks, I didn't know about that.? I was opted in to internalization.? I'll try opting out and see how it goes.


Re: C++ ReqHistoricalData() Request to Response Time Data

 

Okay, nice to hear I am not alone in seeing this. I am requesting for NQ future e-mini data w/ these request details -> reqHistoricalData(m_reqHistDataId++, m_contract, "", "1 D", "1 min", "TRADES", 1, 1, true, TagValueListSPtr());
I am wondering if I should do the 5sec cancel idea, or your #2 suggestion, but only if this is a consistent problem going forward. I want to collect more data on this before doing something crazy. Not that your idea is crazy, but rabbit holes are always deepest when they do not need to be dug.


Re: conflicting orderStatus messages

 

On Thu, Nov 9, 2023 at 05:08 AM, bespalex wrote:
This is not manual. Of course, pre-submitted and submitted statuses are also coming when relevant.
What happened is a failed price modification attempt. PlaceOrder is sent with a new price and the same id for modification, it happens to be just executed, hence the rejection. But there should not be a cancellation as the original orderId is executed, but the 'new' duplicate one is rejected. This is my understanding it should be submitted first before it can be cancelled. And if we assume the order was somehow considered 'new' by IB there should be a duplicate id error instead.

You mean error 103 ?
I recommend that you have to live with the reported 'error' "Cancelled",? The reported error-ed situation from IB seems sound and usable.

Interesting case!
What you try to do is requiring a very complex analysis to decide that it is a 103 and at pretty fast pace considering all the thing that happens during resolution of the execution.

I am not surprised and I won't expect better. Here you dealt within less than 150 ms to: transmit/receive, reroute, choose BBO with a reservation process, execute and clear the order, with delay including exchange servers, while in the middle of that you call for a reprocessing of the same order.
Way faster is feasible but it's way more expensive.

Programmatically sound and easy (and this is even IB recommendation) would be that you cancel your order then do a new one with new pricing.
Consider that, it that may fit your need. And look at the flexibility IB offer to modify an existing as a way to improve performance (Update is cheaper than Delete+Insert), but with the risk that it create complex situation for IB like what you do which is a modification sent while execution is probably in the clearing phase.
Hence an error message as a rejection that I can understand as reflecting exactly what IB experienced, and not a clear cut 103.
BTW can even be more perilous with some order kind that require slack, look at error 440 or 129 (never tried an order that may generate it)


Re: C++ ReqHistoricalData() Request to Response Time Data

 

I does experienced similar things.
While being very difficult to frame it, I would say that it seems more hectic when you call explicitly for last bar.
It does behave like if IB take a long time to build the final 1min bar (If I have to guess, I would say that VWAP is the part that are difficult to gather to make it accurate enough)
Also seems stock specific. Some are harder to get.

I get the feeling that regularity seems better when either:
1- I call for bar, and set the keepUpToDate=true
or
2- I make my call at least 5 second after the end of the minute I specify trough enddate (5 sec or more using the time shifted as IB reqCurrentTime will suggest it),
otherwise said not trying to ask too early for the latest 1min bar (Which should be available straightaway I agree)
or
3- (decommissioned) If not answered within 1 sec I cancel the request and wait 5 sec then initiate a new request,
It behave like if you miss the opportunity to be serviced right-away, IB does put your request in limbo and service it as part of a 'garbage collection'.

Some answers are so slow that I can open a TWS chart and generally does experience the same sloppy display, so I guess that IB is struggling for IO or DB.

Note:
#3 above is pretty complex just to grab bars,
I discarded the code and switch to aggregation of data from a couple of other feeds to get bars in a timely manner, keeping IB requests running and overwriting result with the IB one when they deliver it.
I trust more IB data and they have their own VWAP formula, I am not sure they are right because I guess they only account full lot, but for sake of consistency I better keep the same data kind

I maybe wrong and miss something, but didn't have time to do further experiments.


Re: conflicting orderStatus messages

 

On Thu, Nov 9, 2023 at 05:20 AM, bespalex wrote:
And yes, I always get openOrder with Filled status after execDetails, even if with a few ms delay.
Ok,
I dodn't have same experience but I my understanding of the documentation doesn't suggest that what your experience is an impossible sequence.
I am under the impression that openOrder is a 'local traffic' (TWS/GTB to Client) hence may answer faster than execDetails which require a lot of remote actions.
I am not surprised that you get a "filled" status for the openOrder after the?execDetails ,
I am surprised that you don't get anything like "PreSubmitted" before, like as soon as you submit the placeOrder as IB doc implied it


Re: C++ ReqHistoricalData() Request to Response Time Data

 

Just looking for a frame of reference, I guess...


C++ ReqHistoricalData() Request to Response Time Data

 

Hi,

So, this morning I was watching my program run and request 1 minute candles using reqHistoricalData() every minute. Some times, the request to response time was small, ~ < 1 second, sometimes it was as much as 35 seconds. Has anyone done an investigation as to try and quantify average request to response times using TWS's C++ version of reqHistoricalData()? I am not hitting pacing limitations either, I am submitting 1 request every minute on the dot, so way less than 60 requests in a 10 minute span. I also am requesting real time hours only for the past day, so at most 500 candles. Usually only the number of candles of the current day so far, so less than 390 (6.5 hours a day).

Thank you


Re: limit orders can have status "Submitted" but not show up in Level2 or NBBO ?

 

Thanks, I didn't know about that. ?I was opted in to internalization. ?I'll try opting out and see how it goes.


Re: conflicting orderStatus messages

 

And yes, I always get openOrder with Filled status after execDetails, even if with a few ms delay.


Re: conflicting orderStatus messages

 

This is not manual. Of course, pre-submitted and submitted statuses are also coming when relevant.
What happened is a failed price modification attempt. PlaceOrder is sent with a new price and the same id for modification, it happens to be just executed, hence the rejection. But there should not be a cancellation as the original orderId is executed, but the 'new' duplicate one is rejected. This is my understanding it should be submitted first before it can be cancelled. And if we assume the order was somehow considered 'new' by IB there should be a duplicate id error instead.


Re: limit orders can have status "Submitted" but not show up in Level2 or NBBO ?

 

maybe it is submitted to IB's "marketplace" and the counterparty could be IB?
Check this:


El mar, 7 nov 2023 a las 17:36, <youngneal@...> escribi¨®:
Hi,

I occasionally have a limit order (for a stock) that IB says has status "Submitted", but that doesn't show up right away in Level 2 data (or NBBO), even though the limit price is strictly between the best bid and ask.? It can take several seconds, or even longer, for the limit order to show up in Level 2. ? Has anybody else seen this behavior, and if so does anybody know why it might be occurring?

I don't think it's that IB Level 2 data and NBBO are delayed at these times, because (a) I'm also monitoring Level 2 data and NBBO on other platforms (and not seeing the order their either), and (b) I think I've never had the order fill when this is happening.? I've seen it with orders submitted to various exchanges, so it doesn't seem to be exchange-specific.

Thanks, ?-Neal

[this post is crossposted here from /g/insync/message/9459,?after nobody replied there]

?

?