Keyboard Shortcuts
Likes
- Twsapi
- Messages
Search
Re: Downloading Large Amounts of Historical Data
Richard,
Thank you for the thoughts here, I will work on implementing this. I am interested in what dictates the length of time for the data to be returned from IB?? In most of my requests I receive a years worth of data in about 2-4s, however periodically it takes as long as 30s.? Is this due to other data requests the server is handling?? Or is there some throttling happening? J G, Thank you, and yes that is a much better solution than an arbitrary sleep time. Mike |
Re: trouble requesting trade data *and* quote data for three futures tickers
Nick
Leaky Buckets typically discard entries if the rate is exceeded which is probably not what you want in a trading application.
toggle quoted message
Show quoted text
If you are making a request you generally want it to be processed - preferably without delay but with a delay if necessary. On 2/16/2021 2:23 AM, Ray Racine wrote:
A simple yet effective rate limiter (if you end up needing here) would be to push requests into a FIFO queue and have a thread pull them out and send them rate limited by a Leaky Bucket Rate Limiter.? Very simple algo. |
Re: Downloading Large Amounts of Historical Data
On Tue, Feb 16, 2021 at 05:38 PM, <msaracena@...> wrote:
Your implementation seems to revolve around using time.sleep(). That is not the best approach as you don't know how long it will take for IB to respond to your request. IB will send an indication when it has sent you all data that you requested (or that it has available). You need to wait until you receive this message from IB and then move on to your next request. |
Re: C# Positions
Sorry I guess I missed that you already did what I suggested in my previous message. There is an obvious error in your code sample since you call?AddTextBoxItemPosition() but implement?AddTextBoxItem(). Other than that it's impossible to tell what is going on in your app. In fact it's not even clear which results you are getting already and which you miss.
|
Re: C# Positions
Your EWrapper interface implementation must include a valid implementation of the callbacks that the TWS uses to communicate positions in response to ClientSocket.reqPositions(), which are EWrapper.position(...) for data and EWrapper.positionEnd() for end of data. You need to use these callbacks to handle the responses.
|
Re: Source of the ticks timestamps
Nick
¿ªÔÆÌåÓýJust an additional tidbit, the sample interval is somewhat of a
moving target. According to
IB now
says the interval is 250ms for stock and futures, 100ms for
options and 5ms for fx pairs. But, you never know with IB what the
current value actually is. Richard has also made some detailed posts on historical data rate
limits which are also a moving target and not always properly
documented by IB. On 2/16/2021 6:56 AM, Richard L King
wrote:
|
Re: Source of the ticks timestamps
¿ªÔÆÌåÓýRealtime data returned by TWS from reqMktData() does not contain a timestamp. ib_insync itself adds the timestamp using your computer clock. So indeed there is no way at all to know when the tick was 'really' produced. ? Data returned from reqTickByTick() DOES include a timestamp, but it's limited to a resolution of 1 second. I don't know offhand if it's added by the IB servers or by TWS, but I think this has been discussed before so try searching. ? IBKR does not aggregate data returned from reqMktData: it samples it. If you compare the data returned from reqMktData with a source that definitely includes every tick, you find that every tick from reqMktData is in the full data, but not vice versa. ? The sampling mechanism is very simple. Time is divided into fixed length intervals of around 300 millisecs for stocks and futures and 100 millisecs for forex. For any given instrument, the IBKR market data server records each tick as it arrives. At the end of each period it sends the current value for each tick type to TWS, but only if it is different from the value set to the user at the end of the previous period. To enable this to happen, it must also keep a record of what it sent last time. ? This has implications: in particular, if the previous value for 'last price' was, say, 1500, and during the next period the exchange sends one or more trade reports with different values, then at the end of the period only the latest 'last price' is sent to the API, and if that happens to be the same as at the start of the period then no 'last price' is sent at all (but a 'last size' might be sent if that is different, and a 'volume' will be sent because that has increased). This mechanism explains how the reqMktData stream sometimes results in highs and lows of bars being incorrect by a tick or two: they just weren't sent because no change from the start of the 300ms period had been detexcted. ? Given that there are many tens of thousands of TWS users at any one time, many of whom will have large numbers of tickers running, this might sound like a massive amount to keep track of, but it's actually quite simple (though this is speculation ¨C I don't know for certain exactly how IBKR manage this). If the time period length is 300 millseconds, you could imagine that the IBKR data farm has 3 servers for a particular stock or future. When a user requests data for that instrument, he is allocated to one of these 3 servers, and all his subsequent data (for that instrument) come from that server. Thus that server only needs to record one set of previously sent ticks, not one for each user. And that server now has 300 milliseconds ?to send out each tick to all the users it services, so the load is fairly constant and predictable. Obviously one server would serve data from many different instruments, and conversely each instrument's data would be disseminated to by multiple servers, but the whole thing can be sliced and diced and load-balanced in such a way that the tick data stream is never more than 300 ms behind the market. ? When I first starting investigating this mechanism back in 2003, every market data line in TWS and every market data request via the API counted against your allocation of 100 tickers, even if you requested the same ticker more than once. This was presumably because the request was allocated to a different server, and in fact the data streams received were different.. But at some stage in the late noughties (I think) this was refined so that the same request from the same user was only counted once: so now, if I request data for a given instrument from several different API client programs, that is only counted as one market data line, and they all receive exactly the same data (and that even applies whether the data is requested via the live or the paper-trading account). ? Sorry for the lengthy reply, but I think it helps to have a clear picture of what is going on here. ? ? ? From: [email protected] <[email protected]> On Behalf Of Alex Gorbachev
Sent: 16 February 2021 04:45 To: [email protected] Subject: [TWS API] Source of the ticks timestamps ? Hi all. ? This must be a very noob question... I was trying to track the delay of real time data I get. I use the ib_insync Python module which makes things much easier in Python. ? First of all, ping to? takes about 27ms. ? For real-time bars I take my current system timestamp and then subtract?the bar time minus 5 seconds (bar size). That gives me 500-1000ms. ? For market data ticks, I subscribe using?reqMktData and the difference between time in the ticker (including all ticks in it - they have the same timestamp) and system time when I get them is only about?0.2ms (that's it - 200 microseconds). Now, that makes me think that ticker timestamp produced through?reqMktData subscription is actually generated in TWS (or IB Gateway which is what I use) and not on IBRK servers. Right? Thus, there is no way to say when those ticks were really produced. Is there? ? I haven't looked at TickByTick yet but I see that it has a very low limit on simultaneous subscriptions (3) so won't be useful for as many contracts as I need. ? Also, could someone point me on how exactly IBRK samples or aggregates ticks for?reqMktData? Do I every time just get aggregated ticks by type and price since the last update? ? Thanks, Alex ? |
Re: Downloading Large Amounts of Historical Data
¿ªÔÆÌåÓýFirst, use a different tickerID for each request. ? Second, don't wait for the result for one request before issuing the next. Just process the results as they arrive. They will quite likely arrive in a different order than the requests. The different ticker ids indicate which is which (you obviously have to maintain a map from ticker id to contract). ? Third, bear in mind that the API allows a number of concurrent historical data requests. I'm not sure offhand what the limit is, but I think it's 50. I tend to limit it to about 20, because I find you get diminishing returns with higher concurrency. ? Fourth, bear in mind that the API limits you to 50 input messages (ie API requests of all types) per second. Note also that that limit applies across all API clients currently running, ie 50 per second in total, not 50 per second per client. ? Put all that together, and you need to end up with something like this: ?
? This approach is non-trivial to code, but will give you maximum throughput. ? The simpler approach is indeed just to wait for each request to complete before submitting the next. But you don't need to actually make a thread sleep. Just send the next request when you receive a historical data end callback. ? Richard ? ? From: [email protected] <[email protected]> On Behalf Of msaracena@...
Sent: 16 February 2021 04:05 To: [email protected] Subject: [TWS API] Downloading Large Amounts of Historical Data ? My strategy requires access to large amounts of historical data - ~1000 symbols at 10 min bar resolution for 3 years that I save locally in csv.? I am pulling data through the reqHistoricalData method in 1 year batches and concatenating the resulting queries. Periodically I get "ERROR 1 322 Error processing request.-'bT' : cause - Duplicate ticker ID for API historical data query", because my script is moving to the next data request before the current one is complete.? app.reqHistoricalData(ticker_id, contract, query_end_iteration.strftime('%Y%m%d %H:%M:%S') + ' EST', query_period, |
Source of the ticks timestamps
Hi all. This must be a very noob question... I was trying to track the delay of real time data I get. I use the ib_insync Python module which makes things much easier in Python. First of all, ping to? takes about 27ms. For real-time bars I take my current system timestamp and then subtract?the bar time minus 5 seconds (bar size). That gives me 500-1000ms. For market data ticks, I subscribe using?reqMktData and the difference between time in the ticker (including all ticks in it - they have the same timestamp) and system time when I get them is only about?0.2ms (that's it - 200 microseconds). Now, that makes me think that ticker timestamp produced through?reqMktData subscription is actually generated in TWS (or IB Gateway which is what I use) and not on IBRK servers. Right? Thus, there is no way to say when those ticks were really produced. Is there? I haven't looked at TickByTick yet but I see that it has a very low limit on simultaneous subscriptions (3) so won't be useful for as many contracts as I need. Also, could someone point me on how exactly IBRK samples or aggregates ticks for?reqMktData? Do I every time just get aggregated ticks by type and price since the last update? Thanks, Alex |
Downloading Large Amounts of Historical Data
My strategy requires access to large amounts of historical data - ~1000 symbols at 10 min bar resolution for 3 years that I save locally in csv.? I am pulling data through the reqHistoricalData method in 1 year batches and concatenating the resulting queries. Periodically I get "ERROR 1 322 Error processing request.-'bT' : cause - Duplicate ticker ID for API historical data query", because my script is moving to the next data request before the current one is complete.?
What approach can be taken to ensure the script waits until the current request is complete before moving to the next? app.reqHistoricalData(ticker_id, contract, query_end_iteration.strftime('%Y%m%d %H:%M:%S') + ' EST', query_period, |
C# Positions
Trying to get current position (#'s of contracts) into a textbox. Working in c#.
Added this to Form1.cs? "ibClient.ClientSocket.reqPositions();" Added this to EWrapperImpl.cs in the position method code "myform.AddTextBoxItemPosition(pos);" Added this to Form1.cs - public void AddTextBoxItem(double pos) ? ? ? ? {
? ? ? ? ? ? if (this.tbConNumber.InvokeRequired)
? ? ? ? ? ? {
? ? ? ? ? ? ? ? SetTextCallback d = new SetTextCallback(AddTextBoxItem);
? ? ? ? ? ? ? ? this.Invoke(d, new object[] { pos });
? ? ? ? ? ? }
? ? ? ? ? ? else
? ? ? ? ? ? ? ? Convert.ToInt32(pos);
? ? ? ? ? ? {
? ? ? ? ? ? ? ? this.tbConNumber.Text = pos;
? ? ? ? ? ? }
? ? ? ? }
?Obviously I don't know what I am doing, I have somehow managed to get a connection, data, place orders but without the "positions" I am stuck.Any help is appreciated. Thanks |
Re: trouble requesting trade data *and* quote data for three futures tickers
A simple yet effective rate limiter (if you end up needing here) would be to push requests into a FIFO queue and have a thread pull them out and send them rate limited by a Leaky Bucket Rate Limiter.? Very simple algo. On Sat, Feb 6, 2021 at 10:39 AM Dmitry Shevkoplyas <shevkoplyas@...> wrote:
|
Re: Fx data
I have not worked with FX instruments, but the reqHeadTimestamp() API call should give you an idea of the earliest data you can expect (according to the documentation).
But don't expect to get data all the way back to what reqHeadTimestamp() indicates. I always read it as the "you will never get data older than that" timestamp. Some instruments reach back that far and some don't. |
Re: Fx data
There probably isn't data available any further back,?at least not in a form that you can easily grab. For most things, IB only goes back to around then. You could also buy historical fx data and similar from several sources, but?probably won't get any earlier than around 2005, either.? Historical data for the particular symbol I'm most concerned with only goes back to 2008, although it's been traded a lot longer than that.? On Mon, Feb 15, 2021 at 8:48 AM <ghelie@...> wrote: Hi, |
Re: Long position in TSCO mysteriously shortened this morning
I may be mistaken as to the ticker so do double check this. I'm also very dubious about Yahoo finance's data, it's generally deadly to rely upon it. The prices they serve may or may not be the intended series even if the ticker is correct. If the stock is adjusted for split then that would explain the historic adjustment, you'd see different price series depending on whether you pulled the data before or after the adjusted price was flushed back. It could also be a quirk of the paper trading system failing to allocate the split, it's an under developed version of the live system, so it does fall short sometimes. That might be worth a chat with a support person, they could tell you how the paper accoutn handles splits. Best wishes, M On Mon, 15 Feb 2021 at 11:19, Graham Bygrave <graham@...> wrote: Thanks. -- +44 (0) 7528 551604? Gulfstream Software - Winner Risk Management Awards 2010 This message is subject to : |
Re: Long position in TSCO mysteriously shortened this morning
Thanks.
That'll likely be it. I can see the split clearly in Yahoo. Looking at Google, there's no indication of the split - the price always appears as diluted, which would just imply they'd back adjusted it, which is fine. However, in my record of prices from last week, I have OHLC for TSCO.L at 244.20125223614,245.00123372173,237.99176624583,238.76299288089. And it wouldn't have been adjusted at that point. So I'm struggling to understand Yahoo's figures. On Thursday last week, Yahoo had the OHLC at 246.10001,247.922,243.52499,244.5,244.5. If there's been a 15 -> 19 split and the previous price was at ~304 then all is well and IB have just failed to allocate me the additional stock, they've just diluted my holding, which is an error by my reckoning (otherwise I'd surely just be able to short pending splits and profit). But that doesn't explain why my price history doesn't show > 300 anywhere for last week. Cheers, G. On Mon, 15 Feb 2021 10:45:40 +0000 "mark collins" <mark.collins@...> wrote: A bit more grubbing around shows a 15:19 split... |
Understanding placeOrder and its relation with used client ID
After some struggling, I think I finally understood why my orders weren't being updated with placeOrder:
- If the order was initially created with a different software using a certain client id (e.g. 11), I will be able to modify it if I use a connection with the same client id. - If I use a different client id, the order is not modified at all. - If I use the master client id, same behavior: order not modified - If the order was created manually at TWS, a new order is created, and then updated in next iterations, but the manually created order persists without changes. So, these are my questions: - Is it impossible to modify or cancel an order manually created at TWS from the API? - Is it impossible to modify or cancel orders initially created through the API but with a different client ID? - Is it impossible to keep two different connections using the same client ID? (almost sure that this is not possible, but just checking) |
Re: Long position in TSCO mysteriously shortened this morning
A bit more grubbing around shows a 15:19 split... M On Mon, 15 Feb 2021 at 10:44, mark collins via <mark.collins=[email protected]> wrote:
-- +44 (0) 7528 551604? Gulfstream Software - Winner Risk Management Awards 2010 This message is subject to : |