¿ªÔÆÌåÓý

ctrl + shift + ? for shortcuts
© 2025 Groups.io
  1. Twsapi
  2. Topics

Requesting Historical Data

 

Heya, I am running a strategy right now on the 30 sec chart. But this question can go with any time period, I can't be the only one wondering. I am just pulling historical candles and loading them into a Pandas Dataframe. Then I run logic over the candles in the dataframe to determine to buy or sell. The software has to be started within 5 secs of the newest candle, to NOT pull the current candle. To pull all the candles up to the previous closed one. If I start the program like at +8 seconds over the 1:00 or 1:30, then the last entry in the dataframe is never correct, like IB is pulling me the current active candle. I think the sweet spot is to get the program to start pulling candles +1,+2,+3 seconds after the 1:00 or 1:30 marks respectively.?

Does anyone else jump through this hoop? Today was super weird with the data, it was like I was getting weird "phantom ticks" that register on the chart but are not registering in my dataframe or historical data I am pulling. Which is making me look a suspicious eye at IB because the program was running flawless after 300+ trades all last week... then today its not reading the candles correctly.?

I posted a few days ago, the IB clock was like 17 secs fast on that specific day. Now after months of running my program, I know the IB clock bounces around with how fast or how slow it is by a few seconds every day.

Anyone else deal with this issues and have some solutions on how to ensure your getting accurate candle ticks in the last "closed" candle??
 

And just cuz I know its gunna be asked, keep up to date is False
 

Well you can always request the server time right after you receive the historical data end flag and then ignore the last candle that is incomplete. Or just set the endDateTime on an even 30 seconds which ensures you get the date up to that exact timestamp. Or am I misunderstanding your question??
 

Yeah, I think your misunderstanding. There's like a window of 1-3 seconds after the start of each candle that is like a chink in my armor that I'm trying to figure out how to get rid of.?

Its like there's 3 seconds... which is a ton of time... for stuff to sneak through. There's gotta be a better way!
 

Care to elaborate on your question then? Regardless of when you request the bars you can easily figure out whether the last returned bar is complete or not.?
 

Hi Patrick,

The logic that you're suggesting with gangling time to send the request sounds like a bad design decision.
It should "just work" (c) no matter when you sent your request.
Matt's both suggestions represent a much better approach to solve your "incomplete last bar" problem.
Say for example you got this sequence:

image.png

and then you checked local time and it is 22:06:51 now, then you simply can calculate the "age" of the last bar, which?in this case is only 21 seconds, so you know your last bar is only 70% mature and you can decide the acceptable threshold (for example you might consider 95% mature bar as complete or if you want to be very strict, then simply require 100% maturity?or discard last bar as 'underaged').

But on the bigger picture: what's happening next?
Are you waiting?for the "perfect request time" to ask IB similar historical data request?for 30-seconds bars and to struggle with the last bar age again?
A Much smoother approach would be to set "keep up to date" to true, then after receiving the historical bars portion (and using the above "mature bar detector" logic) IB will continue sending you updates even after you received?historicalDataEnd(). Now you'll baby-sit a new baby-bar and you can simply keep track of the "most recent bar update" bar object stored locally. IB will keep sending you more bar updates with the same bar.time value within 30 seconds time window. Eventually after another 30-sec bar is complete you'll get a bar update with a new bar.time value, which you haven't seen before. You can use this event to determine that the previous bar that you've been tracking is now complete (no more updates from IB will come) and here you can engage your app logic to process freshly baked new bar (update charts, run your trading algo logic, update databases,?call friends, etc.)?
In this approach you:
? - don't even have to check your local time (nor ask IB's time, which is always worse idea),
? - don't have to even think about timing of your original request (you can start anytime),
? - don't need to create multiple requests since there is only one request
? - don't even care about your local time, you simply rely on IB's bar's timestamps (I'm not sure what happen for thin-traded instruments, but I guess you are after something liquid)
? - don't miss bar?completion due to the bad request timing and even have a chance to work on 95% complete bar before millions of other users waiting for the bar to end :-))

This should work for all bar sizes and also you now have much more granular knowledge of "real time" price, which opens a bunch of other possibilities.

Cheers,
Dmitry

On Mon, Jun 14, 2021 at 11:15 PM Matt173 <matt.wolf74@...> wrote:
Care to elaborate on your question then? Regardless of when you request the bars you can easily figure out whether the last returned bar is complete or not.?

Nick
 

You could also use reqRealTimeBars which sends you a completed bar every 5 seconds. You just combine them into 30-second intervals and take action on the completed bars.

As usual IB makes it harder than needed due to the subscription dropping sometimes so you have to monitor error messages and restart if needed.

I would also double check that your local clock is stable. PC clocks are notoriously bad and will not stay stable unless you force them to be, via updating from a reliable time server.

On 6/14/2021 9:55 PM, Patrick C wrote:
Anyone else deal with this issues and have some solutions on how to ensure your getting accurate candle ticks in the last "closed" candle?
 

Hmm I think I didn't explain what I am working with correctly.. or maybe I cannot explain what I am working with correctly.

But, lets start over and try this!

#1: Is there any way that once the current candle closes and becomes finished, my program receives it and the previous 100 candles automatically no matter what the time is, no matter if the candle closes 500ms faster or slower then its "designated" time. No matter anything. Current candle closes, boom, last 100 candles gets pulled.?

Instead of doing: #2: At a specific time interval, I am pulling all the candles and going from there.

Am I correct in thinking those two are totally different things?

That #1 is based off "closure" of the candle and #2 is based off time?
Nick
 

No, there is no way to do this.

You have to save the previous bars and add a new bar as it completes.

On 6/15/2021 9:10 AM, Patrick C wrote:
Is there any way that once the current candle closes and becomes finished, my program receives it and the previous 100 candles automatically
 

Ok, let me get back into the work shed and see what I can figure out. Thank you so much everyone
 

So I want to try and do Dmitry's idea, and I've talked to him privately and he said he's a java guy. Can someone help me set this up with Python??

I tried to mess around with the UpToDate=True. It seems to me that when you do a request you?

1. Pull all the old historical data (once and done)

then seperately

2. Pull the current candle uptodate data into its own function. (endless)

When I tried to put the uptodate into a dataframe. It was just an endless spam of repeat time stamps and everything.?

And I'm not gunna lie, it melted my brain and made me want to just go back to the way I was doing it and just maybe try and work out these glitches by hand.?
Nick
 

Historical keep up to date will certainly work but it's rather chatty as you saw.. Try using one of the other realtime data streams instead. It's not that hard to accumulate data into 30sec bar and detect a new bar.

You don't need to save each tick to a database. Save the ticks until you get a new bar and then add the bar to the database.

So first request the historical data and then start the realtime data going.

On 6/15/2021 3:41 PM, Patrick C wrote:
1. Pull all the old historical data (once and done)

then seperately

2. Pull the current candle uptodate data into its own function. (endless)
 

The IB API has grown over the years and while they would not design it exactly the way if they'd start over, it is no different than so many other modern asynchronous APIs. That's the way distributed collaborative computing is done.

So you don't "pull things" but you request individual data items or streams of data and continue with your other work (if any). Some time later you get a callback with the data you requested and, if it is a stream, callbacks keep coming until you cancel the request or your program terminates.

You will have to learn basic principles of asynchronous programming regardless of the language you are using. One place to start is the complete Python client application IB ships with the API. It does everything you want to do. Fire it up, play with it, and look at the supplied Python source to see how they are doing the things you want to do.

And, if you have not done so, read the API Reference Guide at . front to back.

You might decide that is too much work, I understand, but then IBKR API is not for you.

JR


On Tue, Jun 15, 2021 at 02:41 PM, Patrick C wrote:
So I want to try and do Dmitry's idea, and I've talked to him privately and he said he's a java guy. Can someone help me set this up with Python??

I tried to mess around with the UpToDate=True. It seems to me that when you do a request you?

1. Pull all the old historical data (once and done)

then seperately

2. Pull the current candle uptodate data into its own function. (endless)

When I tried to put the uptodate into a dataframe. It was just an endless spam of repeat time stamps and everything.?

And I'm not gunna lie, it melted my brain and made me want to just go back to the way I was doing it and just maybe try and work out these glitches by hand.?
Nick
 

To clarify, you don't need to save the ticks. You just need to keep the one incomplete bar and adjust high/low/close with each tick. After that you don't need the ticks anymore.

On 6/15/2021 5:07 PM, Nick wrote:
Historical keep up to date will certainly work but it's rather chatty as you saw.. Try using one of the other realtime data streams instead. It's not that hard to accumulate data into 30sec bar and detect a new bar.

You don't need to save each tick to a database. Save the ticks until you get a new bar and then add the bar to the database.

So first request the historical data and then start the realtime data going.

On 6/15/2021 3:41 PM, Patrick C wrote:
1. Pull all the old historical data (once and done)

then seperately

2. Pull the current candle uptodate data into its own function. (endless)
 

also, consider?reqHistoricalData with?with keepUpToDate = true:??

as a result, you don't need to update your candles.

?

?

1 - 15 of 15