¿ªÔÆÌåÓý

ctrl + shift + ? for shortcuts
© 2025 Groups.io
  1. Twsapi
  2. Topics

Tips for working the the 5 second real time bars when using IBApi.EClient.reqHistoricalData with keepUpToDate = TRUE

 

Dear all,

I'm using the latest ibapi version 9.81.1 post1 and running a script using python in visual studio code.

Context:
After I receive all the historical bar data I request, I then start receiving the real time 5 second bar data because of the KeepUpToDate argument being set to TRUE. I am using 1 minute bar size.

The logic of my code is such:
When bar.date is different by 1 minute or more from the last bar.date, I take the bar data of the previous minute that just passed which is made of twelve 5 second bars and store the resulting bar.high, bar.close, bar.low in a dictionary.

The problem:
When the minute passes and i store the bar data in the dictionary, I am actually storing the data of the 1st second of the NEXT 1-minute-bar which is currently being formed and not storing the bar data of the just elapsed minute. So I end up with bar.close = bar.high = bar.low = bar.open which is the bar data of even less than 1 second.

My solution:
Take the bar data of the PRE-LAST 5-second bar in the minute which is from the 50th second to the 55th second aka the eleventh 5-second bar. Now you can probably see the shortcoming: How do I get the bar data of the last 5-seconds of the minute?

The workaround I am currently attempting:
use IBApi.EClient.reqHistoricalData with?keepUpToDate = FALSE every minute to request the bar data of the previous minute. I haven't really gotten this to work yet but I am searching for some insight because this problem feels like a mistake I am making rather a problem with ibapi because it is hard to believe otherwise.

Regards,
Nader.
 

First of all, 9.81.1 is not the latest TWS API version. :
  • Stable API version: is 10.19 (Release Date:Nov 16 2022)
  • Latest API version is 10.20 (Release Date Nov 4 2022)

You might want to move towards one of the Version 10 APIs.

I am not following your approach completely and am wondering whether you make it too complicated.

The API documentation does not say so explicitly, but "bar time" is the time for the open value of the bar (the start of the bar). The bar is assembled by IBKR at (or shortly after) "bar time + 5 seconds" and takes a few 100 milliseconds (up to a second) until it reaches you:

If bar time for a 5 second bar is is 14:44:55, for example:
  • OPEN price for the bar is the first trade at or after 14:44:55
  • CLOSE price is the last trade before 14:45:00
  • You receive the bar very shortly after 14:45:00 and this is the end of the 14:44 minute.
If you want to make your own 1minute bar from 5second bars, you'd:
  • Start with a bar that has a bar time of a full minute (1676321100 or 14:45:00 below) and use the OPEN price of that bar as the open price for your 1 minute bar
  • Remember low and high from the first bar as well as volume and count if you are interested in those.
  • For the next eleven bars (e.g. bars with a time of < first bar time + 60 seconds) check whether their low is lower or high is higher and add up volume and count.
  • Once you have received bar #12 you can finish your 1 minute bar from OPEN, the lowest low you found, the highest high you found, the sum of volumes and counts, and the CLOSE price of bar #12

You should have that 1minute bar in hand a second or less after the end of the minute.

Hope this helps,

´³¨¹°ù²µ±ð²Ô
 

I completely agree with you and I updated TWS to 10.19 although it took me 6 hours to figure out how to do that but nevertheless a success. However I am still facing the same problem but your answer gave me insight to the following:
?
I am in fact receiving the entire 1 minute bar data, however it is only being received for such a short time that by the time my code runs, it would have skipped the 1 minute data and instead use the millisecond bar data of the next minute.

An example to the problem:
The following python code is only a small piece of the entire code. This part runs when I begin receiving real time data. it is pretty self explanatory. When bar.date changes aka a new minute begins, append the dictionary with the bar data and await a new bar.date.

if self.bar_date != bar.date:
? ? ? ? ? ? ? ? self.bar_date = bar.date #self.bar_date is the bar.date of the current minute, so when bar.date changes, the code runs.
? ? ? ? ? ? ? ? #New bar append
? ? ? ? ? ? ? ? self.currentBar.close = bar.close
? ? ? ? ? ? ? ? self.currentBar.high = bar.high
? ? ? ? ? ? ? ? self.currentBar.low = bar.low
? ? ? ? ? ? ? ? self.currentBar.date = bar.date
? ? ? ? ? ? ? ? add_to_bars = {'close': self.currentBar.close, 'high': self.currentBar.high, 'low': self.currentBar.low, 'date': self.currentBar.date}

Printing the current time using the datetime function and the realtime 5 second bar data on an infinite loop just to figure out what was happening resulted in the following:

New Bar!
TSLA
2023-02-15 21:34:00.042966
{'close': 212.78, 'high': 212.86, 'low': 212.63}

New Bar!
TSLA
2023-02-15 21:34:00.888854
{'close': 212.8, 'high': 212.86, 'low': 212.63}

New Bar!
TSLA
2023-02-15 21:34:01.044003
{'close': 212.8, 'high': 212.8, 'low': 212.8}

New Bar!
TSLA
2023-02-15 21:34:01.294376
{'close': 212.8, 'high': 212.8, 'low': 212.8}

So the problem here is that my dictionary is containing {'close': 212.8, 'high': 212.8, 'low': 212.8} instead of {'close': 212.8, 'high': 212.86, 'low': 212.63}.

My solution to this problem:
?At the 55th second of every minute, start infinitely appending the bar data to a list. When bar.close = bar.low = bar.high, stop the infinite loop and take the previous bar data and THAT would contain the correct bar data for the full minute.

Does someone have a better solution?
 

You can also just ask for 1 minute bars.? Here's an example,?https://stackoverflow.com/a/62800202/2855515
 

to naderharb.m@
Not sure I understand where you get your time from when you write "2023-02-15 21:34:00.888854",? look to me like your machine time ?
would have been good to see also the Bar time supplied by IB in the print log of your run.

Sorry but still not sure I understand your issue.
Because the date of the bar is supplied with the Bar and uniquely identify the Bar (even with KeepUpToDate) from the experience I have with it.
This can be seen in the post from Jurgen, then just listen to bar_date. The trigger event to close your dict with previous one is when you get the xx:xx:00 bar, then assume your last xx:xx:55 is the correct last one for the last 5 sec. As Jurgen point it out, the math does work.

Avoid checking h=l=c which is an unsafe practice. I am even surprised that it works on instrument like TSLA (If you did it at 2100 EST then yes there is not that many transactions)
I do not assume that IB deliver an updated RT bar with every ticks.

Aside of calling for 1m bar, an alternative is to use Ticks.? (you are already dealing with 5 sec Bar with KeepUpToDate so pacing limitation start to be an issue anyway)
Ticks looks like arriving reasonably ordered with market, within the supplied time in second that mark them, so your not very wrong if you timestamp them using machine time truncated from each second.
With Ticks you can filter it doing whatever you want to build your bars set.

But somewhere it's back to same issue, you need to trust IB time marker (and it seems that they build it as it should)
 

look to me like your machine time ?
yes it is the machine's time.

Responding to everything else:
My original method was exactly as you said which is to listen to bar.date and when a new bar.date is received, take the bar data with it. And here lied the problem I am facing which was that the bar data coming with the new bar.date every minute would have bar.close = bar.high = bar.open.?So the reason for printing out my computer's time alongside the bar data in a continuous loop was to diagnose the problem to see what was happening. What I found out was that the minute's bar data was being received sometime in the 59th second or the 1st second of the next minute. But that of course depended on my machine's time and how much it differed with ib servers. Regardless though, you can see the problem from my post with the example or in this example where I am printing out bar data alongside my computer's time:

2023-02-16 19:41:58.769470
{'close': 211.04, 'high': 211.51, 'low': 210.93, 'date': '20230216 19:41:00 Europe/Berlin'}

2023-02-16 19:41:59.221647
{'close': 211.17, 'high': 211.51, 'low': 210.93, 'date': '20230216 19:41:00 Europe/Berlin'}

2023-02-16 19:41:59.270578
{'close': 211.2, 'high': 211.2, 'low': 211.2, 'date': '20230216 19:42:00 Europe/Berlin'}


59.270578 - 59.221647
0.05 second window to grab the 1 minute bar before it starts building the next bar. Irrelevant though but just an observation.

So with this method, I verified my problem that when I receive a new bar.date, I also receive with it wrong bar data regardless of my computer's time.

So now I just collect bar data between the 59th second and the 1st second of the next minute and when I receive a bar with high = close = low, then the correct bar for the minute is the previous bar. Now I am just worried about data limitations in the future since this solution feels like a workaround and requests a lot of ticks in the small 3 second window.
But somewhere it's back to same issue, you need to trust IB time marker (and it seems that they build it as it should)
I don't trust it.
 

Thanks for the print out, It clarifies that I still don't understand.
What I see in your print seems relevant to me.

1- What is wrong with the 2023-02-16 19:41:59.221647 bar?
It closes at 211.17 as it is event driven, it's safe and should also be the close of the 1m bar.
Are you assuming here that the 1m bar @2023-02-16 19:41:00? would have a different close than the close of the last 5Sec bar time stamped as 2023-02-16 19:41:55 ?
Is it so?
That would be surprising.

2-?? ?The Bar API is event driven,
So, your code is activated (or is supposed to) on EVERY bar that IB send.? So putting the 5S bar on hold is easy.

3- You don't have to care about the 50 ms available to catch the Bar, can even a lot less than 50ms, IB do the heavy lifting it for you (assuming you can listen reasonably fast to their events)
Seems to me that you expect too much from Bar data and should consider using Ticks that are the instrument specially tailored to access a higher level of granularity and precision.

4- Assuming you read your machine time AFTER you catch the bar then I am surprised that you catch a first 20230216 19:42:00 bar at 2023-02-16 19:41:59.270578 (your machine time)
Your clock is probably off by 700 ms (which is already very good even with sync without hardware RT clock)

In my experience IB is relevant as much on the close value, as on the time base they use.
And unfortunately, you should trust IB because IMHO trying to workaround to extract more juice from these data may lead to problems.
You are already dealing with an approximation as the way sub-second Bar are built, across multiple exchanges, is more an art than a science as it's subject to many approx.

Try trusting IB, Bar API is a fundamental and basic, they have this working for decade
or/and
Try Ticks
 

1- What is wrong with the 2023-02-16 19:41:59.221647 bar?
I do not know how to ONLY listen and await it.

In the example in my previous post, initially, I would constantly listen for a new bar.date and append the bar data into a dictionary which would result in?{'close': 211.2, 'high': 211.2, 'low': 211.2, 'date': '20230216 19:42:00 Europe/Berlin'} instead of {'close': 211.17, 'high': 211.51, 'low': 210.93, 'date': '20230216 19:41:00 Europe/Berlin'}

Emphasizing on 'date', you could see why the logic of awaiting a new bar.date would fail because the minute's bar data in this example is associated with 19:41:00 and not 19:42:00 and I always thought before making this thread that the 19:42:00 bar would have??{'close': 211.17, 'high': 211.51, 'low': 210.93 ... } which is not the case.

So, I see that my amateur skills in python are the problem here and my question has changed:
How should the logic of my code be so that I would ONLY receive the 1 minute bar instead of my workaround solution? What are the relevant ibapi variables I should use here? Because it is apparent now that RELYING ON bar.date is not the way to go about it.
 

1- Every new Bar is a new era.
It may happens that no Bar are available immediately (immediately is a concept) understand that it may take couple of ms or even seconds until a real first transact occurs
(rare case during RTH)
Anyway: nothing of a previous Bar like the Bar xx:xx:41 should be expected in first bar xx:xx:42.

2- Yes IB API is "Event Driven" and you should use this feature.
It is extremely powerful , way more powerful? than a loop or a REST API. as it only bother your processor when something relevant occur.
If not using multi-thread you should do a maximum of processing when the event triggers. Not a a relax mode loop.

I am not familiar enough with Python to give safe advice with it. Structurally your facing a situation that require multi-threading, however IB does already hide one thread for you.
So you can do a lot of thing linearly.
Technically it seems that speed is of the essence for you, I guess Python is not suited for such a case.
People doing fast thing use C or C# or Java, using multi-threading capabilities.

Suggestion:
A- Restore your confidence in IB Bar by doing a dumb but fast save of 100% of Bars in a CSV file.
Then do like what Jurgen did, use Excel to understand that it works, how it works, and think about your options.

B- I assume that you don't' want to use the self.reqHistoricalData probably because you feel it would take more time to get it than being subscriber.
In your situation with Python I am not sure about that, you should try. but Yes, In my experience sometime IB take ages to answer reqHistoricalData . Probably because the underlying concept is to use it within Human UI, like in TWS where you may wait 10 seconds to see a chart but once displayed, price is updated RT.

C- I guess that even in Python you might be able to store all previous Bar in a dict and take decision(action) when triggered by the new Bar event when you detect a date change.
The date IB use in a Bar is as powerful as an index. You can use it as a signature of the Bar you look at.
Even if you lose a 100ms doing this aggregation this is not so critical, you won't miss so many next Bar or the essence of it.

D- You can split your code in 2 codes that should run concurrently using a DB as a data bridge.
a "Client" code that only subscribe to Bars and feed a DataBase on every event (there are many Python DB solution for that).
a "Processor" code that look in the DB and does take decision when a Bar with a new date arrive. (like assembling another Bar from the whole previous min)
This approach use the DB internal lock mechanism to synchronize access. (every DB? have internal lock for that) so that you don't have to learn "mutex" and "cond_var".
The flaw of this approach is that in your Processor cod you don't have access to the event, you will need an Infamous "Loop", but that's not so bad for starting experimentation.

Roger on that for me,I think that this thread goes beyond a direct answer to the issue raised.

1 - 9 of 9