¿ªÔÆÌåÓý

ctrl + shift + ? for shortcuts
© 2025 Groups.io
  1. Twsapi
  2. Messages
Expanded Topics Messages Polls
Date
Re: Historical hourly bars, why is 17:15 EST timestamped bar always included?

 

Whenever I request intraday historical data for currencies, there is always a 17:15 EST bar included, regardless of whether I request 1 hour, 4 hour or other compressions. Do I need to manually filter them out or am I doing something wrong with my request?

?? ? ? ? ? ? ? ? ? ? ? Open? ? ? High? ? ? ?Low? ? ?Close
Date? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
2021-05-25 17:15:00? 1.225025? 1.225775? 1.224920? 1.224985
2021-05-25 18:00:00? 1.224985? 1.225515? 1.224955? 1.225355
2021-05-25 19:00:00? 1.225355? 1.225390? 1.224610? 1.224700
2021-05-25 20:00:00? 1.224700? 1.225385? 1.224460? 1.224525
2021-05-25 21:00:00? 1.224525? 1.224845? 1.224235? 1.224380
2021-05-25 22:00:00? 1.224380? 1.225310? 1.224190? 1.224990
2021-05-25 23:00:00? 1.224990? 1.226060? 1.224975? 1.225805
2021-05-26 00:00:00? 1.225805? 1.226115? 1.225605? 1.225965
2021-05-26 01:00:00? 1.225965? 1.226295? 1.225550? 1.225585
2021-05-26 02:00:00? 1.225585? 1.225665? 1.224330? 1.224915
2021-05-26 03:00:00? 1.224915? 1.224985? 1.223275? 1.224430
2021-05-26 04:00:00? 1.224430? 1.224850? 1.223265? 1.224090
2021-05-26 05:00:00? 1.224090? 1.224710? 1.223755? 1.224505
2021-05-26 06:00:00? 1.224505? 1.224505? 1.223290? 1.223855
2021-05-26 07:00:00? 1.223855? 1.223975? 1.222515? 1.222690
2021-05-26 08:00:00? 1.222690? 1.222985? 1.221700? 1.222005
2021-05-26 09:00:00? 1.222005? 1.223485? 1.221770? 1.222830
2021-05-26 10:00:00? 1.222830? 1.223980? 1.221655? 1.221690
2021-05-26 11:00:00? 1.221690? 1.222125? 1.219735? 1.220345
2021-05-26 12:00:00? 1.220345? 1.220545? 1.219565? 1.219990
2021-05-26 13:00:00? 1.219990? 1.220195? 1.218195? 1.218750
2021-05-26 14:00:00? 1.218750? 1.219325? 1.218460? 1.219305
2021-05-26 15:00:00? 1.219305? 1.219625? 1.219075? 1.219210
2021-05-26 16:00:00? 1.219210? 1.219725? 1.219140? 1.219190
2021-05-26 17:15:00? 1.219075? 1.220005? 1.219020? 1.219450
2021-05-26 18:00:00? 1.219450? 1.219705? 1.219235? 1.219290
2021-05-26 19:00:00? 1.219290? 1.219440? 1.218885? 1.219210
2021-05-26 20:00:00? 1.219210? 1.219275? 1.218000? 1.218830
2021-05-26 21:00:00? 1.218830? 1.218840? 1.217525? 1.218505
2021-05-26 22:00:00? 1.218505? 1.219285? 1.218440? 1.219040
2021-05-26 23:00:00? 1.219040? 1.219455? 1.218640? 1.219365
2021-05-27 00:00:00? 1.219365? 1.219580? 1.219085? 1.219345
2021-05-27 01:00:00? 1.219345? 1.219695? 1.218825? 1.219565
2021-05-27 02:00:00? 1.219565? 1.220265? 1.218695? 1.219805
2021-05-27 03:00:00? 1.219805? 1.220840? 1.219490? 1.220580
2021-05-27 04:00:00? 1.220580? 1.221515? 1.218005? 1.218255
2021-05-27 05:00:00? 1.218255? 1.219625? 1.218220? 1.219315
2021-05-27 06:00:00? 1.219315? 1.220845? 1.219315? 1.220520
2021-05-27 07:00:00? 1.220520? 1.221075? 1.219260? 1.219625
2021-05-27 08:00:00? 1.219625? 1.220225? 1.218440? 1.218615
2021-05-27 09:00:00? 1.218615? 1.219935? 1.217970? 1.219755
2021-05-27 10:00:00? 1.219755? 1.221165? 1.219205? 1.220145
2021-05-27 11:00:00? 1.220145? 1.220355? 1.218575? 1.219365
2021-05-27 12:00:00? 1.219365? 1.219625? 1.218530? 1.218770
Re: reqHistoricalData() "formatDate" set to 2 does not return Unix Epochs

 

Thanks Josh
Re: reqHistoricalData() "formatDate" set to 2 does not return Unix Epochs

 

Unix (epoch) time is only available with the smaller bar sizes.?
reqHistoricalData() "formatDate" set to 2 does not return Unix Epochs

 

I am requesting historical data in python and for some reason setting "formatDate" to 2 does not return Unix epochs, setting it to either 1 or 2 returns the same string date, "YYYYMMDD" when requesting daily data. Is that intended to be like this?
Re: Black-Scholes-Merton - Risk Free Interest Rate and which Volatility value do I use?

 

Inflation rate is irrelevant, all you want (ideally) for European options like SPX, RUT, NDX are the actual interest rates being used by the market makers and the actual PVDivs (present-value-of-dividends) for each contract. You can use futures as you are saying for the quarterlies, but using that data to infer other expirations involves ignoring the true PVDivs, which can be quite lumpy so it's essential to do them expiration-wise if you are serious.
?
But, accurate rates and PVDivs turn out to be secondary for European options ... your target is really the forwards, and you can get accurate forwards using rather poor interest rates (like Libor) and with no assumptions about, or knowledge of, dividends. Good forwards are 90% of the problem, after that you can worry about improving your interest rates for the discount factors if you want to.
?
For American options, accurate interest rates are more important as they are key to the early exercise boundary determination.? ?
Re: Black-Scholes-Merton - Risk Free Interest Rate and which Volatility value do I use?

Matthias Frener
 

Might be off-topic, but G-Money seems to be someone that might be able to ansert, so I'll ask... :D :P

> Not sure what IB uses for risk free rates, however, industry standard is to create bootstrap a curve from various sources (LIBOR, futures, Swaps). This is how the Bloomberg Terminal creates their RFR curve for default option pricing.?
> A decent alternative is just to pull treasury par yields from the??(they also have an XML public API), bootstrap a zero coupon curve from these rates, and finally interpolate on this curve for the given maturity of your the options being modeled.?

Where does the inflation rate come into this game, or is just irgnored? I mean.. for a risk free rate I want the (treasury rate) - (inflation rate), as inflation is a risk.
We should have negate RFRs since months already, but we have not. Don't really get why... I buy the 10T with 1.4 rate at 5% infltation, I'm lossing money (risk free rate is negative in relativ).

>??Depending on your product of choice, arguably a more important input than rates is a decently accurate, forward-looking dividend rate ?(discrete or continuous).?
I actually use the Future contract closes to the Opton expiry, to derive all that values all once, as it should be priced arround (spot price) + (risk free rate + forward-looking dividend rate) as long as market conditions or good and arbitragers do their job.
Re: Requesting Historical Data

Nick
 

To clarify, you don't need to save the ticks. You just need to keep the one incomplete bar and adjust high/low/close with each tick. After that you don't need the ticks anymore.

On 6/15/2021 5:07 PM, Nick wrote:
Historical keep up to date will certainly work but it's rather chatty as you saw.. Try using one of the other realtime data streams instead. It's not that hard to accumulate data into 30sec bar and detect a new bar.

You don't need to save each tick to a database. Save the ticks until you get a new bar and then add the bar to the database.

So first request the historical data and then start the realtime data going.

On 6/15/2021 3:41 PM, Patrick C wrote:
1. Pull all the old historical data (once and done)

then seperately

2. Pull the current candle uptodate data into its own function. (endless)
Re: Requesting Historical Data

 

The IB API has grown over the years and while they would not design it exactly the way if they'd start over, it is no different than so many other modern asynchronous APIs. That's the way distributed collaborative computing is done.

So you don't "pull things" but you request individual data items or streams of data and continue with your other work (if any). Some time later you get a callback with the data you requested and, if it is a stream, callbacks keep coming until you cancel the request or your program terminates.

You will have to learn basic principles of asynchronous programming regardless of the language you are using. One place to start is the complete Python client application IB ships with the API. It does everything you want to do. Fire it up, play with it, and look at the supplied Python source to see how they are doing the things you want to do.

And, if you have not done so, read the API Reference Guide at . front to back.

You might decide that is too much work, I understand, but then IBKR API is not for you.

JR


On Tue, Jun 15, 2021 at 02:41 PM, Patrick C wrote:
So I want to try and do Dmitry's idea, and I've talked to him privately and he said he's a java guy. Can someone help me set this up with Python??

I tried to mess around with the UpToDate=True. It seems to me that when you do a request you?

1. Pull all the old historical data (once and done)

then seperately

2. Pull the current candle uptodate data into its own function. (endless)

When I tried to put the uptodate into a dataframe. It was just an endless spam of repeat time stamps and everything.?

And I'm not gunna lie, it melted my brain and made me want to just go back to the way I was doing it and just maybe try and work out these glitches by hand.?
Re: Requesting Historical Data

Nick
 

Historical keep up to date will certainly work but it's rather chatty as you saw.. Try using one of the other realtime data streams instead. It's not that hard to accumulate data into 30sec bar and detect a new bar.

You don't need to save each tick to a database. Save the ticks until you get a new bar and then add the bar to the database.

So first request the historical data and then start the realtime data going.

On 6/15/2021 3:41 PM, Patrick C wrote:
1. Pull all the old historical data (once and done)

then seperately

2. Pull the current candle uptodate data into its own function. (endless)
Re: Black-Scholes-Merton - Risk Free Interest Rate and which Volatility value do I use?

 

Treasuries are too low. The best easy, cheap thing is Libor, but that also tends to be too low when the Fed is near zero. Probably most people just use Libor ... download for free, translate the Act/360 to continuously compounded rates, and interpolate linearly. Libor only goes to 12 months though, after that you need to bootstrap or just get the rates directly from the options chains.
?
You can also get the IB/TWS rates directly ... those used to be pretty good, but got kind of lousy after they sold Timber Hill.
Re: Requesting Historical Data

 

So I want to try and do Dmitry's idea, and I've talked to him privately and he said he's a java guy. Can someone help me set this up with Python??

I tried to mess around with the UpToDate=True. It seems to me that when you do a request you?

1. Pull all the old historical data (once and done)

then seperately

2. Pull the current candle uptodate data into its own function. (endless)

When I tried to put the uptodate into a dataframe. It was just an endless spam of repeat time stamps and everything.?

And I'm not gunna lie, it melted my brain and made me want to just go back to the way I was doing it and just maybe try and work out these glitches by hand.?
Re: Black-Scholes-Merton - Risk Free Interest Rate and which Volatility value do I use?

 

Not sure what IB uses for risk free rates, however, industry standard is to create bootstrap a curve from various sources (LIBOR, futures, Swaps). This is how the Bloomberg Terminal creates their RFR curve for default option pricing.?

A decent alternative is just to pull treasury par yields from the??(they also have an XML public API), bootstrap a zero coupon curve from these rates, and finally interpolate on this curve for the given maturity of your the options being modeled.?

You will find, however, that rates are almost meaningless for many options, especially those with less than 6 months maturity. Depending on your product of choice, arguably a more important input than rates is a decently accurate, forward-looking dividend rate ?(discrete or continuous).?
Re: Black-Scholes-Merton - Risk Free Interest Rate and which Volatility value do I use?

 

¿ªÔÆÌåÓý

I believe most people use the 3-month Treasury Bills for the risk free rate.? That data is available for free from many data sources including

US Treasury Dept:?

Federal Reserve:

Quandl:? Don't have the link in front of me but it used to be there.

Yahoo:

I'd start with whatever source you generally use for data and go from there. ??



On 6/15/2021 10:27 AM, Don (KN3N) wrote:

I still need help, where do I get the??'Risk Free Interest Rate'?from?

Bruno indicated to pull it from online somewhere:?"I pull rates from the Internet (from overnight to 6 months or longer)"
I am guessing he was referring to the interest rate of the 0-6+ month us treasury bills?
And then calculates the Rate: "...fit a curve and I use the given rate for the expiration of interest."
I can handle the rate curve part once I get confirmation about the data source.

Can anyone help?

?
Re: Black-Scholes-Merton - Risk Free Interest Rate and which Volatility value do I use?

 

I still need help, where do I get the??'Risk Free Interest Rate'?from?

Bruno indicated to pull it from online somewhere:?"I pull rates from the Internet (from overnight to 6 months or longer)"
I am guessing he was referring to the interest rate of the 0-6+ month us treasury bills?
And then calculates the Rate: "...fit a curve and I use the given rate for the expiration of interest."
I can handle the rate curve part once I get confirmation about the data source.

Can anyone help?

?
Re: Requesting Historical Data

 

Ok, let me get back into the work shed and see what I can figure out. Thank you so much everyone
Re: Requesting Historical Data

Nick
 

No, there is no way to do this.

You have to save the previous bars and add a new bar as it completes.

On 6/15/2021 9:10 AM, Patrick C wrote:
Is there any way that once the current candle closes and becomes finished, my program receives it and the previous 100 candles automatically
Re: Requesting Historical Data

 

Hmm I think I didn't explain what I am working with correctly.. or maybe I cannot explain what I am working with correctly.

But, lets start over and try this!

#1: Is there any way that once the current candle closes and becomes finished, my program receives it and the previous 100 candles automatically no matter what the time is, no matter if the candle closes 500ms faster or slower then its "designated" time. No matter anything. Current candle closes, boom, last 100 candles gets pulled.?

Instead of doing: #2: At a specific time interval, I am pulling all the candles and going from there.

Am I correct in thinking those two are totally different things?

That #1 is based off "closure" of the candle and #2 is based off time?
Re: Getting Option Call Open Interest (tickId 27) and Option Call Volume (tickId 29)

 

Hello.

Is?there any other way to get Option Volume?


El s¨¢b, 29 de may. de 2021 a la(s) 03:20, Bruno Voisin (voisin.bruno@...) escribi¨®:
Same here.? I do receive OI and not Volume.? This issue is not a new one...

Re: Requesting Historical Data

Nick
 

You could also use reqRealTimeBars which sends you a completed bar every 5 seconds. You just combine them into 30-second intervals and take action on the completed bars.

As usual IB makes it harder than needed due to the subscription dropping sometimes so you have to monitor error messages and restart if needed.

I would also double check that your local clock is stable. PC clocks are notoriously bad and will not stay stable unless you force them to be, via updating from a reliable time server.

On 6/14/2021 9:55 PM, Patrick C wrote:
Anyone else deal with this issues and have some solutions on how to ensure your getting accurate candle ticks in the last "closed" candle?
Re: Requesting Historical Data

 

Hi Patrick,

The logic that you're suggesting with gangling time to send the request sounds like a bad design decision.
It should "just work" (c) no matter when you sent your request.
Matt's both suggestions represent a much better approach to solve your "incomplete last bar" problem.
Say for example you got this sequence:

image.png

and then you checked local time and it is 22:06:51 now, then you simply can calculate the "age" of the last bar, which?in this case is only 21 seconds, so you know your last bar is only 70% mature and you can decide the acceptable threshold (for example you might consider 95% mature bar as complete or if you want to be very strict, then simply require 100% maturity?or discard last bar as 'underaged').

But on the bigger picture: what's happening next?
Are you waiting?for the "perfect request time" to ask IB similar historical data request?for 30-seconds bars and to struggle with the last bar age again?
A Much smoother approach would be to set "keep up to date" to true, then after receiving the historical bars portion (and using the above "mature bar detector" logic) IB will continue sending you updates even after you received?historicalDataEnd(). Now you'll baby-sit a new baby-bar and you can simply keep track of the "most recent bar update" bar object stored locally. IB will keep sending you more bar updates with the same bar.time value within 30 seconds time window. Eventually after another 30-sec bar is complete you'll get a bar update with a new bar.time value, which you haven't seen before. You can use this event to determine that the previous bar that you've been tracking is now complete (no more updates from IB will come) and here you can engage your app logic to process freshly baked new bar (update charts, run your trading algo logic, update databases,?call friends, etc.)?
In this approach you:
? - don't even have to check your local time (nor ask IB's time, which is always worse idea),
? - don't have to even think about timing of your original request (you can start anytime),
? - don't need to create multiple requests since there is only one request
? - don't even care about your local time, you simply rely on IB's bar's timestamps (I'm not sure what happen for thin-traded instruments, but I guess you are after something liquid)
? - don't miss bar?completion due to the bad request timing and even have a chance to work on 95% complete bar before millions of other users waiting for the bar to end :-))

This should work for all bar sizes and also you now have much more granular knowledge of "real time" price, which opens a bunch of other possibilities.

Cheers,
Dmitry

On Mon, Jun 14, 2021 at 11:15 PM Matt173 <matt.wolf74@...> wrote:
Care to elaborate on your question then? Regardless of when you request the bars you can easily figure out whether the last returned bar is complete or not.?

6541 - 6560 of 53765