¿ªÔÆÌåÓý

ctrl + shift + ? for shortcuts
© 2025 Groups.io

How to get longer historical data?


 

Hi,guys,I'd like to get more than 1 week 3min historical FX data to backtest.
According to documents, there's only 1 week long of 3min historical data.?
Is anyone who solved the problem?

thanks,
--


------
Forex trader
David Liao


 

David,

Simply repeat your?() call with "endDateTime" parameter value changed to the time of the oldest bar you received - you will get next chunk of bars. You can repeat your requests (mind: request rate limits) until you get enough (or hit "no data" error from IB).

May be also see:


Cheers,
Dmitry Shevkoplyas

On Tue, Dec 29, 2020 at 2:23 PM <david03kimo@...> wrote:
Hi,guys,I'd like to get more than 1 week 3min historical FX data to backtest.
According to documents, there's only 1 week long of 3min historical data.?
Is anyone who solved the problem?

thanks,
--


------
Forex trader
David Liao


 

Hi,Dmitry,

It¡¯s great.Thanks for your great help.
--


------
Forex trader
David Liao


 

Hi David,

For the??() call I would recommend to use "formatDate" argument value 2
This will make incoming bars timestamp to be in the format, which is same for all the time zones on the planet and epoch time does not care about summer daylight saving time jumps and it brings other sanity aspects and order back into our lives.

Find some way to convert epoch time to whatever format you need, for example for the next reqHistoricalData() you'll need "endDateTime" argument?to have format "yyyyMMdd HH:mm:ss {TMZ}" (example: "20210113 23:59:59 GMT").

Cheers,
Dmitry

On Wed, Jan 13, 2021 at 4:50 AM <david03kimo@...> wrote:
Hi,Dmitry,

I got a problem and can you guide me?
I used the code below and everything went well except the timezone.I queried my local time and sent back UTC time.How shall I modify the timezone?I cannot find in the API document.
self.QueryTime=datetime.strftime(self.FX_df[reqId]['DateTime'][0],'%Y%m%d %H:%M:%S %Z')
--

Thanks and Cheers,
------
Forex trader
David Liao


 

Hi,Dmitry,

Thanks for your recommendation of value 2.I checked my formateDate and glad to find mine is value 2 copied from IB's video courses.
And I used oldest timestamp of data-frame in?"20210113 23:59:59 GMT" format to call next?reqHistoricalData() and I got 42 times data and idled.
How can I get a feedback of no data to close it or change another pair of FX to call?reqHistoricalData()?Now I use 42 to get the end.

Cheers,
--


------
Forex trader
David Liao


 

David,

Why IB returns only 42 historical bars on your request - that depends. It might be one of the cases:
? - you only asked for that much, so they deliver
? - you've reached end of available historical data for given bar size
? - your app is crashing somewhere along the way and didn't have a chance to process the rest

What are the "bar size" and "duration" values you use for reqHistoricalData() request? There is a table of the max possible duration values for different bar sizes that IB would allow for your request in this question:?

IB API never left my reqHistoricalData hanging or unanswered.
The call to?reqHistoricalData() will lead to the?resulting bars returned in form of?int reqId, Bar bar
callbacks. The very last callback from IB will deliver Bar object with time string which?starts with "finished", which will trigger
??callback.?

Basically you need to implement 2 EWrapper methods:
? 1) historicalData() - to get next bar and store it somehow
? 2)?historicalDataEnd() - to consider request complete / done

Well and of cause "3)" for the error case when IB server response to your?reqHistoricalData() request: error number 162 with text "HMDS query returned no data", which would mean there's no point to try to dig any deeper in time.

Also note: you can use the reqHeadTimeStamp method to find the timestamp for the earliest data available for the contract.

Cheers,
Dmitry Shevkoplyas

On Wed, Jan 13, 2021 at 2:55 PM <david03kimo@...> wrote:
Hi,Dmitry,

Thanks for your recommendation of value 2.I checked my formateDate and glad to find mine is value 2 copied from IB's video courses.
And I used oldest timestamp of data-frame in?"20210113 23:59:59 GMT" format to call next?reqHistoricalData() and I got 42 times data and idled.
How can I get a feedback of no data to close it or change another pair of FX to call?reqHistoricalData()?Now I use 42 to get the end.

Cheers,
--


------
Forex trader
David Liao


 

Hi,Dmitry,

It works,I got 706 times reqHistoricalData() from now till 2007/07 3K bars data and get 162 errorCode to reversal append together,save it and plot it.
I am so surprised that IB provide so much data.I need 28 pairs of FX total historical data to backtest.It need 2 hours to get 1 pair total data.

Is IB API able to run two api concurrently to trade in paper account while getting data in the same paper account?I read the document that is possible to have different socket port number and client ID.But I didn't find any posts or goole about how to do it in practice.Is it to add two port 7497,7498 in the TWS API configuration and use different client id in API as app.connect('127.0.0.1',7497,0) and?? app.connect('127.0.0.1',7498,1) to concurrent 2 API run?So I can test my strategies in one API and get data from another API.

Thanks for your great help.
Cheers,
--
------
Forex trader
David Liao


 

David,

I'm glad it worked out for you.
Re: Is IB API able to run two api concurrently to trade in paper account while getting data in the same paper account?I?
- yes, you can run both paper and live (2 separate TWS applications) and configure them to use different API ports, but note - both TWS'es must run on the same computer.

Cheers,
Dmitry Shevkoplyas

On Fri, Jan 15, 2021 at 5:34 AM <david03kimo@...> wrote:
Hi,Dmitry,

It works,I got 706 times reqHistoricalData() from now till 2007/07 3K bars data and get 162 errorCode to reversal append together,save it and plot it.
I am so surprised that IB provide so much data.I need 28 pairs of FX total historical data to backtest.It need 2 hours to get 1 pair total data.

Is IB API able to run two api concurrently to trade in paper account while getting data in the same paper account?I read the document that is possible to have different socket port number and client ID.But I didn't find any posts or goole about how to do it in practice.Is it to add two port 7497,7498 in the TWS API configuration and use different client id in API as app.connect('127.0.0.1',7497,0) and?? app.connect('127.0.0.1',7498,1) to concurrent 2 API run?So I can test my strategies in one API and get data from another API.

Thanks for your great help.
Cheers,
--
------
Forex trader
David Liao


 

Hi,Dmitry,

It works.Thank you for your guidance.
I use TWS and IB gateway to run API concurrently.One is live account and the other is paper account with different port numbers and client ID, although?I don't know how to open 2 TWS on my mac.
But I can now test strategies and get historical data concurrently.


?Thanks and cheers,
------
Forex trader
David Liao


 

Hi David,

[Q] I don't know how to open 2 TWS on my mac.
[A] Try officially documented solution:




They suggest installing 2 TWS flavours on your mac:?
? ? - one "auto-updating"?
? ? - and another "offline",
then they show how to run both at the same time.?

If you want exact same TWS apps running in parallel: I recall on mac (I quit using it a couple years ago) I was able to run a few TWS apps in parallel and all of them were based on the same "offline" flavour downloaded only once, but if I recall correctly - I had to create a recursive copy of the folder where TWS was installed and tweak some of the TWS start bash scripts. My point is - imho it is also doable, but I don't remember all the exact steps/words from that spell..

Cheers,
Dmitry Shevkoplyas

On Sun, Jan 17, 2021 at 10:41 PM <david03kimo@...> wrote:
Hi,Dmitry,

It works.Thank you for your guidance.
I use TWS and IB gateway to run API concurrently.One is live account and the other is paper account with different port numbers and client ID, although?I don't know how to open 2 TWS on my mac.
But I can now test strategies and get historical data concurrently.


?Thanks and cheers,
------
Forex trader
David Liao


 

Hi,Dmitry,

Thank you for your sharing about details of how to run 2 TWS.
I learn a lot from your sharing.
I have downloaded whole 28 pairs of FX 3m from 2005 or 2008 till now.It's great help to backtest.
But I met trouble in get ticks or shall be 1sec history_data.Is it not allowed?

self.reqHistoricalData(self.count,contract,self.QueryTime,'30 mins','1 sec','MIDPOINT',0,2,False,[])

So I downed 1m,and it works.
? ? ? ? self.reqHistoricalData(self.count,contract,self.QueryTime,'1 D','1 min','MIDPOINT',0,2,False,[])

Thanks and Cheers,
?
------
Forex trader
David Liao