Hi,
I was trying to pull historical fx data yesterday and I was wondering if the data does not go back further than 2005 or if I did something incorrectly. I tried to pull all the fx data for around 7 currency pairs,
and for each of them, the gateway stopped returning bars around 2005.
Thanks
|
Re: Long position in TSCO mysteriously shortened this morning
I may be mistaken as to the ticker so do double check this.
I'm also very dubious about Yahoo finance's data, it's generally deadly to rely upon it. The prices they serve may or may not be the intended series even if the ticker is correct.
If the stock is adjusted for split then that would explain the historic adjustment, you'd see different price series depending on whether you pulled the data before or after the adjusted price was flushed back.
It could also be a quirk of the paper trading system failing to allocate the split, it's an under developed version of the live system, so it does fall short sometimes. That might be worth a chat with a support person, they could tell you how the paper accoutn handles splits.
Best wishes,
M
On Mon, 15 Feb 2021 at 11:19, Graham Bygrave < graham@...> wrote: Thanks.
That'll likely be it.? I can see the split clearly in Yahoo. Looking at
Google, there's no indication of the split - the price always appears
as diluted, which would just imply they'd back adjusted it, which is
fine.
However, in my record of prices from last week, I have OHLC for TSCO.L
at 244.20125223614,245.00123372173,237.99176624583,238.76299288089.
And it wouldn't have been adjusted at that point.? So I'm struggling to
understand Yahoo's figures.
On Thursday last week, Yahoo had the OHLC at
246.10001,247.922,243.52499,244.5,244.5.
If there's been a 15 -> 19 split and the previous price was at ~304
then all is well and IB have just failed to allocate me the additional
stock, they've just diluted my holding, which is an error by my
reckoning (otherwise I'd surely just be able to short pending splits
and profit).
But that doesn't explain why my price history doesn't show > 300
anywhere for last week.
Cheers,
G.
On Mon, 15 Feb 2021 10:45:40 +0000
"mark collins" <mark.collins@...> wrote:
> A bit more grubbing around shows a 15:19 split...
>
> M
>
> On Mon, 15 Feb 2021 at 10:44, mark collins via
> <mark.collins= [email protected]> wrote:
>
> > I attached a png of the series I thought you were speaking of. It
> > closed on the 12th north of 300, opened far lower.
> > I'm still not certain whether you are reporting position or P&L,
> > what was the P&L on the trade?
> >
> > M
> >
> >
> >
> > On Mon, 15 Feb 2021 at 10:09, Graham Bygrave <graham@...>
> > wrote:
> >> Hey,
> >>
> >> Not sure where you're seeing that fall.? My price stream shows me
> >> the close on Friday being around 239.3/245.2 and the open today
> >> being around 243.15/243.53.
> >>
> >> In any case, I didn't trade, so the position shouldn't have
> >> changed. Why would the position change?? I'm miles away from being
> >> called.
> >>
> >> Cheers,
> >> G.
> >>
> >>
> >> On Mon, 15 Feb 2021 09:59:15 +0000
> >> "mark collins" <mark.collins@...> wrote:
> >>?
> >> > A quick Google showed a large fall over the weekend. Could it be
> >> > that you are seeing the p&l change?
> >> >
> >> > M
> >> >
> >> > On Mon, 15 Feb 2021, 09:01 Graham Bygrave, <graham@...>
> >> > wrote:
> >> > > I held 2143 of TSCO.L last night (paper trading account) at the
> >> > > close. This morning on the open I have 1691 with the shortfall
> >> > > being booked in P&L as a loss (i.e. as though I'd just given
> >> > > them away).
> >> > >
> >> > > What's that about?
> >> > >
> >> > > Thanks,
> >> > > G.
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >?
> >> >
> >> >
> >> >
> >> >
> >> >?
> >>
> >>
> >>
> >>
> >>
> >>
> >>?
> >
> > --
> > +44 (0) 7528 551604?
> > Gulfstream Software - Winner Risk Management Awards 2010
> > This message is subject to :
> >
> >
> >
> >?
>
-- +44 (0) 7528 551604?
Gulfstream Software - Winner Risk Management Awards 2010
This message is subject to :
|
Re: Long position in TSCO mysteriously shortened this morning
Thanks. That'll likely be it. I can see the split clearly in Yahoo. Looking at Google, there's no indication of the split - the price always appears as diluted, which would just imply they'd back adjusted it, which is fine. However, in my record of prices from last week, I have OHLC for TSCO.L at 244.20125223614,245.00123372173,237.99176624583,238.76299288089. And it wouldn't have been adjusted at that point. So I'm struggling to understand Yahoo's figures. On Thursday last week, Yahoo had the OHLC at 246.10001,247.922,243.52499,244.5,244.5. If there's been a 15 -> 19 split and the previous price was at ~304 then all is well and IB have just failed to allocate me the additional stock, they've just diluted my holding, which is an error by my reckoning (otherwise I'd surely just be able to short pending splits and profit). But that doesn't explain why my price history doesn't show > 300 anywhere for last week. Cheers, G. On Mon, 15 Feb 2021 10:45:40 +0000 "mark collins" <mark.collins@...> wrote: A bit more grubbing around shows a 15:19 split...
M
On Mon, 15 Feb 2021 at 10:44, mark collins via groups.io
<mark.collins= [email protected]> wrote:
I attached a png of the series I thought you were speaking of. It
closed on the 12th north of 300, opened far lower.
I'm still not certain whether you are reporting position or P&L,
what was the P&L on the trade?
M
On Mon, 15 Feb 2021 at 10:09, Graham Bygrave <graham@...>
wrote:
Hey,
Not sure where you're seeing that fall. My price stream shows me
the close on Friday being around 239.3/245.2 and the open today
being around 243.15/243.53.
In any case, I didn't trade, so the position shouldn't have
changed. Why would the position change? I'm miles away from being
called.
Cheers,
G.
On Mon, 15 Feb 2021 09:59:15 +0000
"mark collins" <mark.collins@...> wrote:
A quick Google showed a large fall over the weekend. Could it be
that you are seeing the p&l change?
M
On Mon, 15 Feb 2021, 09:01 Graham Bygrave, <graham@...>
wrote:
I held 2143 of TSCO.L last night (paper trading account) at the
close. This morning on the open I have 1691 with the shortfall
being booked in P&L as a loss (i.e. as though I'd just given
them away).
What's that about?
Thanks,
G.
--
+44 (0) 7528 551604 gulfstream-software.com
Gulfstream Software - Winner Risk Management Awards 2010
This message is subject to :
|
Understanding placeOrder and its relation with used client ID
After some struggling, I think I finally understood why my orders weren't being updated with placeOrder:
- If the order was initially created with a different software using a certain client id (e.g. 11), I will be able to modify it if I use a connection with the same client id.
- If I use a different client id, the order is not modified at all.
- If I use the master client id, same behavior: order not modified
- If the order was created manually at TWS, a new order is created, and then updated in next iterations, but the manually created order persists without changes.
So, these are my questions:
- Is it impossible to modify or cancel an order manually created at TWS from the API?
- Is it impossible to modify or cancel orders initially created through the API but with a different client ID?
- Is it impossible to keep two different connections using the same client ID? (almost sure that this is not possible, but just checking)
|
Re: Long position in TSCO mysteriously shortened this morning
A bit more grubbing around shows a 15:19 split...
M
On Mon, 15 Feb 2021 at 10:44, mark collins via <mark.collins= [email protected]> wrote:
I attached a png of the series I thought you were speaking of. It closed on the 12th north of 300, opened far lower.
I'm still not certain whether you are reporting position or P&L, what was the P&L on the trade?
M
On Mon, 15 Feb 2021 at 10:09, Graham Bygrave < graham@...> wrote: Hey,
Not sure where you're seeing that fall.? My price stream shows me the
close on Friday being around 239.3/245.2 and the open today being
around 243.15/243.53.
In any case, I didn't trade, so the position shouldn't have changed.
Why would the position change?? I'm miles away from being called.
Cheers,
G.
On Mon, 15 Feb 2021 09:59:15 +0000
"mark collins" <mark.collins@...> wrote:
> A quick Google showed a large fall over the weekend. Could it be that
> you are seeing the p&l change?
>
> M
>
> On Mon, 15 Feb 2021, 09:01 Graham Bygrave, <graham@...> wrote:
>
> > I held 2143 of TSCO.L last night (paper trading account) at the
> > close. This morning on the open I have 1691 with the shortfall
> > being booked in P&L as a loss (i.e. as though I'd just given them
> > away).
> >
> > What's that about?
> >
> > Thanks,
> > G.
> >
> >
> >
> >
> >
> >?
>
>
>
>
>
--
+44 (0) 7528 551604?
Gulfstream Software - Winner Risk Management Awards 2010
This message is subject to :
-- +44 (0) 7528 551604?
Gulfstream Software - Winner Risk Management Awards 2010
This message is subject to :
|
Re: Long position in TSCO mysteriously shortened this morning
I attached a png of the series I thought you were speaking of. It closed on the 12th north of 300, opened far lower.
I'm still not certain whether you are reporting position or P&L, what was the P&L on the trade?
M
On Mon, 15 Feb 2021 at 10:09, Graham Bygrave < graham@...> wrote: Hey,
Not sure where you're seeing that fall.? My price stream shows me the
close on Friday being around 239.3/245.2 and the open today being
around 243.15/243.53.
In any case, I didn't trade, so the position shouldn't have changed.
Why would the position change?? I'm miles away from being called.
Cheers,
G.
On Mon, 15 Feb 2021 09:59:15 +0000
"mark collins" <mark.collins@...> wrote:
> A quick Google showed a large fall over the weekend. Could it be that
> you are seeing the p&l change?
>
> M
>
> On Mon, 15 Feb 2021, 09:01 Graham Bygrave, <graham@...> wrote:
>
> > I held 2143 of TSCO.L last night (paper trading account) at the
> > close. This morning on the open I have 1691 with the shortfall
> > being booked in P&L as a loss (i.e. as though I'd just given them
> > away).
> >
> > What's that about?
> >
> > Thanks,
> > G.
> >
> >
> >
> >
> >
> >?
>
>
>
>
>
-- +44 (0) 7528 551604?
Gulfstream Software - Winner Risk Management Awards 2010
This message is subject to :
|
Re: Long position in TSCO mysteriously shortened this morning
Hey, Not sure where you're seeing that fall. My price stream shows me the close on Friday being around 239.3/245.2 and the open today being around 243.15/243.53. In any case, I didn't trade, so the position shouldn't have changed. Why would the position change? I'm miles away from being called. Cheers, G. On Mon, 15 Feb 2021 09:59:15 +0000 "mark collins" <mark.collins@...> wrote: A quick Google showed a large fall over the weekend. Could it be that
you are seeing the p&l change?
M
On Mon, 15 Feb 2021, 09:01 Graham Bygrave, <graham@...> wrote:
I held 2143 of TSCO.L last night (paper trading account) at the
close. This morning on the open I have 1691 with the shortfall
being booked in P&L as a loss (i.e. as though I'd just given them
away).
What's that about?
Thanks,
G.
|
Re: Long position in TSCO mysteriously shortened this morning
A quick Google showed a large fall over the weekend. Could it be that you are seeing the p&l change?
M
On Mon, 15 Feb 2021, 09:01 Graham Bygrave, < graham@...> wrote: I held 2143 of TSCO.L last night (paper trading account) at the close.
This morning on the open I have 1691 with the shortfall being booked in
P&L as a loss (i.e. as though I'd just given them away).
What's that about?
Thanks,
G.
|
Re: Long position in TSCO mysteriously shortened this morning
Can you provide any more details?
Best wishes,
M
On Mon, 15 Feb 2021, 09:01 Graham Bygrave, < graham@...> wrote: I held 2143 of TSCO.L last night (paper trading account) at the close.
This morning on the open I have 1691 with the shortfall being booked in
P&L as a loss (i.e. as though I'd just given them away).
What's that about?
Thanks,
G.
|
Long position in TSCO mysteriously shortened this morning
I held 2143 of TSCO.L last night (paper trading account) at the close.
This morning on the open I have 1691 with the shortfall being booked in
P&L as a loss (i.e. as though I'd just given them away).
What's that about?
Thanks,
G.
|
Re: bad volume numbers using in history
If you want to capture most historical data available from ibkr make sure you specify SMART for the exchange in the contract object in you request. If you need to specify the exchange as part of the intrinsic definition of the contract, use its primary exchange field additionally. §á§ß, 15 §æ§Ö§Ó§â. 2021 §Ô., 02:57 Hany Danial < shdtech@...>:
i'm working TWS API, trying to get get historical data of a stock using:
date_time_obj = datetime.datetime.strptime('14/11/19', '%d/%m/%y')
queryTime0 = date_time_obj.strftime("%Y%m%d %H:%M:%S")
reqHistoricalData(id, contract, queryTime0, "52 W", "1 day", "TRADES", 1, 1, False, [])
example:
self.reqHistoricalData(id, contract, queryTime0, "3 D", "1 day", "TRADES", 1, 1, False, [])
HistoricalData. ReqId: 1 DNN BarData. Date: 20191112, Open: 0.450100, High: 0.450100, Low: 0.450100, Close: 0.450100, Volume: 5, Average: 0.450100, BarCount: 3
comparing the volume to yahoo chart, the volume in that day is more than 300k shares, it doesn't make any sense, even if i multiply it by 100. some dates will not return data for specified date by using TRADES, only if i used MIDPOINT. is there another api to use or am i using it incorrectly ?
|
List of error codes not documented?
Hello, This is the third error that I come across which is not documented in API documentation. Has anyone collected or listed these undocumented error codes somewhere? Following error code shows with three different descriptions. At least the first one makes sense to me but the second and third are still ambigous. Error Code: 10189, Msg: Failed to request tick-by-tick data:No historical market data for EUR/CASH@FXSUBPIP Last 0Error Code: 10189, Msg: Failed to request tick-by-tick data.Trading TWS session is connected from a different IP address Error Code: 10189, Msg: Failed to request tick-by-tick data.No market data permissions for NYSE STK
More undocumented codes from this group:
Error Code 10187:?/g/twsapi/message/44164?p=,,,20,0,0,0::Created,,undocumented+error,20,2,0,6076358
Error Code 505:?/g/twsapi/topic/4040516#4368
Thanks,
|
bad volume numbers using in history
i'm working TWS API, trying to get get historical data of a stock using:
date_time_obj = datetime.datetime.strptime('14/11/19', '%d/%m/%y')
queryTime0 = date_time_obj.strftime("%Y%m%d %H:%M:%S")
reqHistoricalData(id, contract, queryTime0, "52 W", "1 day", "TRADES", 1, 1, False, [])
example:
self.reqHistoricalData(id, contract, queryTime0, "3 D", "1 day", "TRADES", 1, 1, False, [])
HistoricalData. ReqId: 1 DNN BarData. Date: 20191112, Open: 0.450100, High: 0.450100, Low: 0.450100, Close: 0.450100, Volume: 5, Average: 0.450100, BarCount: 3
comparing the volume to yahoo chart, the volume in that day is more than 300k shares, it doesn't make any sense, even if i multiply it by 100. some dates will not return data for specified date by using TRADES, only if i used MIDPOINT. is there another api to use or am i using it incorrectly ?
|
Locked
Re: What type of storage or db is best for a lot of trading data that streams to you?
I'm locking this thread now.
It's really off-topic, and there are probably as many answers to it as there are members of the group!
Please, folks, try to remember what the purpose of this group is: to help people who are having problems with the TWS API. It's not a free-for-all discussion on any topic that's in some way related (or even unrelated) to the API. You need to make your own decisions about what technologies to use.
Members who persist in making off-topic posts or who appear not to be willing to make an effort to find answers themselves before asking the group, are like to be put back on moderation.
Richard King
Group Owner and Moderator
|
Locked
Re: What type of storage or db is best for a lot of trading data that streams to you?
Hi Bruce,
? I use pandas(python) to and from csv and that works for me. My datasets are not in the GB size and hence ymmv depending on whether you are trading?at the tick level.
-Ajay
|
Locked
Re: What type of storage or db is best for a lot of trading data that streams to you?
|
Locked
Re: What type of storage or db is best for a lot of trading data that streams to you?
Hi Bruce,
I've never played with "billions lines of data" and use only? ?for storage (I use sqlite3 and framework written in java).I keep it simple - one file per one table. So if for example I download a few months of 1-sec bars with "what to show" set to "trades", then it will end up in one file, if it is "what to show" == something else, then it will be a different file. It works for me and covers all my needs. The most complex use case for that storage was to replay few ears of all 500+ components of S&P 500. It took about 2-3 weeks back then to download all the 1-min history bars for more than 10 years (for some ticker more for some less). And I end up with ~40-50GB of db files or so (won't fit ram:). Then I solved the challenge of reading historical data?from?multiple?DB/tables in parallel without making "join" on those huge datasets. I put "prototype" here as a self-sufficient standalone project:? And then spent couple evenings to integrate this into my java "speculant" framework, so it is now part of the "replay" mechanizm.
I'll put here and example of generated files for 1 particular contract:
I put all the scanners result in one file (one table), then have "symbol_store.db" for all the contract details I've ever seen. When run scanners and getting back another 50 conid's I automatically query ContractDetails on all of those. Also recently added "tags.db" to simply be able to associate a tag (string) to contract id - this is useful to group things around etc. If interested - PM me and I'll send all the "create table" statements for all the mentioned DBs.
Cheers,
Dmitry Shevkoplyas
On Sun, Feb 14, 2021 at 2:06 PM Sean McNamara < tank@...> wrote: If you are doing large scale backtesting with the full set sets of data then the suggestion from ds-avatar is a good one as the Parquet file format is quite nice.?
If you are looking for a more database-centric approach, I've had great luck using PostgreSQL () with the TimescaleDB () module enabled.? I like the fact that you get excellent compression of data, that you can generate on-demand subsets of history (vs. file-based persistence?like Parquet), and that the interface to the data is normal SQL queries.
?
I've played around with a few other options such as InfluxDB, but found that for my personal use-case TimescaleDB was preferable.
?
It's not clear how you intend to interact with the data, and those specifics will most likely point you in the right direction.
|
Locked
Re: What type of storage or db is best for a lot of trading data that streams to you?
You would be better off using 4K blocks as that's the size of an inode on disk.? The next size a filesystem uses is 2MB which is 1 parent inode filled with 64 bit pointers to full 4K inodes.? Those are the fastest sizes of data to write to most filesystems.
Hunter
On Sunday, February 14, 2021, 12:43:16 PM PST, btw <newguyanon@...> wrote:
I use gzipped text files, basically just as they come from IB.? Gzip can compress on the fly so I append 32k blocks when I have enough data so it doesn't write all the time.? I get about 7* compression on my data which increases to 8* if I recompress an existing file.? Reading is faster for compressed files.? I use a normal ssd.
? ? ? ? ? ? if (rtvBytes > 16000 || isForce){//32k block size but prefer more writes than less, force on close, still good compression
? ? ? ? ? ? ? ? try (GZIPOutputStream gzout = new GZIPOutputStream(new FileOutputStream(outf, true))) {//true append
? ? ? ? ? ? ? ? ? ? for (; idxF < indVals.size(); idxF++) {
? ? ? ? ? ? ? ? ? ? ? ? RTV rtv = (RTV) indVals.get(idxF).misc;
? ? ? ? ? ? ? ? ? ? ? ? gzout.write(rtv.toSCSV(fmtStr).getBytes());
? ? ? ? ? ? ? ? ? ? }
?
?oops maybe I use 16k blocks I don't need any specific database queries since I just ask for a full month/year/contract etc. at a time.? If you need one symbol for one week and another for another week in reverse order or something then a database makes sense.? ?Obviously this never happens. For historical 5 sec bars which I get some every week I gzip them all at the end.? If you screw up and get too many or forget a week and get them out of order the file is screwed up.? So when loading I put them in a set then sort (all by date).? I was going to write code to detect errors and save the fixed file but I never bothered as it takes just a few msecs to fix every time. I think they would all be called timeseries since I save a timestamp.
|
Locked
Re: What type of storage or db is best for a lot of trading data that streams to you?
I use gzipped text files, basically just as they come from IB.? Gzip can compress on the fly so I append 32k blocks when I have enough data so it doesn't write all the time.? I get about 7* compression on my data which increases to 8* if I recompress an existing file.? Reading is faster for compressed files.? I use a normal ssd.
? ? ? ? ? ? if (rtvBytes > 16000 || isForce){//32k block size but prefer more writes than less, force on close, still good compression
? ? ? ? ? ? ? ? try (GZIPOutputStream gzout = new GZIPOutputStream(new FileOutputStream(outf, true))) {//true append
? ? ? ? ? ? ? ? ? ? for (; idxF < indVals.size(); idxF++) {
? ? ? ? ? ? ? ? ? ? ? ? RTV rtv = (RTV) indVals.get(idxF).misc;
? ? ? ? ? ? ? ? ? ? ? ? gzout.write(rtv.toSCSV(fmtStr).getBytes());
? ? ? ? ? ? ? ? ? ? }
?
?oops maybe I use 16k blocks I don't need any specific database queries since I just ask for a full month/year/contract etc. at a time.? If you need one symbol for one week and another for another week in reverse order or something then a database makes sense.? ?Obviously this never happens. For historical 5 sec bars which I get some every week I gzip them all at the end.? If you screw up and get too many or forget a week and get them out of order the file is screwed up.? So when loading I put them in a set then sort (all by date).? I was going to write code to detect errors and save the fixed file but I never bothered as it takes just a few msecs to fix every time. I think they would all be called timeseries since I save a timestamp.
|
Locked
Re: What type of storage or db is best for a lot of trading data that streams to you?
If you are doing large scale backtesting with the full set sets of data then the suggestion from ds-avatar is a good one as the Parquet file format is quite nice.?
If you are looking for a more database-centric approach, I've had great luck using PostgreSQL () with the TimescaleDB () module enabled.? I like the fact that you get excellent compression of data, that you can generate on-demand subsets of history (vs. file-based persistence?like Parquet), and that the interface to the data is normal SQL queries.
?
I've played around with a few other options such as InfluxDB, but found that for my personal use-case TimescaleDB was preferable.
?
It's not clear how you intend to interact with the data, and those specifics will most likely point you in the right direction.
|
Amaganset
Dmitry Shevkoplyas