开云体育

ctrl + shift + ? for shortcuts
© 2025 开云体育
  1. Twsapi
  2. Topics

historical and live recording of tick data

 

Hi all,
I'm trying to request historical ticks data for trades but i also want to see the bid/offer at the time of the trades in order to reconstruct the time and sales. However ib has this in 2 different functions, I've no idea why. Can anyone help with how can I piece info from these two streams together in a way that is identical to the time and sales screen, so the little red and green dots are also constructed.

If this is not possible,? can i live record the market data to recreate this effect? how does tws do it?
thank you for any assistance
Dave
Nick
 

开云体育

You can't do it from historical download.

In the live stream my observation so far is that you'll get the events in the proper order. I don't know how it lines up with the T&S display in TWS. You'll have to try it and see if it meets your needs.


On 9/2/2018 10:09 AM, hannnnny via Groups.Io wrote:

Hi all,
I'm trying to request historical ticks data for trades but i also want to see the bid/offer at the time of the trades in order to reconstruct the time and sales. However ib has this in 2 different functions, I've no idea why. Can anyone help with how can I piece info from these two streams together in a way that is identical to the time and sales screen, so the little red and green dots are also constructed.

If this is not possible,? can i live record the market data to recreate this effect? how does tws do it?
thank you for any assistance
Dave

 

I'm thinking of a kind of lookup algorithm based on timestamp to slot the trade ticks into the bid/offer ticks, I was just hoping someone has done this already but do you know some reason why this can't be done??
 

Hi Hannnnny,

As described your algorithm will work if the ticks start and end times are synchronisable across the two streams, otherwise you'll have spans of bid/offer that are overlapping multiple spans of trades or vice versa, which will take some effort if you wish to go about correcting it.

Do you know if they are sync'd?

Best wishes,


M

On 3 September 2018 at 11:37, hannnnny via Groups.Io <hannnnny@...> wrote:
I'm thinking of a kind of lookup algorithm based on timestamp to slot the trade ticks into the bid/offer ticks, I was just hoping someone has done this already but do you know some reason why this can't be done??




--
(T) +44 (0) 131 2202413 ?? (M) +44 (0) 7528 551604 ? (W)
Gulfstream Software - Winner Risk Management Awards 2010
This message is subject to :

Nick
 

开云体育

The timestamp is only in seconds so when there are multiple ticks per second you have no way to determine the order of events.

For active instruments there will be multiple ticks per second so your approach will not work.


On 9/3/2018 6:37 AM, hannnnny via Groups.Io wrote:

I'm thinking of a kind of lookup algorithm based on timestamp to slot the trade ticks into the bid/offer ticks, I was just hoping someone has done this already but do you know some reason why this can't be done??

 

i dont know if they are synced. both feed have unix timestamp so i assume trades can be matched to bid/offer in a giant looping lookup? is there any reason they would not be synced coming from IB??
Nick
 

开云体育

I have told you why you can't do it with historical tick data.

You have to use live data.

If you can't understand IB's data then you will have to hire someone to do it for you.


On 9/3/2018 9:39 AM, hannnnny via Groups.Io wrote:

i dont know if they are synced. both feed have unix timestamp so i assume trades can be matched to bid/offer in a giant looping lookup? is there any reason they would not be synced coming from IB??

 

was responding to Mark, didnt see your answer somehow, easy fella...

Now live recording is up, next question,?I apply a unix timestamp on my side whenever a volume ticktype 5 comes through and comparing to time and sales of wts there are about usually 5-6 duplicates to each T_S reported trade.?

Is the IB server sending out multiple messages of the same trade? Is there a easy way to filter out the duplicates??
Also a lot of trades seem to be removed from time and sales, i assume due to status, is there a way for me to remove also from live stream market data?

thanks guys
 

Hi Hannnny,

Duplicates are certainly possible, there are many other events in the IB api that produce duplications and omissions of messaging, so you're better off preparing for them than not. As for removing duplicates, the traditional thing to do would be to use a hash over some relevant part of the data to identify whether it is a duplicate, whatever you chose for that will have to be able to tell the difference between the duplicates and two identical trades. I'm sorry I can't help much, I've not traveled that particular journey, I think you are in an area of the IB api that most people don't venture into.

Best wishes,


M



On 3 September 2018 at 17:18, hannnnny via Groups.Io <hannnnny@...> wrote:
was responding to Mark, didnt see your answer somehow, easy fella...

Now live recording is up, next question,?I apply a unix timestamp on my side whenever a volume ticktype 5 comes through and comparing to time and sales of wts there are about usually 5-6 duplicates to each T_S reported trade.?

Is the IB server sending out multiple messages of the same trade? Is there a easy way to filter out the duplicates??
Also a lot of trades seem to be removed from time and sales, i assume due to status, is there a way for me to remove also from live stream market data?

thanks guys




--
(T) +44 (0) 131 2202413 ?? (M) +44 (0) 7528 551604 ? (W)
Gulfstream Software - Winner Risk Management Awards 2010
This message is subject to :

 

I'm pretty sure when this question came up years ago it was concluded that the timestamps on tick data are put there by TWS/Gateway, based on time of arrival. ?They are not real timestamps based on market events. ?They are a "convenience" for API programming.

Therefore you are just as good generating your own timestamps based on when you receive the ticks, or if order is all that matters, to simply make use of order of arrival. ?You can therefore use any time resolution you want. ?However simultaneity will always be a little ambiguous. ?For sure you will need to make sure your reading of the stream is not blocked significantly waiting for a timer, etc., and you will want to discern the degree of delay between ticks to determine whether you treat two events as being "at the same time".

-Kurt


On 9/3/18, 6:27:14 AM, "Nick" <[email protected] on behalf of news1000@...> wrote:

The timestamp is only in seconds so when there are multiple ticks per second you have no way to determine the order of events.

For active instruments there will be multiple ticks per second so your approach will not work.


On 9/3/2018 6:37 AM, hannnnny via Groups.Io wrote:
I'm thinking of a kind of lookup algorithm based on timestamp to slot the trade ticks into the bid/offer ticks, I was just hoping someone has done this already but do you know some reason why this can't be done??

Nick
 

开云体育

Due to recent additions to the api it is important to distinguish between all the different data streams that are available.

In the old days 'tick data' referred to the live data in reqMktData, which is sampled and not individual ticks.

Then realtime volume came along which was still sampled (but in a different way) and included millisecond timestamps which were, as Kurt said, bogus and derived from the local clock.

Now there is reqTickByTick data which is close to actual non-sampled tick data (no one with access to a high quality data feed has posted here as to how 'un-sampled' IB's data is).

Obviously trades-at-bid and trades-at-ask can't be done on a sampled feed. IB's historical TickByTick data is also not useful since there is no way to reliably combine the separate Last and BidAsk streams.

From what I have seen so far the live TickByTick data updates such that trades-at-bid and trades-at-ask can be determined since the events arrive in a coherent order. I do not have access to a high quality data feed so I can't make any definitive statements about IB's TickByTick data other than it might be good enough. Each person has to research it and see if it meets their needs.

On 9/3/2018 12:47 PM, Kurt Bigler wrote:

I'm pretty sure when this question came up years ago it was concluded that the timestamps on tick data are put there by TWS/Gateway, based on time of arrival. ?They are not real timestamps based on market events. ?They are a "convenience" for API programming.

Therefore you are just as good generating your own timestamps based on when you receive the ticks, or if order is all that matters, to simply make use of order of arrival. ?You can therefore use any time resolution you want. ?However simultaneity will always be a little ambiguous. ?For sure you will need to make sure your reading of the stream is not blocked significantly waiting for a timer, etc., and you will want to discern the degree of delay between ticks to determine whether you treat two events as being "at the same time".

-Kurt


On 9/3/18, 6:27:14 AM, "Nick" <[email protected] on behalf of news1000@...> wrote:

The timestamp is only in seconds so when there are multiple ticks per second you have no way to determine the order of events.

For active instruments there will be multiple ticks per second so your approach will not work.


On 9/3/2018 6:37 AM, hannnnny via Groups.Io wrote:
I'm thinking of a kind of lookup algorithm based on timestamp to slot the trade ticks into the bid/offer ticks, I was just hoping someone has done this already but do you know some reason why this can't be done??


 

great overview, thats what i needed to understand. thank you. so tickbytick with timestamp is the way forward.
 

Thanks. ?Is this TIckByTick thing really only for people who have a dedicated line to IB?

-Kurt


On 9/3/18, 10:28:15 AM, "Nick" <[email protected] on behalf of news1000@...> wrote:

Due to recent additions to the api it is important to distinguish between all the different data streams that are available.

In the old days 'tick data' referred to the live data in reqMktData, which is sampled and not individual ticks.

Then realtime volume came along which was still sampled (but in a different way) and included millisecond timestamps which were, as Kurt said, bogus and derived from the local clock.

Now there is reqTickByTick data which is close to actual non-sampled tick data (no one with access to a high quality data feed has posted here as to how 'un-sampled' IB's data is).

Obviously trades-at-bid and trades-at-ask can't be done on a sampled feed. IB's historical TickByTick data is also not useful since there is no way to reliably combine the separate Last and BidAsk streams.

From what I have seen so far the live TickByTick data updates such that trades-at-bid and trades-at-ask can be determined since the events arrive in a coherent order. I do not have access to a high quality data feed so I can't make any definitive statements about IB's TickByTick data other than it might be good enough. Each person has to research it and see if it meets their needs.

On 9/3/2018 12:47 PM, Kurt Bigler wrote:
I'm pretty sure when this question came up years ago it was concluded that the timestamps on tick data are put there by TWS/Gateway, based on time of arrival. ?They are not real timestamps based on market events. ?They are a "convenience" for API programming.

Therefore you are just as good generating your own timestamps based on when you receive the ticks, or if order is all that matters, to simply make use of order of arrival. ?You can therefore use any time resolution you want. ?However simultaneity will always be a little ambiguous. ?For sure you will need to make sure your reading of the stream is not blocked significantly waiting for a timer, etc., and you will want to discern the degree of delay between ticks to determine whether you treat two events as being "at the same time".

-Kurt


On 9/3/18, 6:27:14 AM, "Nick" <[email protected] on behalf of news1000@...> wrote:

The timestamp is only in seconds so when there are multiple ticks per second you have no way to determine the order of events.

For active instruments there will be multiple ticks per second so your approach will not work.


On 9/3/2018 6:37 AM, hannnnny via Groups.Io wrote:
I'm thinking of a kind of lookup algorithm based on timestamp to slot the trade ticks into the bid/offer ticks, I was just hoping someone has done this already but do you know some reason why this can't be done??


Nick
 

No, it's available to everyone although I think it is only described in the beta release notes and has not made it to the main documentation (at least I couldn't find it).

It is similar to the DOM where you can only have a max of 5 concurrent TickByTick streams. This probably keeps the bandwidth requirements within what they can handle.

On 9/3/2018 3:46 PM, Kurt Bigler wrote:
Thanks. ?Is this TIckByTick thing really only for people who have a dedicated line to IB?

-Kurt

1 - 14 of 14