Re: historical and live recording of tick data
Thanks. ?Is this TIckByTick thing really only for people who have a dedicated line to IB?
-Kurt
Due to recent additions to the api it is important to distinguish
between all the different data streams that are available.
In the old days 'tick data' referred to the live data in reqMktData,
which is sampled and not individual ticks.
Then realtime volume came along which was still sampled (but in a
different way) and included millisecond timestamps which were, as
Kurt said, bogus and derived from the local clock.
Now there is reqTickByTick data which is close to actual non-sampled
tick data (no one with access to a high quality data feed has posted
here as to how 'un-sampled' IB's data is).
Obviously trades-at-bid and trades-at-ask can't be done on a sampled
feed. IB's historical TickByTick data is also not useful since there
is no way to reliably combine the separate Last and BidAsk streams.
From what I have seen so far the live TickByTick data updates such
that trades-at-bid and trades-at-ask can be determined since the
events arrive in a coherent order. I do not have access to a high
quality data feed so I can't make any definitive statements about
IB's TickByTick data other than it might be good enough. Each person
has to research it and see if it meets their needs.
On 9/3/2018 12:47 PM, Kurt Bigler
wrote:
I'm pretty sure when this question came up years ago it was
concluded that the timestamps on tick data are put there by
TWS/Gateway, based on time of arrival. ?They are not real
timestamps based on market events. ?They are a "convenience" for
API programming.
Therefore you are just as good generating your own timestamps
based on when you receive the ticks, or if order is all that
matters, to simply make use of order of arrival. ?You can
therefore use any time resolution you want. ?However
simultaneity will always be a little ambiguous. ?For sure you
will need to make sure your reading of the stream is not blocked
significantly waiting for a timer, etc., and you will want to
discern the degree of delay between ticks to determine whether
you treat two events as being "at the same time".
-Kurt
The timestamp is only in seconds so when there are
multiple ticks per second you have no way to determine
the order of events.
For active instruments there will be multiple ticks per
second so your approach will not work.
On 9/3/2018 6:37 AM, hannnnny
via Groups.Io wrote:
I'm
thinking of a kind of lookup algorithm based on
timestamp to slot the trade ticks into the bid/offer
ticks, I was just hoping someone has done this already
but do you know some reason why this can't be done??
|
Re: historical and live recording of tick data
great overview, thats what i needed to understand. thank you. so tickbytick with timestamp is the way forward.
|
Re: historical and live recording of tick data
Due to recent additions to the api it is important to distinguish
between all the different data streams that are available.
In the old days 'tick data' referred to the live data in reqMktData,
which is sampled and not individual ticks.
Then realtime volume came along which was still sampled (but in a
different way) and included millisecond timestamps which were, as
Kurt said, bogus and derived from the local clock.
Now there is reqTickByTick data which is close to actual non-sampled
tick data (no one with access to a high quality data feed has posted
here as to how 'un-sampled' IB's data is).
Obviously trades-at-bid and trades-at-ask can't be done on a sampled
feed. IB's historical TickByTick data is also not useful since there
is no way to reliably combine the separate Last and BidAsk streams.
From what I have seen so far the live TickByTick data updates such
that trades-at-bid and trades-at-ask can be determined since the
events arrive in a coherent order. I do not have access to a high
quality data feed so I can't make any definitive statements about
IB's TickByTick data other than it might be good enough. Each person
has to research it and see if it meets their needs.
On 9/3/2018 12:47 PM, Kurt Bigler
wrote:
toggle quoted message
Show quoted text
I'm pretty sure when this question came up years ago it was
concluded that the timestamps on tick data are put there by
TWS/Gateway, based on time of arrival. ?They are not real
timestamps based on market events. ?They are a "convenience" for
API programming.
Therefore you are just as good generating your own timestamps
based on when you receive the ticks, or if order is all that
matters, to simply make use of order of arrival. ?You can
therefore use any time resolution you want. ?However
simultaneity will always be a little ambiguous. ?For sure you
will need to make sure your reading of the stream is not blocked
significantly waiting for a timer, etc., and you will want to
discern the degree of delay between ticks to determine whether
you treat two events as being "at the same time".
-Kurt
The timestamp is only in seconds so when there are
multiple ticks per second you have no way to determine
the order of events.
For active instruments there will be multiple ticks per
second so your approach will not work.
On 9/3/2018 6:37 AM, hannnnny
via Groups.Io wrote:
I'm
thinking of a kind of lookup algorithm based on
timestamp to slot the trade ticks into the bid/offer
ticks, I was just hoping someone has done this already
but do you know some reason why this can't be done??
|
Re: Install TWS or IB gateway on Ubuntu and ARM64 bit processor
I think the problem here is that the TWS and Gateway installers now include their own ‘hidden’ copy of Java, this being an Oracle Java that I pretty much guaranteed to work well with that particular version of TWS/Gateway. The rationale for this is that a) it means the user doesn’t have to take the trouble to install and maintain a suitable Java version; and b) this should reduce support problems for IB because in theory the user will not accidentally use an incompatible Java version. ? Now, the actual Java compiled class files, being pure Java, are I believe identical on all platforms for TWS and Gateway, and should run with any Java VM. So in theory you should be able to install TWS/Gateway on another machine (Ubuntu perhaps or even Windows) and then just copy the relevant jar files over to the ARM machine. You’d then need to install a suitable Java on the ARM machine: I don’t know whether Oracle produce a Java version for the ARM architecture, but if they don’t you might be able to find another implementation. Ah, I’ve just discovered that indeed Oracle do support ARM. ? Once you’ve got the JAR files and the Java runtime in place, it should just be a matter of using an appropriate Java command with the correct arguments to start the thing off. I can help with that because that’s exactly what IBC/IBController do under the covers. ? So if you think this might be an avenue worth pursuing, then please let me know and get Oracle Java installed . I’ll then post the details of which jar files to copy, and the appropriate Java command (I can’t be bothered to go and look this up unless you’re actually going to use it). ? Needless to say, there’s no guarantee this will work, but there’s also no obvious reason why it won’t. ? Richard ? ? ?
toggle quoted message
Show quoted text
From: [email protected] <[email protected]> On Behalf Of Derek Fung
Sent: 03 September 2018 17:56
To: [email protected]
Subject: Re: [TWS API] Install TWS or IB gateway on Ubuntu and ARM64 bit processor? This is the error screen I have.? Seems it is related to Java.? Hey Derek,
I use ubuntu all the time for my gateway and tws machines, the ARM64 thing might be a bit odd though, I've no experience with that.
Does it state that your java version is wrong or is that a guess? If its as statement then it's a fairly good thing to look at updating, if it's a guess, well... ummm... maybe?! Best wishes, M ? On 3 September 2018 at 17:06, Derek Fung <ibmderekfung@...> wrote: Hi, Has anyone installed TWS or IB gateway successfully on ARM64 bit processor and Ubuntu?? I have problem in installation. Guess it is because the JVM is not supported.? Any suggestion?
--
(T) +44 (0) 131 2202413 ?? (M) +44 (0) 7528 551604 ? (W)
Gulfstream Software - Winner Risk Management Awards 2010
This message is subject to :
|
Re: Install TWS or IB gateway on Ubuntu and ARM64 bit processor
This is the error screen I have.?
Seems it is related to Java.?
toggle quoted message
Show quoted text
On 4 Sep 2018, at 00:23, mark collins < mark.collins@...> wrote: Hey Derek,
I use ubuntu all the time for my gateway and tws machines, the ARM64 thing might be a bit odd though, I've no experience with that.
Does it state that your java version is wrong or is that a guess? If its as statement then it's a fairly good thing to look at updating, if it's a guess, well... ummm... maybe?!
Best wishes, M
|
Re: historical and live recording of tick data
I'm pretty sure when this question came up years ago it was concluded that the timestamps on tick data are put there by TWS/Gateway, based on time of arrival. ?They are not real timestamps based on market events. ?They are a "convenience" for API programming.
Therefore you are just as good generating your own timestamps based on when you receive the ticks, or if order is all that matters, to simply make use of order of arrival. ?You can therefore use any time resolution you want. ?However simultaneity will always be a little ambiguous. ?For sure you will need to make sure your reading of the stream is not blocked significantly waiting for a timer, etc., and you will want to discern the degree of delay between ticks to determine whether you treat two events as being "at the same time".
-Kurt
The timestamp is only in seconds so when there are multiple ticks
per second you have no way to determine the order of events.
For active instruments there will be multiple ticks per second so
your approach will not work.
On 9/3/2018 6:37 AM, hannnnny via
Groups.Io wrote:
I'm thinking of a
kind of lookup algorithm based on timestamp to slot the trade
ticks into the bid/offer ticks, I was just hoping someone has done
this already but do you know some reason why this can't be done??
|
Re: historical and live recording of tick data
Hi Hannnny,
Duplicates are certainly possible, there are many other events in the IB api that produce duplications and omissions of messaging, so you're better off preparing for them than not. As for removing duplicates, the traditional thing to do would be to use a hash over some relevant part of the data to identify whether it is a duplicate, whatever you chose for that will have to be able to tell the difference between the duplicates and two identical trades. I'm sorry I can't help much, I've not traveled that particular journey, I think you are in an area of the IB api that most people don't venture into. Best wishes, M
On 3 September 2018 at 17:18, hannnnny via Groups.Io <hannnnny@...> wrote: was responding to Mark, didnt see your answer somehow, easy fella...
Now live recording is up, next question,?I apply a unix timestamp on my side whenever a volume ticktype 5 comes through and comparing to time and sales of wts there are about usually 5-6 duplicates to each T_S reported trade.?
Is the IB server sending out multiple messages of the same trade? Is there a easy way to filter out the duplicates??
Also a lot of trades seem to be removed from time and sales, i assume due to status, is there a way for me to remove also from live stream market data?
thanks guys
-- (T) +44 (0) 131 2202413 ?? (M) +44 (0) 7528 551604 ? (W)
Gulfstream Software - Winner Risk Management Awards 2010
This message is subject to :
|
Re: Install TWS or IB gateway on Ubuntu and ARM64 bit processor
Hey Derek,
I use ubuntu all the time for my gateway and tws machines, the ARM64 thing might be a bit odd though, I've no experience with that.
Does it state that your java version is wrong or is that a guess? If its as statement then it's a fairly good thing to look at updating, if it's a guess, well... ummm... maybe?!
Best wishes, M
On 3 September 2018 at 17:06, Derek Fung <ibmderekfung@...> wrote: Hi,
Has anyone installed TWS or IB gateway successfully on ARM64 bit processor and Ubuntu?? I have problem in installation. Guess it is because the JVM is not supported.? Any suggestion?
Derek?
-- (T) +44 (0) 131 2202413 ?? (M) +44 (0) 7528 551604 ? (W)
Gulfstream Software - Winner Risk Management Awards 2010
This message is subject to :
|
Re: historical and live recording of tick data
was responding to Mark, didnt see your answer somehow, easy fella...
Now live recording is up, next question,?I apply a unix timestamp on my side whenever a volume ticktype 5 comes through and comparing to time and sales of wts there are about usually 5-6 duplicates to each T_S reported trade.?
Is the IB server sending out multiple messages of the same trade? Is there a easy way to filter out the duplicates??
Also a lot of trades seem to be removed from time and sales, i assume due to status, is there a way for me to remove also from live stream market data?
thanks guys
|
Install TWS or IB gateway on Ubuntu and ARM64 bit processor
Hi,
Has anyone installed TWS or IB gateway successfully on ARM64 bit processor and Ubuntu?? I have problem in installation. Guess it is because the JVM is not supported.? Any suggestion?
Derek?
|
Re: historical and live recording of tick data
I have told you why you can't do it with historical tick data.
You have to use live data.
If you can't understand IB's data then you will have to hire
someone to do it for you.
On 9/3/2018 9:39 AM, hannnnny via
Groups.Io wrote:
toggle quoted message
Show quoted text
i dont know if they
are synced. both feed have unix timestamp so i assume trades can
be matched to bid/offer in a giant looping lookup? is there any
reason they would not be synced coming from IB??
|
Re: historical and live recording of tick data
i dont know if they are synced. both feed have unix timestamp so i assume trades can be matched to bid/offer in a giant looping lookup? is there any reason they would not be synced coming from IB??
|
Re: historical and live recording of tick data
The timestamp is only in seconds so when there are multiple ticks
per second you have no way to determine the order of events.
For active instruments there will be multiple ticks per second so
your approach will not work.
On 9/3/2018 6:37 AM, hannnnny via
Groups.Io wrote:
toggle quoted message
Show quoted text
I'm thinking of a
kind of lookup algorithm based on timestamp to slot the trade
ticks into the bid/offer ticks, I was just hoping someone has done
this already but do you know some reason why this can't be done??
|
Re: historical and live recording of tick data
Hi Hannnnny,
As described your algorithm will work if the ticks start and end times are synchronisable across the two streams, otherwise you'll have spans of bid/offer that are overlapping multiple spans of trades or vice versa, which will take some effort if you wish to go about correcting it.
Do you know if they are sync'd?
Best wishes,
M
On 3 September 2018 at 11:37, hannnnny via Groups.Io <hannnnny@...> wrote: I'm thinking of a kind of lookup algorithm based on timestamp to slot the trade ticks into the bid/offer ticks, I was just hoping someone has done this already but do you know some reason why this can't be done??
-- (T) +44 (0) 131 2202413 ?? (M) +44 (0) 7528 551604 ? (W)
Gulfstream Software - Winner Risk Management Awards 2010
This message is subject to :
|
Re: historical and live recording of tick data
I'm thinking of a kind of lookup algorithm based on timestamp to slot the trade ticks into the bid/offer ticks, I was just hoping someone has done this already but do you know some reason why this can't be done??
|
Re: historical and live recording of tick data
You can't do it from historical download.
In the live stream my observation so far is that you'll get the
events in the proper order. I don't know how it lines up with the
T&S display in TWS. You'll have to try it and see if it meets
your needs.
On 9/2/2018 10:09 AM, hannnnny via
Groups.Io wrote:
toggle quoted message
Show quoted text
Hi all,
I'm trying to request historical ticks data for trades but i also
want to see the bid/offer at the time of the trades in order to
reconstruct the time and sales. However ib has this in 2 different
functions, I've no idea why. Can anyone help with how can I piece
info from these two streams together in a way that is identical to
the time and sales screen, so the little red and green dots are
also constructed.
If this is not possible,? can i live record the market data to
recreate this effect? how does tws do it?
thank you for any assistance
Dave
|
Re: TWS API Error 2019: Order Event Warning - got when trying to submit Bracket order for execution outside RTH. Entry, profit OK. Stop NG.
When I have encountered this it was due to the rthOnly setting (or whatever it is called now). If you reverse the rthOnly setting for the stop order the warning will probably go away.
I never figured out the actual rules but when I got error 2109 I noted the symbol, order type and exchange. Within my order placement routine I added a check for these combinations and forced rthOnly to be whatever didn't cause the error.
Not? helpful in explaining the root cause I know, but it let order processing happen without warnings.
toggle quoted message
Show quoted text
On 8/31/2018 11:46 AM, loududka@... wrote: I'm trying to submit a Backet order,(in this case, for SQ) to fill outside of regular trading hours.? It successfully submits EXCEPT for this warning on the StopLoss:
"Error: 2109 Order Event Warning:Attribute 'Outside Regular Trading Hours' is ignored based on the order type and destination. PlaceOrder is now being processed."
|
historical and live recording of tick data
Hi all,
I'm trying to request historical ticks data for trades but i also want to see the bid/offer at the time of the trades in order to reconstruct the time and sales. However ib has this in 2 different functions, I've no idea why. Can anyone help with how can I piece info from these two streams together in a way that is identical to the time and sales screen, so the little red and green dots are also constructed.
If this is not possible,? can i live record the market data to recreate this effect? how does tws do it?
thank you for any assistance
Dave
|
Re: Ibrokers package in R - Get option prices
I found the functions eWrapper.data.Last and snapShot?on web so I'm not truly able to understand correctly what is the problem. For the moment, I solved using the basic functions:
? ? opt=twsOption(local="",symbol="SPX",expiry=exp,strike=k,right="P") ? ? optPrice=reqMktData(tws,opt,snapshot = T)
It takes some time to get option prices (about 10-15 seconds for each contract) but at least works. I'll try to find an alternative way to speeding up the process.?
Il giorno ven 31 ago 2018 alle ore 17:23 mark collins < mark.collins@...> ha scritto: Well, your R code does have a while(TRUE) loop control so I'd not expect it to stop unless something hits the conditions for breaking the loop.... it's probably stuck there or somewhere similar. Are you printing out the process steps as it goes?
Best wishes,
M
|
Are there any known issues about running the older TWS.OCX 9.71 along with a new 9.73 build on the same PC ?
Thanks
|
