开云体育

I'm trying to encode tick data from IB, and wanted to use TickUtils.jar contributed here. But it seems to work with standard tick formats for example a Trade tick would have format (%time %price %trade size).

IB data shows up as (%time %tick type %value), so IB would send two tick responses for a trade tick, instead of one.

I was wondering if anyone was using the encoding utility for IB and if they would have a converter for IB tick data into the the format the encoder needs and are willing to share it.

It's not clear from your question what you're trying to accomplish. If you're interested in market microstructure, i.e. very short term, you might have a look at TickString. See this old post:


[rwk]



--- "sc1447" <sc1447@> wrote:

I'm trying to encode tick data from IB, and wanted to use TickUtils.jar contributed here. But it seems to work with standard tick formats for example a Trade tick would have format (%time %price %trade size).

IB data shows up as (%time %tick type %value), so IB would send two tick responses for a trade tick, instead of one.

I was wondering if anyone was using the encoding utility for IB and if they would have a converter for IB tick data into the the format the encoder needs and are willing to share it.

1. I am making data request as: mClientSocket.reqMktData(id, contract, "", false);
which returns the "last price and quantity in separate callbacks".

So, using the post #24461 that you suggested I made the request as: mClientSocket.reqMktData(id, contract, "233", false);

which returns Last Trade tick as:
id=0 RTVolume=3066.00;4;1373851042866;1108;3065.15771661;false

I can now construct the LastTick object instance used in TickUtils.jar.

(Not yet sure if BidTick, AskTick can be done this way, but I'll investigate further, I wanted to reply at the earliest).

++++++
2a. I also noticed an interesting thing. Request made with only "233" still gives me IB response for tickPrice, tickSize methods, so I'm actually using more bandwidth.

2b. Also, please take a look at the data below (I tested $NQ_F)

2013.07.14 20:17:14:836,0,6.0
2013.07.14 20:17:15:759,0,7.0
2013.07.14 20:17:20:208,0,6.0
id=0 lastTimestamp=1373851042 <-tickString() response
2013.07.14 20:17:21:989,4,3066.0 <-TRADE_PRICE
2013.07.14 20:17:21:993,5,1.0 <-TRADE_SIZE
2013.07.14 20:17:21:999,8,1105.0 <-VOLUME
2013.07.14 20:17:22:004,1,3065.75
2013.07.14 20:17:22:014,0,4.0
2013.07.14 20:17:22:021,0,4.0
2013.07.14 20:17:22:025,3,6.0
id=0 RTVolume=3066.00;4;1373851042866;1108;3065.15771661;false
2013.07.14 20:17:23:132,5,3.0 <-TRADE_SIZE
2013.07.14 20:17:23:233,8,1108.0 <-VOLUME
2013.07.14 20:17:23:324,0,5.0
2013.07.14 20:17:23:328,3,7.0
2013.07.14 20:17:24:052,0,6.0

i) There is almost a 030ms (average seems to be like 040ms) lag between when trade is executed and when we get RTVolume response.

ii) I get a tickString response timestamp, then I get a trade, trade size, tickString Tick data, then trade size and then volume upto that point. Seems 2nd trade_size is not preceded with TRADE_PRICE if the price is same as last traded price.

iii) Seems we can construct TRADE ticks using tickPrice stream ourselves, although VWAP might get messed up if data breaks in between.

rwk, your 'understanding' that Windows cannot accurately measure time
intervals shorter than 65 msec is completely erroneous.

If you're programming directly to the Windows API, you can use
QueryPerformanceCounter and QueryPerformanceFrequency. In .Net, use the
Stopwatch class. With these, you can measure pretty accurately down to the
microsecond level.

These all depend on the presence of a high-resolution timer in your PC's
hardware. This doesn't appear to be much of a limitation - I've never come
across a computer that doesn't have one.

Also if you want to do timer-based actions, you can do these with very
small
intervals by using the timeBeginPeriod to set the minimum timer resolution
(down to 1 millisec). Use createTimerQueue and CreateTimerQueueTimer to
initiate the timers (can't remember offhand what the corresponding .Net
timer facility is, and I don't have time to look it up).

Richard

From: TWSAPI@... <mailto:TWSAPI%40yahoogroups.com>
[mailto:TWSAPI@... <mailto:TWSAPI%40yahoogroups.com>] On
Behalf Of
rwk2095
Sent: 15 July 2013 15:18
To: TWSAPI@... <mailto:TWSAPI%40yahoogroups.com>
Subject: [TWS API] Re: Tick Encoding/Decoding

Your original post (OP) makes more sense now. I was unfamiliar with
TickUtils.jar because it appears to be Java, and I don't speak Java.

It sounds like I am doing something similar to TickUtils.jar. I write
quotes
and trades into a file of fixed-length records, 20 bytes each, which I
refer
to as a blob (binary large object). I then load the blob into memory for
detailed analysis at my leisure using a single read command.

As Richard mentioned, a blob is not directly readable, but I have
tools that
extract pieces and put them out in .csv format for analysis in Excel.
I also
have a playback tool that can replay a segment of the day at whatever
speed
I select.

If you're using the RTVolume generic, you might also specify the "mdoff"
flag to suppress quotes in the regular quote stream so that you're seeing
only trades. This saves bandwidth. If you need quotes too, you can get
them
from reqMktDepthEx().

I don't understand the time lag you mentioned. How are you measuring the
lag? It's my understanding that Windows cannot accurately measure time
intervals shorter than about 65 msec. I believe the regular data
stream from
reqMktDataEx() is unsuitable for understanding market microstructure.

[rwk]

--- unatnahs57 <no_reply@...> wrote:

1. I am making data request as: mClientSocket.reqMktData(id,

contract, "",
false);

which returns the "last price and quantity in separate callbacks".

So, using the post #24461 that you suggested I made the request as:

mClientSocket.reqMktData(id, contract, "233", false);

which returns Last Trade tick as:
id=0 RTVolume=3066.00;4;1373851042866;1108;3065.15771661;false

I can now construct the LastTick object instance used in TickUtils.jar.

(Not yet sure if BidTick, AskTick can be done this way, but I'll

investigate further, I wanted to reply at the earliest).

++++++
2a. I also noticed an interesting thing. Request made with only "233"

still gives me IB response for tickPrice, tickSize methods, so I'm
actually
using more bandwidth.

2b. Also, please take a look at the data below (I tested $NQ_F)

2013.07.14 20:17:14:836,0,6.0
2013.07.14 20:17:15:759,0,7.0
2013.07.14 20:17:20:208,0,6.0
id=0 lastTimestamp=1373851042 <-tickString() response
2013.07.14 20:17:21:989,4,3066.0 <-TRADE_PRICE
2013.07.14 20:17:21:993,5,1.0 <-TRADE_SIZE
2013.07.14 20:17:21:999,8,1105.0 <-VOLUME
2013.07.14 20:17:22:004,1,3065.75
2013.07.14 20:17:22:014,0,4.0
2013.07.14 20:17:22:021,0,4.0
2013.07.14 20:17:22:025,3,6.0
id=0 RTVolume=3066.00;4;1373851042866;1108;3065.15771661;false
2013.07.14 20:17:23:132,5,3.0 <-TRADE_SIZE
2013.07.14 20:17:23:233,8,1108.0 <-VOLUME
2013.07.14 20:17:23:324,0,5.0
2013.07.14 20:17:23:328,3,7.0
2013.07.14 20:17:24:052,0,6.0

i) There is almost a 030ms (average seems to be like 040ms) lag between

when trade is executed and when we get RTVolume response.

ii) I get a tickString response timestamp, then I get a trade, trade

size,
tickString Tick data, then trade size and then volume upto that point.
Seems
2nd trade_size is not preceded with TRADE_PRICE if the price is same
as last
traded price.

iii) Seems we can construct TRADE ticks using tickPrice stream

ourselves,
although VWAP might get messed up if data breaks in between.

I'm the author of the TickUtils.jar you're referring to.



(For others reading this thread, this jar can be found in the Files section
of this group. It provides encoding and decoding facilities to store tick
data in a much smaller space than either plain text or some kind of fixed
binary format. I use it for storing tick data in an SQL database. At the
time I originally wrote it, I needed to minimize the space occupied by the
data, because disk space was still expensive back then and database backups
were interminable over a 10Mb/s network. I also didn't have the performance
necessary to write a row for each tick, so I batch up the ticks for each
minute into a blob using the encoding routines, and write the blobs at a
predictable rate. Nowadays disk space is almost trivially cheap. My database
is now around 25GB, and would be about 150GB without this encoding - not a
huge file by modern standards. However the performance aspect may still be
relevant, given that at some times I can be recording several hundred ticks
per second. By the way, the encoding/decoding overhead is completely
insignificant.)



If you want to use this jar with IB's Java API, you need to understand a few
things.



1. IB actually sends trade price and the corresponding trade size in a
single message from TWS to the API client, but the API code then splits that
into two separate callbacks - a tickPrice and a tickSize. (The same applies
to bids and asks.)



2. This combined message is only sent when the price changes - otherwise
just the size is sent, and a tickSize callback occurs.



3. After sending the combined price/size message, TWS then sends the exact
same size again as a separate message, and the API generates another
identical tickSize callback from this (there are historical reasons for this
apparently nonsensical 'feature', though I suspect they are no longer
valid).



4. If a trade occurs at the same price and size as the previously reported
trade, not even the tick size is sent - just a volume update. (Note that the
'previously reported trade' isn't necessarily the same as the most recent
actual trade, because the API doesn't necessarily report every trade.)



From this information, we can make the following deductions:



- when a tickPrice is received, the next tickSize (for the same tick type
obviously) will be the size for that price (and will in fact occur
immediately after, as it happens during the processing of the same socket
message from TWS).



- when a tickSize is received that is identical to the previous tickSize, it
should be ignored, because it is just the duplicate that TWS sends after
sending the combined price/size message - it doesn't represent a separate
trade.



You should now have enough understanding to be able to generate the formats
required by the TickUtils.jar.



By the way, please think carefully about whether you actually need to use
these encoding/decoding routines. For example, if you just need to store
tick data in ordinary files, it's best to simply write it out as text. The
problem with the encoded format is that it's not human-readable. I actually
do this in addition to my database recording, as a sort of backup - the
files are compressed (by the operating system), which reduces their size to
about one-sixth of their uncompressed size.



Richard





From: TWSAPI@... [mailto:TWSAPI@...] On Behalf Of
sc1447
Sent: 14 July 2013 20:55
To: TWSAPI@...
Subject: [TWS API] Tick Encoding/Decoding






Hi All,

I'm trying to encode tick data from IB, and wanted to use TickUtils.jar
contributed here. But it seems to work with standard tick formats for
example a Trade tick would have format (%time %price %trade size).

IB data shows up as (%time %tick type %value), so IB would send two tick
responses for a trade tick, instead of one.

I was wondering if anyone was using the encoding utility for IB and if they
would have a converter for IB tick data into the the format the encoder
needs and are willing to share it.

Thanks

S





[Non-text portions of this message have been removed]

Your original post (OP) makes more sense now. I was unfamiliar with TickUtils.jar because it appears to be Java, and I don't speak Java.

It sounds like I am doing something similar to TickUtils.jar. I write quotes and trades into a file of fixed-length records, 20 bytes each, which I refer to as a blob (binary large object). I then load the blob into memory for detailed analysis at my leisure using a single read command.

As Richard mentioned, a blob is not directly readable, but I have tools that extract pieces and put them out in .csv format for analysis in Excel. I also have a playback tool that can replay a segment of the day at whatever speed I select.

If you're using the RTVolume generic, you might also specify the "mdoff" flag to suppress quotes in the regular quote stream so that you're seeing only trades. This saves bandwidth. If you need quotes too, you can get them from reqMktDepthEx().

I don't understand the time lag you mentioned. How are you measuring the lag? It's my understanding that Windows cannot accurately measure time intervals shorter than about 65 msec. I believe the regular data stream from reqMktDataEx() is unsuitable for understanding market microstructure.

[rwk]



--- unatnahs57 <no_reply@> wrote:

1. I am making data request as: mClientSocket.reqMktData(id, contract, "", false);
which returns the "last price and quantity in separate callbacks".

So, using the post #24461 that you suggested I made the request as: mClientSocket.reqMktData(id, contract, "233", false);

which returns Last Trade tick as:
id=0 RTVolume=3066.00;4;1373851042866;1108;3065.15771661;false

I can now construct the LastTick object instance used in TickUtils.jar.

(Not yet sure if BidTick, AskTick can be done this way, but I'll investigate further, I wanted to reply at the earliest).

++++++
2a. I also noticed an interesting thing. Request made with only "233" still gives me IB response for tickPrice, tickSize methods, so I'm actually using more bandwidth.

2b. Also, please take a look at the data below (I tested $NQ_F)

2013.07.14 20:17:14:836,0,6.0
2013.07.14 20:17:15:759,0,7.0
2013.07.14 20:17:20:208,0,6.0
id=0 lastTimestamp=1373851042 <-tickString() response
2013.07.14 20:17:21:989,4,3066.0 <-TRADE_PRICE
2013.07.14 20:17:21:993,5,1.0 <-TRADE_SIZE
2013.07.14 20:17:21:999,8,1105.0 <-VOLUME
2013.07.14 20:17:22:004,1,3065.75
2013.07.14 20:17:22:014,0,4.0
2013.07.14 20:17:22:021,0,4.0
2013.07.14 20:17:22:025,3,6.0
id=0 RTVolume=3066.00;4;1373851042866;1108;3065.15771661;false
2013.07.14 20:17:23:132,5,3.0 <-TRADE_SIZE
2013.07.14 20:17:23:233,8,1108.0 <-VOLUME
2013.07.14 20:17:23:324,0,5.0
2013.07.14 20:17:23:328,3,7.0
2013.07.14 20:17:24:052,0,6.0

i) There is almost a 030ms (average seems to be like 040ms) lag between when trade is executed and when we get RTVolume response.

ii) I get a tickString response timestamp, then I get a trade, trade size, tickString Tick data, then trade size and then volume upto that point. Seems 2nd trade_size is not preceded with TRADE_PRICE if the price is same as last traded price.

iii) Seems we can construct TRADE ticks using tickPrice stream ourselves, although VWAP might get messed up if data breaks in between.

Yes its java, your tools sound quite interesting..

The time lag was simply the local time when I get a IB response, in
milliseconds. The lag I meant was the time lag between when i get
tickPrice and RTVolume. Seems first I get "id=0
lastTimestamp=1373851042" response, then tickPrice/tickSize response
and then I get RTVolume. Also, appears that tickSize response can come
even after RTVolume, as if two threads are writing the callback response.

--- In TWSAPI@... <mailto:TWSAPI%40yahoogroups.com>,
"rwk2095" <r@...> wrote:

Your original post (OP) makes more sense now. I was unfamiliar with

TickUtils.jar because it appears to be Java, and I don't speak Java.

It sounds like I am doing something similar to TickUtils.jar. I

write quotes and trades into a file of fixed-length records, 20 bytes
each, which I refer to as a blob (binary large object). I then load
the blob into memory for detailed analysis at my leisure using a
single read command.

As Richard mentioned, a blob is not directly readable, but I have

tools that extract pieces and put them out in .csv format for analysis
in Excel. I also have a playback tool that can replay a segment of the
day at whatever speed I select.

If you're using the RTVolume generic, you might also specify the

"mdoff" flag to suppress quotes in the regular quote stream so that
you're seeing only trades. This saves bandwidth. If you need quotes
too, you can get them from reqMktDepthEx().

I don't understand the time lag you mentioned. How are you measuring

the lag? It's my understanding that Windows cannot accurately measure
time intervals shorter than about 65 msec. I believe the regular data
stream from reqMktDataEx() is unsuitable for understanding market
microstructure.

[rwk]



--- unatnahs57 <no_reply@> wrote:

1. I am making data request as: mClientSocket.reqMktData(id,

contract, "", false);

which returns the "last price and quantity in separate callbacks".

So, using the post #24461 that you suggested I made the request

as: mClientSocket.reqMktData(id, contract, "233", false);

which returns Last Trade tick as:
id=0 RTVolume=3066.00;4;1373851042866;1108;3065.15771661;false

I can now construct the LastTick object instance used in

TickUtils.jar.

(Not yet sure if BidTick, AskTick can be done this way, but I'll

investigate further, I wanted to reply at the earliest).

++++++
2a. I also noticed an interesting thing. Request made with only

"233" still gives me IB response for tickPrice, tickSize methods, so
I'm actually using more bandwidth.

2b. Also, please take a look at the data below (I tested $NQ_F)

2013.07.14 20:17:14:836,0,6.0
2013.07.14 20:17:15:759,0,7.0
2013.07.14 20:17:20:208,0,6.0
id=0 lastTimestamp=1373851042 <-tickString() response
2013.07.14 20:17:21:989,4,3066.0 <-TRADE_PRICE
2013.07.14 20:17:21:993,5,1.0 <-TRADE_SIZE
2013.07.14 20:17:21:999,8,1105.0 <-VOLUME
2013.07.14 20:17:22:004,1,3065.75
2013.07.14 20:17:22:014,0,4.0
2013.07.14 20:17:22:021,0,4.0
2013.07.14 20:17:22:025,3,6.0
id=0 RTVolume=3066.00;4;1373851042866;1108;3065.15771661;false
2013.07.14 20:17:23:132,5,3.0 <-TRADE_SIZE
2013.07.14 20:17:23:233,8,1108.0 <-VOLUME
2013.07.14 20:17:23:324,0,5.0
2013.07.14 20:17:23:328,3,7.0
2013.07.14 20:17:24:052,0,6.0

i) There is almost a 030ms (average seems to be like 040ms) lag

between when trade is executed and when we get RTVolume response.

ii) I get a tickString response timestamp, then I get a trade,

trade size, tickString Tick data, then trade size and then volume upto
that point. Seems 2nd trade_size is not preceded with TRADE_PRICE if
the price is same as last traded price.

iii) Seems we can construct TRADE ticks using tickPrice stream

ourselves, although VWAP might get messed up if data breaks in between.