Richard,
Thank you for your detailed reply and for sharing your encoder
1. It seems to me that we get the last traded price(on a change as u say) and the all tickSize are useful and represent trade sizes at the last traded price. This checks out with the RTVolume output, as in most cases I find the tickSize add up to the volume reported in the RTVolume tick. I can construct the classes with this information, although was hoping I could get it from someone in the forum. I'll post a java class I've written so far.
2. I've started recording every every tick I get from tickPrice/tickSize methods in the format of the data I posted, one text file per instrument. files sizes for emini ES,YM,NQ are about 100Mb with depth information (also because of the long timestamp i use) rest are smaller daily. Have a 2TB network drive so its not an issue.
3. Problem is I don't use a data base yet and a fast Java disk file read for the futures file is still quite slow for that file size when running a backtest. I think compression would greatly speed things up for me, especially if decoding isn't much of an overhead. I will continue to save text files that are readable in say "R".
toggle quoted message
Show quoted text
--- In TWSAPI@..., "Richard L King" <rlking@...> wrote:
I'm the author of the TickUtils.jar you're referring to.
(For others reading this thread, this jar can be found in the Files section
of this group. It provides encoding and decoding facilities to store tick
data in a much smaller space than either plain text or some kind of fixed
binary format. I use it for storing tick data in an SQL database. At the
time I originally wrote it, I needed to minimize the space occupied by the
data, because disk space was still expensive back then and database backups
were interminable over a 10Mb/s network. I also didn't have the performance
necessary to write a row for each tick, so I batch up the ticks for each
minute into a blob using the encoding routines, and write the blobs at a
predictable rate. Nowadays disk space is almost trivially cheap. My database
is now around 25GB, and would be about 150GB without this encoding - not a
huge file by modern standards. However the performance aspect may still be
relevant, given that at some times I can be recording several hundred ticks
per second. By the way, the encoding/decoding overhead is completely
insignificant.)
If you want to use this jar with IB's Java API, you need to understand a few
things.
1. IB actually sends trade price and the corresponding trade size in a
single message from TWS to the API client, but the API code then splits that
into two separate callbacks - a tickPrice and a tickSize. (The same applies
to bids and asks.)
2. This combined message is only sent when the price changes - otherwise
just the size is sent, and a tickSize callback occurs.
3. After sending the combined price/size message, TWS then sends the exact
same size again as a separate message, and the API generates another
identical tickSize callback from this (there are historical reasons for this
apparently nonsensical 'feature', though I suspect they are no longer
valid).
4. If a trade occurs at the same price and size as the previously reported
trade, not even the tick size is sent - just a volume update. (Note that the
'previously reported trade' isn't necessarily the same as the most recent
actual trade, because the API doesn't necessarily report every trade.)
From this information, we can make the following deductions:
- when a tickPrice is received, the next tickSize (for the same tick type
obviously) will be the size for that price (and will in fact occur
immediately after, as it happens during the processing of the same socket
message from TWS).
- when a tickSize is received that is identical to the previous tickSize, it
should be ignored, because it is just the duplicate that TWS sends after
sending the combined price/size message - it doesn't represent a separate
trade.
You should now have enough understanding to be able to generate the formats
required by the TickUtils.jar.
By the way, please think carefully about whether you actually need to use
these encoding/decoding routines. For example, if you just need to store
tick data in ordinary files, it's best to simply write it out as text. The
problem with the encoded format is that it's not human-readable. I actually
do this in addition to my database recording, as a sort of backup - the
files are compressed (by the operating system), which reduces their size to
about one-sixth of their uncompressed size.
Richard
From: TWSAPI@... [mailto:TWSAPI@...] On Behalf Of
sc1447
Sent: 14 July 2013 20:55
To: TWSAPI@...
Subject: [TWS API] Tick Encoding/Decoding
Hi All,
I'm trying to encode tick data from IB, and wanted to use TickUtils.jar
contributed here. But it seems to work with standard tick formats for
example a Trade tick would have format (%time %price %trade size).
IB data shows up as (%time %tick type %value), so IB would send two tick
responses for a trade tick, instead of one.
I was wondering if anyone was using the encoding utility for IB and if they
would have a converter for IB tick data into the the format the encoder
needs and are willing to share it.
Thanks
S
[Non-text portions of this message have been removed]
