Re: Tick Encoding/Decoding
Yes its java, your tools sound quite interesting..
The time lag was simply the local time when I get a IB response, in milliseconds. The lag I meant was the time lag between when i get tickPrice and RTVolume. Seems first I get "id=0 lastTimestamp=1373851042" response, then tickPrice/tickSize response and then I get RTVolume. Also, appears that tickSize response can come even after RTVolume, as if two threads are writing the callback response.
toggle quoted message
Show quoted text
--- In TWSAPI@..., "rwk2095" <r@...> wrote:
Your original post (OP) makes more sense now. I was unfamiliar with TickUtils.jar because it appears to be Java, and I don't speak Java.
It sounds like I am doing something similar to TickUtils.jar. I write quotes and trades into a file of fixed-length records, 20 bytes each, which I refer to as a blob (binary large object). I then load the blob into memory for detailed analysis at my leisure using a single read command.
As Richard mentioned, a blob is not directly readable, but I have tools that extract pieces and put them out in .csv format for analysis in Excel. I also have a playback tool that can replay a segment of the day at whatever speed I select.
If you're using the RTVolume generic, you might also specify the "mdoff" flag to suppress quotes in the regular quote stream so that you're seeing only trades. This saves bandwidth. If you need quotes too, you can get them from reqMktDepthEx().
I don't understand the time lag you mentioned. How are you measuring the lag? It's my understanding that Windows cannot accurately measure time intervals shorter than about 65 msec. I believe the regular data stream from reqMktDataEx() is unsuitable for understanding market microstructure.
[rwk]
--- unatnahs57 <no_reply@> wrote:
1. I am making data request as: mClientSocket.reqMktData(id, contract, "", false);
which returns the "last price and quantity in separate callbacks".
So, using the post #24461 that you suggested I made the request as: mClientSocket.reqMktData(id, contract, "233", false);
which returns Last Trade tick as:
id=0 RTVolume=3066.00;4;1373851042866;1108;3065.15771661;false
I can now construct the LastTick object instance used in TickUtils.jar.
(Not yet sure if BidTick, AskTick can be done this way, but I'll investigate further, I wanted to reply at the earliest).
++++++
2a. I also noticed an interesting thing. Request made with only "233" still gives me IB response for tickPrice, tickSize methods, so I'm actually using more bandwidth.
2b. Also, please take a look at the data below (I tested $NQ_F)
2013.07.14 20:17:14:836,0,6.0
2013.07.14 20:17:15:759,0,7.0
2013.07.14 20:17:20:208,0,6.0
id=0 lastTimestamp=1373851042 <-tickString() response
2013.07.14 20:17:21:989,4,3066.0 <-TRADE_PRICE
2013.07.14 20:17:21:993,5,1.0 <-TRADE_SIZE
2013.07.14 20:17:21:999,8,1105.0 <-VOLUME
2013.07.14 20:17:22:004,1,3065.75
2013.07.14 20:17:22:014,0,4.0
2013.07.14 20:17:22:021,0,4.0
2013.07.14 20:17:22:025,3,6.0
id=0 RTVolume=3066.00;4;1373851042866;1108;3065.15771661;false
2013.07.14 20:17:23:132,5,3.0 <-TRADE_SIZE
2013.07.14 20:17:23:233,8,1108.0 <-VOLUME
2013.07.14 20:17:23:324,0,5.0
2013.07.14 20:17:23:328,3,7.0
2013.07.14 20:17:24:052,0,6.0
i) There is almost a 030ms (average seems to be like 040ms) lag between when trade is executed and when we get RTVolume response.
ii) I get a tickString response timestamp, then I get a trade, trade size, tickString Tick data, then trade size and then volume upto that point. Seems 2nd trade_size is not preceded with TRADE_PRICE if the price is same as last traded price.
iii) Seems we can construct TRADE ticks using tickPrice stream ourselves, although VWAP might get messed up if data breaks in between.
|
Re: Tick Encoding/Decoding
Richard,
Thank you for your detailed reply and for sharing your encoder
1. It seems to me that we get the last traded price(on a change as u say) and the all tickSize are useful and represent trade sizes at the last traded price. This checks out with the RTVolume output, as in most cases I find the tickSize add up to the volume reported in the RTVolume tick. I can construct the classes with this information, although was hoping I could get it from someone in the forum. I'll post a java class I've written so far.
2. I've started recording every every tick I get from tickPrice/tickSize methods in the format of the data I posted, one text file per instrument. files sizes for emini ES,YM,NQ are about 100Mb with depth information (also because of the long timestamp i use) rest are smaller daily. Have a 2TB network drive so its not an issue.
3. Problem is I don't use a data base yet and a fast Java disk file read for the futures file is still quite slow for that file size when running a backtest. I think compression would greatly speed things up for me, especially if decoding isn't much of an overhead. I will continue to save text files that are readable in say "R".
toggle quoted message
Show quoted text
--- In TWSAPI@..., "Richard L King" <rlking@...> wrote:
I'm the author of the TickUtils.jar you're referring to.
(For others reading this thread, this jar can be found in the Files section
of this group. It provides encoding and decoding facilities to store tick
data in a much smaller space than either plain text or some kind of fixed
binary format. I use it for storing tick data in an SQL database. At the
time I originally wrote it, I needed to minimize the space occupied by the
data, because disk space was still expensive back then and database backups
were interminable over a 10Mb/s network. I also didn't have the performance
necessary to write a row for each tick, so I batch up the ticks for each
minute into a blob using the encoding routines, and write the blobs at a
predictable rate. Nowadays disk space is almost trivially cheap. My database
is now around 25GB, and would be about 150GB without this encoding - not a
huge file by modern standards. However the performance aspect may still be
relevant, given that at some times I can be recording several hundred ticks
per second. By the way, the encoding/decoding overhead is completely
insignificant.)
If you want to use this jar with IB's Java API, you need to understand a few
things.
1. IB actually sends trade price and the corresponding trade size in a
single message from TWS to the API client, but the API code then splits that
into two separate callbacks - a tickPrice and a tickSize. (The same applies
to bids and asks.)
2. This combined message is only sent when the price changes - otherwise
just the size is sent, and a tickSize callback occurs.
3. After sending the combined price/size message, TWS then sends the exact
same size again as a separate message, and the API generates another
identical tickSize callback from this (there are historical reasons for this
apparently nonsensical 'feature', though I suspect they are no longer
valid).
4. If a trade occurs at the same price and size as the previously reported
trade, not even the tick size is sent - just a volume update. (Note that the
'previously reported trade' isn't necessarily the same as the most recent
actual trade, because the API doesn't necessarily report every trade.)
From this information, we can make the following deductions:
- when a tickPrice is received, the next tickSize (for the same tick type
obviously) will be the size for that price (and will in fact occur
immediately after, as it happens during the processing of the same socket
message from TWS).
- when a tickSize is received that is identical to the previous tickSize, it
should be ignored, because it is just the duplicate that TWS sends after
sending the combined price/size message - it doesn't represent a separate
trade.
You should now have enough understanding to be able to generate the formats
required by the TickUtils.jar.
By the way, please think carefully about whether you actually need to use
these encoding/decoding routines. For example, if you just need to store
tick data in ordinary files, it's best to simply write it out as text. The
problem with the encoded format is that it's not human-readable. I actually
do this in addition to my database recording, as a sort of backup - the
files are compressed (by the operating system), which reduces their size to
about one-sixth of their uncompressed size.
Richard
From: TWSAPI@... [mailto:TWSAPI@...] On Behalf Of
sc1447
Sent: 14 July 2013 20:55
To: TWSAPI@...
Subject: [TWS API] Tick Encoding/Decoding
Hi All,
I'm trying to encode tick data from IB, and wanted to use TickUtils.jar
contributed here. But it seems to work with standard tick formats for
example a Trade tick would have format (%time %price %trade size).
IB data shows up as (%time %tick type %value), so IB would send two tick
responses for a trade tick, instead of one.
I was wondering if anyone was using the encoding utility for IB and if they
would have a converter for IB tick data into the the format the encoder
needs and are willing to share it.
Thanks
S
[Non-text portions of this message have been removed]
|
Re: Connect to TWS in main not in Thread (Java)
Hi btw12342001,
thanks so much for the tip! That actually helps me a lot. I still did not completely understood your earlier reply on how to handle returned data. In which method is the returned data handled? I guess if the connection is not limited to just one thread you also run into synchronization issues between the threads. But as far as I understood you can initiate 6 or 8 connections but IB recommends to use as little as possible, is that correct?
Regards
toggle quoted message
Show quoted text
--- In TWSAPI@..., "btw12342001" <newguy@...> wrote:
Not much of automation when I need to authorize it manually every time :) To fix the authorizing part you just need to set trusted IP addresses in TWS. Add 127.0.0.1 for the local machine.
|
Re: Upcoming stock earnings dates
I found that subscribing to Wall Street Horizon (2.5usd) there are available upcoming earnings dates in TWS. They are also available for the API.
Though testing this shows it seems not really functional. The TWS data lacks dates for many stocks. The API only returns symbols, no detail of type of event or date, and it also returns maximum 50 symbols, where TWS shows 250 (I assume the first upcoming 250).
So not very functional, but may still may help some. Like Japanese stocks can be checked, and you may then enter manually in TWS to get accurate date. Haven't found Japanese earnings anywhere else on the web. If someone has, please share..
Have questioned API support about the problems with this feature, so will see if they have some clarifications.
|
Re: Tick Encoding/Decoding
id=0 RTVolume=3066.00;4;1373851042866;1108;3065.15771661;false
2013.07.14 20:17:23:132,5,3.0 <-TRADE_SIZE
2013.07.14 20:17:23:233,8,1108.0 <-VOLUME
2013.07.14 20:17:23:324,0,5.0
2013.07.14 20:17:23:328,3,7.0
2013.07.14 20:17:24:052,0,6.0
i) There is almost a 030ms (average seems to be like 040ms) lag between when trade is executed and when we get RTVolume response.
The time in your log is from your computer and the RTV time is a timestamp, I'm guessing from the exchange. You have to have your clock accurate in order to measure the delay till when you get the tick. When you set your clock don't forget the ping time from the NIST server or wherever you get it from. Frankly I don't think you can measure that accurately enough to matter even though XP and later OSes can measure very fine time intervals nowadays (much better than the old 1/18 sec). My guess is RTV arrives at home much later than 40 msec after the actual exchange trade. My ping time just to IB's server is 60msec. iii) Seems we can construct TRADE ticks using tickPrice stream ourselves, although VWAP might get messed up if data breaks in between. No, you need tickSize for when price doesn't change. You can get everything useful from the RTV tick except bid and ask prices, for that you need tick price.
|
Re: Tick Encoding/Decoding
I would just add that the whole issue of timing under Windows is
rather a mess. If you want to get resolution higher than the default
you'll have to research a bit to find the implications and gotcha's.
toggle quoted message
Show quoted text
On 7/15/2013 11:07 AM, Richard L King wrote:
rwk, your 'understanding' that Windows cannot accurately measure time
intervals shorter than 65 msec is completely erroneous.
If you're programming directly to the Windows API, you can use
QueryPerformanceCounter and QueryPerformanceFrequency. In .Net, use the
Stopwatch class. With these, you can measure pretty accurately down to the
microsecond level.
These all depend on the presence of a high-resolution timer in your PC's
hardware. This doesn't appear to be much of a limitation - I've never come
across a computer that doesn't have one.
Also if you want to do timer-based actions, you can do these with very
small
intervals by using the timeBeginPeriod to set the minimum timer resolution
(down to 1 millisec). Use createTimerQueue and CreateTimerQueueTimer to
initiate the timers (can't remember offhand what the corresponding .Net
timer facility is, and I don't have time to look it up).
Richard
From: TWSAPI@... <mailto:TWSAPI%40yahoogroups.com>
[mailto:TWSAPI@... <mailto:TWSAPI%40yahoogroups.com>] On
Behalf Of
rwk2095
Sent: 15 July 2013 15:18
To: TWSAPI@... <mailto:TWSAPI%40yahoogroups.com>
Subject: [TWS API] Re: Tick Encoding/Decoding
Your original post (OP) makes more sense now. I was unfamiliar with
TickUtils.jar because it appears to be Java, and I don't speak Java.
It sounds like I am doing something similar to TickUtils.jar. I write
quotes
and trades into a file of fixed-length records, 20 bytes each, which I
refer
to as a blob (binary large object). I then load the blob into memory for
detailed analysis at my leisure using a single read command.
As Richard mentioned, a blob is not directly readable, but I have
tools that
extract pieces and put them out in .csv format for analysis in Excel.
I also
have a playback tool that can replay a segment of the day at whatever
speed
I select.
If you're using the RTVolume generic, you might also specify the "mdoff"
flag to suppress quotes in the regular quote stream so that you're seeing
only trades. This saves bandwidth. If you need quotes too, you can get
them
from reqMktDepthEx().
I don't understand the time lag you mentioned. How are you measuring the
lag? It's my understanding that Windows cannot accurately measure time
intervals shorter than about 65 msec. I believe the regular data
stream from
reqMktDataEx() is unsuitable for understanding market microstructure.
[rwk]
--- unatnahs57 <no_reply@...> wrote:
1. I am making data request as: mClientSocket.reqMktData(id, contract, "",
false);
which returns the "last price and quantity in separate callbacks".
So, using the post #24461 that you suggested I made the request as: mClientSocket.reqMktData(id, contract, "233", false);
which returns Last Trade tick as:
id=0 RTVolume=3066.00;4;1373851042866;1108;3065.15771661;false
I can now construct the LastTick object instance used in TickUtils.jar.
(Not yet sure if BidTick, AskTick can be done this way, but I'll
investigate further, I wanted to reply at the earliest).
++++++
2a. I also noticed an interesting thing. Request made with only "233"
still gives me IB response for tickPrice, tickSize methods, so I'm
actually
using more bandwidth.
2b. Also, please take a look at the data below (I tested $NQ_F)
2013.07.14 20:17:14:836,0,6.0
2013.07.14 20:17:15:759,0,7.0
2013.07.14 20:17:20:208,0,6.0
id=0 lastTimestamp=1373851042 <-tickString() response
2013.07.14 20:17:21:989,4,3066.0 <-TRADE_PRICE
2013.07.14 20:17:21:993,5,1.0 <-TRADE_SIZE
2013.07.14 20:17:21:999,8,1105.0 <-VOLUME
2013.07.14 20:17:22:004,1,3065.75
2013.07.14 20:17:22:014,0,4.0
2013.07.14 20:17:22:021,0,4.0
2013.07.14 20:17:22:025,3,6.0
id=0 RTVolume=3066.00;4;1373851042866;1108;3065.15771661;false
2013.07.14 20:17:23:132,5,3.0 <-TRADE_SIZE
2013.07.14 20:17:23:233,8,1108.0 <-VOLUME
2013.07.14 20:17:23:324,0,5.0
2013.07.14 20:17:23:328,3,7.0
2013.07.14 20:17:24:052,0,6.0
i) There is almost a 030ms (average seems to be like 040ms) lag between
when trade is executed and when we get RTVolume response.
ii) I get a tickString response timestamp, then I get a trade, trade
size,
tickString Tick data, then trade size and then volume upto that point.
Seems
2nd trade_size is not preceded with TRADE_PRICE if the price is same
as last
traded price.
iii) Seems we can construct TRADE ticks using tickPrice stream
ourselves,
although VWAP might get messed up if data breaks in between.
|
Re: Tick Encoding/Decoding
rwk, your 'understanding' that Windows cannot accurately measure time intervals shorter than 65 msec is completely erroneous. If you're programming directly to the Windows API, you can use QueryPerformanceCounter and QueryPerformanceFrequency. In .Net, use the Stopwatch class. With these, you can measure pretty accurately down to the microsecond level. These all depend on the presence of a high-resolution timer in your PC's hardware. This doesn't appear to be much of a limitation - I've never come across a computer that doesn't have one. Also if you want to do timer-based actions, you can do these with very small intervals by using the timeBeginPeriod to set the minimum timer resolution (down to 1 millisec). Use createTimerQueue and CreateTimerQueueTimer to initiate the timers (can't remember offhand what the corresponding .Net timer facility is, and I don't have time to look it up). Richard From: TWSAPI@... [mailto:TWSAPI@...] On Behalf Of rwk2095 Sent: 15 July 2013 15:18 To: TWSAPI@... Subject: [TWS API] Re: Tick Encoding/Decoding Your original post (OP) makes more sense now. I was unfamiliar with TickUtils.jar because it appears to be Java, and I don't speak Java. It sounds like I am doing something similar to TickUtils.jar. I write quotes and trades into a file of fixed-length records, 20 bytes each, which I refer to as a blob (binary large object). I then load the blob into memory for detailed analysis at my leisure using a single read command. As Richard mentioned, a blob is not directly readable, but I have tools that extract pieces and put them out in .csv format for analysis in Excel. I also have a playback tool that can replay a segment of the day at whatever speed I select. If you're using the RTVolume generic, you might also specify the "mdoff" flag to suppress quotes in the regular quote stream so that you're seeing only trades. This saves bandwidth. If you need quotes too, you can get them from reqMktDepthEx(). I don't understand the time lag you mentioned. How are you measuring the lag? It's my understanding that Windows cannot accurately measure time intervals shorter than about 65 msec. I believe the regular data stream from reqMktDataEx() is unsuitable for understanding market microstructure. [rwk] --- unatnahs57 <no_reply@...> wrote: 1. I am making data request as: mClientSocket.reqMktData(id, contract, "", false); which returns the "last price and quantity in separate callbacks".
So, using the post #24461 that you suggested I made the request as: mClientSocket.reqMktData(id, contract, "233", false);
which returns Last Trade tick as:
id=0 RTVolume=3066.00;4;1373851042866;1108;3065.15771661;false
I can now construct the LastTick object instance used in TickUtils.jar.
(Not yet sure if BidTick, AskTick can be done this way, but I'll
investigate further, I wanted to reply at the earliest).
++++++
2a. I also noticed an interesting thing. Request made with only "233"
still gives me IB response for tickPrice, tickSize methods, so I'm actually using more bandwidth.
2b. Also, please take a look at the data below (I tested $NQ_F)
2013.07.14 20:17:14:836,0,6.0
2013.07.14 20:17:15:759,0,7.0
2013.07.14 20:17:20:208,0,6.0
id=0 lastTimestamp=1373851042 <-tickString() response
2013.07.14 20:17:21:989,4,3066.0 <-TRADE_PRICE
2013.07.14 20:17:21:993,5,1.0 <-TRADE_SIZE
2013.07.14 20:17:21:999,8,1105.0 <-VOLUME
2013.07.14 20:17:22:004,1,3065.75
2013.07.14 20:17:22:014,0,4.0
2013.07.14 20:17:22:021,0,4.0
2013.07.14 20:17:22:025,3,6.0
id=0 RTVolume=3066.00;4;1373851042866;1108;3065.15771661;false
2013.07.14 20:17:23:132,5,3.0 <-TRADE_SIZE
2013.07.14 20:17:23:233,8,1108.0 <-VOLUME
2013.07.14 20:17:23:324,0,5.0
2013.07.14 20:17:23:328,3,7.0
2013.07.14 20:17:24:052,0,6.0
i) There is almost a 030ms (average seems to be like 040ms) lag between
when trade is executed and when we get RTVolume response.
ii) I get a tickString response timestamp, then I get a trade, trade size,
tickString Tick data, then trade size and then volume upto that point. Seems 2nd trade_size is not preceded with TRADE_PRICE if the price is same as last traded price.
iii) Seems we can construct TRADE ticks using tickPrice stream ourselves,
although VWAP might get messed up if data breaks in between.
|
C# Option Contract Details
You should be able to get all options by calling reqContractDetailsEx(). Just leave strike, right, and possibly expiry blank/zero. [rwk] --- "andrewcmeier" <andrewcmeier@...> wrote:
Still not sure how to get all options for a stock but will keep trying.
|
Re: Tick Encoding/Decoding
Your original post (OP) makes more sense now. I was unfamiliar with TickUtils.jar because it appears to be Java, and I don't speak Java.
It sounds like I am doing something similar to TickUtils.jar. I write quotes and trades into a file of fixed-length records, 20 bytes each, which I refer to as a blob (binary large object). I then load the blob into memory for detailed analysis at my leisure using a single read command.
As Richard mentioned, a blob is not directly readable, but I have tools that extract pieces and put them out in .csv format for analysis in Excel. I also have a playback tool that can replay a segment of the day at whatever speed I select.
If you're using the RTVolume generic, you might also specify the "mdoff" flag to suppress quotes in the regular quote stream so that you're seeing only trades. This saves bandwidth. If you need quotes too, you can get them from reqMktDepthEx().
I don't understand the time lag you mentioned. How are you measuring the lag? It's my understanding that Windows cannot accurately measure time intervals shorter than about 65 msec. I believe the regular data stream from reqMktDataEx() is unsuitable for understanding market microstructure.
[rwk]
--- unatnahs57 <no_reply@...> wrote:
toggle quoted message
Show quoted text
1. I am making data request as: mClientSocket.reqMktData(id, contract, "", false);
which returns the "last price and quantity in separate callbacks".
So, using the post #24461 that you suggested I made the request as: mClientSocket.reqMktData(id, contract, "233", false);
which returns Last Trade tick as:
id=0 RTVolume=3066.00;4;1373851042866;1108;3065.15771661;false
I can now construct the LastTick object instance used in TickUtils.jar.
(Not yet sure if BidTick, AskTick can be done this way, but I'll investigate further, I wanted to reply at the earliest).
++++++
2a. I also noticed an interesting thing. Request made with only "233" still gives me IB response for tickPrice, tickSize methods, so I'm actually using more bandwidth.
2b. Also, please take a look at the data below (I tested $NQ_F)
2013.07.14 20:17:14:836,0,6.0
2013.07.14 20:17:15:759,0,7.0
2013.07.14 20:17:20:208,0,6.0
id=0 lastTimestamp=1373851042 <-tickString() response
2013.07.14 20:17:21:989,4,3066.0 <-TRADE_PRICE
2013.07.14 20:17:21:993,5,1.0 <-TRADE_SIZE
2013.07.14 20:17:21:999,8,1105.0 <-VOLUME
2013.07.14 20:17:22:004,1,3065.75
2013.07.14 20:17:22:014,0,4.0
2013.07.14 20:17:22:021,0,4.0
2013.07.14 20:17:22:025,3,6.0
id=0 RTVolume=3066.00;4;1373851042866;1108;3065.15771661;false
2013.07.14 20:17:23:132,5,3.0 <-TRADE_SIZE
2013.07.14 20:17:23:233,8,1108.0 <-VOLUME
2013.07.14 20:17:23:324,0,5.0
2013.07.14 20:17:23:328,3,7.0
2013.07.14 20:17:24:052,0,6.0
i) There is almost a 030ms (average seems to be like 040ms) lag between when trade is executed and when we get RTVolume response.
ii) I get a tickString response timestamp, then I get a trade, trade size, tickString Tick data, then trade size and then volume upto that point. Seems 2nd trade_size is not preceded with TRADE_PRICE if the price is same as last traded price.
iii) Seems we can construct TRADE ticks using tickPrice stream ourselves, although VWAP might get messed up if data breaks in between.
|
C# Option Contract Details
I am not sure if anyone has answered your question to your satisfaction yet but I wanted to give you what I have figured out for options.
using request market data you would have something like this in c#
client.RequestMarketData(i, new Option("BAC", "BAC130720C00012000", "", RightType.Undefined, 0), some collection, false, false);
toggle quoted message
Show quoted text
--- In TWSAPI@..., "andrewcmeier" <andrewcmeier@...> wrote:
I've tried both dates, no luck.
--- In TWSAPI@..., Peter Gum <petergum@> wrote:
Should '20130719' be '20130720'? I.e. 19 -> 20 to match the tws market line. (Although IB will expire the option on the 19th. Confusing.)
Pete
On 07/14/13, andrewcmeier<andrewcmeier@> wrote:
Oops, Tester and A1 were meant to be the same variable in the options lines, change Tester to A1 and that is what I have.
--- In TWSAPI@..., "andrewcmeier" <andrewcmeier@> wrote:
Hi there,
I've started using the Krs.Ats.IBNet software () to develop TWS software in C#.
Setting up equities is easy enough:
Equity Google = new Equity("GOOG");
client.RequestMarketData(14, Google, null, true, false);
But when I try to adapt it to options, I can't get it to work. This is what I'm using:
Option A1 = new Option("AA", "AA", "20130719", RightType.Call, 8.0m);
client.RequestMarketData(15, Tester, null, false, false);
This is what it has in its class):
public Option(string equitySymbol, string optionSymbol, string expiry,
RightType right, decimal strike)
This is the error I get when I raise e.ErrorCode:
"No security definition has been found for the request"
Does anyone know what I'm entering wrong, or just how to set up an option in C#? I also then wan't to be able to leave fields out to do things such as return all the contracts in a month, does anyone have any examples how to do this?
I'm finding that C# should be a good way to download and analyse data but don't have any documentation, I've only worked out what to do from searching through lots of the code.
Thanks for your help.
|
Re: problem with contract details
Good news... the problem will be fixed on 7/18. We just need to include the full expiration in the format of YYYYMMDD when invoking the reqContractDeta ils( ) method.
toggle quoted message
Show quoted text
--- In TWSAPI@..., dairen62 <no_reply@...> wrote:
i received exactly the same answer... hope for a quick solution :)
--- In TWSAPI@..., clove2hitch <no_reply@> wrote:
just got an answer from IB API support.
1 minute issue should be solved in near future(when???)
Unfortunately IB imposes limitations on the amount of contracts
that can be returned when invoking the reqContractDetails( ) method.
In this particular case it would be best to include additional
contract parameters which will narrow your scope.
Example if you include m_expiration and m_right then you can
determine all the available strikes for that particular expiration.
Alternatively if you include m_strike and m_right you can determine
all the available expirations.
But please note there are currently an issue that development is
working on with respect to Option Contracts. It seems even though
you include the m_expiry there still is a 1 minute buffer for
additional requests. Once this has been resolved I will provide
you a follow-up comment on this ticket with additional details,
--- In TWSAPI@..., clove2hitch <no_reply@> wrote:
Hi, i experience same issues!
the first request answers quickly then a second after 1 minute
then nothing... :-(
I updated TWS and API to use latest stuff but even worse
with the <>clients.exe crash (previous API clients versions don't crash).
I'm going to log the issue to IB.
Regards
CH
--- In TWSAPI@..., dairen62 <no_reply@> wrote:
Yes, when i try to take the options chain every call to reContractDetailsEx (with strike set to zero) lasts 1 minute instead the normal 6 seconds. I also notice that the demo program of the API (client3.exe) crash when you use "reqContractData". Can you try if you have the same problem?
thanks
--- In TWSAPI@..., "rwk2095" <r@...> wrote:
Is this still a problem? I just ran a program at 8:25 EST on 11 JUL, and everything appears normal on Globex (E6).
[rwk]
--- dairen62 <no_reply@> wrote:
hi everybody, i've a little program that that takes the chain of the options using the method reqContractDetailsEx (ActiveX). It worked fine till yesterday: every call to reqContractDetailsEx lasts 1 minute versus the 6 seconds of before yesterday. In this way the program is dramaticaly slow. Anyone experienced the same?
|
Re: Tick Encoding/Decoding
I'm the author of the TickUtils.jar you're referring to.
(For others reading this thread, this jar can be found in the Files section
of this group. It provides encoding and decoding facilities to store tick
data in a much smaller space than either plain text or some kind of fixed
binary format. I use it for storing tick data in an SQL database. At the
time I originally wrote it, I needed to minimize the space occupied by the
data, because disk space was still expensive back then and database backups
were interminable over a 10Mb/s network. I also didn't have the performance
necessary to write a row for each tick, so I batch up the ticks for each
minute into a blob using the encoding routines, and write the blobs at a
predictable rate. Nowadays disk space is almost trivially cheap. My database
is now around 25GB, and would be about 150GB without this encoding - not a
huge file by modern standards. However the performance aspect may still be
relevant, given that at some times I can be recording several hundred ticks
per second. By the way, the encoding/decoding overhead is completely
insignificant.)
If you want to use this jar with IB's Java API, you need to understand a few
things.
1. IB actually sends trade price and the corresponding trade size in a
single message from TWS to the API client, but the API code then splits that
into two separate callbacks - a tickPrice and a tickSize. (The same applies
to bids and asks.)
2. This combined message is only sent when the price changes - otherwise
just the size is sent, and a tickSize callback occurs.
3. After sending the combined price/size message, TWS then sends the exact
same size again as a separate message, and the API generates another
identical tickSize callback from this (there are historical reasons for this
apparently nonsensical 'feature', though I suspect they are no longer
valid).
4. If a trade occurs at the same price and size as the previously reported
trade, not even the tick size is sent - just a volume update. (Note that the
'previously reported trade' isn't necessarily the same as the most recent
actual trade, because the API doesn't necessarily report every trade.)
From this information, we can make the following deductions:
- when a tickPrice is received, the next tickSize (for the same tick type
obviously) will be the size for that price (and will in fact occur
immediately after, as it happens during the processing of the same socket
message from TWS).
- when a tickSize is received that is identical to the previous tickSize, it
should be ignored, because it is just the duplicate that TWS sends after
sending the combined price/size message - it doesn't represent a separate
trade.
You should now have enough understanding to be able to generate the formats
required by the TickUtils.jar.
By the way, please think carefully about whether you actually need to use
these encoding/decoding routines. For example, if you just need to store
tick data in ordinary files, it's best to simply write it out as text. The
problem with the encoded format is that it's not human-readable. I actually
do this in addition to my database recording, as a sort of backup - the
files are compressed (by the operating system), which reduces their size to
about one-sixth of their uncompressed size.
Richard
From: TWSAPI@... [mailto:TWSAPI@...] On Behalf Of
sc1447
Sent: 14 July 2013 20:55
To: TWSAPI@...
Subject: [TWS API] Tick Encoding/Decoding
Hi All,
I'm trying to encode tick data from IB, and wanted to use TickUtils.jar
contributed here. But it seems to work with standard tick formats for
example a Trade tick would have format (%time %price %trade size).
IB data shows up as (%time %tick type %value), so IB would send two tick
responses for a trade tick, instead of one.
I was wondering if anyone was using the encoding utility for IB and if they
would have a converter for IB tick data into the the format the encoder
needs and are willing to share it.
Thanks
S
|
C# Option Contract Details
Excellent, it worked, had to change a couple of other fields but it was definitely the contractID, thank you. The final code I used is:
AAAA = new Contract(0, "AAPL", SecurityType.Option, "20130719", 420.0, RightType.Call, "100", "SMART", "USD", "", "", SecurityIdType.None, "");
Still not sure how to get all options for a stock but will keep trying.
toggle quoted message
Show quoted text
--- In TWSAPI@..., "Richard L King" <rlking@...> wrote:
contractId is IB's own unique identifier for a contract. If you know this id
for a specific contract, you can use it in a contract details request and
not supply any of the other fields.
Setting this to 22 as you have done, and then supplying all the other fields
for a contract whose id is almost certainly not 22, is probably the source
of your error.
So try setting contractId to 0.
Richard
From: TWSAPI@... [mailto:TWSAPI@...] On Behalf Of
andrewcmeier
Sent: 15 July 2013 04:47
To: TWSAPI@...
Subject: [TWS API] Re: C# Option Contract Details
I've had a look, it looks great, however, I still can't seem to fix my
problem. The way you set up a contract in VB is:
contract.conId = Convert.ToInt32(ss(2))
contract.symbol = TextBox1.Text
contract.secType = "OPT"
contract.expiry = ss(0).Trim.ToString
contract.strike = Convert.ToDouble(ss(1))
contract.right = "C"
contract.multiplier = ""
contract.exchange = "SMART"
contract.primaryExchange = ""
contract.currency = "USD"
contract.localSymbol = ""
contract.includeExpired = 0
tickcount += 1
AxTws1.reqMktDataEx(rnumber, contract, "", 1) '0 =streaming 1=snapshot
However, in C# I don't seem to have any of the contract.x and can't even
seem to use .reqMktDataEx()
I've tried another method which uses:
AAAA = new Contract(22, "AAPL", SecurityType.Option, "201307", 420.0,
RightType.Call, "", "SMART", "USD", "", "", SecurityIdType.None, "");
The constructor for this is:
public Contract(int contractId, String symbol, SecurityType securityType,
String expiry, double strike, RightType right,
String multiplier, string exchange, string currency, string localSymbol,
string primaryExchange,
SecurityIdType secIdType, string secId)
It still says no security definition found.
Any idea of what is wrong here?
Thanks again.
--- In TWSAPI@... <mailto:TWSAPI%40yahoogroups.com> , Nick
<nickhere@> wrote:
In the file section of this group
I wrote a option chain getter
I included the source.it in vb
and conversion is easy to c#
i had very little problems with it
it get the chain of any stock ticker
gets all the quotes
and draws vertical spread graphs
nick
On 7/14/2013 6:23 AM, andrewcmeier wrote:
Oops, Tester and A1 were meant to be the same variable in the options
lines, change Tester to A1 and that is what I have.
--- In TWSAPI@... <mailto:TWSAPI%40yahoogroups.com> ,
"andrewcmeier" <andrewcmeier@> wrote:
Hi there,
I've started using the Krs.Ats.IBNet software
(.
0.21.Full.zip) to develop TWS software in C#.
Setting up equities is easy enough:
Equity Google = new Equity("GOOG");
client.RequestMarketData(14, Google, null, true, false);
But when I try to adapt it to options, I can't get it to work. This is
what I'm using:
Option A1 = new Option("AA", "AA", "20130719", RightType.Call, 8.0m);
client.RequestMarketData(15, Tester, null, false, false);
This is what it has in its class):
public Option(string equitySymbol, string optionSymbol, string expiry,
RightType right, decimal strike)
This is the error I get when I raise e.ErrorCode:
"No security definition has been found for the request"
Does anyone know what I'm entering wrong, or just how to set up an
option in C#? I also then wan't to be able to leave fields out to do things
such as return all the contracts in a month, does anyone have any examples
how to do this?
I'm finding that C# should be a good way to download and analyse data
but don't have any documentation, I've only worked out what to do from
searching through lots of the code.
Thanks for your help.
------------------------------------
Yahoo! Groups Links
|
Re: C# Option Contract Details
contractId is IB's own unique identifier for a contract. If you know this id for a specific contract, you can use it in a contract details request and not supply any of the other fields. Setting this to 22 as you have done, and then supplying all the other fields for a contract whose id is almost certainly not 22, is probably the source of your error. So try setting contractId to 0. Richard From: TWSAPI@... [mailto:TWSAPI@...] On Behalf Of andrewcmeier Sent: 15 July 2013 04:47 To: TWSAPI@... Subject: [TWS API] Re: C# Option Contract Details I've had a look, it looks great, however, I still can't seem to fix my problem. The way you set up a contract in VB is: contract.conId = Convert.ToInt32(ss(2)) contract.symbol = TextBox1.Text contract.secType = "OPT" contract.expiry = ss(0).Trim.ToString contract.strike = Convert.ToDouble(ss(1)) contract.right = "C" contract.multiplier = "" contract.exchange = "SMART" contract.primaryExchange = "" contract.currency = "USD" contract.localSymbol = "" contract.includeExpired = 0 tickcount += 1 AxTws1.reqMktDataEx(rnumber, contract, "", 1) '0 =streaming 1=snapshot However, in C# I don't seem to have any of the contract.x and can't even seem to use .reqMktDataEx() I've tried another method which uses: AAAA = new Contract(22, "AAPL", SecurityType.Option, "201307", 420.0, RightType.Call, "", "SMART", "USD", "", "", SecurityIdType.None, ""); The constructor for this is: public Contract(int contractId, String symbol, SecurityType securityType, String expiry, double strike, RightType right, String multiplier, string exchange, string currency, string localSymbol, string primaryExchange, SecurityIdType secIdType, string secId) It still says no security definition found. Any idea of what is wrong here? Thanks again. --- In TWSAPI@... <mailto:TWSAPI%40yahoogroups.com> , Nick <nickhere@...> wrote:
In the file section of this group
I wrote a option chain getter
I included the source.it in vb
and conversion is easy to c#
i had very little problems with it
it get the chain of any stock ticker
gets all the quotes
and draws vertical spread graphs
nick
On 7/14/2013 6:23 AM, andrewcmeier wrote:
Oops, Tester and A1 were meant to be the same variable in the options
lines, change Tester to A1 and that is what I have.
--- In TWSAPI@... <mailto:TWSAPI%40yahoogroups.com> ,
"andrewcmeier" <andrewcmeier@> wrote: Hi there,
I've started using the Krs.Ats.IBNet software
(. 0.21.Full.zip) to develop TWS software in C#.
Setting up equities is easy enough:
Equity Google = new Equity("GOOG");
client.RequestMarketData(14, Google, null, true, false);
But when I try to adapt it to options, I can't get it to work. This is
what I'm using: Option A1 = new Option("AA", "AA", "20130719", RightType.Call, 8.0m);
client.RequestMarketData(15, Tester, null, false, false);
This is what it has in its class):
public Option(string equitySymbol, string optionSymbol, string expiry,
RightType right, decimal strike)
This is the error I get when I raise e.ErrorCode:
"No security definition has been found for the request"
Does anyone know what I'm entering wrong, or just how to set up an
option in C#? I also then wan't to be able to leave fields out to do things such as return all the contracts in a month, does anyone have any examples how to do this?
I'm finding that C# should be a good way to download and analyse data
but don't have any documentation, I've only worked out what to do from searching through lots of the code.
Thanks for your help.
------------------------------------
Yahoo! Groups Links
|
Re: Tick Encoding/Decoding
1. I am making data request as: mClientSocket.reqMktData(id, contract, "", false);
which returns the "last price and quantity in separate callbacks".
So, using the post #24461 that you suggested I made the request as: mClientSocket.reqMktData(id, contract, "233", false);
which returns Last Trade tick as:
id=0 RTVolume=3066.00;4;1373851042866;1108;3065.15771661;false
I can now construct the LastTick object instance used in TickUtils.jar.
(Not yet sure if BidTick, AskTick can be done this way, but I'll investigate further, I wanted to reply at the earliest).
++++++
2a. I also noticed an interesting thing. Request made with only "233" still gives me IB response for tickPrice, tickSize methods, so I'm actually using more bandwidth.
2b. Also, please take a look at the data below (I tested $NQ_F)
2013.07.14 20:17:14:836,0,6.0
2013.07.14 20:17:15:759,0,7.0
2013.07.14 20:17:20:208,0,6.0
id=0 lastTimestamp=1373851042 <-tickString() response
2013.07.14 20:17:21:989,4,3066.0 <-TRADE_PRICE
2013.07.14 20:17:21:993,5,1.0 <-TRADE_SIZE
2013.07.14 20:17:21:999,8,1105.0 <-VOLUME
2013.07.14 20:17:22:004,1,3065.75
2013.07.14 20:17:22:014,0,4.0
2013.07.14 20:17:22:021,0,4.0
2013.07.14 20:17:22:025,3,6.0
id=0 RTVolume=3066.00;4;1373851042866;1108;3065.15771661;false
2013.07.14 20:17:23:132,5,3.0 <-TRADE_SIZE
2013.07.14 20:17:23:233,8,1108.0 <-VOLUME
2013.07.14 20:17:23:324,0,5.0
2013.07.14 20:17:23:328,3,7.0
2013.07.14 20:17:24:052,0,6.0
i) There is almost a 030ms (average seems to be like 040ms) lag between when trade is executed and when we get RTVolume response.
ii) I get a tickString response timestamp, then I get a trade, trade size, tickString Tick data, then trade size and then volume upto that point. Seems 2nd trade_size is not preceded with TRADE_PRICE if the price is same as last traded price.
iii) Seems we can construct TRADE ticks using tickPrice stream ourselves, although VWAP might get messed up if data breaks in between.
I'll be looking at this for a bit, but thanks a lot for your reply.
S.
toggle quoted message
Show quoted text
--- In TWSAPI@..., "rwk2095" <r@...> wrote:
It's not clear from your question what you're trying to accomplish. If you're interested in market microstructure, i.e. very short term, you might have a look at TickString. See this old post:
[rwk]
--- "sc1447" <sc1447@> wrote:
I'm trying to encode tick data from IB, and wanted to use TickUtils.jar contributed here. But it seems to work with standard tick formats for example a Trade tick would have format (%time %price %trade size).
IB data shows up as (%time %tick type %value), so IB would send two tick responses for a trade tick, instead of one.
I was wondering if anyone was using the encoding utility for IB and if they would have a converter for IB tick data into the the format the encoder needs and are willing to share it.
|
Re: C# Option Contract Details
I've given this a quick try, no luck yet, will try again when I get a chance.
toggle quoted message
Show quoted text
--- In TWSAPI@..., Shane Castle <shane.castle@...> wrote:
You're running an older version of ib-csharp.
The latest code is now hosted here:
Download that and give it a try.
Shane
On Sunday, July 14, 2013, andrewcmeier wrote:
Oops, Tester and A1 were meant to be the same variable in the options
lines, change Tester to A1 and that is what I have.
--- In TWSAPI@... <javascript:;>, "andrewcmeier"
<andrewcmeier@> wrote:
Hi there,
I've started using the Krs.Ats.IBNet software (
)
to develop TWS software in C#.
Setting up equities is easy enough:
Equity Google = new Equity("GOOG");
client.RequestMarketData(14, Google, null, true, false);
But when I try to adapt it to options, I can't get it to work. This is
what I'm using:
Option A1 = new Option("AA", "AA", "20130719", RightType.Call, 8.0m);
client.RequestMarketData(15, Tester, null, false, false);
This is what it has in its class):
public Option(string equitySymbol, string optionSymbol, string expiry,
RightType right, decimal strike)
This is the error I get when I raise e.ErrorCode:
"No security definition has been found for the request"
Does anyone know what I'm entering wrong, or just how to set up an option in C#? I also then wan't to be able to leave fields out to do
things such as return all the contracts in a month, does anyone have any
examples how to do this?
I'm finding that C# should be a good way to download and analyse data
but don't have any documentation, I've only worked out what to do from
searching through lots of the code.
Thanks for your help.
------------------------------------
Yahoo! Groups Links
|
C# Option Contract Details
I've tried both dates, no luck.
toggle quoted message
Show quoted text
--- In TWSAPI@..., Peter Gum <petergum@...> wrote:
Should '20130719' be '20130720'? I.e. 19 -> 20 to match the tws market line. (Although IB will expire the option on the 19th. Confusing.)
Pete
On 07/14/13, andrewcmeier<andrewcmeier@...> wrote:
Oops, Tester and A1 were meant to be the same variable in the options lines, change Tester to A1 and that is what I have.
--- In TWSAPI@..., "andrewcmeier" <andrewcmeier@> wrote:
Hi there,
I've started using the Krs.Ats.IBNet software () to develop TWS software in C#.
Setting up equities is easy enough:
Equity Google = new Equity("GOOG");
client.RequestMarketData(14, Google, null, true, false);
But when I try to adapt it to options, I can't get it to work. This is what I'm using:
Option A1 = new Option("AA", "AA", "20130719", RightType.Call, 8.0m);
client.RequestMarketData(15, Tester, null, false, false);
This is what it has in its class):
public Option(string equitySymbol, string optionSymbol, string expiry,
RightType right, decimal strike)
This is the error I get when I raise e.ErrorCode:
"No security definition has been found for the request"
Does anyone know what I'm entering wrong, or just how to set up an option in C#? I also then wan't to be able to leave fields out to do things such as return all the contracts in a month, does anyone have any examples how to do this?
I'm finding that C# should be a good way to download and analyse data but don't have any documentation, I've only worked out what to do from searching through lots of the code.
Thanks for your help.
|
C# Option Contract Details
I've had a look, it looks great, however, I still can't seem to fix my problem. The way you set up a contract in VB is:
contract.conId = Convert.ToInt32(ss(2))
contract.symbol = TextBox1.Text
contract.secType = "OPT"
contract.expiry = ss(0).Trim.ToString
contract.strike = Convert.ToDouble(ss(1))
contract.right = "C"
contract.multiplier = ""
contract.exchange = "SMART"
contract.primaryExchange = ""
contract.currency = "USD"
contract.localSymbol = ""
contract.includeExpired = 0
tickcount += 1
AxTws1.reqMktDataEx(rnumber, contract, "", 1) '0 =streaming 1=snapshot
However, in C# I don't seem to have any of the contract.x and can't even seem to use .reqMktDataEx()
I've tried another method which uses:
AAAA = new Contract(22, "AAPL", SecurityType.Option, "201307", 420.0, RightType.Call, "", "SMART", "USD", "", "", SecurityIdType.None, "");
The constructor for this is:
public Contract(int contractId, String symbol, SecurityType securityType, String expiry, double strike, RightType right,
String multiplier, string exchange, string currency, string localSymbol, string primaryExchange,
SecurityIdType secIdType, string secId)
It still says no security definition found.
Any idea of what is wrong here?
Thanks again.
toggle quoted message
Show quoted text
--- In TWSAPI@..., Nick <nickhere@...> wrote:
In the file section of this group
I wrote a option chain getter
I included the source.it in vb
and conversion is easy to c#
i had very little problems with it
it get the chain of any stock ticker
gets all the quotes
and draws vertical spread graphs
nick
On 7/14/2013 6:23 AM, andrewcmeier wrote:
Oops, Tester and A1 were meant to be the same variable in the options lines, change Tester to A1 and that is what I have.
--- In TWSAPI@..., "andrewcmeier" <andrewcmeier@> wrote:
Hi there,
I've started using the Krs.Ats.IBNet software () to develop TWS software in C#.
Setting up equities is easy enough:
Equity Google = new Equity("GOOG");
client.RequestMarketData(14, Google, null, true, false);
But when I try to adapt it to options, I can't get it to work. This is what I'm using:
Option A1 = new Option("AA", "AA", "20130719", RightType.Call, 8.0m);
client.RequestMarketData(15, Tester, null, false, false);
This is what it has in its class):
public Option(string equitySymbol, string optionSymbol, string expiry,
RightType right, decimal strike)
This is the error I get when I raise e.ErrorCode:
"No security definition has been found for the request"
Does anyone know what I'm entering wrong, or just how to set up an option in C#? I also then wan't to be able to leave fields out to do things such as return all the contracts in a month, does anyone have any examples how to do this?
I'm finding that C# should be a good way to download and analyse data but don't have any documentation, I've only worked out what to do from searching through lots of the code.
Thanks for your help.
------------------------------------
Yahoo! Groups Links
|
Re: Tick Encoding/Decoding
It's not clear from your question what you're trying to accomplish. If you're interested in market microstructure, i.e. very short term, you might have a look at TickString. See this old post:
[rwk]
--- "sc1447" <sc1447@...> wrote:
toggle quoted message
Show quoted text
I'm trying to encode tick data from IB, and wanted to use TickUtils.jar contributed here. But it seems to work with standard tick formats for example a Trade tick would have format (%time %price %trade size).
IB data shows up as (%time %tick type %value), so IB would send two tick responses for a trade tick, instead of one.
I was wondering if anyone was using the encoding utility for IB and if they would have a converter for IB tick data into the the format the encoder needs and are willing to share it.
|
option expiry [was: C# Option Contract Details]
The expiry is as it always was, a friday for US stock options. The OSI
symbol, introduced a couple years ago, is always one day later than the
expiry. If you are using the OSI symbol use saturday; if you are specifying
the expiry field use friday.
-Kurt
toggle quoted message
Show quoted text
On 7/14/13 8:51 AM, "Peter Gum" <petergum@...> wrote:
Should '20130719' be '20130720'? I.e. 19 -> 20 to match the tws market line.
(Although IB will expire the option on the 19th. Confusing.)
Pete
On 07/14/13, andrewcmeier<andrewcmeier@...> wrote:
Oops, Tester and A1 were meant to be the same variable in the options lines,
change Tester to A1 and that is what I have.
--- In TWSAPI@..., "andrewcmeier" <andrewcmeier@...> wrote:
Hi there,
I've started using the Krs.Ats.IBNet software
(
.21.Full.zip) to develop TWS software in C#.
Setting up equities is easy enough:
Equity Google = new Equity("GOOG");
client.RequestMarketData(14, Google, null, true, false);
But when I try to adapt it to options, I can't get it to work. This is what
I'm using:
Option A1 = new Option("AA", "AA", "20130719", RightType.Call, 8.0m);
client.RequestMarketData(15, Tester, null, false, false);
This is what it has in its class):
public Option(string equitySymbol, string optionSymbol, string expiry,
RightType right, decimal strike)
This is the error I get when I raise e.ErrorCode:
"No security definition has been found for the request"
Does anyone know what I'm entering wrong, or just how to set up an option in
C#? I also then wan't to be able to leave fields out to do things such as
return all the contracts in a month, does anyone have any examples how to do
this?
I'm finding that C# should be a good way to download and analyse data but
don't have any documentation, I've only worked out what to do from searching
through lots of the code.
Thanks for your help.
------------------------------------
Yahoo! Groups Links
|
