|
Hi,
I am trying to monitor a list of tickers (lets say the list is big).
I want to know when one of the tickers reaches a certain price. The target price is generated by an algo so it's not just a %-change.
I tried subscribing to realtime data to all of the instruments but this generates HUGE amount of network traffic to the point that my code is not really able to process all of the events.
Is this possible to do with scanner subscriptions?
I dont find where I can specify that scanner subscriptions to work only with a specific ticker or a list of tickers.
|
Can you be more specific what you consider huge? Did you buy quote boosters so that you can subscribe to more than 100 contracts? From what you are doing, I'd say you'd only need to subscribe to monitor the Last Price and Last Size ticks. So you would look at roughly 4 - 5 call backs per contract per second. You might want to review your approach and code. I am subscribing to a pretty massive amount of data (process and log them in a Json format) and my server does not break a sweat. And it's a very small server for today's standards. Just to give you some idea:
- The total volume is between 30million and 60million ticks per day and sometimes more than 1,000 callbacks per second for busy days
- That is all available market data for ~75 contracts
- Plus TickByTickLast and TickByTickBidAsk for five contracts (yes five. I ask for five and most of the days get five)
- Plus Level II for three contracts
We use the standard Java API.. ´³¨¹°ù²µ±ð²Ô
On Wed, Mar 17, 2021 at 03:54 PM, corneliu maftuleac wrote:
Hi,
I am trying to monitor a list of tickers (lets say the list is big).
I want to know when one of the tickers reaches a certain price. The target price is generated by an algo so it's not just a %-change.
I tried subscribing to realtime data to all of the instruments but this generates HUGE amount of network traffic to the point that my code is not really able to process all of the events.
Is this possible to do with scanner subscriptions?
I dont find where I can specify that scanner subscriptions to work only with a specific ticker or a list of tickers.
|
I think the list of tickers is around 800 (I am using ibinsync python api, but the general idea is the same).
So, your suggestion is to just use regular market data (ex: reqMktData(conId, contract, 221)) and it should be fine?
I was thinking there should be a more advanced solution like a scanner subscription or something, because in fact I dont need the data itself rather just a notification when price reaches certain level.
|
I am not aware of a scanner that does what you are looking for. Please keep in mind the IB API is mostly for the support af TWS and if TWS does not need it, it's probably not there. If you are only interested in price changes, subscribing to tick id 4 (Last Price) should suffice.That tick will only be sent when the trade price changes, Take a look at . Even at 800 contracts, data volume will be quite reasonable. But your 800 contracts simultaneously will run against the data lines limitation. Take a look at the ":How Market Data is Allocated" in For 800 market lines you'd need:
- $6,400 in commission monthly
- $8,000,000 in equity
- or purchase 7 quote boosters at $30 each per month (210 per month)
´³¨¹°ù²µ±ð²Ô
On Wed, Mar 17, 2021 at 07:28 PM, corneliu maftuleac wrote:
I think the list of tickers is around 800 (I am using ibinsync python api, but the general idea is the same).
So, your suggestion is to just use regular market data (ex: reqMktData(conId, contract, 221)) and it should be fine?
I was thinking there should be a more advanced solution like a scanner subscription or something, because in fact I dont need the data itself rather just a notification when price reaches certain level.
|
JR,
You must have 8 market depth level 2 subscriptions.
I seem to have 6 subscriptions and can pull 6 full book depth level 2 but if I use?TickByTickBidAsk?that also counts towards marketdepthlevel2. Do you see the same?
By "That is *all available market data for ~75 contracts" I think you mean partial data as you do not get market depth level 2 for all 75.
What you mean by "I ask for five and most of the days I get five"? Some days you intentionally dont ask for five or something is not available on some days or not worth pulling based on your strategies?
Sounds like you have a good system running. I am interested to know how you store all this streaming data or if you do not see a point in storing it?
- Bruce
Can you be more specific what you consider huge? Did you buy quote boosters so that you can subscribe to more than 100 contracts?
From what you are doing, I'd say you'd only need to subscribe to monitor the Last Price and Last Size ticks. So you would look at roughly 4 - 5 call backs per contract per second.
You might want to review your approach and code. I am subscribing to a pretty massive amount of data (process and log them in a Json format) and my server does not break a sweat. And it's a very small server for today's standards. Just to give you some idea:
- The total volume is between 30million and 60million ticks per day and sometimes more than 1,000 callbacks per second for busy days
- That is all available market data for ~75 contracts
- Plus TickByTickLast and TickByTickBidAsk for five contracts (yes five. I ask for five and most of the days get five)
- Plus Level II for three contracts
We use the standard Java API..
´³¨¹°ù²µ±ð²Ô
On Wed, Mar 17, 2021 at 03:54 PM, corneliu maftuleac wrote:
Hi,
I am trying to monitor a list of tickers (lets say the list is big).
I want to know when one of the tickers reaches a certain price. The target price is generated by an algo so it's not just a %-change.
I tried subscribing to realtime data to all of the instruments but this generates HUGE amount of network traffic to the point that my code is not really able to process all of the events.
Is this possible to do with scanner subscriptions?
I dont find where I can specify that scanner subscriptions to work only with a specific ticker or a list of tickers.
|
So each of your ticker has its own trigger price or your trigger criteria is generic?
|
> So each of your ticker has its own trigger price
That is correct, each ticker has its own trigger price. Criteria is not generic and is absolute (not in %).
|
Thank you for the reference, that is good info to know.
|
Instead of subscribing, why not poll each ticker to get the last price?
I think IB has a limit of 50 messages per second (correct me if I am wrong)
For 800 tickers, that would be 16 seconds. So you would poll each ticker about every 16 seconds.
Would that work for you?
|
You might have just solved that riddle for me, Bruce, why sometimes only four of the five TickByTick subscriptions succeed. I don't always subscribe to Level II data, so that seems to free up extra TickByTicks. I tried to stay with IB terminology, so when I referred to market data for the ~75 contracts it means data streams from reqTopMktData() requests. No TickByTick or Level II data. But we determine all valid tick ids for each equity type (STK, FUT, IND, ..) and request everything they are willing to give. So while there is no TickByTick or Level II for those 75, the amount of data is quite substantial. A long time ago we switched to mostly immutable objects, flow processing, and a logging strategy that favors logging complete objects over human readable text. That way we can drill down after the fact and are not limited to deciphering incomplete information in the log messages. Objects are being serialized into Json strings and saved with some rich extra context. Log files are being rolled every 15min and compressed with "gzip --best" n the background at low system priorities. That all is handled asynchronously in a couple threads and does not slow down IB API callback processing even under the heaviest of loads. I just grabbed a random day that had 35 million IB callbacks:
- With all the extra context and object serialization overhead, the average uncompressed object is 224 bytes long for a total of just shy of 8GB
- But Json serialized objects compress very nicely so that that we store less than 12 bytes per object in average after compression or just over 400MB
This may sound like a lot of unnecessary overhead and long delays but it actually is not. Data stream serialization/compression and decompression/deserialization happen on the fly and it is quite liberating that you can insert any object type into the streams and get high-level language objects back into your business logic (without any extra work). Processing can takes place in parallel on multi-processors and most systems have ample idle and otherwise unused CPU power for this. ´³¨¹°ù²µ±ð²Ô
On Wed, Mar 17, 2021 at 08:22 PM, Bruce B wrote:
JR,
?
You must have 8 market depth level 2 subscriptions.
?
I seem to have 6 subscriptions and can pull 6 full book depth level 2 but if I use?TickByTickBidAsk?that also counts towards marketdepthlevel2. Do you see the same?
?
By "That is *all available market data for ~75 contracts" I think you mean partial data as you do not get market depth level 2 for all 75.
?
What you mean by "I ask for five and most of the days I get five"? Some days you intentionally dont ask for five or something is not available on some days or not worth pulling based on your strategies?
?
Sounds like you have a good system running. I am interested to know how you store all this streaming data or if you do not see a point in storing it?
?
- Bruce
Can you be more specific what you consider huge? Did you buy quote boosters so that you can subscribe to more than 100 contracts?
From what you are doing, I'd say you'd only need to subscribe to monitor the Last Price and Last Size ticks. So you would look at roughly 4 - 5 call backs per contract per second.
You might want to review your approach and code. I am subscribing to a pretty massive amount of data (process and log them in a Json format) and my server does not break a sweat. And it's a very small server for today's standards. Just to give you some idea:
- The total volume is between 30million and 60million ticks per day and sometimes more than 1,000 callbacks per second for busy days
- That is all available market data for ~75 contracts
- Plus TickByTickLast and TickByTickBidAsk for five contracts (yes five. I ask for five and most of the days get five)
- Plus Level II for three contracts
We use the standard Java API..
´³¨¹°ù²µ±ð²Ô
On Wed, Mar 17, 2021 at 03:54 PM, corneliu maftuleac wrote:
Hi,
I am trying to monitor a list of tickers (lets say the list is big).
I want to know when one of the tickers reaches a certain price. The target price is generated by an algo so it's not just a %-change.
I tried subscribing to realtime data to all of the instruments but this generates HUGE amount of network traffic to the point that my code is not really able to process all of the events.
Is this possible to do with scanner subscriptions?
I dont find where I can specify that scanner subscriptions to work only with a specific ticker or a list of tickers.
?
?
|
50 messages per second do not apply to this. However, there is some other limit, based on available market data lines. But polling snapshots have another problem (at least it had a couple of years ago) - snapshot could be cached if polled too frequently. So instead of new prices, you are getting something stale.
Instead of subscribing, why not poll each ticker to get the last price?
I think IB has a limit of 50 messages per second (correct me if I am wrong)
For 800 tickers, that would be 16 seconds. So you would poll each ticker about every 16 seconds.
Would that work for you?
|
Chernikov,
Cached where? And what do you mean by *stale? Missing data points or delivered too late that may not be useful for strategies?
Thanks,
On Thu, Mar 18, 2021, 4:55 AM §¡§Ý§Ö§Ü§ã§Ñ§ß§Õ§â §¹§Ö§â§ß§Ú§Ü§à§Ó < me@...> wrote: 50 messages per second do not apply to this. However, there is some other limit, based on available market data lines. But polling snapshots have another problem (at least it had a couple of years ago) - snapshot could be cached if polled too frequently. So instead of new prices, you are getting something stale.
Instead of subscribing, why not poll each ticker to get the last price?
I think IB has a limit of 50 messages per second (correct me if I am wrong)
For 800 tickers, that would be 16 seconds. So you would poll each ticker about every 16 seconds.
Would that work for you?
--
|
Where - I do not know (never bothered to check traffic between TWS and IB servers). From what I remember - you receive the same value over and over again. And since there are no timestamps on data - no way to know what you got.
Chernikov,
Cached where? And what do you mean by *stale? Missing data points or delivered too late that may not be useful for strategies?
Thanks, On Thu, Mar 18, 2021, 4:55 AM §¡§Ý§Ö§Ü§ã§Ñ§ß§Õ§â §¹§Ö§â§ß§Ú§Ü§à§Ó < me@...> wrote: 50 messages per second do not apply to this. However, there is some other limit, based on available market data lines. But polling snapshots have another problem (at least it had a couple of years ago) - snapshot could be cached if polled too frequently. So instead of new prices, you are getting something stale.
Instead of subscribing, why not poll each ticker to get the last price?
I think IB has a limit of 50 messages per second (correct me if I am wrong)
For 800 tickers, that would be 16 seconds. So you would poll each ticker about every 16 seconds.
Would that work for you?
--
|
Agreed - most things don't from IBKR do not? include timestamps and some may include fake timestamps from TWS. Dimitry I think has looked into this and posted before.
Is it also possible that maybe your requests were duplicates OR that price didn't change in between your multiple requests?
You can compare to TWS data or to other sources to confirm maybe.
I would be surprised if they send stale data over a live account as this has to be a serious? programming error. Can't be a network cache issue either as TCP ACKS would take care of those but maybe a different type of cache.
On Thu, Mar 18, 2021, 2:15 PM §¡§Ý§Ö§Ü§ã§Ñ§ß§Õ§â §¹§Ö§â§ß§Ú§Ü§à§Ó < me@...> wrote: Where - I do not know (never bothered to check traffic between TWS and IB servers). From what I remember - you receive the same value over and over again. And since there are no timestamps on data - no way to know what you got.
Chernikov,
Cached where? And what do you mean by *stale? Missing data points or delivered too late that may not be useful for strategies?
Thanks, On Thu, Mar 18, 2021, 4:55 AM §¡§Ý§Ö§Ü§ã§Ñ§ß§Õ§â §¹§Ö§â§ß§Ú§Ü§à§Ó < me@...> wrote: 50 messages per second do not apply to this. However, there is some other limit, based on available market data lines. But polling snapshots have another problem (at least it had a couple of years ago) - snapshot could be cached if polled too frequently. So instead of new prices, you are getting something stale.
Instead of subscribing, why not poll each ticker to get the last price?
I think IB has a limit of 50 messages per second (correct me if I am wrong)
For 800 tickers, that would be 16 seconds. So you would poll each ticker about every 16 seconds.
Would that work for you?
--
--
|
My request in fact were duplicates - since I requested snapshots for 1500 instruments to get "almost real-time" data. Request ids were never repeated. Price did changed and if I had real-time subscription at the same time - it worked fine. Actually, I worked around it and used real-time data for important data and round-robin for everything else. From IB docs - " It is important to note that a snapshot request will only return available data over the 11 second span; in some cases values may not be returned for all tick types."
So maybe this is expected behavior.?
Agreed - most things don't from IBKR do not? include timestamps and some may include fake timestamps from TWS. Dimitry I think has looked into this and posted before.
Is it also possible that maybe your requests were duplicates OR that price didn't change in between your multiple requests?
You can compare to TWS data or to other sources to confirm maybe.
I would be surprised if they send stale data over a live account as this has to be a serious? programming error. Can't be a network cache issue either as TCP ACKS would take care of those but maybe a different type of cache. On Thu, Mar 18, 2021, 2:15 PM §¡§Ý§Ö§Ü§ã§Ñ§ß§Õ§â §¹§Ö§â§ß§Ú§Ü§à§Ó < me@...> wrote: Where - I do not know (never bothered to check traffic between TWS and IB servers). From what I remember - you receive the same value over and over again. And since there are no timestamps on data - no way to know what you got.
Chernikov,
Cached where? And what do you mean by *stale? Missing data points or delivered too late that may not be useful for strategies?
Thanks, On Thu, Mar 18, 2021, 4:55 AM §¡§Ý§Ö§Ü§ã§Ñ§ß§Õ§â §¹§Ö§â§ß§Ú§Ü§à§Ó < me@...> wrote: 50 messages per second do not apply to this. However, there is some other limit, based on available market data lines. But polling snapshots have another problem (at least it had a couple of years ago) - snapshot could be cached if polled too frequently. So instead of new prices, you are getting something stale.
Instead of subscribing, why not poll each ticker to get the last price?
I think IB has a limit of 50 messages per second (correct me if I am wrong)
For 800 tickers, that would be 16 seconds. So you would poll each ticker about every 16 seconds.
Would that work for you?
--
--
|
Their last sentence is vague. Bad documentation like this I have seen in other places too.
Yes, I think the delay is in effect and so your quick requests were responded again and again with same data because your 11 seconds bottleneck was not up yet.
Most likely those quick - over time - requests were not even sent to IBKR at all and TWS decides and responds with "stale" data based on 11 second rule. So that could be the "cache".
- Bruce
On Thu, Mar 18, 2021, 2:34 PM §¡§Ý§Ö§Ü§ã§Ñ§ß§Õ§â §¹§Ö§â§ß§Ú§Ü§à§Ó < me@...> wrote: My request in fact were duplicates - since I requested snapshots for 1500 instruments to get "almost real-time" data. Request ids were never repeated. Price did changed and if I had real-time subscription at the same time - it worked fine. Actually, I worked around it and used real-time data for important data and round-robin for everything else. From IB docs - " It is important to note that a snapshot request will only return available data over the 11 second span; in some cases values may not be returned for all tick types."
So maybe this is expected behavior.?
Agreed - most things don't from IBKR do not? include timestamps and some may include fake timestamps from TWS. Dimitry I think has looked into this and posted before.
Is it also possible that maybe your requests were duplicates OR that price didn't change in between your multiple requests?
You can compare to TWS data or to other sources to confirm maybe.
I would be surprised if they send stale data over a live account as this has to be a serious? programming error. Can't be a network cache issue either as TCP ACKS would take care of those but maybe a different type of cache. On Thu, Mar 18, 2021, 2:15 PM §¡§Ý§Ö§Ü§ã§Ñ§ß§Õ§â §¹§Ö§â§ß§Ú§Ü§à§Ó < me@...> wrote: Where - I do not know (never bothered to check traffic between TWS and IB servers). From what I remember - you receive the same value over and over again. And since there are no timestamps on data - no way to know what you got.
Chernikov,
Cached where? And what do you mean by *stale? Missing data points or delivered too late that may not be useful for strategies?
Thanks, On Thu, Mar 18, 2021, 4:55 AM §¡§Ý§Ö§Ü§ã§Ñ§ß§Õ§â §¹§Ö§â§ß§Ú§Ü§à§Ó < me@...> wrote: 50 messages per second do not apply to this. However, there is some other limit, based on available market data lines. But polling snapshots have another problem (at least it had a couple of years ago) - snapshot could be cached if polled too frequently. So instead of new prices, you are getting something stale.
Instead of subscribing, why not poll each ticker to get the last price?
I think IB has a limit of 50 messages per second (correct me if I am wrong)
For 800 tickers, that would be 16 seconds. So you would poll each ticker about every 16 seconds.
Would that work for you?
--
--
--
|
