reqMktData subscription is not continuous
Trying to subscribe to live Bid / Ask prices for ES futures using "reqMktData".?
?
var contract = new Contract { Symbol = "ES", Exchange = "CME", ConId = 495512563, LastTradeDateOrContractMonth = "20251219" };
_client.TickPrice += o => Console.WriteLine(JsonSerializer.Serialize(o));
_client.ClientSocket.reqMktData(id, contract, string.Empty, false, false, null);
?
As a result, I receive one round of messages below, and then notifications stop.??
?
{"Attribs":{"CanAutoExecute":true,"PastLimit":true,"PreOpen":false,"Unreported":false,"BidPastLow":false,"AskPastHigh":false},"Value":5801,"Data":5801,"RequestId":3,"Field":1}
{"Attribs":null,"Value":1,"Data":1,"RequestId":3,"Field":0}
{"Attribs":{"CanAutoExecute":true,"PastLimit":false,"PreOpen":false,"Unreported":false,"BidPastLow":false,"AskPastHigh":false},"Value":6300,"Data":6300,"RequestId":3,"Field":2}
{"Attribs":null,"Value":1,"Data":1,"RequestId":3,"Field":3}
{"Attribs":null,"Value":1,"Data":1,"RequestId":3,"Field":0}
{"Attribs":null,"Value":1,"Data":1,"RequestId":3,"Field":3}
{"Attribs":null,"Value":0,"Data":0,"RequestId":3,"Field":5}
{"Attribs":{"CanAutoExecute":false,"PastLimit":false,"PreOpen":false,"Unreported":false,"BidPastLow":false,"AskPastHigh":false},"Value":6274,"Data":6274,"RequestId":3,"Field":9}
{"Attribs":{"CanAutoExecute":false,"PastLimit":false,"PreOpen":false,"Unreported":false,"BidPastLow":false,"AskPastHigh":false},"Value":6050.5,"Data":6050.5,"RequestId":3,"Field":14}
?
Are there any settings, like "Return only snapshots instead of continuous subscriptions"??
Why notifications stop after receiving only one quote??
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
I still believe there is no bug or anything wrong with reqMktData real time feeds. And snapshots are probably what you are looking for in the first place.
TWS API documentation says about reqMktData snapshots that: ... it is possible to request a snapshot of the current state of the market once instead of requesting a stream of updates continuously as market values change.
What you experience is likely that, at times, not all of the market values you are interested in change during the 20 seconds you are looking at the market. You could easily verify that yourselves by downloading TickByTick historical data for the time periods that fail your test.
I took a quick look at the "missing" BID price for MTDR on 20241217:
- You request real-time data for MTDR@NYSE only, instead of MTDR@SMART. On that day, NYSE handled only 22% of all trades. They handled 51% of the total volume, but I bet most of it was related to the MOO and MOC auctions.
- And then there were only 563 trades during the entire trading session (RTH and outside of RTH) on NYSE. That means there were definitely long periods of time that day (way longer than 20 seconds) where the BID price (or other "market values") just did not change and, therefore, no reports would have taken place in the real-time stream. But that would be no bug, just "nothing to report".
- Requesting data for SMART instead of only for NYSE may increase your chance of seeing "market values change" while you look. But even then, MTDR had only 2,563 trades (compared with 177,164 for APPL) and long quiet periods would have to be expected.
I don't have enough data about your 10.19 vs 10.30+ comparisons to explain the differences your are seeing. It may very well be that IBGW/TWS 10.19, upon subscription, send some "recent historical values" and 10.30+ are now cleaner and send "only market values that changed". But that is just a guess and there may be other explanations.
闯ü谤驳别苍
?
?
?
On Thu, Dec 19, 2024 at 11:56 AM, ajn wrote:
Just run a couple of more tests (using the app I posted, just changed False to True and commented cancel request). Snapshot works 100% reliable as Daniel experiences i.e. snapshot delivers all that is expected and no errors printed
self.reqMktData(orderId,?contract,?"", True,?False, [])
live data (4th argument is False) oth is the case which has the problems
self.reqMktData(orderId,?contract,?"",?False,?False, [])
It is live data is bit broken in gateway 10.30+ . We are making some progress :)
?
|
Re: Entry that triggers OCO with TP and SL
toggle quoted message
Show quoted text
On Sat, Dec 21, 2024 at 12:27?PM Richard L King via <rlking= [email protected]> wrote: This is not correct: all three orders must have unique order ids. But the TP and SL orders must have “ParentId” set to the “OrderId” of the main order. ? ? ? Create 3 separate orders. TP and SL orders will have property "OrderId" that should be the same as the main order ID.? The first two orders will have property "Transmit" set to false, the last one is true.?
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
On Mon, Dec 23, 2024 at 10:31 PM, 闯ü谤驳别苍 Reinold wrote:
Just curious whether the data eventually arrives if you give it enough time.
Twenty seconds seems to be a pretty long time (see attached).
How long do you think we should wait before crossing over from thinking this is a "feature" to "bug"?
Ah... good old Sorites paradox, never fails to amuse :-)
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
On Tue, Dec 24, 2024 at 06:53 AM, 闯ü谤驳别苍 Reinold wrote:
I still would not call this a bug
Well, for practical purposes, whether it's a "bug" or not is a moot point. But, just as you wouldn't call it a bug, I wouldn't necessarily call it a great improvement either ;-)
That said, I understand your viewpoint, and would simply remind future readers that it's wise to distinguish between so-called "hard" vs "soft" real-time requirements when building these types of programs.
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
We are talking about entirely different cases
- when the data never arrives in 10.30+ or
- when it does arrive but not within the time @ajn is willing to wait for. Even if data always arrives within that time with IBGW 10.19.
The first case smells like a bug (as long as data should be available), while the second case is simply a change in systemic behavior. Even if that behavior change, in the eyes of the user, is a degradation and causes previously working timeouts to now fail. That is the crux with timeouts - they are always too long until they are suddenly not long enough.
During my due diligence before switching from V9 TWS/IBGW to the first stable V10 (probably around TWS 10.10) I found that timing for certain operations had changed. Some for the better and some for the worse. So it is not unreasonable to assume that something similar happened between 10.19 and 10.30+.
To quickly check that hypothesis I went back to my records and compared all reqMkt() subscriptions for SPY ETF in December 2023 (when I used stable TWS/IBGW 10.19) with those from MTD December 2024 (with stable TWS/IBGW 10.30). And guess what:
- The average time from reqMktData() to the first arrival of a data tick was 0.3 seconds (300ms) with TWS 10.19 last year?
- but it is now 3.2 seconds (3,200ms) with TWS 10.30 this month
Small things other than the TWS version have changed on my side since December 2023, but I cannot imagine that any of those changes could cause the longer time to first tick. But I will run tests with 10.19 to confirm when the markets are closed later in the week. Attached a quick xlsx with the raw results.
I still would not call this a bug (though this change of behavior may have not been intended by the developers)
闯ü谤驳别苍
?
On Mon, Dec 23, 2024 at 09:40 PM, buddy wrote:
On Mon, Dec 23, 2024 at 10:31 PM, 闯ü谤驳别苍 Reinold wrote:
Just curious whether the data eventually arrives if you give it enough time.
Even if it did, that wouldn't explain why it arrives in time when using 10.19. If you run the test case as prescribed, and compare the results of 10.19 vs 10.30+, you'll probably see what I mean.
I suppose it could be classified as "not a bug" if IBKR wants to intentionally degrade the service they once provided. And, as a skeptic, I wouldn't put that past them. Nevertheless, from a technical user's perspective, this is at best a [massive] shortcoming IMHO.
After all, without any SLA they could just hold data back completely, or provide the same QOS for DELAYED and REALTIME. It's an absurd proposition so I'm leaning toward "this is a bug (full stop)".
Playing that much faster and looser w/ their specifications would leave a bad taste in an engineer's mouth... although a lawyer might love it.
:-/
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
On Tue, Dec 24, 2024 at 03:40 AM, buddy wrote:
"this is a bug (full stop)"
Of course, what's also interesting, is this could actually be a bug w/ 10.19 instead. That is, 10.19 might be providing bad data in a timely manner instead of timing out.
Unfortunately, it's difficult to be certain without digging much much deeper into the IBKR infra and the TWS/IBGW code.
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
On Mon, Dec 23, 2024 at 10:31 PM, 闯ü谤驳别苍 Reinold wrote:
Just curious whether the data eventually arrives if you give it enough time.
Even if it did, that wouldn't explain why it arrives in time when using 10.19. If you run the test case as prescribed, and compare the results of 10.19 vs 10.30+, you'll probably see what I mean.
I suppose it could be classified as "not a bug" if IBKR wants to intentionally degrade the service they once provided. And, as a skeptic, I wouldn't put that past them. Nevertheless, from a technical user's perspective, this is at best a [massive] shortcoming IMHO.
After all, without any SLA they could just hold data back completely, or provide the same QOS for DELAYED and REALTIME. It's an absurd proposition so I'm leaning toward "this is a bug (full stop)".
Playing that much faster and looser w/ their specifications would leave a bad taste in an engineer's mouth... although a lawyer might love it.
:-/
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
Rarely is lack of "last hop" bandwidth the root cause for network issues. In my experience it is pretty much always latency. That is latency from processing, end-to-end "flight time", congestion along the route, traffic shaping games played by the ISP, or packet loss and retransmission. But since you can reproduce what you are seeing with a t3.2xlarge AWS instance, I am pretty sure that network issues are not the root cause of what you are seeing. Having said that, latency (or changes of latency) could very well make your experience better or worse.
I am still not convinced that there is a "bug" here and I was wondering, whether you have tried (or could try) some runs without timeouts. At least the Python code currently gives up and cancels the subscription after 20 seconds, right? Maybe it just takes that long occasionally when you send massive numbers of subscriptions and cancellations.
There are a few reasons why that may be the case:
- There is no interface contract or service level agreement that specifies how long it will take for the first market data ticks to arrive after a successful subscription. It may just be that it sometimes takes a while. Especially since in many cases a set of historical tick (last close, open, ...) is sent before the real-time data starts.
- The API documentation for market data snapshot says, that the tickSnapshotEnd() callback "... will indicate the snapshot reception is finished. Expected to occur 11 seconds after beginning of request." It does not guarantee that any data has been sent before the end of the 11 second period.
- I checked my records for the SPY ETF how long it takes from the market data subscription to the first data arriving. This comparison is not perfect, since my client subscribes between 16:00 and 17:00 US/Central, keeps the subscriptions active for 24 hours, and only makes few subscriptions compared to your 1,000++ runs. For about 2,500 subscriptions over the last few years, max time for the arrival of the first data tick was 6.3s with an average of 510ms and stdev of 794ms. That does not include quite a few cases where the first tick did not arrive for more than a minute and was a "LastTimeStamp" callback and not a data tick.
Just curious whether the data eventually arrives if you give it enough time.
闯ü谤驳别苍
PS. Just as an FYI attached recent latency and packet loss measurements for the IBKR entry points as experienced by a server outside of Chicago.
?
?
On Fri, Dec 20, 2024 at 10:10 AM, ajn wrote:
Uh, my sanity is restored :)
?
As for your comment about the network, yes my dev setup is in EU (yet its 250Mbit up/down fiber, typically no issues here with Internet otherwise, but of course anything can happen :) and again switching back and forth between 10.19 and 10.30 all the time, where 10.19 does't have issues) However, I can reproduce it from my test site in US just as well. Test site is in AWS with pretty biffy setup on the East Coast? (us-east-1, i..e supposed to be closest to IB servers, ?t3.2xlarge instance aws claims 5gbit network, ok it is probably less in reality but itis not likelly a bottleneck even it is "just" 500Mbit to the Internet ).?
|
Re: reqIds hangs when recereating IBClient - EWrapper
Each time when you connect you apparently use client ID zero: eConnect(Host, Port, 0). When you disconnect this client you must allow IBKR time to close all processes related to this client on their side. According to an undocumented recommendation from IBKR do you need to wait approximately one minute before you try to connect again to the same client ID number (zero, in your case). Your post does not mention how long you are waiting after closing before trying to reconnect to that client.
If you can't wait one minute before reconnecting then you should simply use a different client ID number. Any integer number can be used, there are no rules for this. So you could use for example a randomly generated ID, or a timestamp as ID (e.g. hhmmss). It is documented that you can have a maximum of 32 clients connected at the same time.
|
Re: Entry that triggers OCO with TP and SL
omg thank you. Didn't realize this was markdown compatible.
|
reqIds hangs when recereating IBClient - EWrapper
I have IBClient that implements EWrapper that I copied from C# samples from official TWS repository.?
When some event happens, I would like to destroy existing client and create new instance.?
This is the method I use to create this client.?
?
void Create()
{
? var signal = new EReaderMonitorSignal();
? _client = new IBClient(signal);
? _client.ClientSocket.SetConnectOptions("+PACEAPI");
? _client.ClientSocket.eConnect(Host, Port, 0);
? var reader = new EReader(_client.ClientSocket, signal);
? var process = new Thread(() =>
? {
? ? while (_client.ClientSocket.IsConnected())
? ? {
? ? ? signal.waitForSignal();
? ? ? reader.processMsgs();
? ? }
? });
? process.Start();
? reader.Start();
}
?
After that, I'm waiting for the next valid ID.
?
async Task Register()
{
? var source = new TaskCompletionSource();
? void subscribe(int o)
? {
? ? _client.NextValidId -= subscribe;
? ? source.TrySetResult();
? }
? _client.NextValidId += subscribe;
? _client.ClientSocket.reqIds(-1);
? await source.Task;
}
?
Then use method to destroy client.?
?
void Destroy()
{
? _client?.ClientSocket?.eDisconnect();
}
?
The first time everything works fine.
When I try to call these 3 methods next time, "reqIds" hangs and I never receive response for "NextValidId".?
Is it some known limitation that only one client can connect to TWS only once or am I missing something??
|
Re: Entry that triggers OCO with TP and SL
This is not correct: all three orders must have unique order ids. But the TP and SL orders must have “ParentId” set to the “OrderId” of the main order. ? ?
toggle quoted message
Show quoted text
From: [email protected] <[email protected]> On Behalf Of Andy Sanders via groups.io
Sent: 19 December 2024 05:28
To: [email protected]
Subject: Re: [TWS API] Entry that triggers OCO with TP and SL? Create 3 separate orders. TP and SL orders will have property "OrderId" that should be the same as the main order ID.? The first two orders will have property "Transmit" set to false, the last one is true.?
|
Re: Entry that triggers OCO with TP and SL
Thank you.? I will try this.
On Thu, Dec 19, 2024 at 12:28?AM Andy Sanders via <arteinvolo= [email protected]> wrote: Create 3 separate orders.
TP and SL orders will have property "OrderId" that should be the same as the main order ID.?
The first two orders will have property "Transmit" set to false, the last one is true.?
?
|
Re: IB's New automated trading system questionnaire
Thanks to everyone for their suggestions.? I flailed around for longer than I would like to admit before finding a send email solution that works for me.
1) I found out gmail offers something called an "app password" that requires you to use 2-factor authentication and then you can create a password that works with SMTP over port 465. 2) I installed MSVC 2010, changed my email program to use the new password and rebuilt.? (It was easier to use MSVC 2010 than to build with MSVC 2019 which had various complaints.)
Voila, I'm once again able to send emails.? The only ongoing cost will be dealing with gmail's two factor log ins.
Frank
|
Re: No definition for SPX option using LocalSymbol
For the future readers.
I was using data coming from Schwab and each SPX contract had two dates, "LastTradingDate" and "ExpirationDate". I was trying to use option contract expiring the same day to place order in IB but it was rejected because even though contract is expiring today, the last trading day for that contract was yesterday.?
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
Uh, my sanity is restored :)
?
As for your comment about the network, yes my dev setup is in EU (yet its 250Mbit up/down fiber, typically no issues here with Internet otherwise, but of course anything can happen :) and again switching back and forth between 10.19 and 10.30 all the time, where 10.19 does't have issues) However, I can reproduce it from my test site in US just as well. Test site is in AWS with pretty biffy setup on the East Coast? (us-east-1, i..e supposed to be closest to IB servers, ?t3.2xlarge instance aws claims 5gbit network, ok it is probably less in reality but itis not likelly a bottleneck even it is "just" 500Mbit to the Internet ).?
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
On Thu, Dec 19, 2024 at 10:23 PM, ajn wrote:
It is often happening on the second loop or later.
Ok, I managed to reproduce the problem.
|
Re: IB's New automated trading system questionnaire
?
Once you've set up the appropriate Telegram bot on your Telegram account, sending a message programmatically is pretty easy. ?Here's my (redacted) code for that:
?
#!/usr/bin/env python3
import requests
def send_message(text, parse_mode=None):
bf_token = "YOUR_TOKEN_HERE" # replace this your Bot's token
user_id = 1234567890 # replace this with your user id
assert parse_mode in (None, "HTML", "MarkdownV2")
# from BotFather
url = f"https://api.telegram.org/bot{bf_token}/sendMessage"
params = {"chat_id": user_id, "text": text}
if parse_mode:
params["parse_mode"] = parse_mode
resp = requests.get(url, params=params, timeout=3)
# Throw exception if Telegram API fails
resp.raise_for_status()
if __name__ == "__main__":
from sys import argv
if len(argv) <= 1:
i = 50 // 3
MESSAGE = "```\n" + (f"{3 * i}." * i) + "\n```"
else:
MESSAGE = " ".join(argv[1:])
send_message(MESSAGE, parse_mode="MarkdownV2")
|
Re: IB's New automated trading system questionnaire
Another option is Telegram. I wouldn't trust too much on email as
it's very difficult these days to have your mail accepted by the
receiving end as non-spam or not get blocked.
For instant notification, Telegram has been a great option for me.
It has? good api documentation. Start here:
Greetings,
Raoul
On 18-12-2024 18:10, 闯ü谤驳别苍 Reinold via
groups.io wrote:
toggle quoted message
Show quoted text
There are many ways you can handle this. @wordd already
mentioned sending messages to slack channels, for example. All
large cloud providers have services that are called something
like "Simple Message Service" or so, that allow you to send
messages through simple API calls.
If you want to stick with email, check how your email provider
supports "SMTP for outgoing messages" and look for an SMTP
library for your preferred environment. Your service provider
may place various restrictions on email delivered to them by
SMTP (such as SSH/TLS encryption or certificates), but generally
I would expect that those do not need two-factor authentication.
For Java, for example, you'd use "JavaMail" directly or through
higher level libraries such as "Apache Commons Email".
闯ü谤驳别苍
?
On Tue, Dec 17, 2024 at 11:31 PM, Frank Bell wrote:
I logged on to my account and found a must complete
questionnaire about how I'm monitoring my automated api.? It
wanted to know
?
1) Did I write it or is it commercial and if so the
vendor?
2) Is the API completely automated?? If not to what
extent it is manual?
3) How are critical errors handled (audible alert, email,
text)?
4) Is it constantly monitored?
5) and more
?
Apparently it is now being required by regulators.
?
Given that, can anyone point to code that will allow me
to send email/texts now that Google and Yahoo are requiring
two factor authentication?? I used to send emails and texts
when single factor was good enough.
?
Frank
|
buddy
wordd