Re: Question on nextValidId function
You are correct, Marco. Each clientId has its own sequence of orderIds and, since nextValidId() has a value of 1 when a clientId is used for the first time, the numeric values for orderIds for different clientIds can (and will be) be identical at times.
The technical requirement your client has to be aware of is that each placeOrder() call shall present an orderId that is higher than all orderIds the calling clientId has ever received in openOrder() or orderStatus() callbacks.
As long as a client receives only openOrder() callbacks for orders it placed itself, keeping a counter that is initialized from nextValidId() at connection time is sufficient. When a client terminates, TWS/IBGW will remember the highest orderId used in placeOrder() calls and assures that nextValidId() returns a value higher than that when it connects again later.
But there are cases where a client can receive openOrder() callbacks for orders placed by other clients (or TWS). In these cases, the client may need to bump the internal nextValidid counter accordingly. Let me quote directly from the TWS API documentation since it nicely describes this:
However if there are multiple client applications connected to one account, it is necessary to use an order ID with new orders which is greater than all previous order IDs returned to the client application in openOrder or orderStatus callbacks. For instance, if the client is set as the Master client, it will automatically receive order status and trade callbacks from orders placed from other clients. In such a case, any orderID used in placeOrder must be greater than the orderIDs returned in these status callbacks.
Alternatively if the function reqAllOpenOrders is used by a client, subsequent orders placed by that client must have order IDs greater than the order IDs of all orders returned because of that function call.?
The second part of that statement is what you need to be aware of. Say your client needs to be know about orders for the instrument is manages that are open when it connects, a call to reqAllOpenOrders() might generate openOrder() callbacks with orderIds that are higher than the nextValidId(). In these cases, simply bump the internal counter to that orderId + 1.
And to make this very simple and safe for all scenarios, just perform that test unconditionally for all openOrder() and orderStatus() callbacks regardless of why your client receives them.
Sounds complicated, but is really not.
There is also an API Global Configuration option called "Use negative numbers to bind automatic orders." Take a look at the tool-tip.
´³¨¹°ù²µ±ð²Ô
?
?
On Fri, Dec 27, 2024 at 09:36 AM, Marco wrote:
Hello ´³¨¹°ù²µ±ð²Ô,
thanks for your detailed reply. Yes I?confirm my counter(s) get initialized with the value from the automatic nextValidId() callback. It is not completely clear to me when you mention "in contact". Does it mean in case different clients can manage same order?
if the clients are independently managing different orders I can have independent?counters so in principle the same numerical orderId can be reused for different clients, am I correct?
thanks
Marco
On Fri, Dec 27, 2024 at 3:04?PM ´³¨¹°ù²µ±ð²Ô Reinold via <TwsApiOnGroupsIo= [email protected]> wrote:
There is nothing wrong with your approach, as long as your internal counter gets initialized with the value of the automatic nextValidId() callback you receive when your client connects to TWS API.
If your client gets "in contact" with orders from other clients, you need to adjust your interal counter so that it is higher than the orderId of all orders your client sees. That is mostly important for client 0 and the master client since they receive openOrder() callback for orders placed by other clients or TWS in real time, but all clients might need to do that in case they request all open orders, for example, and the list contains order from other clients.
The simplest approach is that your openOrder() callback makes sure that your internal counter is always higher than all orderIds it sees.
´³¨¹°ù²µ±ð²Ô
?
On Fri, Dec 27, 2024 at 08:03 AM, Marco wrote:
Hi All,
one question on the orderId and the necessity of nextValidId function. From the documentation () it is stated that ¡°if there are multiple client applications connected to one account, it is necessary to use an order ID with new orders which is greater than all previous order IDs returned to the client application in openOrder or orderStatus callbacks?¡±. so I expect that the nextValidId function is needed in case of multiple clients for one account. However i have created a program in python that handles orderIds internally without relying on the nextvalidid function. By launching multiple instances of it (each with different client id) I see no problems in handling orders (each client autonomously updating its orderId). What am I missing? Is nextValidId really needed?
thanks,
Marco
?
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
?I have executed the short file you posted multiple times through IB_Gateway.
Final result in prompt: ?
done symbols: 1283
Number of not done symbols: 0
In about 52 seconds.
Also, gateway is sending with each ticker
2024-12-27 13:09:32.976 [OQ] INFO ?[JTS-usfarmDispatcherS3-52S3-53] - Sending for ticker=1001373 market data info: bboExchangeAndSecType=a60001, snapshot perms=REALTIME_TOP
?
I noticed that I don't have any output files from this script. Could you send the code you used to write the bid data to file.
I am on TWS 10.30.1, did you recently update to 10.30 or have you been fighting with this for a while?
|
Re: Question on nextValidId function
Hello ´³¨¹°ù²µ±ð²Ô, thanks for your detailed reply. Yes I?confirm my counter(s) get initialized with the value from the automatic nextValidId() callback. It is not completely clear to me when you mention "in contact". Does it mean in case different clients can manage same order? if the clients are independently managing different orders I can have independent?counters so in principle the same numerical orderId can be reused for different clients, am I correct? thanks Marco On Fri, Dec 27, 2024 at 3:04?PM ´³¨¹°ù²µ±ð²Ô Reinold via <TwsApiOnGroupsIo= [email protected]> wrote:
There is nothing wrong with your approach, as long as your internal counter gets initialized with the value of the automatic nextValidId() callback you receive when your client connects to TWS API.
If your client gets "in contact" with orders from other clients, you need to adjust your interal counter so that it is higher than the orderId of all orders your client sees. That is mostly important for client 0 and the master client since they receive openOrder() callback for orders placed by other clients or TWS in real time, but all clients might need to do that in case they request all open orders, for example, and the list contains order from other clients.
The simplest approach is that your openOrder() callback makes sure that your internal counter is always higher than all orderIds it sees.
´³¨¹°ù²µ±ð²Ô
?
On Fri, Dec 27, 2024 at 08:03 AM, Marco wrote:
Hi All,
one question on the orderId and the necessity of nextValidId function. From the documentation () it is stated that ¡°if there are multiple client applications connected to one account, it is necessary to use an order ID with new orders which is greater than all previous order IDs returned to the client application in openOrder or orderStatus callbacks?¡±. so I expect that the nextValidId function is needed in case of multiple clients for one account. However i have created a program in python that handles orderIds internally without relying on the nextvalidid function. By launching multiple instances of it (each with different client id) I see no problems in handling orders (each client autonomously updating its orderId). What am I missing? Is nextValidId really needed?
thanks,
Marco
?
|
Re: Question on nextValidId function
There is nothing wrong with your approach, as long as your internal counter gets initialized with the value of the automatic nextValidId() callback you receive when your client connects to TWS API.
If your client gets "in contact" with orders from other clients, you need to adjust your interal counter so that it is higher than the orderId of all orders your client sees. That is mostly important for client 0 and the master client since they receive openOrder() callback for orders placed by other clients or TWS in real time, but all clients might need to do that in case they request all open orders, for example, and the list contains order from other clients.
The simplest approach is that your openOrder() callback makes sure that your internal counter is always higher than all orderIds it sees.
´³¨¹°ù²µ±ð²Ô
?
On Fri, Dec 27, 2024 at 08:03 AM, Marco wrote:
Hi All,
one question on the orderId and the necessity of nextValidId function. From the documentation () it is stated that ¡°if there are multiple client applications connected to one account, it is necessary to use an order ID with new orders which is greater than all previous order IDs returned to the client application in openOrder or orderStatus callbacks?¡±. so I expect that the nextValidId function is needed in case of multiple clients for one account. However i have created a program in python that handles orderIds internally without relying on the nextvalidid function. By launching multiple instances of it (each with different client id) I see no problems in handling orders (each client autonomously updating its orderId). What am I missing? Is nextValidId really needed?
thanks,
Marco
?
|
Question on nextValidId function
Hi All, one question on the orderId and the necessity of nextValidId function. From the documentation () it is stated that ¡°if there are multiple client applications connected to one account, it is necessary to use an order ID with new orders which is greater than all previous order IDs returned to the client application in openOrder or orderStatus callbacks?¡±. so I expect that the nextValidId function is needed in case of multiple clients for one account. However i have created a program in python that handles orderIds internally without relying on the nextvalidid function. By launching multiple instances of it (each with different client id) I see no problems in handling orders (each client autonomously updating its orderId). What am I missing? Is nextValidId really needed? thanks, Marco
|
reqExecution failed to get executions
I have a Java program using TWS API to get market data and place orders. After it made a trade, it gets "ExecDetails" and "CommissionReport" callbacks. But after I restarted the Java program and called "clientSocket.reqExecutions(reqId, new ExecutionFilter());", it only gets "ExecDetailsEnd" message without any executions that were done before the restart. I tried to set the "clientID" and "time" field to start of the today in the ExecutionFilter object but no use. Do I miss anything in the TWS setting?? Could anyone shed some light?
|
Re: NQ Futures Security Definition
Sorry to bother.? I figured it out.? I need to use March 2025.
When I try to place an order on NQ futures, I have defined the contract the following way, but i get the following error: Error: ?32 ? 200 ? No security definition has been found for the request
contract = Contract()
contract.symbol = 'NQ'
contract.secType = 'FUT'
contract.exchange = 'CME'
contract.currency = 'USD'
contract.lastTradeDateOrContractMonth = "202412"
|
NQ Futures Security Definition
When I try to place an order on NQ futures, I have defined the contract the following way, but i get the following error: Error: ?32 ? 200 ? No security definition has been found for the request
contract = Contract()
contract.symbol = 'NQ'
contract.secType = 'FUT'
contract.exchange = 'CME'
contract.currency = 'USD'
contract.lastTradeDateOrContractMonth = "202412"
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
On Thu, Dec 26, 2024 at 05:36 AM, ´³¨¹°ù²µ±ð²Ô Reinold wrote:
Requesting data for SMART instead
I thought this was an interesting observation as well. So, I changed the input list to just 800 of the most liquid stocks (see attached) and specified SMART as the exchange instead. Additionally, I changed the timeout to 45 seconds. Failure occurred in the same manner.
Of course, your theory that the semantics w.r.t. reqMktData have mysteriously changed still holds water. And, even though the professional way to distribute such a change would have been to implement reqMktData2 and optionally deprecate reqMktData (so that there were no unpleasant surprises), at least there appears to be some form of sour logic to what's going on.
However, without confirmation from IBKR there's no way to know this with absolute certainty, and that's not great. Regardless... it seems the best advice, at the present time, might be as previously suggested, to first initialize things with a snapshot.
If IBKR takes responsibility for any of this I'd absolutely love to know about it. Keep us posted @ajn.
|
Re: reqMktData subscription is not continuous
Snapshot parameter is set to false.
|
Re: reqMktData subscription is not continuous
You are requesting a snapshot, hence its expected to only receive one quote. If you want to subscribe permanently you can notify it on the reqMktData request.?
Trying to subscribe to live Bid / Ask prices for ES futures using "reqMktData".?
?
var contract = new Contract { Symbol = "ES", Exchange = "CME", ConId = 495512563, LastTradeDateOrContractMonth = "20251219" };
_client.TickPrice += o => Console.WriteLine(JsonSerializer.Serialize(o));
_client.ClientSocket.reqMktData(id, contract, string.Empty, false, false, null);
?
As a result, I receive one round of messages below, and then notifications stop.??
?
{"Attribs":{"CanAutoExecute":true,"PastLimit":true,"PreOpen":false,"Unreported":false,"BidPastLow":false,"AskPastHigh":false},"Value":5801,"Data":5801,"RequestId":3,"Field":1}
{"Attribs":null,"Value":1,"Data":1,"RequestId":3,"Field":0}
{"Attribs":{"CanAutoExecute":true,"PastLimit":false,"PreOpen":false,"Unreported":false,"BidPastLow":false,"AskPastHigh":false},"Value":6300,"Data":6300,"RequestId":3,"Field":2}
{"Attribs":null,"Value":1,"Data":1,"RequestId":3,"Field":3}
{"Attribs":null,"Value":1,"Data":1,"RequestId":3,"Field":0}
{"Attribs":null,"Value":1,"Data":1,"RequestId":3,"Field":3}
{"Attribs":null,"Value":0,"Data":0,"RequestId":3,"Field":5}
{"Attribs":{"CanAutoExecute":false,"PastLimit":false,"PreOpen":false,"Unreported":false,"BidPastLow":false,"AskPastHigh":false},"Value":6274,"Data":6274,"RequestId":3,"Field":9}
{"Attribs":{"CanAutoExecute":false,"PastLimit":false,"PreOpen":false,"Unreported":false,"BidPastLow":false,"AskPastHigh":false},"Value":6050.5,"Data":6050.5,"RequestId":3,"Field":14}
?
Are there any settings, like "Return only snapshots instead of continuous subscriptions"??
Why notifications stop after receiving only one quote??
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
On Thu, Dec 26, 2024 at 05:36 AM, ´³¨¹°ù²µ±ð²Ô Reinold wrote:
It may very well be that IBGW/TWS 10.19, upon subscription, send some "recent
historical values" and 10.30+ are now cleaner and send "only market values
that changed".
The above is an interesting thought. But, I feel obligated to note it would be a breaking change to the interface. So basically still a pretty big "bug/problem"... just not in the implementation.
And even if it's "cleaner" over the wire, changing an interface without also changing its signature is terrible practice because it silently breaks end user applications (as @ajn may have found out).
With all due respect ´³¨¹°ù²µ±ð²Ô, dismissing this or trying to spin it as a good thing(tm) strikes me as extremely forgiving or just naive.
Finally, if this speculation is correct, it makes me think that extra caution should be exercised prior to using 10.30+ in production. Since, I wouldn't want to have any code running the relied on defaults being set to "recent historical values" and didn't immediately fail loudly when such an assumption quietly disappeared; which seems like a possibility.
|
reqMktData subscription is not continuous
Trying to subscribe to live Bid / Ask prices for ES futures using "reqMktData".?
?
var contract = new Contract { Symbol = "ES", Exchange = "CME", ConId = 495512563, LastTradeDateOrContractMonth = "20251219" };
_client.TickPrice += o => Console.WriteLine(JsonSerializer.Serialize(o));
_client.ClientSocket.reqMktData(id, contract, string.Empty, false, false, null);
?
As a result, I receive one round of messages below, and then notifications stop.??
?
{"Attribs":{"CanAutoExecute":true,"PastLimit":true,"PreOpen":false,"Unreported":false,"BidPastLow":false,"AskPastHigh":false},"Value":5801,"Data":5801,"RequestId":3,"Field":1}
{"Attribs":null,"Value":1,"Data":1,"RequestId":3,"Field":0}
{"Attribs":{"CanAutoExecute":true,"PastLimit":false,"PreOpen":false,"Unreported":false,"BidPastLow":false,"AskPastHigh":false},"Value":6300,"Data":6300,"RequestId":3,"Field":2}
{"Attribs":null,"Value":1,"Data":1,"RequestId":3,"Field":3}
{"Attribs":null,"Value":1,"Data":1,"RequestId":3,"Field":0}
{"Attribs":null,"Value":1,"Data":1,"RequestId":3,"Field":3}
{"Attribs":null,"Value":0,"Data":0,"RequestId":3,"Field":5}
{"Attribs":{"CanAutoExecute":false,"PastLimit":false,"PreOpen":false,"Unreported":false,"BidPastLow":false,"AskPastHigh":false},"Value":6274,"Data":6274,"RequestId":3,"Field":9}
{"Attribs":{"CanAutoExecute":false,"PastLimit":false,"PreOpen":false,"Unreported":false,"BidPastLow":false,"AskPastHigh":false},"Value":6050.5,"Data":6050.5,"RequestId":3,"Field":14}
?
Are there any settings, like "Return only snapshots instead of continuous subscriptions"??
Why notifications stop after receiving only one quote??
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
I still believe there is no bug or anything wrong with reqMktData real time feeds. And snapshots are probably what you are looking for in the first place.
TWS API documentation says about reqMktData snapshots that: ... it is possible to request a snapshot of the current state of the market once instead of requesting a stream of updates continuously as market values change.
What you experience is likely that, at times, not all of the market values you are interested in change during the 20 seconds you are looking at the market. You could easily verify that yourselves by downloading TickByTick historical data for the time periods that fail your test.
I took a quick look at the "missing" BID price for MTDR on 20241217:
- You request real-time data for MTDR@NYSE only, instead of MTDR@SMART. On that day, NYSE handled only 22% of all trades. They handled 51% of the total volume, but I bet most of it was related to the MOO and MOC auctions.
- And then there were only 563 trades during the entire trading session (RTH and outside of RTH) on NYSE. That means there were definitely long periods of time that day (way longer than 20 seconds) where the BID price (or other "market values") just did not change and, therefore, no reports would have taken place in the real-time stream. But that would be no bug, just "nothing to report".
- Requesting data for SMART instead of only for NYSE may increase your chance of seeing "market values change" while you look. But even then, MTDR had only 2,563 trades (compared with 177,164 for APPL) and long quiet periods would have to be expected.
I don't have enough data about your 10.19 vs 10.30+ comparisons to explain the differences your are seeing. It may very well be that IBGW/TWS 10.19, upon subscription, send some "recent historical values" and 10.30+ are now cleaner and send "only market values that changed". But that is just a guess and there may be other explanations.
´³¨¹°ù²µ±ð²Ô
?
?
?
On Thu, Dec 19, 2024 at 11:56 AM, ajn wrote:
Just run a couple of more tests (using the app I posted, just changed False to True and commented cancel request). Snapshot works 100% reliable as Daniel experiences i.e. snapshot delivers all that is expected and no errors printed
self.reqMktData(orderId,?contract,?"", True,?False, [])
live data (4th argument is False) oth is the case which has the problems
self.reqMktData(orderId,?contract,?"",?False,?False, [])
It is live data is bit broken in gateway 10.30+ . We are making some progress :)
?
|
Re: Entry that triggers OCO with TP and SL
toggle quoted message
Show quoted text
On Sat, Dec 21, 2024 at 12:27?PM Richard L King via <rlking= [email protected]> wrote: This is not correct: all three orders must have unique order ids. But the TP and SL orders must have ¡°ParentId¡± set to the ¡°OrderId¡± of the main order. ? ? ? Create 3 separate orders. TP and SL orders will have property "OrderId" that should be the same as the main order ID.? The first two orders will have property "Transmit" set to false, the last one is true.?
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
On Mon, Dec 23, 2024 at 10:31 PM, ´³¨¹°ù²µ±ð²Ô Reinold wrote:
Just curious whether the data eventually arrives if you give it enough time.
Twenty seconds seems to be a pretty long time (see attached).
How long do you think we should wait before crossing over from thinking this is a "feature" to "bug"?
Ah... good old Sorites paradox, never fails to amuse :-)
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
On Tue, Dec 24, 2024 at 06:53 AM, ´³¨¹°ù²µ±ð²Ô Reinold wrote:
I still would not call this a bug
Well, for practical purposes, whether it's a "bug" or not is a moot point. But, just as you wouldn't call it a bug, I wouldn't necessarily call it a great improvement either ;-)
That said, I understand your viewpoint, and would simply remind future readers that it's wise to distinguish between so-called "hard" vs "soft" real-time requirements when building these types of programs.
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
We are talking about entirely different cases
- when the data never arrives in 10.30+ or
- when it does arrive but not within the time @ajn is willing to wait for. Even if data always arrives within that time with IBGW 10.19.
The first case smells like a bug (as long as data should be available), while the second case is simply a change in systemic behavior. Even if that behavior change, in the eyes of the user, is a degradation and causes previously working timeouts to now fail. That is the crux with timeouts - they are always too long until they are suddenly not long enough.
During my due diligence before switching from V9 TWS/IBGW to the first stable V10 (probably around TWS 10.10) I found that timing for certain operations had changed. Some for the better and some for the worse. So it is not unreasonable to assume that something similar happened between 10.19 and 10.30+.
To quickly check that hypothesis I went back to my records and compared all reqMkt() subscriptions for SPY ETF in December 2023 (when I used stable TWS/IBGW 10.19) with those from MTD December 2024 (with stable TWS/IBGW 10.30). And guess what:
- The average time from reqMktData() to the first arrival of a data tick was 0.3 seconds (300ms) with TWS 10.19 last year?
- but it is now 3.2 seconds (3,200ms) with TWS 10.30 this month
Small things other than the TWS version have changed on my side since December 2023, but I cannot imagine that any of those changes could cause the longer time to first tick. But I will run tests with 10.19 to confirm when the markets are closed later in the week. Attached a quick xlsx with the raw results.
I still would not call this a bug (though this change of behavior may have not been intended by the developers)
´³¨¹°ù²µ±ð²Ô
?
On Mon, Dec 23, 2024 at 09:40 PM, buddy wrote:
On Mon, Dec 23, 2024 at 10:31 PM, ´³¨¹°ù²µ±ð²Ô Reinold wrote:
Just curious whether the data eventually arrives if you give it enough time.
Even if it did, that wouldn't explain why it arrives in time when using 10.19. If you run the test case as prescribed, and compare the results of 10.19 vs 10.30+, you'll probably see what I mean.
I suppose it could be classified as "not a bug" if IBKR wants to intentionally degrade the service they once provided. And, as a skeptic, I wouldn't put that past them. Nevertheless, from a technical user's perspective, this is at best a [massive] shortcoming IMHO.
After all, without any SLA they could just hold data back completely, or provide the same QOS for DELAYED and REALTIME. It's an absurd proposition so I'm leaning toward "this is a bug (full stop)".
Playing that much faster and looser w/ their specifications would leave a bad taste in an engineer's mouth... although a lawyer might love it.
:-/
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
On Tue, Dec 24, 2024 at 03:40 AM, buddy wrote:
"this is a bug (full stop)"
Of course, what's also interesting, is this could actually be a bug w/ 10.19 instead. That is, 10.19 might be providing bad data in a timely manner instead of timing out.
Unfortunately, it's difficult to be certain without digging much much deeper into the IBKR infra and the TWS/IBGW code.
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
On Mon, Dec 23, 2024 at 10:31 PM, ´³¨¹°ù²µ±ð²Ô Reinold wrote:
Just curious whether the data eventually arrives if you give it enough time.
Even if it did, that wouldn't explain why it arrives in time when using 10.19. If you run the test case as prescribed, and compare the results of 10.19 vs 10.30+, you'll probably see what I mean.
I suppose it could be classified as "not a bug" if IBKR wants to intentionally degrade the service they once provided. And, as a skeptic, I wouldn't put that past them. Nevertheless, from a technical user's perspective, this is at best a [massive] shortcoming IMHO.
After all, without any SLA they could just hold data back completely, or provide the same QOS for DELAYED and REALTIME. It's an absurd proposition so I'm leaning toward "this is a bug (full stop)".
Playing that much faster and looser w/ their specifications would leave a bad taste in an engineer's mouth... although a lawyer might love it.
:-/
|
buddy
