Keyboard Shortcuts
Likes
Search
Java tws api 972.03 too old? Causing errorCode 505 not found
开云体育Hello,Starting today, after placing an order I'm getting this strange error all of a sudden: id=-1 errorCode=505 msg=Fatal Error: Unknown message id. It comes back via the general error method??public void error(Exception e) in the implementation of EWrapper. Nothing has changed in the code, it's the same binary jar. The order is placed correctly though, but after the error, tws disconnects.? From the 972 api doc, there does not seem to be an error code 505. And the more current documenation (IBKR Campus) this error says indeed Fatal Error: Unknown message id. My TWS is on 10.30.1l of Aug 13 2024. My feeling is that the 972 api is just too old, but I may have missed something. I believe tws would show a warning when an certain api is getting outdated. Is anyone else working succesfully with api 972.03 and tws 10.30.1l ? Changing everything to api 1019.04 does have a big impact so it seems. It's not just a few parameters and types, but reqMktData and depth seem to work in a different way. The demo client app 1019.04 works fine, so I'll just have step through that I think. Or is there an api version that does work, but with lower impact? So that I have more time to make the big jump to 1019.04 which will happen some day of course. Thanks for any advice! Greetings, Raoul |
|
What is most amazing to me is that API 972.03 has worked for you this long, Raoul. And proof that TWS API messages can really be understood by clients and TWS/IBGW that are years apart. API 972.03 must be somewhere around API level 105, just shortly after the magic V100 level that made it possible for clients and TWS/IBGW to understand vastly different protocol levels.? API 10.30.1 is level 187 and has undergone more than 80 sometimes radical changes. Among them the chance from int to long for size fields, the subsequent change to a Decimal type, and the introduction of may new fields in Contract and Order objects. Eventually you should consider adopting a more modern API version, but I can see at least two scenarios that could buy you some time:
But it sound like there is a "long weekend and several six packs" event coming up for you. 闯ü谤驳别苍
?
On Tue, Oct 22, 2024 at 03:22 PM, Raoul Suurmeijer wrote:
Hello, |
开云体育On 22-10-2024 23:05, 闯ü谤驳别苍 Reinold via groups.io wrote:Looking back at all the changes, it is indeed surprising to see that things just kept running without any issues. I vaguely remember that in the past, tws would warn if the api version is no? longer supported or upgrade is strongly suggested. Some people on the mailinglist have seen this message pop up as well. Eventually you should consider adopting a more modern API version, but I can see at least two scenarios that could buy you some time: That should probably work for a while, but I like how tws is running nicely with the auto update and all.
Nice suggestion you found there, but unfortunately it didn't prevent the disconnect (all my tws interactions are within try catch). The weird thing is that the order is placed correctly, which seems like the most critical part to me, but yet there's some other fatality. I'll take the big jump to the current stable version instead and start reading through the (updated) manual.
Oh I like those weekends, even more so with an api upgrade ;-) As always, thanks again for your help 闯ü谤驳别苍! Greetings, Raoul |
|
Don't use version 10.30 (even if it's the current stable version). This issue is fixed in versions 10.31 and newer. It didn't exist with versions 10.29 and older.
The problem started with TWS version 10.30 and since stable version rolled from 10.19 to 10.30 a few days ago, you have started experiencing the problem just now. This is why one should always prefer the offline versions of TWS that do not automatically update (even while using the stable version). Here's a list of my findings. I have experienced this issue myself with API clients that use older versions of the API. The problem triggers in the orderStatus callback. In the newest APIs, the orderStatus callback sends Decimal fields (instead of Integer) for filled and remaining quantities (openOrder does too but in the Order object). Months ago, I have contacted IB to ask if a TWS API setting could be introduced to maintain compatibility with older API clients but my request was denied due to lack of demand. I decided to start upgrading my applications to use the newest API. Later, I found that TWS version 10.31 and 10.32, for some unknown reason, do not trigger this error. Stable version is currently 10.30 but it has this issue with older API clients, specifically, in the orderStatus callback. I have also found that there is a thread in this group that mentioned this issue: /g/twsapi/topic/107023187 Hope this helps. -- |
开云体育Thanks for the warning. I just finished the api upgrade to 1019.04 which went a lot smoother than I thought. Everything is back up and running. The download page still has 1019 as stable and 1032 as latest. Maybe they're jumping from 1019 right to 1032.Or maybe you're referring to tws? I have done some test in paper with the same tws and new api, all seems to be fine. But it is most likely the reason my previous ancient api 'suddenly' stopped working ;-) I will probably follow up on making more frequent, but smaller, api upgrades to prevent such problems in the future and also stay in-sync with my auto-updated tws. Take care, Raoul On 25-10-2024 12:04, Orionn wrote:
Don't use version 10.30 (even if it's the current stable version). This issue is fixed in versions 10.31 and newer. It didn't exist with versions 10.29 and older. The problem started with TWS version 10.30 and since stable version rolled from 10.19 to 10.30 a few days ago, you have started experiencing the problem just now. This is why one should always prefer the offline versions of TWS that do not automatically update (even while using the stable version). Here's a list of my findings. I have experienced this issue myself with API clients that use older versions of the API. The problem triggers in the orderStatus callback. In the newest APIs, the orderStatus callback sends Decimal fields (instead of Integer) for filled and remaining quantities (openOrder does too but in the Order object). Months ago, I have contacted IB to ask if a TWS API setting could be introduced to maintain compatibility with older API clients but my request was denied due to lack of demand. I decided to start upgrading my applications to use the newest API. Later, I found that TWS version 10.31 and 10.32, for some unknown reason, do not trigger this error. Stable version is currently 10.30 but it has this issue with older API clients, specifically, in the orderStatus callback. I have also found that there is a thread in this group that mentioned this issue: /g/twsapi/topic/107023187 Hope this helps. |
|
Sorry to clutter the Post but as it looks like issue is raised about old lib 10.11, hhereafter is a past of an internal memo about it. ? ONLY of interest (aside of curiosity) to users using c++ language (while it may look like it’s could help Java user too) AND API rev >= 10.11 and somewhere < 10.25 (earlier version may work but not checked, as the hereafter patch worked for 10.11 version, we jumped to 1.25 afterwards) The IB lib work as expected in C# and most probably in Python (not tested) And did work well in C++ with rev 9.xx ? ? The internal Post: ? ? ? ? FYI about 10.11 The IB lib work as expected in C# and most probably in Python (not tested) And did work well in C++ with rev 9.xx ? In EClient.cpp, ?the following method as delivered by IB, doesn’t work (at least for me, tell me if I am alone with that issue) and need a mandatory patch (from my perspective) ? void EClient::placeOrder( OrderId id, const Contract& contract, const Order& order) This method is very fundamental and have no alternative, Environment: VStudio 2019 c++ default compiler, however I have reason to believe that using gcc would raise same issue ? The issue: 1- When upgrading the API to support Decimal, (API_Version>= 10.10.01), IB did introduce a new method to handle Encoding of Decimal when building the msg Message to local server. ?template<> void EClient::EncodeField<Decimal>(std::ostream& os, Decimal); ? 2- Decimal type is defined through a typedef unsigned long long Decimal; However in EClient.cpp, the methode EClient::placeOrder while preparing it’s buffer does need run code: ENCODE_FIELD(order.conditions.size()); ? 3-? As .size() deliver a size_t type which is also a unsigned __int64 (on X64) compiler see the same basic type than Decimal, because being define as typedef unsigned long long Decimal; ? Consequence: The compiler does call the ::EncodeField<Decimal>(std::ostream& os, Decimal); template to encode the order.conditions.size(); ? This is legitimate from compiler standpoint; however, Decimal is NOT at all able to encode a size_t properly. In particular Decimal 0 is not at all delivering "0" in the stream but a crazy value. Such messy message with a wrong size, are rejected by the server (and the connection brutally severed) ? ? The workaround: patch the IB code in EClient::placeOrder ?call with a cast (double) to force usage of another method un-equivocally, like: ENCODE_FIELD((double)order.conditions.size());? ? This force polymorphism to call the ‘double’ method instead of the 'Decimal' And "voila" it works, as message size have no known reason to be bigger than 2GB and IEEE double is properly encoding int, this casting is safe enough. ? ? => Updated as of 10/19/2023 (maybe earlier): In Lib like 10.25 IBKR seems to have done about the same thing but casting to a long Which is safer because long is a better guaranty from compiler stand point to be an integer type. So you better use 10.25 and discard 10.11 |
|
@Orionn
toggle quoted message
Show quoted text
I can confirm that this problem still exists in the supposedly 'stable' TWS 10.30.1o. However there's no reason for users not to use 10.30.1o if they use any of IBKR's current API versions, which are immune. But they may hit the problem if they're using a very old pre-V100 API version, or a V100 version (9.72 onwards) with DisableV100 set, or a home-grown API implementation that doesn't use the message length headers wisely. I hit this problem myself with an earlier version of TWS 10.30, using my own API implementation. This did use the V100 protocol, but it didn't make use of the API message length header to limit the parsing of the API messages. It just parsed the incoming data as one continuous stream, and skipped over the next message length header when it thought it had got to the end of each message (ie it had processed all the expected fields). But this caused a problem if the message actually contained an unexpected field at the end, which is indeed the case for the open order message. To fix this, I simply used the message length header to skip directly to the start of the next message: any irrelevant data at the end of a message is simply ignored. Here's the TLDR bit: Your diagnosis of the issue is not correct. The problem is actually caused by the open order API message, not the order status message. The open order message contains one extra zero byte at the end, effectively an empty field Here's an open order message from earlier today (underscores represent zero bytes): __ ?5_268575564_471204140_TSCO_STK__0_?__LSE_GBP_TSCO_SET1_SELL_100_MKT_0.0_ 0.0_DAY__DU5263__0__49723415_1900775028_0_0_0__1900775028.0/DU5263/100______ ___0__-1_0______2147483647_0_0_0__3_0_0__0_0__0_None__0____?_0_0__0_0______0 _0_0_2147483647_2147483647___0__IB_0_0__0_0_PreSubmitted_1.7976931348623157E 308_1.7976931348623157E308_1.7976931348623157E308_1.7976931348623157E308_1.7 976931348623157E308_1.7976931348623157E308_1.7976931348623157E308_1.79769313 48623157E308_1.7976931348623157E308______0_0_0_None_1.7976931348623157E308_1 .7976931348623157E308_1.7976931348623157E308_1.7976931348623157E308_1.797693 1348623157E308_1.7976931348623157E308_0____0_1_0_0_0___0_______ And here's the order status message: ___<3_268575564_PreSubmitted_0_100_0_1900775028_0_0_49723415__0_ Here's how the order status message is interpreted (I've left out a lot of the fields in the middle because they're not relevant): Id=268575564 ContractId=471204140 Symbol=TSCO Sec type=STK Expiry= . . . omitted fields . . Duration= PostToAts= AutoCancelParent=0 MinimumTradeQuantity= MinimumCompeteSize= CompeteAgainstBestOffset= MidOffsetAtWhole= MidOffsetAtHalf= So the last zero in the open order message above is the value of the AutoCancelParent field, and it's followed by its zero-byte field terminator. There should then be another 5 field terminators for the empty following fields. But in fact there 6 field terminators. If the API implementation simply extracts fields consecutively according to the expected message structure, and then takes the next 4 bytes to be the message-length header of the next message (but doesn't actually use the message length to control the parsing), it will now get out of sync. It will take the length header of the next message to be the last zero byte of the open order message and the first three bytes of the next message, ie ____ (which is zero, but that doesn't matter if we're not using the message length). And the message type of the next message will be "<3", which is not a valid message type. This might explain the "Unknown message id" error message received by the original poster in this thread: note that he was using API 9.72, which would only be subject to this flaw if he had called DisableV100. Richard -----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Orionn Sent: 25 October 2024 11:05 To: [email protected] Subject: Re: [TWS API] Java tws api 972.03 too old? Causing errorCode 505 not found Don't use version 10.30 (even if it's the current stable version). This issue is fixed in versions 10.31 and newer. It didn't exist with versions 10.29 and older. The problem started with TWS version 10.30 and since stable version rolled from 10.19 to 10.30 a few days ago, you have started experiencing the problem just now. This is why one should always prefer the offline versions of TWS that do not automatically update (even while using the stable version). Here's a list of my findings. I have experienced this issue myself with API clients that use older versions of the API. The problem triggers in the orderStatus callback. In the newest APIs, the orderStatus callback sends Decimal fields (instead of Integer) for filled and remaining quantities (openOrder does too but in the Order object). Months ago, I have contacted IB to ask if a TWS API setting could be introduced to maintain compatibility with older API clients but my request was denied due to lack of demand. I decided to start upgrading my applications to use the newest API. Later, I found that TWS version 10.31 and 10.32, for some unknown reason, do not trigger this error. Stable version is currently 10.30 but it has this issue with older API clients, specifically, in the orderStatus callback. I have also found that there is a thread in this group that mentioned this issue: /g/twsapi/topic/107023187 Hope this helps. -- |
|
Let me do a little train conducting here. The various posts describe API situations correctly, but they are from different time frames and vastly different versions of the API and Trader Workstation. Let;s start with Trader Workstation TWS 10.30.1.e from July 1, 2024 (a "latest" version at that time). There were several reports about an extra byte in messages clients received from TWS. That caucused issues when that specific TWS talked with clients that implement certain lower version APIs (such as version 177 as Richard reported). In July, closely related Trader Workstations 10.29 and 10.31 did not show the issue. Very recently, Trader Workstation 10.30.1l replaced 10.19.2p as the new "stab;e" version. Between the faulty version 10.30.1e in July and 10.30.1l now, Trader Workstation had several minor releases that may have addressed the Julz issue. And as of right now, the "stable" TWS has actually moved to 10.30.1o. After upgrading to TWS API 10.19.04 (which is API version 176) , It looks like Raoul is able to run a client successfully with "stable" Trader Workstation 10.30.1l. I also have not seen any "extra byte" issues when my stable Trader Workstation 10.30.1l talked with clients of TWS API version 173 all this week. So right now it looks to me as if "stable" Trader Workstation 10.30.1l is, indeed, stable and does not show the issues reported in July. I will try out "stable" Trader Workstation 10.30.1o over the weekend as well. 闯ü谤驳别苍 ? ?
?
On Fri, Oct 25, 2024 at 05:04 AM, Orionn wrote:
Don't use version 10.30 (even if it's the current stable version). This issue is fixed in versions 10.31 and newer. It didn't exist with versions 10.29 and older. |
开云体育闯ü谤驳别苍 I think what you’re missing here is that the ‘extra byte issue’ was never a problem for any of IBKR’s APIs, at any version, that use V100 protocol (and didn’t have DisableV1000 set). ? This is because of the way they work. They split the incoming message stream into individual messages using the message length header, and these individual messages are queued. Each message is then eventually removed from the queue, and EDecoder parses it according to the expected structure. Once it has dealt with the expected last field, any remaining bytes are simply ignored, job done. ? But if you have a pre-V100 API implementation, the only way the parser knows when it has got to the end of a message is that it has processed the relevant number of fields for that message, and the very next field is the message type for the next message. There is no possibility here of breaking the overall byte stream into individual messages first, because there is nothing to indicate where the breaks between messages are without actually parsing the whole lot. Note that if you set DisableV100 for a V100 implementation, then the individual messages are extracted in precisely this way by parsing the byte stream – see the readSingleMessage() function in EDecoder: so in this case it’s vulnerable to the extra byte problem. ? So the only people who have had a problem with the extra byte are those using pre-V100 APIs, (ie API 971 or earlier) ?or using API 9.72 or later versions with DisableV100 explicitly turned off. This included me, Ross Hemingway, and the others who voiced their experience with the faulty version of TWS 10.30. ? As far as 10.30.1o is concerned, it is simply indisputable that it still adds this extra byte to the open order message, as my example from earlier today showed. My example was produced with my own .Net API implementation, which I haven’t fixed yet for this issue (it doesn’t use the message length headers). But my COM-based implementation that has been enhanced to use the message length headers works perfectly. ? So I repeat, anyone using one of the IBKR APIs from 9.72 onwards will have no problems ?with TWS 10.30.1o (at least, not from the extra byte issue). Just for completeness, I still have the installer for API 9.72 (and for other versions going right back to 2003). I installed it today and ran a test with it: the order went through fine and the open order and order status messages were processed without a problem. ? Richard ? ? From: [email protected] <[email protected]> On Behalf Of 闯ü谤驳别苍 Reinold via groups.io
Sent: 25 October 2024 20:10 To: [email protected] Subject: Re: [TWS API] Java tws api 972.03 too old? Causing errorCode 505 not found ? Let me do a little train conducting here. The various posts describe API situations correctly, but they are from different time frames and vastly different versions of the API and Trader Workstation. Let;s start with Trader Workstation TWS 10.30.1.e from July 1, 2024 (a "latest" version at that time). There were several reports about an extra byte in messages clients received from TWS. That caucused issues when that specific TWS talked with clients that implement certain lower version APIs (such as version 177 as Richard reported). In July, closely related Trader Workstations 10.29 and 10.31 did not show the issue. Very recently, Trader Workstation 10.30.1l replaced 10.19.2p as the new "stab;e" version. Between the faulty version 10.30.1e in July and 10.30.1l now, Trader Workstation had several minor releases that may have addressed the Julz issue. And as of right now, the "stable" TWS has actually moved to 10.30.1o. After upgrading to TWS API 10.19.04 (which is API version 176) , It looks like Raoul is able to run a client successfully with "stable" Trader Workstation 10.30.1l. I also have not seen any "extra byte" issues when my stable Trader Workstation 10.30.1l talked with clients of TWS API version 173 all this week. So right now it looks to me as if "stable" Trader Workstation 10.30.1l is, indeed, stable and does not show the issues reported in July. I will try out "stable" Trader Workstation 10.30.1o over the weekend as well. 闯ü谤驳别苍 ? ? ? On Fri, Oct 25, 2024 at 05:04 AM, Orionn wrote:
|
|
Thank you for the clarification, Richard. I had been multitasking this afternoon and missed your detailed description while I was crafting mine. Needless to say I saw it right after hitting send. It makes sense that users of IBKR supplied version 100++TWS APIs (such as Raoul and myself) have no issues with TWS 10.30.1l. The IBKR message decoders take the "message length" field into consideration and read the correct number of bytes for each message from the socket. Even if there are more bytes than what the message actually needs. So I guess this is an "all clear" the 10.30.1l or 10.30.1o for most of us. 闯ü谤驳别苍 ?
On Fri, Oct 25, 2024 at 04:24 PM, Richard L King wrote:
|
|
Yes indeed, you said "Don't use version 10.30 (even if it's the current stable version)."
My post is disputing this assertion. As I thought I said very clearly, there is no reason not to use TWS/Gateway 10.30, provided you are using any of IBKR's current API versions. In other words, I think your assertion is completely wrong without some qualification and it's misleading for group members, who should ignore your suggestion. |
|
Nowhere was it suggested not to use TWS/Gateway 10.30 with recent API versions.
Of course there is no reason not to use TWS/Gateway 10.30 provided you are using any of IBKR's current API versions. My assertion not to use TWS/Gateway version 10.30 was in reply to Raoul who, like me, was using a very old API version simultaneously with TWS/Gateway 10.30. For reference, Raoul's initial post: /g/twsapi/message/53378 And my reply to his post:: /g/twsapi/message/53389 -- |
|
Fair enough.
toggle quoted message
Show quoted text
Apologies if I misunderstood what you were saying. Richard -----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Orionn Sent: 28 October 2024 14:51 To: [email protected] Subject: Re: [TWS API] Java tws api 972.03 too old? Causing errorCode 505 not found Nowhere was it suggested not to use TWS/Gateway 10.30 with recent API versions. Of course there is no reason not to use TWS/Gateway 10.30 provided you are using any of IBKR's current API versions. My assertion not to use TWS/Gateway version 10.30 was in reply to Raoul who, like me, was using a very old API version simultaneously with TWS/Gateway 10.30. For reference, Raoul's initial post: /g/twsapi/message/53378 And my reply to his post:: /g/twsapi/message/53389 -- |