Keyboard Shortcuts
Likes
Search
Question regarding reboot of trader workstation
Arthur
Hi all,
i read somewhere in the documentation that the trader workstation is designed to reboot every day. If this is the case, will then all results be gone ? No historical Profits ? The more important question is the variable that is incremented for every request, can it then start with value '1' again ?? Because if this isnt the case (the datatype is long ) theres a limit for the number of requests that one can send. Please clarify regards, Arthur |
When you connect you will get a call back that fire: nextValidId(OrderId orderId)
Which give you the next valid id you can use (seems more to be used as an index, so you are free to use anything higher as long as you commit yourself to be consistent in numbering. I can even guess that you can recycle old number, just a guess, it's a bad practice) So you will never reach the 'long' limit. BTW: Hypothetical case if reset did not existed, 10 call/sec non stop 24 h/days then you can run ~7 Years before you hit the limit. Not sure I understand your question about "will then all results be gone ?", yes all pending request is aborted (and that is good as it avoid to reconcile requestid), This happens around 23:00 ~adjusted to your time zone so unlikely you are doing anything critical. As per existing order/position if I understand your question there, no impact, it's unrelated, and it work as you would do it with TWS |
To be precise, the TWS is designed to shut down some time (24h?) after being launched. It can auto-restart after the shutdown but only if it's specially configured to do so.
The NextValidId variable you are referring to needs not be incremented for __every__ request, as you seem to believe, it only needs to be incremented to define the Order Id when placing orders. Request Ids used for querying statuses and getting market data, news and whatnot have nothing to do with that and can be any numbers at the user's discretion. Surely it's sensible to use different request ids for different requests, but it's no problem reusing retired request ids. The same is not the case with Order Ids as they always need to be greater than any previously used order id, until the sequence is manually reset back to 0 in the TWS. Contrary to what Gordon pondered, I think it's not just some naughty "bad practice" to recycle order ids, it's completely useless since it will strictly result in an error and the order placement will be aborted; to make matters even worse, I used to get some cryptic and seemingly unrelated error messages from TWS when learning, and failing, to synchronize the order id sequence in a highly concurrent system, making the issue somewhat hard to debug. But I concur that the amount of orders needed to be placed to run out of order ids is so huge it's not worth a bother. To get some peace of mind it would seem adequate simply to reset the sequence once a year.? -- Best, DS |
I wouldn't dismiss Gordon's comments outright, though. Granted, orderIds and requestIds serve very different purposes, and it is entirely legitimate if you have a market data subscription with requestId=10 as well as an order with orderId=10, but orderIds and requestIds are only independent until something goes wrong. In that example, where would you route an ( id=10, .... ) callback? To the order manager that handles orderId=10 or the market data manager that handles requestId=10? We follow the lead of the ApiController class that ships within the Java API and are very happy with this approach. We take the orderId value returned by
This approach assures that requestIds and orderIds are unique within each client session, that requestIds and orderIds never collide, and that all error callbacks can be associated with the request or order placement it relates to. ´³¨¹°ù²µ±ð²Ô On Fri, Aug 4, 2023 at 02:30 PM, ds-avatar wrote:
To be precise, the TWS is designed to shut down some time (24h?) after being launched. It can auto-restart after the shutdown but only if it's specially configured to do so. |
To add to the previous comments, and make it more complicated: the order ID sequence is linked to the connection number. If you always connect only one client, and always use the same connection number then this remark is not relevant. However, if you first connect client A (eConnect(..,..,A)) and place several orders, and then connect client B (eConnect(..,..,B)) then the order ID sequence for B starts again at 1 (one). This makes the recommendation to use reqIds() to get an initial order ID even more valuable.
|
Arthur
I consider the theoretical limit of orders (used long type) as a design flaw from IB, because this is a point where something can break (here its highly unlikely i only agree on that). If you have dozens of line like that the software is crap. As i experienced developer i like to avoid such pitfalls, if possible. In this particular case i can't avoid it but have to keep it in mind.
By "will then all results be gone ?" i meant that the trading history of already executed orders will be gone or not. As a matter of fact currently i only have demo access to tws and my trading results always will be cleared if i start the application |
Arthur
@´³¨¹°ù²µ±ð²Ô
If you start counting the RequestId as "orderId + 10_000_000" you will get the problem when the overlap occurs (if the session is not "that long" this problem doesnt occur, so i have to calculate what "that long" exactly means). So the safest approach would be to get the m_orderId incremented by tws server when calling "m_pClient->reqIds(-1);" and using the callback nextValidId(). Further i will avoid using the request() functions that need an ID to be properly executed as much as possible if there exists an alternative to it. |
Just because you don't like a design detail does not make it a design flaw, Arthur. Many of our group members are running rock stable automated trading systems based upon TWS API, some of them do that for more than 20 years, and Richard King created this mail group in 2002. There are aspects of TWS API that I personally would have designed differently, but it appears to me that TWS API is not fundamentally flawed. You still have to get a much better understanding of TWS API, use it in a real (paper) account, and spend more time studying and understanding the . You clearly don¡¯t have a grasp of TWS API yet if you seriously believe you ¡°will avoid using the request() functions that need an ID ¡¡±. TWS API is a low level, simple, and very fast asynchronous real-time protocol that is only concerned with the exchange of messages between your client app and TWS/IBGW within the current session (the time between connect() and disconnect() or when TWS/IBGW closes the connection). It connects your client with a complex trading world (any imaginable combination of accounts at IBKR, hundreds of exchanges, virtually unlimited number of tradable instruments) and relays to your client app(s) in real-time all events that trading world generates. The API uses ephemeral IDs to relate one or more responses with a specific request and the actual ID values have no relevance once the session ends. IDs that fall into this class are orderIds, reqIds, and tickerIds. Due to the real-time, asynchronous, and event-driven nature of the protocol, the design encourages maximum autonomy for clients and TWS/IBGW each. Therefore, your client application is in charge of and free to select the actual ID values with very few limiting requirements:
In a regime where TWS/IBGW restart daily, a client session cannot be longer than approx. 24 hours. At a rate limit of 50 requests per second, no client can generate more than 4,320,000 valid requests or orders during each session (practically only a tiny fraction thereof, though):
Hope this helps, ´³¨¹°ù²µ±ð²Ô I consider the theoretical limit of orders (used long type) as a design flaw from IB, because this is a point where something can break (here its highly unlikely i only agree on that). If you have dozens of line like that the software is crap. As i experienced developer i like to avoid such pitfalls, if possible. In this particular case i can't avoid it but have to keep it in mind. On Sat, Aug 5, 2023 at 03:47 AM, Arthur wrote: @´³¨¹°ù²µ±ð²Ô |
¿ªÔÆÌåÓýThanks for another great post ´³¨¹°ù²µ±ð²Ô ¨C I don¡¯t know how you find the time! Are you aware that there¡¯s this thing that humans are supposed to do, called ¡®sleep¡¯ if I remember correctly? ? I just wanted to mention that in fact it wasn¡¯t me that started this Group. It was created, within the now-defunct Yahoo Groups, by a user known as ¡®marinindextrader¡¯ (you can see his very first post, dated 8 June 2002, at /g/twsapi/message/1). I joined the Group in early May 2003, and I was invited to become a Moderator some time mid-noughties. Eventually ownership of the Group was transferred to me. I was responsible for the move from Yahoo Groups to Groups.io in January 2017 (the first post after the transfer is here: /g/twsapi/message/36779). In September 2021 I invited ´³¨¹°ù²µ±ð²Ô to become co-owner of the Group to ensure its continuity in case of any personal mishaps ¨C and I¡¯m delighted to say that this was an excellent choice. ? I absolutely underscore everything that ´³¨¹°ù²µ±ð²Ô says. The API is mature, reliable, performant and has tremendous depth and scope of functionality. I got my first autotrading system running live on 24 September 2003. I¡¯ve been using the API all day every trading day since then. Back then it had a lot of rough edges and missing functionality (though few actual bugs), and it was necessary to use a number of tricks to get round some limitations in the API (for example, if you requested contract details, there was no ¡®end¡¯ callback, so you couldn¡¯t directly know when you¡¯d got all the contracts resulting from a request, but I managed to find a workaround). ? You can get some idea of how far IB have come since then by the fact that the both TWS and the API libraries are very roughly ten times the size they were back in 2003. ? So, please let¡¯s not hear suggestions that IB¡¯s software is crap. It¡¯s not. ? Richard ? ? From: [email protected] <[email protected]> On Behalf Of ´³¨¹°ù²µ±ð²Ô Reinold via groups.io
Sent: Saturday, August 5, 2023 8:06 PM To: [email protected] Subject: Re: [TWS API] Question regarding reboot of trader workstation ? Just because you don't like a design detail does not make it a design flaw, Arthur. Many of our group members are running rock stable automated trading systems based upon TWS API, some of them do that for more than 20 years, and Richard King created this mail group in 2002. There are aspects of TWS API that I personally would have designed differently, but it appears to me that TWS API is not fundamentally flawed. You still have to get a much better understanding of TWS API, use it in a real (paper) account, and spend more time studying and understanding the . You clearly don¡¯t have a grasp of TWS API yet if you seriously believe you ¡°will avoid using the request() functions that need an ID ¡¡±. TWS API is a low level, simple, and very fast asynchronous real-time protocol that is only concerned with the exchange of messages between your client app and TWS/IBGW within the current session (the time between connect() and disconnect() or when TWS/IBGW closes the connection). It connects your client with a complex trading world (any imaginable combination of accounts at IBKR, hundreds of exchanges, virtually unlimited number of tradable instruments) and relays to your client app(s) in real-time all events that trading world generates. The API uses ephemeral IDs to relate one or more responses with a specific request and the actual ID values have no relevance once the session ends. IDs that fall into this class are orderIds, reqIds, and tickerIds. Due to the real-time, asynchronous, and event-driven nature of the protocol, the design encourages maximum autonomy for clients and TWS/IBGW each. Therefore, your client application is in charge of and free to select the actual ID values with very few limiting requirements:
In a regime where TWS/IBGW restart daily, a client session cannot be longer than approx. 24 hours. At a rate limit of 50 requests per second, no client can generate more than 4,320,000 valid requests or orders during each session (practically only a tiny fraction thereof, though):
As I said before, the scope of orderIds, reqIds, and tickerIds is one session for one client. In other words, identical numeric IDs can (and probably will) be used when a client reconnects for a new session or when several parallel clients have their own sessions. Objects with life spans of more than one session, therefore, have permanent identifiers that are different from ephemeral session IDs:
My suggestion is you forget about the IDs since there will be many and more important challenges ahead of you. Apply the ¡°set and forget¡± and ¡°keep it simple¡± principals, add a few lines of code to the interface between your clients and TWS API, and manage IDs just like we and many others do successfully for years:
All ephemeral data (such as orderIds, reqIds, tickerIds) will be gone once your client disconnects or TWS/IBGW restart (since they are obsolete), but any account level information is retained within the IBKR infrastructure and can be retrieved by your client app as needed. Please refer to the TWS API guide, but among others, you can request:
Hope this helps, ´³¨¹°ù²µ±ð²Ô On Sat, Aug 5, 2023 at 03:32 AM, Arthur wrote:
|
Arthur
I didn't say that TWS Api is fundamentally flawed in some way. To make this clear, i DO NOT have the audacity to state that something is "crap" that i worked with only 21 days. That would be ignorant and rude from my side. As a matter of fact i am currently evaluating if my trading approach is compatible with all that is provided in tws api or not. The provided API has to be taken as is, either it fits my needs or not. The only thing I can complain about right now is, that the lack of "clean code principles" / pImpl idiom (one can argue if its outdated practise) in TestClient.cpp as the class is too big.
I see that "I consider the theoretical limit of orders (used long type) as a design flaw from IB, because this is a point where something can break (here its highly unlikely i only agree on that). If you have dozens of line like that the software is crap." can be misinterpreted. What I meant is, that if I compose MY (!!) software (NOT the TWS API itself) of functions that have certain restrictions it can possibly lead to problems on my side. So i brought up the (theoretical) question here, if this can be avoided. I don't mind to use an ever incrementing ID only for the orders as they are limited by their "nature" as you pointed out. This doesnt necessarily apply to requests for data or other information (as you don't know my already existing algorithms). I would like to avoid maintaining multiple IDs because it adds complexity and I am very very careful about adding complexity (murphys law, see example below). For example theres a request for pnl that needs an ID and there is updatePortfolio function where i can get the pnl information without using an ID. So here i prefer the function without the ID requirement to keep my datastructures updated. The statement of "rock-stability" has to be proven to me as well in the months to come. I will not rely on advertised properties (no offense or what so ever). I simply have been working in the software industry for 15 years now in various fields (military, aerospace safety control, milling, sound systems, CAD,...) and from my experience, you can break software systems if you touch them in some unusal way (no matter who writes the code). Again I am currently evaluating which of the functions provided "deliver in 100% of time", to find out if my approach is possible. Example: I give you an example what i encountered: In my previous posts i mentioned that i worked with the MetaTrader 4. I can confirm that MT4 is rock solid. BUT i encountered a critical error: There exists a function where you can get the symbol name for the chart that is currently displayed. I configured the trading system to trade only the euro dollar and it executed a trade in the "US dollar Index" because somehow it switched the charts in the night. Therefore i prefer to hardcode the symbol name to avoid such dangerous problems. I can agree on the technical part of your reply (-> highly unlikely that the limit will be reached in this case), but i will not deviate from my safety principles that i developed over time if its not necessary and of course thank you for the enlighting reply regards, Arthur |
Arthur
@ ´³¨¹°ù²µ±ð²Ô
I had to read your last reply a couple of times due to its length and richness of detail. The solution with nextRequestId() with nextValidId() + someLargeOffset where someLargeOffset > 4,320,000 sounds feasible to me. I'll give it a try. From tests i did earlier today, it seems that the tws server doesn't increment the orderID by itself it only deliver the max that it received from N Client at any particular point during the session ? Is this correct ? |
Not sure what needs to be tested here since the documentation at is crystal clear.? Also not clear why you are still trying to force the API into some kind of a "server increments orderIds"? mode. That is not how it works and you just need to accept that. For your convenience, below and annotated copy of the first paragraph of Perhaps the most important event received after successfully connecting to the TWS is the [...] As its name indicates, the nextValidId event provides the next valid identifier needed to place an order. This identifier is nothing more than the next number in the sequence. This means that if there is a single client application submitting orders to an account, it does not have to obtain a new valid identifier every time it needs to submit a new order. It is enough to increase the last value received from the nextValidId method by one. For example, if the valid identifier for your first API order is 1, the next valid identifier would be 2 and so on. This is what you should do for now. The situation may become a little more complicated as JG has suggested when orders can simultaneously be placed by multiple clients and TWS. Particularly if one of those clients is the "Master Client" (received order status for all orders placed by all clients) or "Client 0" (receives orders order status updates for all orders placed in TWS). The documentation is again crystal clear for that case: However if there are multiple client applications connected to one account, it is necessary to use an order ID with new orders which is greater than all previous order IDs returned to the client application in openOrder or orderStatus callbacks. In your TWS API adapter (the small layer of code that shields your business logic from any and all TWS API details), you'd provide a very simple nextOrderId() service that works as follows:
You obviously want to put some synchronization in place to make sure increment and set operations are atomic. The final piece of documentation says: You can always use the method in the event that your client application loses track of the sequence. I hope this settles it.´³¨¹°ù²µ±ð²Ô On Mon, Aug 7, 2023 at 08:18 AM, Arthur wrote:
@ ´³¨¹°ù²µ±ð²Ô |
Thanks for the very informative posts.I was looking into orderId recently and it solved my problem.One little comment about the following cited piece in the document.
I think it is inherently a race condition. Think about the situation that an orderStatus callback is en route from TWS to the master client while at the same time the master client is placing an order. In this event, there may be order id collision. |
That is correct. There is a TWS API Global Setting that can eliminate the race condition for a large class of orders. It is called "Use negative numbers to bind automatic orders". When selected, automatic orders get bound to monotonously decreasing negative numbers and, therefore, will not interfere with the client's requirement of using monotonously increasing orderIds for new orders. The remaining cases need to be eliminated by design. When several independent clients place orders in the same account, there is probably a need for coordination amongst them anyway (account margin management, avoiding opposing orders for the same instrument, ...)? so that one shared orderId assignment strategy could be used. And the measure of last resort is monitoring the "Duplicate OrderID" error 103. The client could assign a new (higher) orderId and retry the placeOrder request. ´³¨¹°ù²µ±ð²Ô On Mon, Nov 6, 2023 at 10:38 PM, Little Trader wrote:
|