I didn't say that TWS Api is fundamentally flawed in some way. To make this clear, i DO NOT have the audacity to state that something is "crap" that i worked with only 21 days. That would be ignorant and rude from my side. As a matter of fact i am currently evaluating if my trading approach is compatible with all that is provided in tws api or not. The provided API has to be taken as is, either it fits my needs or not. The only thing I can complain about right now is, that the lack of "clean code principles" / pImpl idiom (one can argue if its outdated practise) in TestClient.cpp as the class is too big.

I see that

"I consider the theoretical limit of orders (used long type) as a design flaw from IB, because this is a point where something can break (here its highly unlikely i only agree on that). If you have dozens of line like that the software is crap."

can be misinterpreted.

What I meant is, that if I compose MY (!!) software (NOT the TWS API itself) of functions that have certain restrictions it can possibly lead to problems on my side. So i brought up the (theoretical) question here, if this can be avoided. I don't mind to use an ever incrementing ID only for the orders as they are limited by their "nature" as you pointed out. This doesnt necessarily apply to requests for data or other information (as you don't know my already existing algorithms). I would like to avoid maintaining multiple IDs because it adds complexity and I am very very careful about adding complexity (murphys law, see example below). For example theres a request for pnl that needs an ID and there is updatePortfolio function where i can get the pnl information without using an ID. So here i prefer the function without the ID requirement to keep my datastructures updated.


The statement of "rock-stability" has to be proven to me as well in the months to come. I will not rely on advertised properties (no offense or what so ever). I simply have been working in the software industry for 15 years now in various fields (military, aerospace safety control, milling, sound systems, CAD,...) and from
my experience, you can break software systems if you touch them in some unusal way (no matter who writes the code). Again I am currently evaluating which of the functions provided "deliver in 100% of time", to find out if my approach is possible.


Example: I give you an example what i encountered: In my previous posts i mentioned that i worked with the MetaTrader 4. I can confirm that MT4 is rock solid. BUT i encountered a critical error: There exists a function where you can get the symbol name for the chart that is currently displayed. I configured the trading system to trade only the euro dollar and it executed a trade in the "US dollar Index" because somehow it switched the charts in the night. Therefore i prefer to hardcode the symbol name to avoid such dangerous problems.



I can agree on the technical part of your reply (-> highly unlikely that the limit will be reached in this case), but i will not deviate from my safety principles
that i developed over time if its not necessary

and of course thank you for the enlighting reply

regards,

Arthur

开云体育

Hi group,
I have been using IBKR API solution with C++? and using Xcode to edit the c++ code.?
I complied and executed it on Terminal on my Macbook pro by Linux commands such as "make" and "./".?
I recently purchased a new Macbook pro with M2 processor(Silicon chip?) . When I run ?the program first time on the new Mac, I had errors like below.??

Do you have any ideas why this errors should happen and what would be the solution to these errors??

Thank you!

=== Error on Terminal ===============================================================================================================================

g++ -DIB_USE_STD_STRING -Wall -Wno-switch -I../source/posixclient/shared/ -I../source/posixclient/src??-o TestPosix EClientSocketBase.o EPosixClientSocket.o PosixTestClient.o Main.o

(base) mbp-4725:TestPosix seoprhim$ ./TestPosix

dyld[24531]: dyld cache '(null)' not loaded: syscall to map cache into shared region failed

dyld[24531]: Library not loaded: /usr/lib/libc++.1.dylib

??Referenced from: <A3EC47BA-EC3F-3737-B3B6-396101A2BB48> /Users/seoprhim/Documents/new API_Beta/API_Beta/IBJts/cpp1/TestPosix/TestPosix

??Reason: tried: '/usr/lib/libc++.1.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/lib/libc++.1.dylib' (no such file), '/usr/lib/libc++.1.dylib' (no such file, no dyld cache), '/usr/local/lib/libc++.1.dylib' (no such file)

Abort trap: 6

?

(base) mbp-4725:TestPosix seoprhim$?

How do you determine that level 2 data falls behind the market? And how large is the lag you experience?

My immediate reaction would be to check whether MySQL database updates cause the lag. For liquid instruments, L2 data feeds easily generate hundreds if not thousands of callbacks per second and SQL database updates are generally very expensive and time consuming operations. In case your client performs any DB update functions within the TWS API callback thread, TWS API may be unable to report L2 updates to you in a timely fashion although your client already has received those messages from TWS/IBGW through the network socket.

闯ü谤驳别苍

Just because you don't like a design detail does not make it a design flaw, Arthur. Many of our group members are running rock stable automated trading systems based upon TWS API, some of them do that for more than 20 years, and Richard King created this mail group in 2002. There are aspects of TWS API that I personally would have designed differently, but it appears to me that TWS API is not fundamentally flawed.

You still have to get a much better understanding of TWS API, use it in a real (paper) account, and spend more time studying and understanding the . You clearly don’t have a grasp of TWS API yet if you seriously believe you “will avoid using the request() functions that need an ID …”.

TWS API is a low level, simple, and very fast asynchronous real-time protocol that is only concerned with the exchange of messages between your client app and TWS/IBGW within the current session (the time between connect() and disconnect() or when TWS/IBGW closes the connection). It connects your client with a complex trading world (any imaginable combination of accounts at IBKR, hundreds of exchanges, virtually unlimited number of tradable instruments) and relays to your client app(s) in real-time all events that trading world generates.

The API uses ephemeral IDs to relate one or more responses with a specific request and the actual ID values have no relevance once the session ends. IDs that fall into this class are orderIds, reqIds, and tickerIds.

Due to the real-time, asynchronous, and event-driven nature of the protocol, the design encourages maximum autonomy for clients and TWS/IBGW each. Therefore, your client application is in charge of and free to select the actual ID values with very few limiting requirements:

• At any point in time during a session, IDs have to be unique in the sense that no two outstanding requests shall use the same ID.
• orderIds shall be monotonously increasing by a value of at least 1
• the first orderId in a session shall have a value of no less than the nextValidId() value announced by TWS/IBGW during the connect() phase of the protocol.
• reqId and tickerId need to be unique but are not required to increase monotonously
• Other than that, your client may divide the available range of integer numbers anyway it likes (32bit signed int not 64bit signed long). There is no hard requirement that reqIds/tickerIds shall be different from orderIds, but since all requests share the same error() callback, practically it makes sense that you assure orderIds, reqIds, and tickerIds are all unique within the same session.

In a regime where TWS/IBGW restart daily, a client session cannot be longer than approx. 24 hours. At a rate limit of 50 requests per second, no client can generate more than 4,320,000 valid requests or orders during each session (practically only a tiny fraction thereof, though):

• The 32bit integer type has a value range of -2^31 to 2^31-1 which is a MAX_VALUE of 2,147,483,647. In other words, the integer data type can hold sufficient unique ID values for at least 500 sessions.
• No client can place more than 4,320,000 orders per session. Dividing the number space such that reqIds start at nextValidId() + 10,000,000 is therefore appropriate and there will never be an overlap in that session. And the initial values are recalculated right after connect() in the next session.
• realistic clients create only tens or hundreds of orders per session so there is really no risk of “overlap”. But if you are concerned, divide the number space such that orderIds increase starting at nextValidId() and reqIds/tickerIds decrease starting at 2,147,483,647 (MAX_VALUE ) each session.

As I said before, the scope of orderIds, reqIds, and tickerIds is one session for one client. In other words, identical numeric IDs can (and probably will) be used when a client reconnects for a new session or when several parallel clients have their own sessions. Objects with life spans of more than one session, therefore, have permanent identifiers that are different from ephemeral session IDs:

• Once your client has placed an order using the next valid orderId and in if that order is not rejected by IBKR, TWS API will send you a new Order object that contains the order’s permId. permId is assigned by IBKR, has a scope of “Account”, and is also an “int” value. This way, each order in each account can be identified uniquely in perpetuity, unless you plan on placing? 2,147,483,647 orders any time soon.
• In case an order fills (entirely or partially), one or more "trades" are recorded in and objects. These objects are identified by very long string based .

My suggestion is you forget about the IDs since there will be many and more important challenges ahead of you. Apply the “set and forget” and “keep it simple” principals, add a few lines of code to the interface between your clients and TWS API, and manage IDs just like we and many others do successfully for years:

• maintain a local nextOrderId() service that is initialized with nextValidId() when the client starts and monotonously increases each time it is called. It would be a waste of time to determine orderId by calling reqIds(-1) and waiting for the asynchronous answer for each individual order.
• maintain a local nextRequestId() service you use for all reqIds and tickerIds. It provides a unique number that is different from any previous nextRequestId() and all possible nextOrderId() values for the current session. It is robust, prudent and safe to simply initialize nextRequestId() with nextValidId() + someLargeOffset where someLargeOffset > 4,320,000.

The short answer to your "will then all results be gone ?" question is “no”.

All ephemeral data (such as orderIds, reqIds, tickerIds) will be gone once your client disconnects or TWS/IBGW restart (since they are obsolete), but any account level information is retained within the IBKR infrastructure and can be retrieved by your client app as needed. Please refer to the TWS API guide, but among others, you can request:

• all orders that are open when your client connects
• the entire trade history for the account for a period of up to seven days (length of history selectable within TWS)
• all current positions in the account
• a long list of account related monetary details including cash positions, net liquidation values, and currently committed levels of margin

Hope this helps,

闯ü谤驳别苍

On Sat, Aug 5, 2023 at 03:32 AM, Arthur wrote:

I consider the theoretical limit of orders (used long type) as a design flaw from IB, because this is a point where something can break (here its highly unlikely i only agree on that). If you have dozens of line like that the software is crap. As i experienced developer i like to avoid such pitfalls, if possible. In this particular case i can't avoid it but have to keep it in mind.

By "will then all results be gone ?" i meant that the trading history of already executed orders will be gone or not. As a matter of fact currently i only have demo access to tws and my trading results always will be cleared if i start the application

On Sat, Aug 5, 2023 at 03:47 AM, Arthur wrote:

@闯ü谤驳别苍
If you start counting the RequestId as "orderId + 10_000_000" you will get the problem when the overlap occurs (if the session is not "that long" this problem doesnt occur, so i have to calculate what "that long" exactly means). So the safest approach would be to get the m_orderId incremented by tws server when calling "m_pClient->reqIds(-1);" and using the callback nextValidId(). Further i will avoid using the request() functions that need an ID to be properly executed as much as possible if there exists an alternative to it.

@闯ü谤驳别苍
If you start counting the RequestId as "orderId + 10_000_000" you will get the problem when the overlap occurs (if the session is not "that long" this problem doesnt occur, so i have to calculate what "that long" exactly means). So the safest approach would be to get the m_orderId incremented by tws server when calling "m_pClient->reqIds(-1);" and using the callback nextValidId(). Further i will avoid using the request() functions that need an ID to be properly executed as much as possible if there exists an alternative to it.

I consider the theoretical limit of orders (used long type) as a design flaw from IB, because this is a point where something can break (here its highly unlikely i only agree on that). If you have dozens of line like that the software is crap. As i experienced developer i like to avoid such pitfalls, if possible. In this particular case i can't avoid it but have to keep it in mind.

By "will then all results be gone ?" i meant that the trading history of already executed orders will be gone or not. As a matter of fact currently i only have demo access to tws and my trading results always will be cleared if i start the application

To add to the previous comments, and make it more complicated: the order ID sequence is linked to the connection number. If you always connect only one client, and always use the same connection number then this remark is not relevant. However, if you first connect client A (eConnect(..,..,A)) and place several orders, and then connect client B (eConnect(..,..,B)) then the order ID sequence for B starts again at 1 (one). This makes the recommendation to use reqIds() to get an initial order ID even more valuable.

I wouldn't dismiss Gordon's comments outright, though. Granted, orderIds and requestIds serve very different purposes, and it is entirely legitimate if you have a market data subscription with requestId=10 as well as an order with orderId=10, but orderIds and requestIds are only independent until something goes wrong.

In that example, where would you route an ( id=10, .... ) callback? To the order manager that handles orderId=10 or the market data manager that handles requestId=10?

We follow the lead of the ApiController class that ships within the Java API and are very happy with this approach. We take the orderId value returned by

• and initialize a nextOrderId( orderId ) counter that monotonously increments for each call

• we also initialize a nextRequestId( orderId + 10_000_000 ) counter that also monotonously increments for each call

This approach assures that requestIds and orderIds are unique within each client session, that requestIds and orderIds never collide, and that all error callbacks can be associated with the request or order placement it relates to.

闯ü谤驳别苍

When I used the IBKR API to stream the level 2 data into a MySQL database, the data seems to fall behind the actual level 2 quotes that are in the market.? Is there any way to avoid this eventual lag that takes place?

ebtrader

Not sure about the "anchored" concept. Seems to me against IB philosophy, it is very difficult for them as well as you to decide what is the reference date of the instant to start computation.
And it is also against general understanding of what VWAP is (which is understood as? the Weighted Average of all transactions DURING the bar)
That way and using Volume and VWAP you can compute aggregation. (if they already aggregate for you it became impossible to compile that info)
I must admit that I don't know I never thought could be anything else than Bar duration, and I don't expect it to be an absolute reference but soundly enough computed so that it allows safe comparisons.

Anyway! A couple of cents is an excellent accuracy! Don't expect better.
There is even no way to know what is the truth. Between data vendors you can find 10% differences. (even Volume is a strange thing in itself nobody report exactly the same)
The main issue is to know what is the exact volume of each transaction. This is a daunting task for the data provider and they frequently also should deal with odd lots (and in fact frequently discard them)

1- I guess that on chart, IB give you the VWAP of the time frame, or of you preferred Time Period.
2- I recommend historicalBar with KeepUpToDate=true because this make your life simpler, just choose the timeframe that suite you the best and leaving IB in charge of this computation which require a lot more care than it looks.
For example fundamentally the VWAP varies by venues, because each exchange may have very different volume transacted during the bar, there are then as many VWAP as there are exchanges trading an instrument.
Seems more relevant that it looks because it allows you in a glimpse to give a clue of the most active market place etc ... a step toward HFT.

In short
Better leave IB give you a safe aggregate.
Seems you master what you need, don't expect better and trust the API.

On Tue, Aug 1, 2023 at 11:53 AM, ds-avatar wrote:

No time wasted, I received a response from IB today. API is likely to be supported in the new app, but no timeline. TWS and Gateway are here to stay.

many thanks. Hopefully Ewald will develop the new API wrapper just like he did with TWS API.

yes.. the 'realtimeBar' returns 5sec bars. In TWS I have 1m,2m,5m,10m,15m,30m (m=minutes) charts and all show the same VWAP value, both in Quote panel and in the chart, down to the penny.?
I tried your suggestion to "ask Historical Day bar for current day with?keepUpToDate =true". (used TSLA ticker for trial), I get a 5-6 cents difference from what's shown on the TWS chart (Classic TWS and Mosaic-chart show same values). This difference stays steady over a period of several minutes! (with AMD ticker, the diff was around 2-3 cents). Not sure why is there a difference as these values come from TWS itself!

I then turned on 'realtimeBar' and the wap value I got was about $4 apart with TSLA! What I think is reported by 'realtimeBar' is the "anchored" VWAP, i.e. starts calculating when I subscribe using 'reqRealTimeBars'. I locally started calculating anchored vwap (calculated on every 5 second bar) to compare the returned value from 'realtimeBar', which starts around same value and then differs as time goes by.?

To be precise, the TWS is designed to shut down some time (24h?) after being launched. It can auto-restart after the shutdown but only if it's specially configured to do so.

The NextValidId variable you are referring to needs not be incremented for __every__ request, as you seem to believe, it only needs to be incremented to define the Order Id when placing orders. Request Ids used for querying statuses and getting market data, news and whatnot have nothing to do with that and can be any numbers at the user's discretion. Surely it's sensible to use different request ids for different requests, but it's no problem reusing retired request ids. The same is not the case with Order Ids as they always need to be greater than any previously used order id, until the sequence is manually reset back to 0 in the TWS. Contrary to what Gordon pondered, I think it's not just some naughty "bad practice" to recycle order ids, it's completely useless since it will strictly result in an error and the order placement will be aborted; to make matters even worse, I used to get some cryptic and seemingly unrelated error messages from TWS when learning, and failing, to synchronize the order id sequence in a highly concurrent system, making the issue somewhat hard to debug. But I concur that the amount of orders needed to be placed to run out of order ids is so huge it's not worth a bother. To get some peace of mind it would seem adequate simply to reset the sequence once a year.?

--
Best,
DS

VWAP involve a duration over which the averaging is done,
Assuming what you call RT bar are 5 sec bar, What time frame is TWS bar ? if it's day then it is perfectly normal they don't show same thing.
Does TWS on chart use use chart time frame ?
suggestion : ask Historical Day bar for current day with keepUpToDate =true? and compare it with TWS in Tab view, or see on chart, using chart time frame with historical bar updated.

When you connect you will get a call back that fire: nextValidId(OrderId orderId)
Which give you the next valid id you can use (seems more to be used as an index, so you are free to use anything higher as long as you commit yourself to be consistent in numbering. I can even guess that you can recycle old number, just a guess, it's a bad practice)
So you will never reach the 'long' limit. BTW: Hypothetical case if reset did not existed, 10 call/sec non stop 24 h/days then you can run ~7 Years before you hit the limit.

Not sure I understand your question about "will then all results be gone ?", yes all pending request is aborted (and that is good as it avoid to reconcile requestid),
This happens around 23:00 ~adjusted to your time zone so unlikely you are doing anything critical.
As per existing order/position if I understand your question there, no impact, it's unrelated, and it work as you would do it with TWS

What you get is the exhaustive list of what you are allowed to use (normally linked to what you subscribed for)
Don't be lured by TWS that seems to work differently than Gateway, allowing you to see in the UI news you can't fetch trough the API.
Anyway even with TWS you cannot catch trough the API all you can read, (or I don't know how to do that)
I guess this is for copyright reasons as there is less risk that theses UI display news are "caught and forwarded" by bots, which is what providers are scared by.

FYI:
Once you get this list you should use it to call contractDetails which will give the exhaustive sub list of all feeds for each providers.
In your case it should come out with something essentially? like that (take what is after the ":" before is raw custom log)

+>IBK? 120: reqInitNews contractDetails: BRFG:BRFG_SMU,'Stock Market Update'
+>IBK? 120: reqInitNews contractDetails: BRFG:BRFG_ALL,'All News'
+>IBK? 120: reqInitNews contractDetails: BRFG:BRFG_STS,'Story Stocks'
+>IBK? 120: reqInitNews contractDetails: BRFG:BRFG_DSW,'Daily Sector Wrap'
+>IBK? 120: reqInitNews contractDetails: BRFG:BRFG_TECS,'Tech Stocks'
+>IBK? 120: reqInitNews contractDetails: BRFG:BRFG_RTB,'Rate Brief'
+>IBK? 120: reqInitNews contractDetails: BRFG:BRFG_FEDB,'Fed Brief'
+>IBK? 120: reqInitNews contractDetails: BRFG:BRFG_WKW,'Weekly Wrap'
+>IBK? 120: reqInitNews contractDetails: BRFG:BRFG_PONE,'Page One'
+>IBK? 120: reqInitNews contractDetails: BRFG:BRFG_TBP,'The Big Picture'
+>IBK? 120: reqInitNews contractDetails: BRFG:BRFG_MV,'Market View'
+>IBK? 120: reqInitNews contractDetails: BRFG:BRFG_WIP,'Week In Perspective'
+>IBK? 120: reqInitNews contractDetails: BRFG:BRFG_BIP,'Bond in Play'
+>IBK? 120: reqInitNews contractDetails: BRFG:BRFG_EDR,'Economic Data Review'
+>IBK? 120: reqInitNews contractDetails: BRFG:BRFG_UPDN,'Up/Downgrades'
+>IBK? 122: reqInitNews contractDetails: DJNL:DJNL_ALL,'All News'
+>IBK? 121: reqInitNews contractDetails: BRFUPDN:BRF_ALL,'All Briefing.com Events'

You are requesting data for Nasdaq only ("ISLAND") and that's probably what you get. I say "probably" because meaningful historical data are not easy to get from IBKR, at least?in terms of specifying what exactly is?required. Equities frequently change symbols and move between exchanges, and to add insult to injury perform all sorts of mergers, splits and spin-offs. Data providers need do some stock forensics to keep track of all of this so as to get all the different varieties of adjusted current data, including gluing together data from different exchanges and for different symbols, all referring to the same stock. I'm not sure IBKR is very good at this.

TLT is likely no stranger to such disruption as Blackrock moved a lot of their iShares ETF's between ARCA, BATS and NMS in the previous decade. Therefore it is completely credible that there must be no Nasdaq (NMS) data for this stock between its inception date in 2002 and some more recent date in the past.

Unfortunately it doesn't get much easier from here. Generally I would suggest just using "SMART" for the exchange so that the API aggregates data from all exchanges for the equity in question. This seems to work with TLT but only partially. Indeed, with SMART we get 5 times as much volume as with ISLAND supporting the suggestion that this time it's data collected from many more venues than just NMS. However the historical period now is even shorter - we only get data from 2016 and later.

I have no sensible interpretation for this other that 2016 was yet another of those years when some significant event took place in the history of this equity and IBKR probably messed up its aggregation. Usually I believe SMART should naturally yield both the biggest volume and the longest historical period.

FWIW I managed experimentally to get data right from the very inception of TLT in 2002 using exchange="ARCA". I guess that was the original exchange where TLT started to trade. But even though this data is available for the whole lifespan of TLT from 2022 right to 2023, the volume is again much smaller than with SMART (when it's available) since, naturally, we only get data for a single trading venue rather than the whole market, and one that has not even been primary for some years now, suggesting that the critical EOD prices in this period may not precisely match the official consolidated tape closing prints widely used for daily price tracking.

I tried to follow the instructions?
on how to get list of news providers codes,?but ended up only with:?

Hi all,
i read somewhere in the documentation that the trader workstation is designed to reboot every day. If this is the case, will then all results be gone ? No historical Profits ? The more important question is the variable that is incremented for every request, can it then start with value '1' again ?? Because if this isnt the case (the datatype is long ) theres a limit for the number of requests that one can send.

Please clarify

regards,
Arthur