On Sun, Oct 1, 2023 at 03:15 PM, David Armour wrote:

I agree with everything you said although I am not entirely happy with my final solution. I do not feel that destroying the EReader object is the correct way to disconnect. It is certainly not intuitive which means I cannot be the only person finding this problem. Is everybody using C++ just ignoring this error?

You may see this as a wart but I, personally, won't comment with regard to how intuitive things are or any design aesthetic. I will say that consideration should be given to backward compatibility and just how much existing code would break (and therefore need to change) if the disconnect where removed from the destructor.

Just some food for thought... since there's usually ripple effects, unintended consequences, and API stability is actually very important. Also, I don't think it's wise to hold off fixing some simple, obvious, and existing issue merely because there's an intention of addressing it in a more significant re-design. As they say, there is no time like the present and I appreciate small and tractable, incremental changes.

However, this may come down to a judgment call. Since, I guess, people have indeed been ignoring it or working around it (or it's simply gone unnoticed). After all, it only happens late in the game when there usually isn't much network activity expected or necessarily happening.

Anyway, I will sleep on it one more night before submitting a bug report.

Is there a special place to report API bugs or do we just raise ticket as usual?

A pull request is probably the best way to go about this:

On Sun, Oct 1, 2023 at 10:02 AM, David Armour wrote:

Thanks for your comment Buddy.

You're welcome. I think you've found an interesting oversight.

I agree that the logic would seem that setting m_isAlive to false during the ~EReader() call should prevent the code trying to read from the socket.

Yeah, superficially and at first blush it would seem that way. But it'd be a mistake to think so. I can understand how that initial thought may have fooled the original author into thinking the code was correct though. All this is post-mortem speculation on my part however.

Nevertheless, once considered more thoughtfully, it becomes apparent that nothing prevents the destructor from being called while the processing thread's readToQueue is in the while block.

The std::atomic will only insure that reading and writing m_isAlive from multiple threads is well-defined; but guarantees nothing w.r.t. the destructor and what's happening in the while loop.

I tried playing around with that. If I pause after m_isAlive is set to false, then continue, the code works successfully without 509 error. It is as if at

"Pause", with a sleep or something? No... that's surely a game of whack-a-mole. It might work sometimes, depending on unpredictable factors, and might not.

It can be fun to experiment with the timing while you investigate though. You can also try playing w/ ordering by running the code on a single core/cpu and see how that affects things.

full speed the atomic operation is not being completed before the call to eDisconnect() happens. On Visual Studio this even happens when I turn off all Optimisations. As eDisconnect() does not use anything related to that atomic, I wonder if the compiler is parallelising the operation as all atomic operations are very slow.

Compiler optimizations fall somewhat into the category of "unpredictable factors" I alluded to (although they'd be deterministic given the compiler source and enough work). It's a bit surprising that a critical section wasn't implemented via mutex, but maybe the author wanted to avoid a performance penalty.

I also noticed that eDisconnect() sets m_fd = -1.? This should prevent the call to processNonBlockingSelect() from trying to call onReceive() which ultimately triggers receive() and recv() causing the error. Again the setting of the atomic variable is not being done in time.

I am still scratching my head over this. I suspect some synchronization issue between the various threads.

Anyway... I think you found a legitimate problem and I wouldn't worry about it too much. I like your proposed solution as well. Joining the thread and then disconnecting makes clear sense to me. And, as long as m_pClientSocket isn't being shared all over the place I don't see a reason to use more complicated locking via mutex.

If I were the reviewer I could see signing off after some confirmation tests... open a ticket and see what they have to say. If you submit a patch remember to change the IB_POSIX ifdef block too :-)

I guess the assumption is that using std::atomic<bool> m_isAlive; turns the following into a critical section:

void EReader::readToQueue() {
	//EMessage *msg = 0;

	while (m_isAlive) {
		if (m_buf.size() == 0 && !processNonBlockingSelect() && m_pClientSocket->isSocketOK())
			continue;

        if (!putMessageToQueue())
			break;
	}

But it does not, so I take your point.

??? if (m_hReadThread) {
??? ? ? ?? m_isAlive = false;
?? ??? ??? m_pClientSocket->eDisconnect();
? ?? ????? WaitForSingleObject(m_hReadThread, INFINITE);
?? ???? }

??? if (m_hReadThread) {
????????? m_isAlive = false;
? ? ????? WaitForSingleObject(m_hReadThread, INFINITE);
?? ?????? m_pClientSocket->eDisconnect();
?? ???? }

????? ftrMsgProcThrd_ = pool_->submit(
????????? [&]()
????????? {
??????????? while (clientSocket_->isConnected())
??????????? {
????????????? signal_.waitForSignal();?? // This waits 2 seconds.
????????????? reader_->processMsgs();
??????????? }
????????? });

I decided to tackle this annoying bug (not the first time) and have found that after the call to EClientSocket::eDisconnect() which calls EClientSocket::SocketClose() which just calls a Windows Sockets closesocket() on the open socket, I am still getting the message processing thread (EReader thread) trying to perform a Windows Sockets recv() messages on the closed socket resulting in a 509 error. I have traced that error to be socket error 10038 which confirms it is an invalid socket (in this case, a closed socket).

Before the line "reader_->processMsgs()"? I have tried checking for the socket still being open with if(clientSocket_->isConnected()) but it does not solve the problem. The EReader is running in its own thread as per the reader_->start() call.

I thought, perhaps I need to close the EReader before calling eDisconnect so I tried deleting the object and removing the call to eDisconnect() because the destructor of the EReader calls eDisconnect() itself but this does not fix the error. i still get the 509 caused by a read on the closed socket.

I am struggling here and would appreciate advise from any C++ coders that use a multi-threaded approach like the above. It is likely a threading issue but if anyone else has faced a similar "disconnect" issue I would be happy to hear what you did to resolve it.

"200ms-400ms? delay in between each order is transmitted by Gateway and has?Submitted status"

• A laptop on which I just did a fresh reinstall of Windows
• A newly-downloaded TWS (version 10.25.1j, which is their 'LATEST' build, but I've also tried the most recent STABLE and BETA builds, same result)
• A newly-downloaded TWS API version 10.25.01?(which is LATEST, but also tested STABLE, which is v 10.19.01, same result)
• And the Excel file that generates the NBBO error is the default 'LegacyTwsDde.xls' file that gets downloaded along with the API. I literally just open it in its default state, put my username is cell D5 of the Basic Orders sheet, and press the 'Place / Modify Order' button on one of the default populated rows (e.g. IBM, row 15)...and it throws the NbboPriceCap error in cell J6, as shown in my screenshot.

@Richard

I know that you were not trying to suggest one should use the same numerical ids for different request types. I just wanted to add another reason why one should not do it.

Looks like we "late comers" have a little advantage since the TWS API source code contains more useful bits than what you had to work with in the early years.

@Gordon

We are meandering away from the original topic, but a couple thoughts on why you get "117" or something similar for nextValidId.

TWS/IBGW actually memorizes the highest orderId for each clientId and each account. I did not know this either for the longest time, but I think it was JG who made a comment related to this in a post a couple months ago. So I went through the logs and looked for the oldest clientId I could find for a client that placed orders (that clientId had not been used in at least three years), used that clientId to connect to the account, and for sure, automagically, nextOrderId upon connection returned a number that was one higher than the last order that client had placed three years ago.

So if you are reusing the same clientId and that client occasionally places orders, nextOrderId will be going up slightly over time. I guess the clientId that gave you a nextValidId of "117" has placed 115 or so orders over time.

You can break that cycle and reset all orderIds for all clientIds for an account back to, I believe, "1" with the "Reset API order ID sequence" button at the bottom of "Global Configuration -> Configuration -> API -> Settings".

We also never call reqIds while a client is running. The nextValidId we receive as part of the connection protocol is sufficient for a sequence of (thread safe) incrementing orderIds.

We recently added a little code that potentially nudges the clients' internal nextValidOrderId counter while they are running. That code assures that the nextValidOrderId counter is always higher than any orderId the client is exposed to by openOrder and orderStatus callbacks. This is most important for the master client and client 0 but we just added it to the framework for all clients based on this comment in :

"However if there are multiple client applications connected to one account, it is necessary to use an order ID with new orders which is greater than all previous order IDs returned to the client application in openOrder or orderStatus callbacks."

闯ü谤驳别苍

I never got a first nextValidID? < 100.?
Got things like "117", why ? never look at the why, I don't know of any zombie process and it works, so ...

Your absolutely right, rules are more relax, and I am a bad guy, I don't abide all rules that I may recommend. However I was burned being too relax.

You have more experience than I on handling IB and have the practical knowledge of the limits in use. As your approach work and fulfill your need, and your right, classifying request is not a must.
To be a better guy ... hereafter sharing what I do. (IMHO, just a use case) not the right thread for that but the djinn is out of the bottle

1- I use a single central requestId 'pump', thread safe,
and another one, separate, for OrderId.
I used to use the same pump for both, but it's not pretty. (very subjective criteria);

2- I always track nextValidId and use it as seed for it.
Taking the highest of next one or IB suggestion.
Handling of nextValidID must be within the protected critical section of the TS pump.
But it's very rare, happens only on connect.

3- I don't call reqIds , not that it's bad, just that I am not ready to wait for the answer.

4- I prefer "overpumping" (wasting ids) than taking the risk to re-use an existing id, even if serviced (even if it may work, I feel it's like playing with fire)

5- I try to keep continuity in OrderId, (I do in fact, but the code is ready for any discontinuity)
But I don't care at all about continuity on 'request'.
Assuming that the higher is the reqid the latest the request was built, was enough for diag.

The side effect is that OrderId grow much slower than reqid, so rapidly you can diagnose what you are looking at just by the range of the number.
@Jurgen, de facto it does something like what you suggest with +100000, (which make more sense, using a fixed value, I agree, as its very unlikely you ever do 100000 orders without a reboot)
However again human-wise: it's difficult to read a huge number.
Most of the time, tests are conducted at beginning of code start, hence small numbers which are easier to memorize or even simply read.

6- I did try to make it bit field on LSB,
In fact in was not so handy as sorting pending/send/received request id manually during debug can become cumbersome.
It help tracking the class of request but you loose the ease to track the sequence.

I dislike ranges, because I always wonder if or when they may overlap. (or require code to handle it as an exception) Not enough experiences with that, probably too dogmatic

This is just a use case.

开云体育

thank you for the clarification. Gives me a better handle on what's going on. I'm going to rationalize my data approach in 2024 as I have MANY experimental scripts ( python, julia, and now mojo) which are run by cron jobs. I just shut one down that's been running for 2 years and there's a 20gb csv with all the data.? Thanks again for all you do and taking the time.

Good luck with log reading. I hope it helps you find the root cause.

• The real-time market data request for a Frankfurt based instrument is handled by eufarm in Switzerland
• My TWS is in the US, did not have an active connection to eufarm but automatically established one when needed, No surprise here.
• The somewhat surprising observation is that the Historical Market Data Farm euhmds is also needed, although my client did not request any historical data. In fact TWS always connects the corresponding HMDS farm before the actual data farm since it provides certain instrument related information it needs for the real-time market data subscription.

Since you push the envelope a little by requesting historical data for 100 instruments I thought it might be good to make sure the client is careful with other requests that inadvertently cause HMDS queries.

闯ü谤驳别苍

开云体育

thank you AGAIN for such a detailed analysis.? I have turned on the HMDS logging feature for extra detail, thanks again for the heads up.? I am always a little nervous when there is a hint of data flow consolidation. I can see why they would want to do it but...

the issue I have with these logs is that I would have expected the log to show the unique identifiers for the request. Your request reqMktData( ADS:STK:FWB )? has no, that I can see, identifier NOR does the log give me a CLEAR indication of which dataserver got the request. I understand that they have load balancers but it would be nice to be able to read the logs more clearly.

I had not thought to look at the internal TWS logs partly because I don't like having to decrypt them all the time. I hoped that I could turn encryption off but I suppose that it's necessary in this day and age.

Based on your wonderful ( as always) overview I am going through all my linux boxes ( scattered over the house) and check the crontab for TWS scripts that might be running that add to my data pacing. I'm decrypting an api log now so I'll try to figure out what's going on. I am not a fan of the TWS log encryption preferring straightforward text so I can ssh into the tws machine and work that way.

thanks again for EVERYTHING you do.

You are welcome.

Generally, IBKR has relaxed historical data pacing rules over the years, but in the spirit of "did something change", here a couple of observations. This is "thin ice territory" in the sense that I'll describe some observations but I have no more insight into or documentation for what is really going on.

During the transition from V9 to V10 TWS/IBGW we detected that Historical Market Data Servers (HMDS farms) had become more prominent and that they must have taken on functions other than simply providing historical market data. They may had been involved before, but as part of our V10 testing and qualification they stood out. Also, TWS/IBGW suddenly had the HMDS logging feature that gives great insight into the kinds of XML database queries TWS/IBGW send to the various HMDS farms.

HMDS and market data farms in each region seem to be responsible for contract related details, real time data, and historical data for the instruments traded in that region. Below a log overview for an attempt to get real-time market data for a Frankfurt FWB traded instrument that we do not have a market data subscription for. You see that our TWS was connected to usfuture.nj, usfuture, usfarm, ushmds, and secdefnj at the moment our client started and that euhmds and eufarm were temporarily connected once we made the call. They were automatically disconnected after 3 minutes of inactivity.

This may be relevant to your problem since most (all ?) HMDS queries trugered by your clients' requests (not just calls) seem to count somehow towards historical market data pacing. At least for TWS around V10.10? to V10.12 when we did our tests.

The funny thing was that we got unexplained TWSAPI errors (similar to your 366) that did not say "pacing violation" in the error message but the corresponding TWS HMDS internal logs clearly did. We restructured the tool that caused the errors, removed calls to that we really did not need (they caused HMDS queries) and the errors when away.

So I guess this was the long way of saying that the combined requests your client sends to TWS/IBGW may trigger a silent pacing violation that leads to error 366 "ticker id was cancelled" situations. After all, you said you make requests for 100 instruments.

Just a thought,

开云体育

开云体育

WOW that's my bad sorry. the last date the program worked was Thu 21 Sep 2023 02:11:09 AM then the next day Fri 22 Sep 2023 02:11:09 AM CDT? it failed to get any data.? Excellent work Holmes :-)?