I understand... I'm looking forward to reading the replies also.

开云体育

not really, just asking for high level approach. The denizens of this forum are usually helpful in this regard.?

Lol, I gave you my opinion... use asyncio. But you seem to expect an extremely detailed explanation of how people implement their own proprietary software. I wouldn't hold out much hope for that. You seem to be walking a fine line here; good luck with that.

?冲(ツ)冲/?

开云体育

I was one of the first users of ib_insync and I have the deepest respect for ewald.? My question is specific to the flow of the code NOT the environment.? I would like to know how others handle, at a high level, the processing of multiple symbols.?

On Wed, Dec 21, 2022 at 11:41 PM, comicpilsen wrote:

Could you tell me how you handle the multiple symbol real time data situation as my thinking is clearly wrong. Don't really mind what the language is I can figure it out

Clearly this question can elicit countless different responses. However, to be honest, if I had to rewrite my application from scratch... I probably would just use . As far as reconsidering your approach is concerned, have a look and try running with it.

I consider this significant because I cut my teeth with C/C++ and Python is rarely my language of choice. Plus, I abhor that it doesn't brace delimit code blocks. Nonetheless, the rest of the world seems to disagree with me. So, the Python ecosystem continues to grow and grow; it cannot be denied. And, at the end of the day, if you're stubborn or must, you can always just do a rslt = os.popen(cmd).read().

Finally, what can be done with , in an afternoon or two, is often jaw-dropping. Anyway, hope you find something that works for you.

There is no greater importance in all the world like knowing you are right and that the wave of the world is wrong, yet the wave crashes upon you. --- Norman Mailer

On Thu, Dec 22, 2022 at 01:26 AM, dent wrote:

it's the same script

That's what I expected. Seems our problems are not related.

开云体育

No it's the same script posting the asynch requests and it has a callback wrapper which hands back the streams from each which has a unique identifier so I know which request is which.

I've left the script in the cron stack just to see what happens. If nothing else it's MAKING me put in the error handling I should have in the first place.

On Wed, Dec 21, 2022 at 11:29 PM, comicpilsen wrote:

I figured that client id would cause issues in the future so I hard coded it into the script (ib = Jib.connect("192.168.1.219", 1234, 31)) so I always know which script is doing what.

Last year I noticed my issue like night and day from 9.x to 10.x and rolled production back because auto-update caught me off guard. Naturally I've stopped auto-updating. Then, out of pure tenacity and curiosity, spent a lot of time working with IB support to pinpoint the issue exactly (that's the reason for the github gist).

But, I'm a little confused... are you saying that one of your scripts makes multiple reqMktData requests yet a different script (with different client ID) receives the tickPrices? Then you match them up via reqId? I'm not sure that's a good idea generally but if it works could see you hitting the same bug that affects me - since each script would have a different client ID.

The alternative, I think, means that you're using the async keyword (I'm not a Julia expert) to process tickPrices via coroutines (so, in the same thread and process) and then I'd think you may not be hitting the bug that affects me; since it'd all be using the same, fixed clientID in the very same process too.

Anyway, I'm sure you looked a the github gist and can judge how closely our issues are related. That Python code is brain-dead simple and the explanation is as clear as can be.

It'd also be interesting if you found a different problem, even if it was just superficially related.

Hi there I was populating an array with a list of symbols and related exchanges. I'd loop through the array and for each row call reqMktData with a new request id. This kicked off an asynch callback which I monitored and when a request completed handled it with a function using the request id and the completion type ie LAST. Somehow this got broken on 12/16/22 but ONLY for me as I am sure you good people do this all the time. Could you tell me how you handle the multiple symbol real time data situation as my thinking is clearly wrong. Don't really mind what the language is I can figure it out EXCEPT APL :-)

thanks cp

I'm concerned that no one else is seeing this. Points to a failure in logic my end. I would have thought posting a bunch of reqMktData requests each with it's own unique identifier THEN handling the asynch completion inside a callback using that identifier was a safe approach. Seems that is WAS but now no longer. Answering your questions

1 Nope, am on the TWS update cycle. Right from day one I have used the TWS desktop running on a I7 laptop with 8gb ram Linux mint. I wanted to do this so I could monitor the markets and make sure I could see that the DATA was flowing. I shut it completely down on Fridays and start the whole laptop up on Sundays which updates to the most recent TWS. In my case 10.20.1f 12/13/22 so I was using it FINE on 12/14 12/15 BUT something happened on the night of 12/15 because the log ( I log the HISTORICAL population of the system NOT the realtime) shows me that. The logs ALSO show me that the historical data population scripts work perfectly and they use the same loop but call Jib.reqHistoricalData. When the script for realtime kicks off and calls Jib.reqMktData during market open the IO PIPE just dies. That's the weird part, MARKET OPEN FIRST CALL kills IO pipe, MARKET CLOSE no problem.

2 Yep Jib is well coded ( not my code) and asynch client id processing is easy to wrapper. I figured that client id would cause issues in the future so I hard coded it into the script (ib = Jib.connect("192.168.1.219", 1234, 31)) so I always know which script is doing what.

SO I am in information gathering mode right now. I am NOT a fan of this encrypted log stuff as I would REALLY REALLY REALLY like to just use bash to help me out in this regard. It would be nice to have the option in TWS config to say NO HELL NO to encrytion but I am sure there is a valid reason for it AND Jurgen has given me a wonderful set of instructions to get what I need. Also it's FORCING me to put error handling and logging into my code.

On Wed, Dec 21, 2022 at 10:01 PM, dent wrote:

REALLY strange is that if I run this script after market close it works as expected

Yeah, the bug I documented is only guaranteed to happen during regular trading hours too. It can be reproduced somewhat reliably with a trial account, but that code path is certainly different and the results become intermittent. Are you able to test your software against a 9.x version of TWS/IBGW?

Part of me really wants to pour over the obfuscated java byte codes. Unfortunately, I'm naively holding out hope that IB fixes it eventually.

Did you upgrade from 9.x to 10.x on 12/16/22? That would be another telltale sign. IDK exactly how Jib works but I assume you can provide the client ID? If so, as a workaround or for confirmation, try fixing the client ID to be the same number all the time... then 10.x may behave as expected; if it's a related issue.

In my case IB confirmed the bug and admitted 10.x was to blame but they wouldn't commit to fixing it :-( Apparently I'm not important enough. Honestly it doesn't matter to me yet... my system code is frozen for now.

Plus, I have a workaround ready if I'm forced off 9.x. I really don't like to put ugly workaround hacks into my stuff to compensate for a multi-billion dollar company's inadequacies though. Still, I'm not taking this to my grave. As they say... perfect is the enemy of the good. Not sure I agree but IRL it seems to hold water occasionally.

Keep us posted.

开云体育

we got the microwave because someone complained about a chocolate bar melting in their pocket.? What is REALLY strange is that if I run this script after market close it works as expected. I get all the LAST prices for my list of 60 symbols and then it just sits on an awaitio as the call to reqMktData callback is waiting for a change which won't come.? This has made it stranger for me. I was HOPING that this was something to do with IB changing the manner in which unique request identifiers are handled.


thanks again

On Wed, Dec 21, 2022 at 09:06 PM, dent wrote:

great heads up with the bug reference which "might" be related.

I was just complaining. But, given how you described your script and the problem, I suppose it actually could be related.

When I was first investigating that issue, the initial symptom was indeed a disconnect for no apparent reason.

thanks J replied to other thread so now can say this one COMPLETE. excellent help thanks

开云体育

Hi 闯ü谤驳别苍

GREAT to hear from you and happy hols to the whole Reinold clan :-)

YEP another steaming pile of opportunity from TWS but I'm sure there's a reason for it.

gotcha on the no client connection issues, that's REALLY helpful as your team are the gold standard for this stuff in my opinion. This means the problem is my end and so back to the drawing board.

In my case I have separate julia scripts for historical data and a single script for reqMktData that sends about 60 symbols to an asynch process which wrappers the callbacks.

the request payload has been working consistantly for 4 months and is pretty standard so I am confident that is not the issue given it's basic structure is the same as historical and market. I am NOT ruling it out though.

I am working to put in REAL error handling in my app which I didn't do given time constraints ( I only can work on this 1 hour a week) and BOY AM I REGRETTING not adding it OR logging. Luckily julia makes this REALLY easy so I'm laying into that in my new code.

OUTSTANDING help on the log guidance and I'll do that next. I SHOULD HAVE THOUGHT OF THAT MYSELF sorry.

I agree with you and buddy that wireshark would help but I wanted to isolate what TWS was saying.? Buddy gave me a great heads up with the bug reference which "might" be related.

thanks again for an EXCELLENT year's work on this forum, you did an excellent job.

comicpilsen

• Go into the TWS configuration menu and disable "Create API message log file" in the API settings
• Locate the folder that is related to the account your are currently logged into. That folder has a very long name of random characters and is located within the TWS/IBGW installation folder, such as the default ~/Jts
• Look for files with names such as "api.1671575462.20221221.142852.ibgzenc" and move them out of that folder
• Enable "Create API message log file" in the API settings"
• Run your client code that exposes the issue
• At this point you could disable API logging again to keep the log file small.

thanks for pointing out your issue. I'm wondering if they aren't related in some way. In my case the code was working fine up until 12/16/22. Calls to historical work as expected BUT calls to reqMktData fail instantly. I'll go back and review my approach as you suggested but I am loathe to do so as it seems the most rational one. Submit all the symbols with an ID and unique identifier reqMktData to an asynch process and then wrapper the callbacks and handle them as they come in.

I'm considering wireshark as a last resort as I am confident that the protocol stack is working. This smacks to me of application level failure.

thanks again for you heads up on the bug.

Well, if you're seeing the same problem with a basic test case across different language APIs, I'd consider reaching for the usual tools like wireshark, strace to troubleshoot further. And, if the problem isn't with just your local configuration... I'm sure it will be confirmed by others [eventually] and probably fixed [at some point] by IB.

Otherwise, I personally, would consider some kind of creative workaround.

Again, good luck! Please keep us posted.