Just started getting these "broken pipe" messages in code that's been working for months. Anyone know how to diagnose TWS socket issues like broken pipes. I don't even know which log to look at!!!

this code has been working for about 3 months without fault so this behavior started 12/16/22. NOW it just fails on the first call to reqMktData

I checked the HELP for the TWS desktop I am using as a gateway and the build is 10.20.1f 12/13/22 so it's NOT pacing problems. I know the connection is working because I have TWS desktop running as my gateway api and I can see the DATA indicator is consistently GREEN. I monitor the ISP feed ( 1GB/s) using a raspberry pi and that seems ok. I'm working on how to look at the TWS logs to see if they tell me anything.

┌ Warning: connection terminated
└ @ Jib.Reader ~/.julia/packages/Jib/JOmEj/src/reader.jl:70
[ Info: reader exiting
ERROR: LoadError: IOError: write: broken pipe (EPIPE)
Stacktrace:
  [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64)
    @ Base ./stream.jl:1064
  [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64)
    @ Base ./stream.jl:1118
  [3] unsafe_write
    @ ./io.jl:683 [inlined]
  [4] write
    @ ./io.jl:706 [inlined]
  [5] write_one(socket::Sockets.TCPSocket, buf::IOBuffer)
    @ Jib.Client ~/.julia/packages/Jib/JOmEj/src/client.jl:28
  [6] sendmsg

what is REALLY weird, other than the fact this JUST broke a few days ago, is that it ONLY happens with reqMktData whereas reqHistoricalData works fine.

for (idx, s) in enumerate(eachrow(symbol_list))

    contract = Contract(symbol=s.Sym, secType="STK" , exchange=s.exchange , currency="USD")
    reqMktData(ib, idx, contract,"106", false) # https://interactivebrokers.github.io/tws-api/tick_types.html
end

BUT this one works properly. They both share the same ZMQ configuration code and worked fine up until 12/16/22.

for (idx, s) in enumerate(eachrow(symbol_list))
  sleep(5)
    contract = Contract(symbol=s.Sym, secType="STK" , exchange=s.exchange , currency="USD")
  try
       reqHistoricalData(ib, idx ,contract,"","1 Y","1 day","TRADES",true,1,false )
  
  catch e
           println("something went wrong with : " , s.Sym)
           stk_send_msg = "STK" *  "~" * s.Sy * "~" * "CLOSE" * "~" * string( 999 ) #show there was an error
           ZMQ.send( stk_socket, stk_send_msg )
           continue
  end
end

Uhhh.... this sounds like a problem w/ Jib. Have you reached out to ?

this isn't a julia problem it's a TWS issue and yes I they have helped me A LOT to narrow the cause down to the IO subsystem.

I'm asking if anyone else has noticed if TWS tcpip IO has been playing up recently. I am NOT asking for code support. Right now I am "TRYING" to use the TWS logs for the first time. What would be GREAT is that if anyone knows how to look at them remotely. I use Linux and not having any luck with the diagnostics on the TWS desktop. I try to export them but it just hangs.

On Wed, Dec 21, 2022 at 06:13 PM, comicpilsen wrote:

this isn't a julia problem it's a TWS issue

You sound pretty sure of this so I'm guessing you must've confirmed the same thing is happening when you create a simple test case using one of the officially supported API/SDKs too... yes?

Anyway, wish I could help more but I'm still using the 9.x branch due to bug introduced in 10.x (specific to my own usage). Therefore... haven't seen the problem you're experiencing with Jib or exporting diags.

Good luck.

开云体育

As I said, I am asking for the forum's guidance in two regards:

1 has anyone else experienced IO issues since 12/16/22 ?

2 has the forum got any guidance as to how to resolve TCPIp type issues with TWS.?

?ANY help would be great setting up the debugging toolkit IRRESPECTIVE of language or api which are not relevant in this thread.

Well, if you're seeing the same problem with a basic test case across different language APIs, I'd consider reaching for the usual tools like wireshark, strace to troubleshoot further. And, if the problem isn't with just your local configuration... I'm sure it will be confirmed by others [eventually] and probably fixed [at some point] by IB.

Otherwise, I personally, would consider some kind of creative workaround.

Again, good luck! Please keep us posted.

thanks for pointing out your issue. I'm wondering if they aren't related in some way. In my case the code was working fine up until 12/16/22. Calls to historical work as expected BUT calls to reqMktData fail instantly. I'll go back and review my approach as you suggested but I am loathe to do so as it seems the most rational one. Submit all the symbols with an ID and unique identifier reqMktData to an asynch process and then wrapper the callbacks and handle them as they come in.

I'm considering wireshark as a last resort as I am confident that the protocol stack is working. This smacks to me of application level failure.

thanks again for you heads up on the bug.

• Go into the TWS configuration menu and disable "Create API message log file" in the API settings
• Locate the folder that is related to the account your are currently logged into. That folder has a very long name of random characters and is located within the TWS/IBGW installation folder, such as the default ~/Jts
• Look for files with names such as "api.1671575462.20221221.142852.ibgzenc" and move them out of that folder
• Enable "Create API message log file" in the API settings"
• Run your client code that exposes the issue
• At this point you could disable API logging again to keep the log file small.

开云体育

Hi 闯ü谤驳别苍

GREAT to hear from you and happy hols to the whole Reinold clan :-)

YEP another steaming pile of opportunity from TWS but I'm sure there's a reason for it.

gotcha on the no client connection issues, that's REALLY helpful as your team are the gold standard for this stuff in my opinion. This means the problem is my end and so back to the drawing board.

In my case I have separate julia scripts for historical data and a single script for reqMktData that sends about 60 symbols to an asynch process which wrappers the callbacks.

the request payload has been working consistantly for 4 months and is pretty standard so I am confident that is not the issue given it's basic structure is the same as historical and market. I am NOT ruling it out though.

I am working to put in REAL error handling in my app which I didn't do given time constraints ( I only can work on this 1 hour a week) and BOY AM I REGRETTING not adding it OR logging. Luckily julia makes this REALLY easy so I'm laying into that in my new code.

OUTSTANDING help on the log guidance and I'll do that next. I SHOULD HAVE THOUGHT OF THAT MYSELF sorry.

I agree with you and buddy that wireshark would help but I wanted to isolate what TWS was saying.? Buddy gave me a great heads up with the bug reference which "might" be related.

thanks again for an EXCELLENT year's work on this forum, you did an excellent job.

comicpilsen

On Wed, Dec 21, 2022 at 09:06 PM, dent wrote:

great heads up with the bug reference which "might" be related.

I was just complaining. But, given how you described your script and the problem, I suppose it actually could be related.

When I was first investigating that issue, the initial symptom was indeed a disconnect for no apparent reason.

开云体育

we got the microwave because someone complained about a chocolate bar melting in their pocket.? What is REALLY strange is that if I run this script after market close it works as expected. I get all the LAST prices for my list of 60 symbols and then it just sits on an awaitio as the call to reqMktData callback is waiting for a change which won't come.? This has made it stranger for me. I was HOPING that this was something to do with IB changing the manner in which unique request identifiers are handled.


thanks again

On Wed, Dec 21, 2022 at 10:01 PM, dent wrote:

REALLY strange is that if I run this script after market close it works as expected

Yeah, the bug I documented is only guaranteed to happen during regular trading hours too. It can be reproduced somewhat reliably with a trial account, but that code path is certainly different and the results become intermittent. Are you able to test your software against a 9.x version of TWS/IBGW?

Part of me really wants to pour over the obfuscated java byte codes. Unfortunately, I'm naively holding out hope that IB fixes it eventually.

Did you upgrade from 9.x to 10.x on 12/16/22? That would be another telltale sign. IDK exactly how Jib works but I assume you can provide the client ID? If so, as a workaround or for confirmation, try fixing the client ID to be the same number all the time... then 10.x may behave as expected; if it's a related issue.

In my case IB confirmed the bug and admitted 10.x was to blame but they wouldn't commit to fixing it :-( Apparently I'm not important enough. Honestly it doesn't matter to me yet... my system code is frozen for now.

Plus, I have a workaround ready if I'm forced off 9.x. I really don't like to put ugly workaround hacks into my stuff to compensate for a multi-billion dollar company's inadequacies though. Still, I'm not taking this to my grave. As they say... perfect is the enemy of the good. Not sure I agree but IRL it seems to hold water occasionally.

Keep us posted.

I'm concerned that no one else is seeing this. Points to a failure in logic my end. I would have thought posting a bunch of reqMktData requests each with it's own unique identifier THEN handling the asynch completion inside a callback using that identifier was a safe approach. Seems that is WAS but now no longer. Answering your questions

1 Nope, am on the TWS update cycle. Right from day one I have used the TWS desktop running on a I7 laptop with 8gb ram Linux mint. I wanted to do this so I could monitor the markets and make sure I could see that the DATA was flowing. I shut it completely down on Fridays and start the whole laptop up on Sundays which updates to the most recent TWS. In my case 10.20.1f 12/13/22 so I was using it FINE on 12/14 12/15 BUT something happened on the night of 12/15 because the log ( I log the HISTORICAL population of the system NOT the realtime) shows me that. The logs ALSO show me that the historical data population scripts work perfectly and they use the same loop but call Jib.reqHistoricalData. When the script for realtime kicks off and calls Jib.reqMktData during market open the IO PIPE just dies. That's the weird part, MARKET OPEN FIRST CALL kills IO pipe, MARKET CLOSE no problem.

2 Yep Jib is well coded ( not my code) and asynch client id processing is easy to wrapper. I figured that client id would cause issues in the future so I hard coded it into the script (ib = Jib.connect("192.168.1.219", 1234, 31)) so I always know which script is doing what.

SO I am in information gathering mode right now. I am NOT a fan of this encrypted log stuff as I would REALLY REALLY REALLY like to just use bash to help me out in this regard. It would be nice to have the option in TWS config to say NO HELL NO to encrytion but I am sure there is a valid reason for it AND Jurgen has given me a wonderful set of instructions to get what I need. Also it's FORCING me to put error handling and logging into my code.

On Wed, Dec 21, 2022 at 11:29 PM, comicpilsen wrote:

I figured that client id would cause issues in the future so I hard coded it into the script (ib = Jib.connect("192.168.1.219", 1234, 31)) so I always know which script is doing what.

Last year I noticed my issue like night and day from 9.x to 10.x and rolled production back because auto-update caught me off guard. Naturally I've stopped auto-updating. Then, out of pure tenacity and curiosity, spent a lot of time working with IB support to pinpoint the issue exactly (that's the reason for the github gist).

But, I'm a little confused... are you saying that one of your scripts makes multiple reqMktData requests yet a different script (with different client ID) receives the tickPrices? Then you match them up via reqId? I'm not sure that's a good idea generally but if it works could see you hitting the same bug that affects me - since each script would have a different client ID.

The alternative, I think, means that you're using the async keyword (I'm not a Julia expert) to process tickPrices via coroutines (so, in the same thread and process) and then I'd think you may not be hitting the bug that affects me; since it'd all be using the same, fixed clientID in the very same process too.

Anyway, I'm sure you looked a the github gist and can judge how closely our issues are related. That Python code is brain-dead simple and the explanation is as clear as can be.

It'd also be interesting if you found a different problem, even if it was just superficially related.

开云体育

No it's the same script posting the asynch requests and it has a callback wrapper which hands back the streams from each which has a unique identifier so I know which request is which.

I've left the script in the cron stack just to see what happens. If nothing else it's MAKING me put in the error handling I should have in the first place.

On Thu, Dec 22, 2022 at 01:26 AM, dent wrote:

it's the same script

That's what I expected. Seems our problems are not related.