¿ªÔÆÌåÓý

ctrl + shift + ? for shortcuts
© 2025 ¿ªÔÆÌåÓý

No real-time data after IB nightly server reset


 

I have been using IB Gateway (v10.19.2o) inside a docker container. After the nightly reset of IB servers at around 0430 UTC, I get the following message on some days:
?
Connectivity between IB and Trader Workstation has been restored - data maintained. The following farms are connected: eufarm; euhmds; secdefeu. The following farms are not connected: ushmds.
Error Code: 1102
?
After this I don't receive any real-time data even if I resubscribe to it using reqRealTimeBars. I have tried resetting the data connection as well as account & the reset seems to be done fine but I still can't get the data. The only way to get the data is to re-login which requires 2FA.
?
Curiously, on some days it just works fine after the nightly reset. In such cases, I get the following message (same error code as before but all farms are connected including ushmds):
?
Connectivity between IBKR and Trader Workstation has been restored - data maintained. All data farms are connected: eufarm; euhmds; ushmds; secdefeu.
Error Code: 1102
?
I was wondering if anyone else has also encountered this problem & found a solution for it that doesn't require re-login.
?
Thanks,
Farhan


 

¿ªÔÆÌåÓý

I¡¯ve searched back through my Data Collector log files for this 1102 message with one or more farms not connected, and since December 2020 it has occurred 13 times (and always usfarm).

?

I didn¡¯t check them all in detail, but the handful I looked at had an ¡°Error code=2104;Error msg=Market data farm connection is OK:usfarm;¡± shortly afterwards. The gap varied from a few seconds to a few minutes, but it always occurred during the reset period. After that, all my realtime data subscriptions served by the usfarm continued without incident, and without my code having to do anything.

?

I don¡¯t use reqRealtimeBars, just reqMktData, but I¡¯m surprised that your realtime bars aren¡¯t continuing in the same way.

?

Perhaps you¡¯re not waiting long enough? It¡¯s certainly my understanding that realtime bars subscriptions do survive the reset period: ´³¨¹°ù²µ±ð²Ô has mentioned this explicitly a couple of times recently.

?

You say that resubscribing to reqRealtimBars doesn¡¯t work: what does happen when you do this? Is there an error message? Are you using the same request id as before? Do you cancel the existing subscription before trying again?

?

?


 

Thanks, Richard for your reply. After I receive "farms not connected ..." 1102? message, I don't get a 2104 message. After the server reset once the connection is re-established, I cancel all my data subscriptions & disconnect. I then connect again (with the same client ID) & subscribe again using reqRealtimeBars with bar length 5 seconds. I wait for 50 seconds to receive market data & if I don't then I logout of IB Gateway & login again (triggering 2FA) & go through the same routine. This always works.
?
The strange thing is that I don't have this problem every single day. On the days when all farms are connected after reset, I do get the real-time data.


 

Don't do that. There is no need for you to cancel subscriptions or to disconnect/reconnect your clients during the nightly reset window and what your client does may be more harmful than helpful. Especially if you have hard-coded wait/sleep/timeout times of 50 seconds.

TWS/IBGW are remarkably reliable through the nightly reset process and all real-time subscriptions survive the reset (and in fact provide data throughout the entire reset process). That includes subscriptions for five second real-time bars but not subscriptions the historical data with keepUpToDate = true.

First, each data farm gets disconnected/reconnected individually and you get a series of "connection broken" and "connection OK" indications. In my case, each of these resets takes only a fraction of a second and all real-time data keeps arriving (MktData, Level 2, TickBytick, Realtime Bars)

Eventually, TWS/IBGW reset their main connection with IBKR and re-authenticate. That is when you may see a series of error code 1100 messages "Connectivity between IB and Trader Workstation has been lost.", count-down pop-up windows, and eventually the indication 1102 "Connectivity between IB and Trader Workstation has been restored - data maintained.". But these have nothing to do with real-time data feeds for existing subscriptions. It is just that you can't make new subscriptions, place orders, or make any API request between 1100 and 1102 that require communication between TWS/IBGW and the IBKR servers.

I often get many 1100 over a period of several minutes before the final 1102, but real-time data continues since the market data farm connections are and all stay OK (where the real-time data comes from).

And the fact that you sometimes get "The following farms are not connected: ushmds" is probably normal and not an error. Since you use IGBW, the HMDS server may simply not be needed at that time. HMD traditionally stood for Historical Market Data but those servers now seem to provide a more general purpose query infrastructure but may not be needed until you request historical data or ContractDetails, establish new real-time subscriptions, or place orders. They would automatically be connected at that point.

And coming back to the 5 second real-time bars for a moment. You did not tell us what kind of instruments and exchanges you are subscribing to but it could very well be, that there are no trades to be reported for 10 or more periods of 5 seconds in the middle of the night. Very active instruments and CME/CBOT traded instruments do get updates even for periods with no activity (volume and count is 0), but I have seen some instruments that only get bars for periods with trades or bid/ask changes.

Just remove the cancel/disconnect/connect logic and your client should get through the reset windows without any problems and with real-time data arriving. It has done so for me for years.

´³¨¹°ù²µ±ð²Ô

?

?
?
?
?
?
?
On Fri, Aug 2, 2024 at 02:19 PM, <farhanferoz@...> wrote:

Thanks, Richard for your reply. After I receive "farms not connected ..." 1102? message, I don't get a 2104 message. After the server reset once the connection is re-established, I cancel all my data subscriptions & disconnect. I then connect again (with the same client ID) & subscribe again using reqRealtimeBars with bar length 5 seconds. I wait for 50 seconds to receive market data & if I don't then I logout of IB Gateway & login again (triggering 2FA) & go through the same routine. This always works.
?
The strange thing is that I don't have this problem every single day. On the days when all farms are connected after reset, I do get the real-time data.


 

Thanks, ´³¨¹°ù²µ±ð²Ô for your detailed reply. I am working with fairly liquid CME & NYMEX instruments (GC, CL etc) for which I am sure there will be trades within 50 seconds I am waiting for. Also, when I log in to IBGW again, I get the trades within 5 seconds of requesting them.
?
I am disconnecting after IB reset as I have live orders in place before the reset & I am unsure if the callbacks for different order events work after the IB reset. I will try your advice next week of not disconnecting & report back.
?
Farhan


 

No worries about the open orders. The reset only relates to the connections with IBKR, TWS maintains state, remembers open orders that were placed before connection is lost, and order callbacks continue when the connection is restored.

In fact, with more recent TWS/IBGW (such as the latest 10.30.1j) it seems to be possible to place orders when the connection is lost. A new API configuration parameter was introduced (and seems to be selected by default):
  • Maintain and resubmit orders when connection is restored.
  • The tool-tip says: When this setting is checked, all orders received while connectivity is lost will be saved in memory and automatically resubmitted when connectivity is restored. You may use this option to prevent order data loss.
´³¨¹°ù²µ±ð²Ô
?
On Sat, Aug 3, 2024 at 08:38 AM, <farhanferoz@...> wrote:

Thanks, ´³¨¹°ù²µ±ð²Ô for your detailed reply. I am working with fairly liquid CME & NYMEX instruments (GC, CL etc) for which I am sure there will be trades within 50 seconds I am waiting for. Also, when I log in to IBGW again, I get the trades within 5 seconds of requesting them.
?
I am disconnecting after IB reset as I have live orders in place before the reset & I am unsure if the callbacks for different order events work after the IB reset. I will try your advice next week of not disconnecting & report back.
?
Farhan


 

Hi ´³¨¹°ù²µ±ð²Ô?,
?
I tried your suggestion of not disconnecting after reestablishing the connection with IB after the nightly reset this morning but had the same problem i.e. no real-time data. My logs are as follows (timestamps are in UTC):
?
08/08/2024 04:29:27.192 ERROR: request id = -1, error code = 1100, error string = Connectivity between IB and Trader Workstation has been lost
08/08/2024 04:29:27.193 IB connection lost.
08/08/2024 04:29:27.194 Found running IBC instance: psutil.Process(pid=42, name='java', status='sleeping', started='21:08:51')
08/08/2024 04:29:35.998 ERROR: request id = -1, error code = 1100, error string = Connectivity between IB and Trader Workstation has been lost.
08/08/2024 04:29:35.999 IB connection lost.
08/08/2024 04:29:36.000 Found running IBC instance: psutil.Process(pid=42, name='java', status='sleeping', started='21:08:51')
08/08/2024 04:29:39.773 ERROR: request id = -1, error code = 1100, error string = Connectivity between IB and Trader Workstation has been lost
08/08/2024 04:29:39.773 IB connection lost.
08/08/2024 04:29:39.775 Found running IBC instance: psutil.Process(pid=42, name='java', status='sleeping', started='21:08:51')
08/08/2024 04:29:42.791 ERROR: request id = -1, error code = 1100, error string = Connectivity between IB and Trader Workstation has been lost.
08/08/2024 04:29:42.791 IB connection lost.
08/08/2024 04:29:42.793 Found running IBC instance: psutil.Process(pid=42, name='java', status='sleeping', started='21:08:51')
08/08/2024 04:29:49.989 ERROR: request id = -1, error code = 1100, error string = Connectivity between IB and Trader Workstation has been lost
08/08/2024 04:29:49.989 IB connection lost.
08/08/2024 04:29:49.991 Found running IBC instance: psutil.Process(pid=42, name='java', status='sleeping', started='21:08:51')
2024-08-08 04:29:58:415 IBC: detected frame entitled: Starting application...; event=Lost focus
2024-08-08 04:29:58:415 IBC: detected frame entitled: Starting application...; event=Deactivated
2024-08-08 04:29:58:415 IBC: detected frame entitled: Starting application...; event=Closed
08/08/2024 04:29:58.532 ERROR: request id = -1, error code = 2105, error string = HMDS data farm connection is broken:ushmds
08/08/2024 04:29:58.533 Data connection broken within IB reset window. Will not attempt connection reset.
08/08/2024 04:29:58.537 Found running IBC instance: psutil.Process(pid=42, name='java', status='sleeping', started='21:08:51')
08/08/2024 04:29:58.549 ERROR: request id = -1, error code = 2106, error string = HMDS data farm connection is OK:euhmds
08/08/2024 04:29:58.550 Found running IBC instance: psutil.Process(pid=42, name='java', status='sleeping', started='21:08:51')
08/08/2024 04:30:28.470 ERROR: request id = -1, error code = 1102, error string = Connectivity between IB and Trader Workstation has been restored - data maintained. The following farms are connected: eufarm; euhmds; secdefeu. The following farms are not connected: ushmds
08/08/2024 04:30:28.470 IB connection reestablished.
?
After the last message, I waited a minute to receive the real-time 5 seconds bars for liquid CME/NYMEX instruments e.g. CL, GC but received nothing. I then restarted IB Gateway & logged in again (after 2FA) & everything was fine.
?
I forgot to mention last time that I am using IB Gateway through IBC () inside Docker. Could that have anything to do with the problems I am experiencing?
?
Thanks,
Farhan


 

I must have read your initial post to quickly and had not realized the fact that you have a docker containerized setup. While I understand how both work, I use neither Docker nor IBC so that I cannot give you any advice based on practical experience. But between IBC and Docker, I'd first look into the Docker network setup.

Docker containers materially change the data flow between applications inside the container, the host, and the external world with many possible configuration options and scenarios. And, depending on your docker network configuration, container isolation may interfere with data routing from IBKR to IBGW inside the container after the nightly reset when the port numbers used by IBGW have changed, but the ports number used by IBKR have not. The IBGW restart would clear that up.

If you can, make a test where IBGW runs without the docker container directly in the host. That works just fineand rock solid for me.

Extensive discussions about docker and IBC are out of scope for our group, though. For IBC you can start a discussion over at the /g/ibcalpha group.

´³¨¹°ù²µ±ð²Ô

?

?
?
?
?
On Thu, Aug 8, 2024 at 02:26 AM, <farhanferoz@...> wrote:

Hi ´³¨¹°ù²µ±ð²Ô?,
?
I tried your suggestion of not disconnecting after reestablishing the connection with IB after the nightly reset this morning but had the same problem i.e. no real-time data. My logs are as follows (timestamps are in UTC):
?
.. log deleted ...
After the last message, I waited a minute to receive the real-time 5 seconds bars for liquid CME/NYMEX instruments e.g. CL, GC but received nothing. I then restarted IB Gateway & logged in again (after 2FA) & everything was fine.
?
I forgot to mention last time that I am using IB Gateway through IBC () inside Docker. Could that have anything to do with the problems I am experiencing?
?
Thanks,
Farhan


 

Hi ´³¨¹°ù²µ±ð²Ô,
?
I managed to run my setup on a Google cloud Linux machine outside Docker but got the same problem as before i.e. no real-time market after IB reset at around 0430 UTC. I am no longer disconnecting after reestablishing the connection with IB after the nightly reset, as you suggested. Can you think of anything else I should try to fix this issue?
?
Thanks,
Farhan


 

Sorry to hear it still does not work for you, Farhan. But unless everyone is still on vacation, it looks like this issue may be unique to your setup.

Just to make sure. When you say that real time subscriptions do not send data after the maintenance window reconnects, that relates to callbacks from one or more of these: reqMktData, reqRealTimeBards, ?reqTickByTickData,, reqMarketDepth. Right? We are not talking about reqHistoricalData with keepUpToDate = true. I have seen those stop after the maintenance window reconnects.

And your code does not (inadvertently) call the corresponding cancel requests upon the various errors/warning/infos the client receives during the reset period?

How about other TWS API callbacks? Do you receive account value updates, position updates, openOrder, execDetails, ... or any of the non-data related calls?

I guess my next step would be to determine whether this is related to the client or to IBGW. Try to get a "second opinion" from one of the clients that ship with the API or All of those allow you to quickly subscribe to a market data feed. If they receive market data after the maintenance window, it's the client you;d have to look at. If not, it is something with IBGW that probably IBKR needs to look at.

´³¨¹°ù²µ±ð²Ô

?

On Wed, Aug 28, 2024 at 05:10 AM, <farhanferoz@...> wrote:

Hi ´³¨¹°ù²µ±ð²Ô,
?
I managed to run my setup on a Google cloud Linux machine outside Docker but got the same problem as before i.e. no real-time market after IB reset at around 0430 UTC. I am no longer disconnecting after reestablishing the connection with IB after the nightly reset, as you suggested. Can you think of anything else I should try to fix this issue?
?
Thanks,
Farhan


 

Hi?´³¨¹°ù²µ±ð²Ô?,
?
I encountered the same issue and still no market data after the connection was restored.
?
Followings are my logs
?
2024-11-27 12:15:54.656112 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:15:55.257818 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:16:06.239496 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:16:12.080815 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:16:16.738522 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:16:21.150339 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:16:31.521287 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:16:42.307221 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:16:56.400287 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:17:12.772813 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:17:38.175326 ERROR 2105 HMDS data farm connection is broken:ushmds
2024-11-27 12:17:38.312416 ERROR 2106 HMDS data farm connection is OK:apachmds
2024-11-27 12:18:08.178669 ERROR 1102 Connectivity between IBKR and Trader Workstation has been restored - data maintained. The following farms are connected: hfarm; apachmds; secdefhk. The following farms are not connected: ushmds.
2024-11-27 15:30:47.702324 ERROR 2157 Sec-def data farm connection is broken:secdefhk
2024-11-27 15:30:48.190508 ERROR 2158 Sec-def data farm connection is OK:secdefhk
2024-11-27 15:45:17.419716 ERROR 2157 Sec-def data farm connection is broken:secdefhk
2024-11-27 15:45:21.453034 ERROR 2158 Sec-def data farm connection is OK:secdefhk
2024-11-27 19:23:05.387361 ERROR 2103 Market data farm connection is broken:hfarm
2024-11-27 19:23:06.315837 ERROR 2104 Market data farm connection is OK:hfarm
?
I was requesting reqRealTimeBars and no data received after Nov 27 12:17, I did not use docker or IBC and no disconnecting / cancel subscription logic.
?
Any thoughts?


 

Nothing jumps at me, except that ushmds disconnected at 12:17:38 and did not reconnect by the time TWS/IBGW had reestablished full connectivity with IBKR at 12:18:08. But that may be fine in case it was not needed any longer.

What kind of symbols were your reqRealTimeBars for? STK, FUT, OPT, ..., US, HK, Asia, ...?

What time zone are these timestamps in?

Have you taken a look at the TWS/IBKR logs? Just to see whether there are some errors related to market data.

´³¨¹°ù²µ±ð²Ô

?

?
?
On Thu, Nov 28, 2024 at 12:52 AM, <xiaobo.li@...> wrote:

Hi?´³¨¹°ù²µ±ð²Ô?,
?
I encountered the same issue and still no market data after the connection was restored.
?
Followings are my logs
?
2024-11-27 12:15:54.656112 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:15:55.257818 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:16:06.239496 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:16:12.080815 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:16:16.738522 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:16:21.150339 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:16:31.521287 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:16:42.307221 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:16:56.400287 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:17:12.772813 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:17:38.175326 ERROR 2105 HMDS data farm connection is broken:ushmds
2024-11-27 12:17:38.312416 ERROR 2106 HMDS data farm connection is OK:apachmds
2024-11-27 12:18:08.178669 ERROR 1102 Connectivity between IBKR and Trader Workstation has been restored - data maintained. The following farms are connected: hfarm; apachmds; secdefhk. The following farms are not connected: ushmds.
2024-11-27 15:30:47.702324 ERROR 2157 Sec-def data farm connection is broken:secdefhk
2024-11-27 15:30:48.190508 ERROR 2158 Sec-def data farm connection is OK:secdefhk
2024-11-27 15:45:17.419716 ERROR 2157 Sec-def data farm connection is broken:secdefhk
2024-11-27 15:45:21.453034 ERROR 2158 Sec-def data farm connection is OK:secdefhk
2024-11-27 19:23:05.387361 ERROR 2103 Market data farm connection is broken:hfarm
2024-11-27 19:23:06.315837 ERROR 2104 Market data farm connection is OK:hfarm
?
I was requesting reqRealTimeBars and no data received after Nov 27 12:17, I did not use docker or IBC and no disconnecting / cancel subscription logic.
?
Any thoughts?


 

Hi?´³¨¹°ù²µ±ð²Ô?,
?
The time in my machine e.g.12:17:38 is in UTC timezone. I was requesting realtime bars and historical data for US stocks only, do you think the failure was due to ushmds not reconnecting successfully at that time???
?
Thanks!
?


 

Hi ´³¨¹°ù²µ±ð²Ô,
?
I think the ushmds might be the cause,?
?
BTW Here's my gateway logs for ushmds part around that time
?
2024-11-27 12:15:57.522 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | cf312@6322834f | Done | cf312;;BQ@SMART Trades;;1;;true;;0;;I | 0
2024-11-27 12:15:57.523 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | adjustments311@2e87e5ab | Done | adjustments311;;BQ@SMART Trades;;1;;true;;0;;I | 0
2024-11-27 12:15:57.523 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | cf312@6322834f | Clear | 0
2024-11-27 12:15:57.523 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | indicativeData310@1c25bb71 | Done | indicativeData310;;BQ@SMART Trades;;1;;true;;0;;I | 0
2024-11-27 12:15:57.523 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | adjustments311@2e87e5ab | Clear | 0
2024-11-27 12:15:57.523 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | cf312@6322834f | Clear | 0
2024-11-27 12:15:57.523 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | agg309@1719520f | Done | agg309;;BQ@SMART Trades;;1;;true;;0;;I | 0
2024-11-27 12:15:57.523 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | indicativeData310@1c25bb71 | Clear | 0
2024-11-27 12:15:57.523 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | adjustments311@2e87e5ab | Clear | 0
2024-11-27 12:15:57.523 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | cf312@6322834f | Clear | 0
2024-11-27 12:15:57.526 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - HMDS query: 78;;BQ@SMART Trades;;1;;true;;0;;I has returned for series: BQ@SMART Trades
2024-11-27 12:15:57.527 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | qm@42f1f33f | all EOQ
2024-11-27 12:15:57.582 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - [50:176:176:1:0:17:3:INFO] Sending historical data for ticker = 78. 182489 items.
2024-11-27 12:15:59.217 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - Start writing to output data. Left to write: 12019494
2024-11-27 12:15:59.219 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - Writing to output data. Written: 0 Left: 12019494
2024-11-27 12:15:59.719 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - Finished writing to output data.
2024-11-27 12:16:01.844 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - [50:176:176:1:0:17:3:INFO] Historical data is sent for ticker = 78. 182489 items.
2024-11-27 12:17:38.170 [UO] INFO ?[JTS-AsyncNonLocked-33] - disconnectAndReconnect due to routing change for ushmds state:NATIVE default:ccp dataType:NONE points:[useccphost] nativeViaCcp:null
2024-11-27 12:17:38.170 [UO] INFO ?[JTS-AsyncNonLocked-33] - SocketTracker[name=name=ushmds(cdc1.ibllc.com:4000),usesCcpConMan=false,lastReadTimestamp=1732709856527,lastReadDateTime=2024-11-27T12:17:36.527,lastFixSendTime=].
2024-11-27 12:17:38.170 [UO] INFO ?[JTS-ushmdsServicePingS14-749] - Terminating ping thread
2024-11-27 12:17:38.170 [UO] INFO ?[JTS-AsyncNonLocked-33] - AuthTimeoutMonitor-ushmds: deactivate
2024-11-27 12:17:38.170 [UO] INFO ?[JTS-ushmdsListenerS14-744] - Socket or stream for connection cdc1.ibllc.com:4000 was closed by another thread.
2024-11-27 12:17:38.170 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - Dispatcher thread terminating [sessionID=14,interrupted=true]...
2024-11-27 12:17:38.170 [UO] INFO ?[JTS-ushmdsListenerS14-744] - Listener thread terminating [sessionID=14] [seen=15024770,totalShifted=15024770,moreAvailable=0]
2024-11-27 12:17:38.172 [UO] INFO ?[JTS-DisconnectedS14-941] - Farm ushmds/NATIVE: Lost active connection with disconnect status DISCONNECT_ON_BROKEN_SOCKET
2024-11-27 12:17:38.173 [UO] INFO ?[JTS-DisconnectedS14-941] - Farm ushmds/NATIVE: Resetting
2024-11-27 12:17:38.174 [UO] INFO ?[JTS-AsyncNonLocked-33] - Keep-alive scheduled for:ushmds
2024-11-27 12:17:38.174 [UO] INFO ?[JTS-AsyncNonLocked-33] - [50:176:176:1:0:4:2:DET] [4;2;-1;2105;HMDS data farm connection is broken:ushmds;]
2024-11-27 12:17:38.175 [UO] INFO ?[JTS-AsyncNonLocked-33] - [112501:187:187:1:0:4:2:DET] [4;2;-1;2105;HMDS data farm connection is broken:ushmds;]
2024-11-27 12:17:38.175 [UO] INFO ?[JTS-AsyncNonLocked-33] - cdebug: ROUTING | connTracker:ushmds@4785586b | Disconnected | 1732709857176 | true
?
°Õ³ó²¹²Ô°ì²õ£¬
xb
?


 

I have posted about more or less the same experience not long ago: /g/twsapi/message/53498
I've come to a solution that I will perform a fresh reconnection after nightly reset as well as I usually do after daily TWS auto restart.
Seems to be working better for me, but I want to monitor for a few more days to be sure if the loss of data subscriptions still happens.
?


 

From the logs is sounds like the intention was to "disconnectAndReconnect due to routing change for ushmds" but the reconnect part never materialized. You'd probably have to reach out to IBKR to understand why the reconnect never happened and what can be done about it.

Before disconnect, your TWS/IBGW was connected to the US Central/Chicago HMDS located at cdc1.ibllc.com. Do you have a decent network connection between your system and cdc1? As in decent packet loss?

´³¨¹°ù²µ±ð²Ô

?

?

On Thu, Nov 28, 2024 at 03:26 AM, xb wrote:


2024-11-27 12:17:38.170 [UO] INFO ?[JTS-AsyncNonLocked-33] - disconnectAndReconnect due to routing change for ushmds state:NATIVE default:ccp dataType:NONE points:[useccphost] nativeViaCcp:null
2024-11-27 12:17:38.170 [UO] INFO ?[JTS-AsyncNonLocked-33] - SocketTracker[name=name=ushmds(cdc1.ibllc.com:4000),usesCcpConMan=false,lastReadTimestamp=1732709856527,lastReadDateTime=2024-11-27T12:17:36.527,lastFixSendTime=].
?


 

I have an open ticket with IBKR about a very similar (even identical?) topic since a couple of weeks. I have experienced the same problems where on some days Gateway would not recover after a daily reset. In all cases was a disconnect from ushmds involved. Initially I did not respond to the opening post of this thread because I am not using Docker.
?
In recent weeks the problem has disappeared. I used to download once per day quite some historical data for a list of US stock tickers (5 years daily data for 230 tickers). Every now and then (once, twice per week?) ushmds would disconnect after I had submitted the data requests, and before I received the results. Immediately after this disconnect Gateway would disconnect all clients and not allow any new clients to connect! Forcing me to kill Gateway and restart it. Four weeks ago I changed my software: now I am downloading a lot less data every day, and merge that with data I already have. Since then has the problem disappeared. What I suspect, but don't have confirmation of, is that somehow I exceeded the "soft limits" IBKR has in place on historical data downloads. In the open ticket I'm having with IBKR I am hoping to get a confirmation that this was indeed the case. I must emphasize though that I never received a warning that I was downloading too much data, or that I had crossed the soft limit.