¿ªÔÆÌåÓý

ctrl + shift + ? for shortcuts
© 2025 ¿ªÔÆÌåÓý
  1. Twsapi
  2. Messages
Expanded Topics Messages Polls
Date
TWS API Stopped accepting incoming connections sometimes

 

TWS sometimes suddenly cannot connect, and it must be re-verified and logged in again to fix it.
?
Belows are gateway logs happened before my connection failure today
?
2024-11-29 08:37:59.254 [EY] INFO ?[JTS-ushmdsServicePingS9-286] - Farm ushmds/NATIVE: Closing dormant connection type:HISTORICAL_DATA
2024-11-29 08:37:59.254 [EY] INFO ?[JTS-ushmdsServicePingS9-286] - Disconnecting cdc1.ibllc.com:4000 [disconnectDetails=DisconnectDetails[sessionID=9,endPoint=cdc1.ibllc.com:4000,reason=DISCONNECT_ON_INACTIVITY,cause=null,systemMessage=null,keepSocketOpen=false]]...
2024-11-29 08:37:59.254 [EY] INFO ?[JTS-ushmdsServicePingS9-286] - Socket closed.
2024-11-29 08:37:59.254 [EY] INFO ?[JTS-ushmdsServicePingS9-286] - SocketTracker[name=name=ushmds(cdc1.ibllc.com:4000),usesCcpConMan=false,lastReadTimestamp=1732869470298,lastReadDateTime=2024-11-29T08:37:50.298,lastFixSendTime=].
2024-11-29 08:37:59.254 [EY] INFO ?[JTS-ushmdsServicePingS9-286] - Interrupting dispatcher [sessionID=9]...
2024-11-29 08:37:59.254 [EY] INFO ?[JTS-ushmdsServicePingS9-286] - Interrupting listener [sessionID=9,disconnectSocket=true]...
2024-11-29 08:37:59.254 [EY] INFO ?[JTS-ushmdsServicePingS9-286] - AuthTimeoutMonitor-ushmds: deactivate
2024-11-29 08:37:59.254 [EY] INFO ?[JTS-ushmdsDispatcherS9-282S9-283] - Dispatcher thread terminating [sessionID=9,interrupted=true]...
2024-11-29 08:37:59.254 [EY] INFO ?[JTS-ushmdsListenerS9-281] - Socket or stream for connection cdc1.ibllc.com:4000 was closed by another thread.
2024-11-29 08:37:59.254 [EY] INFO ?[JTS-ushmdsServicePingS9-286] - Terminating ping thread
2024-11-29 08:37:59.254 [EY] INFO ?[JTS-ushmdsListenerS9-281] - Listener thread terminating [sessionID=9] [seen=36342294,totalShifted=36342294,moreAvailable=0]
2024-11-29 08:37:59.254 [EY] INFO ?[JTS-DisconnectedS9-551] - Farm ushmds/NATIVE: Lost active connection with disconnect status DISCONNECT_ON_INACTIVITY
2024-11-29 08:37:59.254 [EY] INFO ?[JTS-DisconnectedS9-551] - Farm ushmds/NATIVE: Not resetting
2024-11-29 08:37:59.255 [EY] INFO ?[JTS-AsyncNonLocked-33] - [50:176:176:1:0:4:2:DET] Sending error.
2024-11-29 08:37:59.255 [EY] INFO ?[JTS-AsyncNonLocked-33] - [50:176:176:1:0:4:2:DET] [4;2;-1;2107;HMDS data farm connection is inactive but should be available upon demand.ushmds;]
2024-11-29 08:37:59.255 [EY] INFO ?[JTS-AsyncNonLocked-33] - [50:176:176:1:0:4:2:DET] Error sent.
2024-11-29 08:38:31.130 [EY] INFO ?[JTS-CCPPingS2-28] - Resetting long time mode
2024-11-29 08:39:32.134 [EY] INFO ?[JTS-CCPPingS2-28] - Resetting long time mode
2024-11-29 08:40:11.939 [EY] INFO ?[JTS-DeadlockMonitor-2] - Memory:total=786,432KB free=277,852KB HeapUsage: Max=786,432KB used=506,764KB committed=786,432KB NonHeapUsage: Max=0KB used=172,722KB committed=176,432KB
2024-11-29 08:40:12.940 [EY] INFO ?[JTS-DeadlockMonitor-2] - CPU:cur=12.52% avg=0.38% 30 min avg=4.46% 10 min avg=12.51% 5 min avg=12.50% 1 min avg=12.52%
GC:called=953 times CPU used=0.03%
Finalizer:cur=0.00% avg=0.00% 30 min avg=0.00% 10 min avg=0.00% 5 min avg=0.00% 1 min avg=0.00%
Threads Count:curr live=72 curr daemon=33
2024-11-29 08:40:33.139 [EY] INFO ?[JTS-CCPPingS2-28] - Resetting long time mode
2024-11-29 08:41:34.144 [EY] INFO ?[JTS-CCPPingS2-28] - Resetting long time mode
2024-11-29 08:42:35.148 [EY] INFO ?[JTS-CCPPingS2-28] - Resetting long time mode
2024-11-29 08:43:36.153 [EY] INFO ?[JTS-CCPPingS2-28] - Resetting long time mode
2024-11-29 08:44:37.158 [EY] INFO ?[JTS-CCPPingS2-28] - Resetting long time mode
2024-11-29 08:45:11.944 [EY] INFO ?[JTS-DeadlockMonitor-2] - Memory:total=786,432KB free=273,763KB HeapUsage: Max=786,432KB used=510,860KB committed=786,432KB NonHeapUsage: Max=0KB used=172,732KB committed=176,432KB
2024-11-29 08:45:12.945 [EY] INFO ?[JTS-DeadlockMonitor-2] - CPU:cur=12.52% avg=0.49% 30 min avg=6.54% 10 min avg=12.51% 5 min avg=12.51% 1 min avg=12.52%
GC:called=953 times CPU used=0.03%
Finalizer:cur=0.00% avg=0.00% 30 min avg=0.00% 10 min avg=0.00% 5 min avg=0.00% 1 min avg=0.00%
Threads Count:curr live=72 curr daemon=33
2024-11-29 08:45:38.162 [EY] INFO ?[JTS-CCPPingS2-28] - Resetting long time mode
2024-11-29 08:46:39.167 [EY] INFO ?[JTS-CCPPingS2-28] - Resetting long time mode
2024-11-29 08:47:40.171 [EY] INFO ?[JTS-CCPPingS2-28] - Resetting long time mode
2024-11-29 08:49:00.405 [EY] INFO ?[JTS-EServerSocket-546] - [112501:187:187:1:0:0:0:ERR] Socket connection for client{112501} has closed. Reason: Connection terminated
2024-11-29 08:49:00.405 [EY] INFO ?[JTS-EServerSocket-546] - [112501:187:187:1:0:0:0:INFO] Stopped processing incoming messages for client{112501}.
?
?
Does anyone encounter this issue and know why is it happending? Appreciate if you can provide some insights.
?
Thanks?
xb
?
Re: No real-time data after IB nightly server reset

 

I have an open ticket with IBKR about a very similar (even identical?) topic since a couple of weeks. I have experienced the same problems where on some days Gateway would not recover after a daily reset. In all cases was a disconnect from ushmds involved. Initially I did not respond to the opening post of this thread because I am not using Docker.
?
In recent weeks the problem has disappeared. I used to download once per day quite some historical data for a list of US stock tickers (5 years daily data for 230 tickers). Every now and then (once, twice per week?) ushmds would disconnect after I had submitted the data requests, and before I received the results. Immediately after this disconnect Gateway would disconnect all clients and not allow any new clients to connect! Forcing me to kill Gateway and restart it. Four weeks ago I changed my software: now I am downloading a lot less data every day, and merge that with data I already have. Since then has the problem disappeared. What I suspect, but don't have confirmation of, is that somehow I exceeded the "soft limits" IBKR has in place on historical data downloads. In the open ticket I'm having with IBKR I am hoping to get a confirmation that this was indeed the case. I must emphasize though that I never received a warning that I was downloading too much data, or that I had crossed the soft limit.
Re: How to get position info in "points"

 

If you are not collecting order executions through the API as Juergen suggested, the other way to approach this is to collect and aggregate IBKR's trade reports or custom designed flex reports from email.


§é§ä, 28 §ß§à§ñ§Ò. 2024?§Ô., 18:04 ´³¨¹°ù²µ±ð²Ô Reinold via <TwsApiOnGroupsIo=[email protected]>:

It sounds like you are looking in the wrong corner.

At any point in time, a "Position" represents the cumulative effect of one or more Trades (Executions) that were caused by one or more Orders for the instrument in the "Position". The attributes you call "open time" and "open price" are actually attributes of the "Execution" not the "Position".

The "Position" only know about sums and averages.

Your client receives real-time order and execution callbacks as the position changes. You could collect and persist at least the executions if you need to know later, how the position changed over time.

´³¨¹°ù²µ±ð²Ô

?

?

On Thu, Nov 28, 2024 at 01:27 PM, Andy Sanders wrote:
What is the most efficient way to get information about position including the fields below.
?
- open time?
- open price
- unrealized PnL?
?
Request "reqPositions" and "reqPositionsMulti" return these?
?
- reqId
- account
- contract
- avgCost
?
1. Instead of average cost, how do I get price of the stock / option / future when position was opened??
2. How do I get time when it was open??
3. There is a separate request for PnL but even if I get PnL, there are some missing pieces, e.g. I can't calculate the range in points between open and current prices of the asset.??

?


--
Best,
DS
Re: How to get position info in "points"

 

It sounds like you are looking in the wrong corner.

At any point in time, a "Position" represents the cumulative effect of one or more Trades (Executions) that were caused by one or more Orders for the instrument in the "Position". The attributes you call "open time" and "open price" are actually attributes of the "Execution" not the "Position".

The "Position" only know about sums and averages.

Your client receives real-time order and execution callbacks as the position changes. You could collect and persist at least the executions if you need to know later, how the position changed over time.

´³¨¹°ù²µ±ð²Ô

?

?

On Thu, Nov 28, 2024 at 01:27 PM, Andy Sanders wrote:

What is the most efficient way to get information about position including the fields below.
?
- open time?
- open price
- unrealized PnL?
?
Request "reqPositions" and "reqPositionsMulti" return these?
?
- reqId
- account
- contract
- avgCost
?
1. Instead of average cost, how do I get price of the stock / option / future when position was opened??
2. How do I get time when it was open??
3. There is a separate request for PnL but even if I get PnL, there are some missing pieces, e.g. I can't calculate the range in points between open and current prices of the asset.??

?
Re: No real-time data after IB nightly server reset

 

From the logs is sounds like the intention was to "disconnectAndReconnect due to routing change for ushmds" but the reconnect part never materialized. You'd probably have to reach out to IBKR to understand why the reconnect never happened and what can be done about it.

Before disconnect, your TWS/IBGW was connected to the US Central/Chicago HMDS located at cdc1.ibllc.com. Do you have a decent network connection between your system and cdc1? As in decent packet loss?

´³¨¹°ù²µ±ð²Ô

?

?

On Thu, Nov 28, 2024 at 03:26 AM, xb wrote:


2024-11-27 12:17:38.170 [UO] INFO ?[JTS-AsyncNonLocked-33] - disconnectAndReconnect due to routing change for ushmds state:NATIVE default:ccp dataType:NONE points:[useccphost] nativeViaCcp:null
2024-11-27 12:17:38.170 [UO] INFO ?[JTS-AsyncNonLocked-33] - SocketTracker[name=name=ushmds(cdc1.ibllc.com:4000),usesCcpConMan=false,lastReadTimestamp=1732709856527,lastReadDateTime=2024-11-27T12:17:36.527,lastFixSendTime=].
?
How to get position info in "points"

 

What is the most efficient way to get information about position including the fields below.
?
- open time?
- open price
- unrealized PnL?
?
Request "reqPositions" and "reqPositionsMulti" return these?
?
- reqId
- account
- contract
- avgCost
?
1. Instead of average cost, how do I get price of the stock / option / future when position was opened??
2. How do I get time when it was open??
3. There is a separate request for PnL but even if I get PnL, there are some missing pieces, e.g. I can't calculate the range in points between open and current prices of the asset.??
Re: No real-time data after IB nightly server reset

 

I have posted about more or less the same experience not long ago: /g/twsapi/message/53498
I've come to a solution that I will perform a fresh reconnection after nightly reset as well as I usually do after daily TWS auto restart.
Seems to be working better for me, but I want to monitor for a few more days to be sure if the loss of data subscriptions still happens.
?
Re: No real-time data after IB nightly server reset

 

Hi ´³¨¹°ù²µ±ð²Ô,
?
I think the ushmds might be the cause,?
?
BTW Here's my gateway logs for ushmds part around that time
?
2024-11-27 12:15:57.522 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | cf312@6322834f | Done | cf312;;BQ@SMART Trades;;1;;true;;0;;I | 0
2024-11-27 12:15:57.523 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | adjustments311@2e87e5ab | Done | adjustments311;;BQ@SMART Trades;;1;;true;;0;;I | 0
2024-11-27 12:15:57.523 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | cf312@6322834f | Clear | 0
2024-11-27 12:15:57.523 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | indicativeData310@1c25bb71 | Done | indicativeData310;;BQ@SMART Trades;;1;;true;;0;;I | 0
2024-11-27 12:15:57.523 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | adjustments311@2e87e5ab | Clear | 0
2024-11-27 12:15:57.523 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | cf312@6322834f | Clear | 0
2024-11-27 12:15:57.523 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | agg309@1719520f | Done | agg309;;BQ@SMART Trades;;1;;true;;0;;I | 0
2024-11-27 12:15:57.523 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | indicativeData310@1c25bb71 | Clear | 0
2024-11-27 12:15:57.523 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | adjustments311@2e87e5ab | Clear | 0
2024-11-27 12:15:57.523 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | cf312@6322834f | Clear | 0
2024-11-27 12:15:57.526 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - HMDS query: 78;;BQ@SMART Trades;;1;;true;;0;;I has returned for series: BQ@SMART Trades
2024-11-27 12:15:57.527 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | qm@42f1f33f | all EOQ
2024-11-27 12:15:57.582 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - [50:176:176:1:0:17:3:INFO] Sending historical data for ticker = 78. 182489 items.
2024-11-27 12:15:59.217 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - Start writing to output data. Left to write: 12019494
2024-11-27 12:15:59.219 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - Writing to output data. Written: 0 Left: 12019494
2024-11-27 12:15:59.719 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - Finished writing to output data.
2024-11-27 12:16:01.844 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - [50:176:176:1:0:17:3:INFO] Historical data is sent for ticker = 78. 182489 items.
2024-11-27 12:17:38.170 [UO] INFO ?[JTS-AsyncNonLocked-33] - disconnectAndReconnect due to routing change for ushmds state:NATIVE default:ccp dataType:NONE points:[useccphost] nativeViaCcp:null
2024-11-27 12:17:38.170 [UO] INFO ?[JTS-AsyncNonLocked-33] - SocketTracker[name=name=ushmds(cdc1.ibllc.com:4000),usesCcpConMan=false,lastReadTimestamp=1732709856527,lastReadDateTime=2024-11-27T12:17:36.527,lastFixSendTime=].
2024-11-27 12:17:38.170 [UO] INFO ?[JTS-ushmdsServicePingS14-749] - Terminating ping thread
2024-11-27 12:17:38.170 [UO] INFO ?[JTS-AsyncNonLocked-33] - AuthTimeoutMonitor-ushmds: deactivate
2024-11-27 12:17:38.170 [UO] INFO ?[JTS-ushmdsListenerS14-744] - Socket or stream for connection cdc1.ibllc.com:4000 was closed by another thread.
2024-11-27 12:17:38.170 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - Dispatcher thread terminating [sessionID=14,interrupted=true]...
2024-11-27 12:17:38.170 [UO] INFO ?[JTS-ushmdsListenerS14-744] - Listener thread terminating [sessionID=14] [seen=15024770,totalShifted=15024770,moreAvailable=0]
2024-11-27 12:17:38.172 [UO] INFO ?[JTS-DisconnectedS14-941] - Farm ushmds/NATIVE: Lost active connection with disconnect status DISCONNECT_ON_BROKEN_SOCKET
2024-11-27 12:17:38.173 [UO] INFO ?[JTS-DisconnectedS14-941] - Farm ushmds/NATIVE: Resetting
2024-11-27 12:17:38.174 [UO] INFO ?[JTS-AsyncNonLocked-33] - Keep-alive scheduled for:ushmds
2024-11-27 12:17:38.174 [UO] INFO ?[JTS-AsyncNonLocked-33] - [50:176:176:1:0:4:2:DET] [4;2;-1;2105;HMDS data farm connection is broken:ushmds;]
2024-11-27 12:17:38.175 [UO] INFO ?[JTS-AsyncNonLocked-33] - [112501:187:187:1:0:4:2:DET] [4;2;-1;2105;HMDS data farm connection is broken:ushmds;]
2024-11-27 12:17:38.175 [UO] INFO ?[JTS-AsyncNonLocked-33] - cdebug: ROUTING | connTracker:ushmds@4785586b | Disconnected | 1732709857176 | true
?
°Õ³ó²¹²Ô°ì²õ£¬
xb
?
Re: No real-time data after IB nightly server reset

 

Hi?´³¨¹°ù²µ±ð²Ô?,
?
The time in my machine e.g.12:17:38 is in UTC timezone. I was requesting realtime bars and historical data for US stocks only, do you think the failure was due to ushmds not reconnecting successfully at that time???
?
Thanks!
?
Re: No real-time data after IB nightly server reset

 

Nothing jumps at me, except that ushmds disconnected at 12:17:38 and did not reconnect by the time TWS/IBGW had reestablished full connectivity with IBKR at 12:18:08. But that may be fine in case it was not needed any longer.

What kind of symbols were your reqRealTimeBars for? STK, FUT, OPT, ..., US, HK, Asia, ...?

What time zone are these timestamps in?

Have you taken a look at the TWS/IBKR logs? Just to see whether there are some errors related to market data.

´³¨¹°ù²µ±ð²Ô

?

?
?
On Thu, Nov 28, 2024 at 12:52 AM, <xiaobo.li@...> wrote:

Hi?´³¨¹°ù²µ±ð²Ô?,
?
I encountered the same issue and still no market data after the connection was restored.
?
Followings are my logs
?
2024-11-27 12:15:54.656112 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:15:55.257818 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:16:06.239496 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:16:12.080815 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:16:16.738522 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:16:21.150339 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:16:31.521287 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:16:42.307221 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:16:56.400287 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:17:12.772813 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:17:38.175326 ERROR 2105 HMDS data farm connection is broken:ushmds
2024-11-27 12:17:38.312416 ERROR 2106 HMDS data farm connection is OK:apachmds
2024-11-27 12:18:08.178669 ERROR 1102 Connectivity between IBKR and Trader Workstation has been restored - data maintained. The following farms are connected: hfarm; apachmds; secdefhk. The following farms are not connected: ushmds.
2024-11-27 15:30:47.702324 ERROR 2157 Sec-def data farm connection is broken:secdefhk
2024-11-27 15:30:48.190508 ERROR 2158 Sec-def data farm connection is OK:secdefhk
2024-11-27 15:45:17.419716 ERROR 2157 Sec-def data farm connection is broken:secdefhk
2024-11-27 15:45:21.453034 ERROR 2158 Sec-def data farm connection is OK:secdefhk
2024-11-27 19:23:05.387361 ERROR 2103 Market data farm connection is broken:hfarm
2024-11-27 19:23:06.315837 ERROR 2104 Market data farm connection is OK:hfarm
?
I was requesting reqRealTimeBars and no data received after Nov 27 12:17, I did not use docker or IBC and no disconnecting / cancel subscription logic.
?
Any thoughts?
Re: No real-time data after IB nightly server reset

 

Hi?´³¨¹°ù²µ±ð²Ô?,
?
I encountered the same issue and still no market data after the connection was restored.
?
Followings are my logs
?
2024-11-27 12:15:54.656112 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:15:55.257818 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:16:06.239496 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:16:12.080815 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:16:16.738522 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:16:21.150339 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:16:31.521287 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:16:42.307221 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:16:56.400287 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:17:12.772813 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:17:38.175326 ERROR 2105 HMDS data farm connection is broken:ushmds
2024-11-27 12:17:38.312416 ERROR 2106 HMDS data farm connection is OK:apachmds
2024-11-27 12:18:08.178669 ERROR 1102 Connectivity between IBKR and Trader Workstation has been restored - data maintained. The following farms are connected: hfarm; apachmds; secdefhk. The following farms are not connected: ushmds.
2024-11-27 15:30:47.702324 ERROR 2157 Sec-def data farm connection is broken:secdefhk
2024-11-27 15:30:48.190508 ERROR 2158 Sec-def data farm connection is OK:secdefhk
2024-11-27 15:45:17.419716 ERROR 2157 Sec-def data farm connection is broken:secdefhk
2024-11-27 15:45:21.453034 ERROR 2158 Sec-def data farm connection is OK:secdefhk
2024-11-27 19:23:05.387361 ERROR 2103 Market data farm connection is broken:hfarm
2024-11-27 19:23:06.315837 ERROR 2104 Market data farm connection is OK:hfarm
?
I was requesting reqRealTimeBars and no data received after Nov 27 12:17, I did not use docker or IBC and no disconnecting / cancel subscription logic.
?
Any thoughts?
Re: Best practice for cancelmktdata

 

¿ªÔÆÌåÓý

I think that given the large number of market data streams you have running, this might just be caused by your application not keeping up with the market data events. If you¡¯re processing several hundred market data streams, you could easily be handling several thousand API callbacks per second at busy times

?

As API messages are read from the socket, they are placed in a queue. A separate thread (the reader thread) takes items off that queue, parses the message and calls the appropriate callback function, then proceeds to the next item. If the processing in the callbacks (which happens on the same reader thread) is such that new items are arriving on the queue faster than the reader thread and the callback function can process them, then the queue will just keep growing. So you could end up with potentially thousands of items on the queue wating to be processed. If you now cancel a ticker, the cancel message will be immediately sent to TWS and it won¡¯t send you any more data for that ticker. But there may already be several, or tens, or hundreds of items for that ticker waiting in the queue, and they will just be processed when the reader thread gets to them, which could be long after the cancel was sent. This is because the cancelMktData method does nothing except send a cancel API message to TWS: it doesn¡¯t attempt to remove any unprocessed messages for that ticker from the queue. This means that the API client program has to simply ignore any market data callbacks for a ticker after it has been cancelled.

?

It¡¯s actually very easy to see this using IBKRs C# sample APP. If you start a number of very active market data streams, such as the ES, NQ, YM and DAX futures, and the major forex pairs (EUR.USD, GBP.USD, JPY.USD, etc) and add some market depth streams (ES, YM, NQ), and tick-by-tick etc, and make sure the busiest ones appear in the various grids, then run it at the US market open, the application will eventually get bogged down. Stop the tickers, and the data grid will continue to update for a while. DISCLAIMER: it¡¯s a long time since I last did this, so it¡¯s possible IBKR have enhanced the program to make it more efficient, but it certainly wasn¡¯t originally written for efficiency (nor was the C# API itself, but that has certainly been improved enormously over the years.)

?

Perhaps you could give a summary of what processing you do for each tick?

?

Richard

?

?

?

From: [email protected] <[email protected]> On Behalf Of H Liu via groups.io
Sent: 24 November 2024 19:01
To: [email protected]
Subject: Re: [TWS API] Best practice for cancelmktdata

?

interesting, In my case I need the cancelmktdata in case I run to Error 300, too many instruments subscribed. My experience is that in lag between cancelmktdata and IBKR cancelling the mktdata is that IBKR PACE doesn't actually cancel the first request, for example if I sent ticker ID 1-300 to be cancelled, I could sometimes get ticker id 245 cancelled before ticker 45,? (I continue getting ticker id 45 callbacks a long time before ticker 245, (which i confirm is cancelled by getting error Can't find EId) has anyone else had this?
Re: Best practice for cancelmktdata

 

interesting, In my case I need the cancelmktdata in case I run to Error 300, too many instruments subscribed. My experience is that in lag between cancelmktdata and IBKR cancelling the mktdata is that IBKR PACE doesn't actually cancel the first request, for example if I sent ticker ID 1-300 to be cancelled, I could sometimes get ticker id 245 cancelled before ticker 45,? (I continue getting ticker id 45 callbacks a long time before ticker 245, (which i confirm is cancelled by getting error Can't find EId) has anyone else had this?
Re: Best practice for cancelmktdata

 

I think JG is correct. You are likely bumping at the 50 requests/sec API limit when you cancel 300 market data feeds.

Traditionally, TWS/IBGW did reject requests at rates of more them 50/sec with an error callback. But there was always the "PACEAPI" connection option that, when specified during your connection requests, would instruct TWS.IBGW to perform request throttling instead of rejecting requests. More recent TWS/IBGW versions now enable throttling by default and you'd have to use an API configuration field in TWS/IBGW to disable it.

I have been using TWS/IBGW throttling for years (first with the PACEAPI option and now just by default) and found it works very well. In the few occasions where I did exceed the 50 requests/sec limit, TWS/IBGW newer throttled below that rate and for short peaks allowed a few more than 50. So I think it is just fine to send requests without implementing any delays on your client side. And should IBKR increase the rate at some time in the future (they tend to relax limits over time), your clients would immediately take advantage of the higher rate.

Keep in mind, that the 50 requests/sec is the cumulative limit for all clients connected to TWS/IBGW at any point in time. To implement throttling on the client side, you'd have to coordinate all of your clients should there be more than one.

But coming back to your question. We don't know all of your requirements, but I can see a three basic approaches (assuming you indeed need to cancel all these feeds simultaneously):

  • You could continue with what you are doing now and simply ignore any data callbacks after you submitted the cancel request for a feed to hide the additional data from your application. But TWS API would be unable to accept additional requests for up to 300/50 = 6 seconds until all cancel requests have been satisfied. But that may be fine.
  • You could simply keep the subscriptions active all the time and never cancel them since there is no limit on the number of messages TWS/IBGW send to your client.
  • Or you could have two connections with TWS/IBGW.
    • One for general requests (such as order placement etc) that you keep established for the entire lifetime of your client
    • One for these temporary market data feeds that you establish when you start the subscriptions and disconnect (instead of calling cancel requests for all data feeds) when you are done with the market data. Closing the connection will cancel all outstanding data feeds. But there may be unintended consequences when closing the connection without cancelling the data feeds (there was a bug in TWS/IBGW in the past that made it important that feeds were cancelled before disconnect) and you'd have to experiment with that.

I assume you also experience a delay when you establish the 300 market data feeds, right? So it might take up to six seconds for the first market data callbacks for the last instrument on the list to arrive. So the best approach depends on the the balance of all of your requirements.

´³¨¹°ù²µ±ð²Ô

?
On Sun, Nov 24, 2024 at 01:21 AM, J G wrote:

It used to be the case that you can send a maximum of 50 messages per second via the API. So you would need to include some waiting time if you need to send more messages. However, latest versions of the API have a pacing function built-in. So it would not help if you include any waiting time when you send your messages. In my case: even though the pacing function is there, I still include some waiting time in case I send a batch of messages that is longer than 50 messages.
Re: How do I find out the margin requirements for a US stock through the TWS API?

 

On Sun, Nov 24, 2024 at 4:14?PM ´³¨¹°ù²µ±ð²Ô Reinold via <TwsApiOnGroupsIo=[email protected]> wrote:

This is where your assumptions are incorrect. The actual margin required for changes in positions (buy or sell) depends on an account level risk model that you do not know all parameters of and that you will not be able to rebuild.


I don't need or want to rebuild this, or re-calculate this, I just want to know what IB has calculated in this regard. these are all based on the account's properties and not related?to the asset I'm trying to take. The accountSummary endpoint seems the closest?to contain this information:?? . Looking at it, AvailableFunds-S seems to come closest in relation to stocks.
?
The information from the instrument description page you mentioned describes only a very small portion of the model, and for US stocks it always says the same:
  • Initial (long) 25.00 %
  • Maintenance (long) 25.00 %
  • Initial (short) 30.00 %
  • Maintenance (short) 30.00 %

So you'd need no API call if you think you can determine your margin requirements from those constant percentages.

It's actually not the same for all stocks or ETFs. For example, for QQQ it's 25% for a long?position, but for TQQQ / SQQQ, it's 75% for a long position. hence the need to have this available via an API call, which apparently doesn't exist

and the size of the trade is about:

<funds available for trading> / margin_requirement, e.g. $100k / 25% = $100k / 0.25 = $400k
?
where the missing link is <funds available for trading>, something akin to AvailableFunds-S in account properties

That TWS screen has a "More" link that points to various margin related resources at IBKR. It also mentions the ability in TWS to monitor your real time margin requirements with the "check margin" feature, which happens to be identical to placing a WhatIf order through the API.

it also has a page that basically displays the account properties for the account
?

Keep in mind that our group has not designed the API nor can we make any changes to it. We can help each other with experience and advice about how to use it. If you do not like our advice and the current API "WhatIf" order placement feature, you need to continue your discussion directly with IBKR.

´³¨¹°ù²µ±ð²Ô

?
On Sun, Nov 24, 2024 at 02:38 AM, ?kos Mar¨®y wrote:
On Sun, Nov 24, 2024 at 8:25?AM J G via <windmill_1965=[email protected]> wrote:
I don't think that it is counter-intuitive. Whether an order would go through may depend on many parameters. For example what other positions are already in your portfolio, and how much free cash you have available. Besides the size of the intended order. The margin required is therefore depending on the situation and can only be analyzed by submitting a "what-if order".
?
?
yes, and all this information is available before entering the trade, and I should be able to deduct the result myself, without a 'what if' order. ideally for example based on values in accountSummary(), such as?AvailableFunds-S
?

?

?

?
Re: How do I find out the margin requirements for a US stock through the TWS API?

 

This is where your assumptions are incorrect. The actual margin required for changes in positions (buy or sell) depends on an account level risk model that you do not know all parameters of and that you will not be able to rebuild.

The information from the instrument description page you mentioned describes only a very small portion of the model, and for US stocks it always says the same:
  • Initial (long) 25.00 %
  • Maintenance (long) 25.00 %
  • Initial (short) 30.00 %
  • Maintenance (short) 30.00 %

So you'd need no API call if you think you can determine your margin requirements from those constant percentages.

That TWS screen has a "More" link that points to various margin related resources at IBKR. It also mentions the ability in TWS to monitor your real time margin requirements with the "check margin" feature, which happens to be identical to placing a WhatIf order through the API.

Keep in mind that our group has not designed the API nor can we make any changes to it. We can help each other with experience and advice about how to use it. If you do not like our advice and the current API "WhatIf" order placement feature, you need to continue your discussion directly with IBKR.

´³¨¹°ù²µ±ð²Ô

?
On Sun, Nov 24, 2024 at 02:38 AM, ?kos Mar¨®y wrote:

On Sun, Nov 24, 2024 at 8:25?AM J G via <windmill_1965=[email protected]> wrote:
I don't think that it is counter-intuitive. Whether an order would go through may depend on many parameters. For example what other positions are already in your portfolio, and how much free cash you have available. Besides the size of the intended order. The margin required is therefore depending on the situation and can only be analyzed by submitting a "what-if order".
?
?
yes, and all this information is available before entering the trade, and I should be able to deduct the result myself, without a 'what if' order. ideally for example based on values in accountSummary(), such as?AvailableFunds-S
?

?

?

?
Re: using EWrapper::fundamentalData

 

Thanks J_G,
now I have studied the old posts in the group and found many references to other sites to get fundamental financial data.
Now I am in data hell. For example, I tried to compare fundamental data on EBIT and EBITDA of CRM (Salesforce) from a few different sites and could not find matching data. It is very frustrating.
For EBIT, I found an acceptable match between SeekingAlpha and TWS (IBKR) and SimFin. However only SimFin has 15 years of historical data.
?
Does anyone have any suggestions on which data is more reliable among:
- SeekingAlpha (historical data not old enough from what I see and not easily reachable via API)
- macrotrends
- SimFin
- Zacks
- hedginglabs
- alphavantage
- eodhd
- financialmodel grep
- polygon
- ...
Re: How do I find out the margin requirements for a US stock through the TWS API?

 

On Sun, Nov 24, 2024 at 8:25?AM J G via <windmill_1965=[email protected]> wrote:
I don't think that it is counter-intuitive. Whether an order would go through may depend on many parameters. For example what other positions are already in your portfolio, and how much free cash you have available. Besides the size of the intended order. The margin required is therefore depending on the situation and can only be analyzed by submitting a "what-if order".

yes, and all this information is available before entering the trade, and I should be able to deduct the result myself, without a 'what if' order. ideally for example based on values in accountSummary(), such as?AvailableFunds-S
?
Re: How do I find out the margin requirements for a US stock through the TWS API?

 

I don't think that it is counter-intuitive. Whether an order would go through may depend on many parameters. For example what other positions are already in your portfolio, and how much free cash you have available. Besides the size of the intended order. The margin required is therefore depending on the situation and can only be analyzed by submitting a "what-if order".
Re: Best practice for cancelmktdata

 

It used to be the case that you can send a maximum of 50 messages per second via the API. So you would need to include some waiting time if you need to send more messages. However, latest versions of the API have a pacing function built-in. So it would not help if you include any waiting time when you send your messages. In my case: even though the pacing function is there, I still include some waiting time in case I send a batch of messages that is longer than 50 messages.
501 - 520 of 53682