Keyboard Shortcuts
Likes
- Twsapi
- Messages
Search
|
Re: No real-time data after IB nightly server reset
I have posted about more or less the same experience not long ago: /g/twsapi/message/53498
I've come to a solution that I will perform a fresh reconnection after nightly reset as well as I usually do after daily TWS auto restart.
Seems to be working better for me, but I want to monitor for a few more days to be sure if the loss of data subscriptions still happens.
? |
|
Re: No real-time data after IB nightly server reset
Hi ´³¨¹°ù²µ±ð²Ô,
?
I think the ushmds might be the cause,?
?
BTW Here's my gateway logs for ushmds part around that time
?
2024-11-27 12:15:57.522 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | cf312@6322834f | Done | cf312;;BQ@SMART Trades;;1;;true;;0;;I | 0
2024-11-27 12:15:57.523 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | adjustments311@2e87e5ab | Done | adjustments311;;BQ@SMART Trades;;1;;true;;0;;I | 0 2024-11-27 12:15:57.523 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | cf312@6322834f | Clear | 0 2024-11-27 12:15:57.523 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | indicativeData310@1c25bb71 | Done | indicativeData310;;BQ@SMART Trades;;1;;true;;0;;I | 0 2024-11-27 12:15:57.523 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | adjustments311@2e87e5ab | Clear | 0 2024-11-27 12:15:57.523 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | cf312@6322834f | Clear | 0 2024-11-27 12:15:57.523 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | agg309@1719520f | Done | agg309;;BQ@SMART Trades;;1;;true;;0;;I | 0 2024-11-27 12:15:57.523 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | indicativeData310@1c25bb71 | Clear | 0 2024-11-27 12:15:57.523 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | adjustments311@2e87e5ab | Clear | 0 2024-11-27 12:15:57.523 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | cf312@6322834f | Clear | 0 2024-11-27 12:15:57.526 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - HMDS query: 78;;BQ@SMART Trades;;1;;true;;0;;I has returned for series: BQ@SMART Trades 2024-11-27 12:15:57.527 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - cdebug: QUERY | qm@42f1f33f | all EOQ 2024-11-27 12:15:57.582 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - [50:176:176:1:0:17:3:INFO] Sending historical data for ticker = 78. 182489 items. 2024-11-27 12:15:59.217 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - Start writing to output data. Left to write: 12019494 2024-11-27 12:15:59.219 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - Writing to output data. Written: 0 Left: 12019494 2024-11-27 12:15:59.719 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - Finished writing to output data. 2024-11-27 12:16:01.844 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - [50:176:176:1:0:17:3:INFO] Historical data is sent for ticker = 78. 182489 items. 2024-11-27 12:17:38.170 [UO] INFO ?[JTS-AsyncNonLocked-33] - disconnectAndReconnect due to routing change for ushmds state:NATIVE default:ccp dataType:NONE points:[useccphost] nativeViaCcp:null 2024-11-27 12:17:38.170 [UO] INFO ?[JTS-AsyncNonLocked-33] - SocketTracker[name=name=ushmds(cdc1.ibllc.com:4000),usesCcpConMan=false,lastReadTimestamp=1732709856527,lastReadDateTime=2024-11-27T12:17:36.527,lastFixSendTime=]. 2024-11-27 12:17:38.170 [UO] INFO ?[JTS-ushmdsServicePingS14-749] - Terminating ping thread 2024-11-27 12:17:38.170 [UO] INFO ?[JTS-AsyncNonLocked-33] - AuthTimeoutMonitor-ushmds: deactivate 2024-11-27 12:17:38.170 [UO] INFO ?[JTS-ushmdsListenerS14-744] - Socket or stream for connection cdc1.ibllc.com:4000 was closed by another thread. 2024-11-27 12:17:38.170 [UO] INFO ?[JTS-ushmdsDispatcherS14-745S14-746] - Dispatcher thread terminating [sessionID=14,interrupted=true]... 2024-11-27 12:17:38.170 [UO] INFO ?[JTS-ushmdsListenerS14-744] - Listener thread terminating [sessionID=14] [seen=15024770,totalShifted=15024770,moreAvailable=0] 2024-11-27 12:17:38.172 [UO] INFO ?[JTS-DisconnectedS14-941] - Farm ushmds/NATIVE: Lost active connection with disconnect status DISCONNECT_ON_BROKEN_SOCKET 2024-11-27 12:17:38.173 [UO] INFO ?[JTS-DisconnectedS14-941] - Farm ushmds/NATIVE: Resetting 2024-11-27 12:17:38.174 [UO] INFO ?[JTS-AsyncNonLocked-33] - Keep-alive scheduled for:ushmds 2024-11-27 12:17:38.174 [UO] INFO ?[JTS-AsyncNonLocked-33] - [50:176:176:1:0:4:2:DET] [4;2;-1;2105;HMDS data farm connection is broken:ushmds;] 2024-11-27 12:17:38.175 [UO] INFO ?[JTS-AsyncNonLocked-33] - [112501:187:187:1:0:4:2:DET] [4;2;-1;2105;HMDS data farm connection is broken:ushmds;] 2024-11-27 12:17:38.175 [UO] INFO ?[JTS-AsyncNonLocked-33] - cdebug: ROUTING | connTracker:ushmds@4785586b | Disconnected | 1732709857176 | true ?
°Õ³ó²¹²Ô°ì²õ£¬
xb
? |
|
Re: No real-time data after IB nightly server reset
Nothing jumps at me, except that ushmds disconnected at 12:17:38 and did not reconnect by the time TWS/IBGW had reestablished full connectivity with IBKR at 12:18:08. But that may be fine in case it was not needed any longer. What kind of symbols were your reqRealTimeBars for? STK, FUT, OPT, ..., US, HK, Asia, ...? What time zone are these timestamps in? Have you taken a look at the TWS/IBKR logs? Just to see whether there are some errors related to market data. ´³¨¹°ù²µ±ð²Ô ? ?
?
On Thu, Nov 28, 2024 at 12:52 AM, <xiaobo.li@...> wrote:
|
|
Re: No real-time data after IB nightly server reset
Hi?´³¨¹°ù²µ±ð²Ô?,
?
I encountered the same issue and still no market data after the connection was restored.
?
Followings are my logs
?
2024-11-27 12:15:54.656112 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost.
2024-11-27 12:15:55.257818 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost. 2024-11-27 12:16:06.239496 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost. 2024-11-27 12:16:12.080815 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost. 2024-11-27 12:16:16.738522 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost. 2024-11-27 12:16:21.150339 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost. 2024-11-27 12:16:31.521287 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost. 2024-11-27 12:16:42.307221 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost. 2024-11-27 12:16:56.400287 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost. 2024-11-27 12:17:12.772813 ERROR 1100 Connectivity between IBKR and Trader Workstation has been lost. 2024-11-27 12:17:38.175326 ERROR 2105 HMDS data farm connection is broken:ushmds 2024-11-27 12:17:38.312416 ERROR 2106 HMDS data farm connection is OK:apachmds 2024-11-27 12:18:08.178669 ERROR 1102 Connectivity between IBKR and Trader Workstation has been restored - data maintained. The following farms are connected: hfarm; apachmds; secdefhk. The following farms are not connected: ushmds. 2024-11-27 15:30:47.702324 ERROR 2157 Sec-def data farm connection is broken:secdefhk
2024-11-27 15:30:48.190508 ERROR 2158 Sec-def data farm connection is OK:secdefhk 2024-11-27 15:45:17.419716 ERROR 2157 Sec-def data farm connection is broken:secdefhk 2024-11-27 15:45:21.453034 ERROR 2158 Sec-def data farm connection is OK:secdefhk 2024-11-27 19:23:05.387361 ERROR 2103 Market data farm connection is broken:hfarm
2024-11-27 19:23:06.315837 ERROR 2104 Market data farm connection is OK:hfarm ?
I was requesting reqRealTimeBars and no data received after Nov 27 12:17, I did not use docker or IBC and no disconnecting / cancel subscription logic.
?
Any thoughts? |
|
Re: Best practice for cancelmktdata
¿ªÔÆÌåÓýI think that given the large number of market data streams you have running, this might just be caused by your application not keeping up with the market data events. If you¡¯re processing several hundred market data streams, you could easily be handling several thousand API callbacks per second at busy times ? As API messages are read from the socket, they are placed in a queue. A separate thread (the reader thread) takes items off that queue, parses the message and calls the appropriate callback function, then proceeds to the next item. If the processing in the callbacks (which happens on the same reader thread) is such that new items are arriving on the queue faster than the reader thread and the callback function can process them, then the queue will just keep growing. So you could end up with potentially thousands of items on the queue wating to be processed. If you now cancel a ticker, the cancel message will be immediately sent to TWS and it won¡¯t send you any more data for that ticker. But there may already be several, or tens, or hundreds of items for that ticker waiting in the queue, and they will just be processed when the reader thread gets to them, which could be long after the cancel was sent. This is because the cancelMktData method does nothing except send a cancel API message to TWS: it doesn¡¯t attempt to remove any unprocessed messages for that ticker from the queue. This means that the API client program has to simply ignore any market data callbacks for a ticker after it has been cancelled. ? It¡¯s actually very easy to see this using IBKRs C# sample APP. If you start a number of very active market data streams, such as the ES, NQ, YM and DAX futures, and the major forex pairs (EUR.USD, GBP.USD, JPY.USD, etc) and add some market depth streams (ES, YM, NQ), and tick-by-tick etc, and make sure the busiest ones appear in the various grids, then run it at the US market open, the application will eventually get bogged down. Stop the tickers, and the data grid will continue to update for a while. DISCLAIMER: it¡¯s a long time since I last did this, so it¡¯s possible IBKR have enhanced the program to make it more efficient, but it certainly wasn¡¯t originally written for efficiency (nor was the C# API itself, but that has certainly been improved enormously over the years.) ? Perhaps you could give a summary of what processing you do for each tick? ? Richard ? ? ? From: [email protected] <[email protected]> On Behalf Of H Liu via groups.io
Sent: 24 November 2024 19:01 To: [email protected] Subject: Re: [TWS API] Best practice for cancelmktdata ? interesting, In my case I need the cancelmktdata in case I run to Error 300, too many instruments subscribed. My experience is that in lag between cancelmktdata and IBKR cancelling the mktdata is that IBKR PACE doesn't actually cancel the first request, for example if I sent ticker ID 1-300 to be cancelled, I could sometimes get ticker id 245 cancelled before ticker 45,? (I continue getting ticker id 45 callbacks a long time before ticker 245, (which i confirm is cancelled by getting error Can't find EId) has anyone else had this? |
|
Re: Best practice for cancelmktdata
interesting, In my case I need the cancelmktdata in case I run to Error 300, too many instruments subscribed. My experience is that in lag between cancelmktdata and IBKR cancelling the mktdata is that IBKR PACE doesn't actually cancel the first request, for example if I sent ticker ID 1-300 to be cancelled, I could sometimes get ticker id 245 cancelled before ticker 45,? (I continue getting ticker id 45 callbacks a long time before ticker 245, (which i confirm is cancelled by getting error Can't find EId) has anyone else had this? |
|
Re: Best practice for cancelmktdata
I think JG is correct. You are likely bumping at the 50 requests/sec API limit when you cancel 300 market data feeds. Traditionally, TWS/IBGW did reject requests at rates of more them 50/sec with an error callback. But there was always the "PACEAPI" connection option that, when specified during your connection requests, would instruct TWS.IBGW to perform request throttling instead of rejecting requests. More recent TWS/IBGW versions now enable throttling by default and you'd have to use an API configuration field in TWS/IBGW to disable it. I have been using TWS/IBGW throttling for years (first with the PACEAPI option and now just by default) and found it works very well. In the few occasions where I did exceed the 50 requests/sec limit, TWS/IBGW newer throttled below that rate and for short peaks allowed a few more than 50. So I think it is just fine to send requests without implementing any delays on your client side. And should IBKR increase the rate at some time in the future (they tend to relax limits over time), your clients would immediately take advantage of the higher rate. Keep in mind, that the 50 requests/sec is the cumulative limit for all clients connected to TWS/IBGW at any point in time. To implement throttling on the client side, you'd have to coordinate all of your clients should there be more than one. But coming back to your question. We don't know all of your requirements, but I can see a three basic approaches (assuming you indeed need to cancel all these feeds simultaneously):
I assume you also experience a delay when you establish the 300 market data feeds, right? So it might take up to six seconds for the first market data callbacks for the last instrument on the list to arrive. So the best approach depends on the the balance of all of your requirements. ´³¨¹°ù²µ±ð²Ô ?
On Sun, Nov 24, 2024 at 01:21 AM, J G wrote:
|
|
Re: How do I find out the margin requirements for a US stock through the TWS API?
On Sun, Nov 24, 2024 at 4:14?PM ´³¨¹°ù²µ±ð²Ô Reinold via <TwsApiOnGroupsIo=[email protected]> wrote:
I don't need or want to rebuild this, or re-calculate this, I just want to know what IB has calculated in this regard. these are all based on the account's properties and not related?to the asset I'm trying to take. The accountSummary endpoint seems the closest?to contain this information:?? . Looking at it, AvailableFunds-S seems to come closest in relation to stocks. ?
It's actually not the same for all stocks or ETFs. For example, for QQQ it's 25% for a long?position, but for TQQQ / SQQQ, it's 75% for a long position. hence the need to have this available via an API call, which apparently doesn't exist and the size of the trade is about: <funds available for trading> / margin_requirement, e.g. $100k / 25% = $100k / 0.25 = $400k ? where the missing link is <funds available for trading>, something akin to AvailableFunds-S in account properties
it also has a page that basically displays the account properties for the account ?
|
|
Re: How do I find out the margin requirements for a US stock through the TWS API?
This is where your assumptions are incorrect. The actual margin required for changes in positions (buy or sell) depends on an account level risk model that you do not know all parameters of and that you will not be able to rebuild. The information from the instrument description page you mentioned describes only a very small portion of the model, and for US stocks it always says the same:
So you'd need no API call if you think you can determine your margin requirements from those constant percentages. That TWS screen has a "More" link that points to various margin related resources at IBKR. It also mentions the ability in TWS to monitor your real time margin requirements with the "check margin" feature, which happens to be identical to placing a WhatIf order through the API. Keep in mind that our group has not designed the API nor can we make any changes to it. We can help each other with experience and advice about how to use it. If you do not like our advice and the current API "WhatIf" order placement feature, you need to continue your discussion directly with IBKR. ´³¨¹°ù²µ±ð²Ô ?
On Sun, Nov 24, 2024 at 02:38 AM, ?kos Mar¨®y wrote:
? |
|
Re: using EWrapper::fundamentalData
Thanks J_G,
now I have studied the old posts in the group and found many references to other sites to get fundamental financial data.
Now I am in data hell. For example, I tried to compare fundamental data on EBIT and EBITDA of CRM (Salesforce) from a few different sites and could not find matching data. It is very frustrating. For EBIT, I found an acceptable match between SeekingAlpha and TWS (IBKR) and SimFin. However only SimFin has 15 years of historical data. ?
Does anyone have any suggestions on which data is more reliable among: - SeekingAlpha (historical data not old enough from what I see and not easily reachable via API) - macrotrends - SimFin - Zacks - hedginglabs - alphavantage - eodhd - financialmodel grep - polygon - ... |
|
Re: How do I find out the margin requirements for a US stock through the TWS API?
On Sun, Nov 24, 2024 at 8:25?AM J G via <windmill_1965=[email protected]> wrote:
yes, and all this information is available before entering the trade, and I should be able to deduct the result myself, without a 'what if' order. ideally for example based on values in accountSummary(), such as?AvailableFunds-S ? |
|
Re: How do I find out the margin requirements for a US stock through the TWS API?
I don't think that it is counter-intuitive. Whether an order would go through may depend on many parameters. For example what other positions are already in your portfolio, and how much free cash you have available. Besides the size of the intended order. The margin required is therefore depending on the situation and can only be analyzed by submitting a "what-if order". |
|
Re: Best practice for cancelmktdata
It used to be the case that you can send a maximum of 50 messages per second via the API. So you would need to include some waiting time if you need to send more messages. However, latest versions of the API have a pacing function built-in. So it would not help if you include any waiting time when you send your messages. In my case: even though the pacing function is there, I still include some waiting time in case I send a batch of messages that is longer than 50 messages. |
|
Re: How do I find out the margin requirements for a US stock through the TWS API?
Well, it sounds very anti-intuitive that one cannot determine if a trade will go through by using data available at the time of entering a trade - but has to 'reverse engineer' by issuing 'what if' trades and seeing their result. Is this really the case? On Sat, Nov 2, 2024 at 6:35?PM ´³¨¹°ù²µ±ð²Ô Reinold via <TwsApiOnGroupsIo=[email protected]> wrote:
|
|
Re: using EWrapper::fundamentalData
You use reqFundamentalData to request certain data. The requested data is then sent to you via fundamentalData. However, as you already mention, is the request reqFundamentalData deprecated. Meaning that if you can't request data, then implicitly no data can be sent to you.
If you search in this group you will find multiple topics, all related to the fact that fundamental data has been deprecated. |
|
Best practice for cancelmktdata
Hello I am currently have this in a foreach loop using
?
?
My question is should i add sleep between each canceltickerid, I am batching about 300 canceltickerids at once and the tick call backs for each canceltickerids keeps firing,? On the log for example I could call canclemktdata for a specific tickerid and for up to 2 minutes I may still get tick call backs for that partciular tickerid, Will adding sleep allow for faster processing or should i do smaller batches? |
|
Re: API call: Error validating request.-'bF' : cause - Please enter a stop price
The TWS documentation on orders is most useful when you want to determine, which order object fields need to be initialized. But that documentation does not give you any details on exactly what the different order types (or algos) do, what thez are intended for, or what their major limitations are. Richard gave you an overview of the STP, STLMT, and "IF TOCUHED" orders, but I recommended that, if you have not done so yet, you spend some time with this IBKR "" explorer. Here the . ´³¨¹°ù²µ±ð²Ô ? ?
On Fri, Nov 22, 2024 at 12:01 PM, Danny wrote:
.. |
|
Re: API call: Error validating request.-'bF' : cause - Please enter a stop price
¿ªÔÆÌåÓýFor a sell order, you only use stop or stoplimit when the market is above the trigger price (ie the AuxPrice). When the price drops to the trigger price, either a market or a limit order is then entered into the exchange. So you can use this to exit from a long position (this is typically a stop-loss, but it can alternatively be a profit taker if the position is currently in profit). ? You can use a buy stop or stoplimit to enter a new position, in which case market price must be below the trigger price. ? Be careful not to confuse a stop or stoplimit ?order with a ¡®stop-loss¡¯ order: stop and stoplimit orders are specific types of order, whereas a stop-loss order is really a mode of operating that protects a position against excessive loss, and can be of several different types. ? Note also that a market- or limit--if-touched order can also be used either to enter or exit a position, but you wouldn¡¯t use it as a stop-loss. I¡¯ll leave you to think about why that is. You need to make sure you have a clear understanding of these different order types. ? But here¡¯s an example: suppose the market is trading in a range, and you want to catch a breakout from that range in either direction: you could use a sell stop with trigger 1-tick below the bottom of the range, and a buy stop with trigger 1-tick above the high of the range. And you would put both orders in an OCA group so if one order fills, the other is cancelled. ? ? ? ? From: [email protected] <[email protected]> On Behalf Of Danny via groups.io
Sent: 22 November 2024 18:01 To: [email protected] Subject: Re: [TWS API] API call: Error validating request.-'bF' : cause - Please enter a stop price ? ? ? thanks ? Can I use a 'stop limit' for my 2 SELL calls? (one for a stop above the current market and another for below the current market, whichever reaches first) Here it says that a 'stop limit' can only be used for a stop at a price below?the current market, and to use 'Limit if Touched' for a SELL for a price above the current market ? ? Limit if TouchedA??is an order to buy (or sell) a contract at a specified price or better, below (or above) the market. This order is held in the system until the trigger price is touched. An LIT order is similar to a stop limit order, except that an LIT sell order is placed above the current market price, and a stop limit sell order is placed below.
¡¤?????????????????????? Order order = new Order(); order.Action = action; order.OrderType = "LIT"; order.TotalQuantity = quantity; order.LmtPrice = limitPrice; order.AuxPrice = triggerPrice; ? ? |