gateway version dependent error 10089

Dear all,

I hope someone can direct me in the right direction to solve this error:

Error 10089, reqId 8: Requested market data requires additional subscription for API. See link in 'Market Data Connections' dialog for more details.Delayed market data is available.SPY ARCA/TOP/ALL, contract: Stock(symbol='SPY', exchange='SMART', currency='USD')

The strange thing is that I do not get this error when requesting live data with gateway 981, but since I upgraded to gateway 1019 or 1023 it does give this error. I check with 981 again, and there it does give me live data. Is it still something related to market data subscription or did something change in how to request data, for example (using python 3.8 and ib_insync):

Contract = Stock('SPY','SMART','USD')

?

Data = ib.reqMktData(Contract)

The gateway is not in read only mode. Does anyone have an idea how I can resolve this issue??

Thanks in advance for any insight.

Re: buy order at open if market open between a minimum and maximum

#51385

开云体育

A minimum indicates a sufficiently strong follow on move (from previous day), a maximum prohibits excessively outsized moves. The trading system does significantly better with both a minimum and a maximum price limit when entering at the open.? Though of course, yes, one option is to take the hit, use only a maximum, and just work with a more reliable LOO order.

?

I’ve come up with three workarounds, which approximate the required order behaviour.? Each of them is fine, even with their inherent faults, for an EOD trading system (which shouldn’t be so fragile as to require trade entry prices to be at exactly the open price); but they compromise/approximate in different ways, and the differences are just techncially interesting (how does IB really handle price/time conditions in these scenarios).

?

The three options I see are:

?

Stream tick data and place a LOO buy order (at the required maximum) if the first trade price is above the required minimum.? Slowest of the three options (I would guess) as it involves my client code on a non IB server; has the advantage of making a decision based on the opening price, but the order will reach the market at some interval (likely just milliseconds) after the open.? I think this is the most robust, but I’m trying to avoid adding streaming architecture to this trading client if I can.

?

Place a limit order (for the required maximum), and add price and time conditioning to cancel the order if the open price is less than the required minimum.? [theoretically, if our order was somehow first in the queue, it could be filled first, before the cancel conditions can trigger; or it may be filled prior to IB’s order handling has time to register another opening trade below the minimum and send the cancel order]

?

order = Order()

order.action = "BUY"
order.orderType = "LMT"
order.totalQuantity = quantity
order.lmtPrice = limit_price
order.outsideRth = False
# PRICE CONDITION to cancel order if opening trade is below the minimum price
price_condition = Create(OrderCondition.Price)
price_condition.conId = t.contract.conId
price_condition.exchange = 'SMART'
price_condition.isMore = False
price_condition.triggerMethod = 2
price_condition.price = minimum_required_open

price_condition.isConjunctionConnection = False
order.conditions.append(price_condition)
# TIME CONDITION – cancel after n seconds so the order is only active at the “open”

time_condition = Create(OrderCondition.Time)
time_condition.time = datetime.strftime(end_datetime, '%Y%m%d %H:%M:%S ')? # end_datetime = n seconds after the 9:30

time_condition.isMore = True
time_condition.isConjunctionConnection = False

order.conditions.append(time_condition)

# Conditions do not apply outside RTH, and any condition being met will cancel the order
order.conditionsIgnoreRth = False
order.conditionsCancelOrder = True

?

Place a limit order (for the required maximum) and add price conditioning to activate the order if the opening price is above the required minimum? [this will activate the order if there is any trade above the required minimum during the GTD period (n seconds), which is not the same as activating on the open price alone]

?

# [Indicative code, untested]

order = Order()

order.action = "BUY"
order.orderType = "LMT"

order.tif = "GTD"

order.goodTillDate = datetime.strftime(end_datetime, '%Y%m%d %H:%M:%S ')? # end_datetime = n seconds after the 9:30

order.totalQuantity = quantity
order.lmtPrice = limit_price
order.outsideRth = False
# PRICE CONDITION to activate the order if a minimum price trade is observed

price_condition = Create(OrderCondition.Price)
price_condition.conId = t.contract.conId
price_condition.exchange = 'SMART'
price_condition.isMore = True
price_condition.triggerMethod = 2
price_condition.price = minimum_required_open

order.conditions.append(price_condition)

# Conditions do not apply outside RTH
order.conditionsIgnoreRth = False

?

Unfortunately IB won’t allow conditions to be added to LOO type orders – otherwise that would be a great way to remove the timing aspect of it.

?

The three options are all a bit rough, and each may be trying too hard! ?I am testing #2 at the moment. ??The system trades infrequently (once or twice a week), so it takes a while to build up a dataset.

?

But are there better ways of doing it?? Would love to get others thoughts.

?

Dave

?

Show quoted text

From: [email protected] <[email protected]> On Behalf Of Hilmar via groups.io
Sent: Friday, July 14, 2023 7:59 AM
To: [email protected]
Subject: Re: [TWS API] buy order at open if market open between a minimum and maximum

?

Why care about a minimum? Your limit-on-open will fill if the current price is below the limit.It's effectively a market order with a maximum.

Re: buy order at open if market open between a minimum and maximum

#51384

Why care about a minimum? Your limit-on-open will fill if the current price is below the limit.It's effectively a market order with a maximum.

Re: Web Portal API - whole Option Chain

#51383

It deems that you have not received any replies.

If using the IB API is also acceptable then there is a solution available that may meet your requirements:

I am the author.

There is a free and fully functional evaluation version available on demand.

--

Re: Intermittent IBGW connection problems 10.19.2a

#51382

That's impressive 闯ü谤驳别苍.

That's good to know for tick by tick data, what about lvl 1 reqMktData? same stats?

Would be interested to know your thoughts on orders and positions tracking please,
as neither work reliably in my humble experience:

- Positions can go wrong all of a sudden requiring a re-subscription or a full restart altogether,
so I just end up proactively cancelling and re-subscribing at regular intervals, which I find dirty.

- Order executions are difficult to track given they can be partially filled or the "Filled" status can not be triggered.
So I essentially track the execution of the order using a "done" Event and a timeout (the order validity end time),
and then check if order.filled has something in it, which I also find convoluted.

Thanks.

class Order_:
    __slots__ = ...
    def __init__(...):
        ...
    def update(self, **kwargs):
        for key, value in kwargs.items():
            setattr(self, key, value)
        if self.status in ('Filled', 'Cancelled', 'Inactive', 'ApiCancelled'):
            if not self.done.is_set():
                self.done.set()

def orderStatus(...):
    ...
    self.orders[orderId].update(status=status, filled=filled, remaining=remaining, avgFillPrice=avgFillPrice)

def execution(self, order: Order_, timeout: int):
    order.done.wait(timeout=timeout)
    if order.filled > 0:
        ...

Re: buy order at open if market open between a minimum and maximum

#51381

On Wed, Jul 12, 2023 at 06:14 AM, Gaether wrote:

Try to find a library for scheduling in your programming language and send an order when you'd like.?

Does not work for exchange auctions, they don't care what parameters IB supplies.

Re: Sent multiple bracket orders (open, takeprofit, stoploss, exit at time), but 1 of them only took (open and exit at time)

#51380

I hope your client properly captures and records error callbacks (there are three different I.callbacks) as well as the update callbacks you receive after placing orders.

You need to go back to those logs and determine:

whether any of the missing orders were rejected outright and what the error codes were
whether TWS/IBGW acknowledged the missing orders and what states those orders went through

Without that kind of detail, a discussion would be based on guess work. Please provide error/order status detail or, in case you don't record them, improve your client to do so in the future.

In TWS you could also go to Account -> Audit Trail and get a detailed view of you order and trade activity, but I am not sure whether orders that were rejected with an error code appear in the audit report.

闯ü谤驳别苍

Sent multiple bracket orders (open, takeprofit, stoploss, exit at time), but 1 of them only took (open and exit at time)

#51379

I send multiple orders to ib gateway everyday, for the last week I have been using the following bracket order.

Open -> long or short an equity Take profit -> price above long entry or below short entry stop loss -> price below long entry or above short entry exit before market close -> price at time.

I sent this bracket order for 3-7 stocks everyday for the last week and it ran fine.

Today, for one of the orders, the take profit and stop loss portion of it was missing. I have no idea why, I called the IB trade desk and they say that portion of the order is missing for this order. I'm sure that this order also had all 4 of the legs above. Any ideas on what could have gone wrong???

Meaning of Message 1102 (2104, 2106)

#51378

开云体育

Thanks to all for the valuable informations responding to my last blog post.
I found another discussion here commenting the interpretation of message codes.

W.r.t. the connectivity topics I ask myself what the following API log entry says:
06:37:11:822 -> ---?4-2--1-1102-Connectivity between IB and Trader Workstation has been restored - data maintained.
The following farms are connected: usfarm; ushmds; secdefnj. The following farms are not connected: euhmds.--

The entry is close to the end of the logfile because RTD transfer ceases, then, and nothing more happened.

The API Manual explains Msg 1102:
"The TWS/IB Gateway has successfully reconnected to IB's servers. Your market data requests have been recovered and there is no need for you to re-submit them."

I would think, if EUHMDS is *not* (re-)connected (and no further EUREX RTD arrives) message 1102 is not appropriate, because not *all* IB servers are reconnected and
the disrupted EUHMDS data supply is sanctioned.

Yes, I see another question: Why is EUREX RTD delivered by EUHMDS and not EUFARM? I have no answer, but it is rather evident from the logfile, that it is so.
The word 'eufarm' does not appear at all in the logfile, but we have:?
00:53:05:375 -> ---64-2--1-2104-Market data farm connection is OK:usfarm--
At 00:59 EUREX RTD is requested:
00:59:02:002 -> ---44-2--1-2106-HMDS data farm connection is OK:euhmds--).

Maybe I'm wrong and I appreciate to be enlightened.
Greg

Re: Intermittent IBGW connection problems 10.19.2a

#51377

If your server has enough memory, running multiple TWS/IBGW simultaneously works just fine. And sharing the same market data subscription between a TWS/IBGW real and paper account is no problem either. But you might want to take a quick look at this article:

Real-time tick-by-tick data feeds and subscriptions are very reliable and completely unaffected by occasional historical market data farm disconnects. We have never experienced any gaps in tick-by-tick data feeds as long as our clients were behaving correctly. You can find several posts related to tick-by-tick data in the group archive, some of them even with performance details (such as this one).

Your questions made me curios and I ran some additional stats on our 2023 tick-by-tick real-time data feeds:

We subscribe to 15 tick-by-tick feeds: three feeds each (Last, Midpoint, BidAsk) for five futures traded at CME in Chicago.
Real-time tick-by-tick messages make up just under 40% of all real-time data messages we receive. The rest of the data volume relates to a number of Level I feeds plus three market book Level II feeds.
Our virtual server is located in the east of the US in a data center with high-speed, high-availability, and low-latency networking.

So here the fun part. Year-to-date 2023 we received:

A total of 5,931,446,980 real-time market data related callbacks. That is an average of 531 callbacks each second for all 23hr CME trading sessions this year.
2,287,378,131 of these were tick-by-tick callbacks (205 callbacks per second)
The arrival time gap between adjacent tick-by-tick callbacks was less than a second for 99.94% of all callbacks (2,286,012,479)
99.999975% of back-to-back callbacks had a time gap of less than 10 seconds.
And only 427 callbacks or 0.00001865% had a gap of 10 seconds or more. These are mostly callbacks that happen during the nightly maintenance window when you receive error 1100 "Connectivity between IB and the TWS has been lost" and error 1102 "Connectivity between IB and TWS has been restored- data maintained" during account resets.

In my book that qualifies for rock-stable, considering the monthly data charge is negligible.

If you do experience any tick-by-tick data interruptions, I'd suggest the following next steps: This is my interpretation of what Gordon called in a different post "Respect the basic API concepts":

Make sure your client always uses TWS API in an asynchronous fashion. There is one or more callback for every request make sure you implement them all.
Respecting the asynchronous nature of the API requires that your client shall never (ever ever) call sleep() for synchronization between requests and their related response callbacks.
The API is very good at error reporting when something goes wrong. So make sure your client properly captures, handles, and possibly logs all TWS API error messages. There are three error callbacks with slightly different signatures.
Make sure your clients perform only minimal processing on the the thread that handles the tick-by-tick callbacks. Perform any time consuming processing on or storage operations of that data on a separate thread so that tick-by-tick callbacks can always be processed right away.
You will very likely get error 1100/1102 pairs during the daily maintenance and reset window. These are generally brief interruptions of data flow (several seconds), can happen more than once during the window, and you do not need to resubscribe your feeds. Just make sure your client is aware of these.
Look out for error 1100/1101 pairs, since 1101 is "Connectivity between IB and TWS has been restored- data lost." and you will need to re-subscribe to your market data feeds. We have never seen an error 1101 in all those years we use TWS API, but your code should be ready to perform re-subscriptions should such an error ever be reported.
You might get 21xx warning codes (via the error callback) that relate to data farm disconnection/connection events. They have no impact on the flow of tick-by-tick data streams and you don't have to engage any special code to keep those farms connected.

Pairs of error 1100/1102 can happen and disrupt real-time data flow outside of the maintenance window, too. In case you experience many of them during regular trading sessions, review the network connection between TWS/IBGW and IBKR. You can find some thoughts on that in this post.

闯ü谤驳别苍

Re: ReqHistoricalData for Stock Giving API Version Error After Forced Upgrade to 10.19.2

#51376

I guess it is time to upgrade to TWS API version 10.19 (stable) or 10.22 (latest), Adam. Just like IBKR support suggests.

The Decimal data type and fractional sizes (API level 163) were introduced two years ago with API versions 10.10.x and have become pretty fundamental for the most recent versions of TWS/IBGW. This was no a sudden change in TWS/IBGW 10.19.2 and you can find a good number of posts about this in our archive.

You may be able to continue using your current client and API version until you finish the upgrade as long as you exclusively work with quantities in multiples of full shares, .But when TWS/IBGW needs to send you a non-integer size value (real-time or historical market data, order updates, commission and execution details), they will truncate the value and issue the warning.

We ran some legacy V9.85 Java clients with V10 TWS/IBGW for a while until we switched entirely over to all V10 clients earlier in the year. Those legacy clients worked just fine but occasionally received these warnings (and value truncation) for real-time market data updates outside of regular trading hours. But that did not interfere with the operation of those clients.

But your mileage will vary and you should start upgrading now.

闯ü谤驳别苍

ReqHistoricalData for Stock Giving API Version Error After Forced Upgrade to 10.19.2

#51375

Has anyone else encountered this and found a solution??

Now when I call ReqHistoricalData on a stock it says "Your API version does not support fractional share size rules. Please upgrade to a minimum version 163. Trimmed value 372.68 to 372".? I've tried turning on forex compatibility mode but that doesn't help, probably because I'm requesting stock data.? Everything worked fine before the forced upgrade and unfortunately I'm using the old IbPy API implementation in python 2.7, so using IBKR's official python 3 API would require rewriting a lot of code and be a giant headache.? ?IBKR support is no help and simply tells me to upgrade.

Any ideas?

Re: buy order at open if market open between a minimum and maximum

#51374

Try to find a library for scheduling in your programming language and send an order when you'd like.?

Re: Updating ibapi on Mac

#51373

My suggestion dont use IB API and use IB_INSYNC?https://github.com/erdewit/ib_insync

Updating ibapi on Mac

#51372

Hi, I've got a new Mac and was trying to install ibapi.?I'm running python 3.11.4, miniconda and spyder 5.3.4.

I downloaded the IB API 10.22 and run "python setup.py install" and got a message warning me not to run setup.py directly. (that worked fine before)
i run "pip install ibapi", but an old version is installed.?

What's the best way to upgrade the ibapi to the latest version?

Re: Intermittent IBGW connection problems 10.19.2a

#51371

Thanks for all informations which I carefully collect.
Some further questions came up which I found no answer for:

1) Is it harmful to operate the Real account and Paper account simultaneously with RTD supply (on different or same assets) with 2 IBGW/TWS operating?
2) The IB support told me, that European exchange data (Market or History) are always delivered from Switzerland server, independent where my account is hosted at.
???? When I request EUREX realtime tick-by-tick data (and nothing else) the IBGW GUI tells me
??? ??? Market Data Farm????? ON:usfarm
??? ??? Historical Data Farm? ON:ushmds, ON:euhmds? or ON:ushmds, OFF:euhmds when disconnected
??? So it seems to me, that realtime tick-by-tick data is delivered by EUHMDS (I guess Switzerland) and not 'EUFARM'
Then, according to Gordon's mail I have to keep (EU)HMDS alive, albeit I do not need historical data.
3) A good idea to test if the TWS API provides a more stable tick-by-tick data supply. The time-and-sales list would be fragmentary, otherwise.

Re: going to have to reconsider my approach to calling reqMktData for multiple symbols how does the forum handle it please?

#51370

Surely some
But I am curious, how did you diagnose that data were wrong ? (I means in real time not postmortem analysis)

Re: Intermittent IBGW connection problems 10.19.2a

#51369

I maybe wrong,but from what I see and practice:

0- You can't compare status of farm with a 30 seconds delay as in your evaluation.
I assume that you probably stop your Real account and start paper but didn't have both running simultaneously.

1- HMDS is for Historical request, it is? doesn't seems the farm that feed ticks (this seems to be Marketdata)

2- Hence HMDS maybe disconnected when not used with no harm or very little (waking it up require ~400ms on the first call as a wake up call, then its back to normal speed for quit sometime, at least 10 min during RTH)

3- If you want, set a watch dog that make a dummy call for a 1 bar AAPL, (or also a DAX stocks as it looks like your concern its the disconnection of farm related to Europe while this maybe related to your server location more than your marketplace)
Do it once every 3 min is enough to keep it alive. But again this may only be an issue for Historical and I don't felt any need for that.
As per Europe: be careful that RTH are earlier than US and it looks to me like IB tolerance to unused connection is lower (faster disconnect) aside of Local RTH.

4- Try using TWS (which have more internal reason to keep and ongoing traffic with IB farms)

So my question was: practically did you experience or have a clue that this impact your code ?
I am not sure it does.
But I would agree that it's pretty difficult to get such a clue.

I did set a receiving time stamp and always compare it to supplied time stamp for sake of consistencies and sometime it show a small discrepancies (< 900ms) but I always find it related to complex market condition.
Otherwise said ticks can be late (they always are but how much you accept is another issue) but not lost.
In general Tick driven algo need to be sure that you are not loosing ticks, but the time critical part will be the time to send and process the order anyway. which is > 1 magnitude slower than any tick discrepancies

Note:
In general people on this forum that are using IB ticks + simultaneously another tick provider, complain that the other one could be couple of second late. Without being a proof it's a suspicion that IB is reliable enough in general.

Re: Intermittent IBGW connection problems 10.19.2a

#51368

开云体育

Well, a trading algorithm based on realtime tick-by-tick data can have a problem if the time series has gaps or even ceases.
While at less liquid market hours the time between two ticks can last a couple of minutes I think a lazy use shutdown here and there is not a good idea.
As I understand the API a realtime tick-by-tick request is active and continues until the subscription is canceled.
However, it would be interesting if the API allows in a timely manner to detect that the realtime tick-by-tick supply is broken so that a new request can be initiated.
All in all the last two days my data supply was flawless and I'm in good hope.

On 08.07.2023 21:33, Gordon Eldest via groups.io wrote:

Show quoted text

You saw this but did impact your code ? Otherwise said did you try to make a request anyway ?
I suspect a pattern where IB disconnect on lazy use, (a connection is expensive as a resource)
Real Account typically are less active than Paper account which allow more experiment hence more activity hiding some difference in timing and traffic, less traffic is not always a good thing.

Also HDMS are for quering Data, this should not impact your trading capabilities

Re: Intermittent IBGW connection problems 10.19.2a

#51367

You saw this but did impact your code ? Otherwise said did you try to make a request anyway ?
I suspect a pattern where IB disconnect on lazy use, (a connection is expensive as a resource)
Real Account typically are less active than Paper account which allow more experiment hence more activity hiding some difference in timing and traffic, less traffic is not always a good thing.

Also HDMS are for quering Data, this should not impact your trading capabilities