Re: CI-testing anyone?

Matthias Frener

Thanks for all the answers so far :)
A simular or any kind of moch is pretty much out of question.
We are build an API client (the API interface and encoder/decoder to process the messages that go to / come from the TCP socket).
The background of this exercise is ensure that out lib works across kind of different TWS versions, ideal on CI, firing up 20 dockers in parallel.
(the root issue the account - we need out paper accounts for local testing, was hoping for some dev-program form IB or a reference server, or idk... =) )

Re: question about server support and login best practice

#46605

开云体育

Hi Richard.?

Never really paid attention to IBC although have seen it mentioned in posts. Went on your GitHub link and from what I gather it is for logging in / maintaining the login around restarts etc.?

It is not meant to be a replacement for IB Api right. More of augmentation to it.?

Thanks for all you do.?

Traderapi?

Show quoted text

On Feb 20, 2021, at 5:25 PM, Richard L King <rlking@...> wrote:

?
No no no!
?
IBController is a defunct project that is no longer maintained.
?
It was superseded three ?years ago by IBC, which you can find at
?
?
IBC is fully maintained (by me as it happens) and works with every version of TWS/Gateway (a new release is imminent to cope with a change by IBKR in the installation location for the macOS version of TWS/Gateway 981.2r).
?
I don't know why the current owner of the old IBController project persists in not marking it as defunct: it is in no-one's interest to leave it in its current state.
?
Richard
?
?
?
From: [email protected] <[email protected]> On Behalf Of Alex Gorbachev
Sent: 20 February 2021 18:37
To: [email protected]
Subject: Re: [TWS API] question about server support and login best practice
?
I think you should look at IBController -?
Official from IBRK.
?
Or check a docker image like this -?
?
I took the latter and adapted for me.
I also managed to hack my way into running IB gateway on Google Colab to experiment.
?

Re: Have you ever noticed that markets match your price but your LMT order doesn't execute?

#46604

This is yet another completely off-topic post.

If you have a complaint about whether some aspect of IBKR's service is working properly, take it up with IBKR. Asking whether anyone else on this has encountered it or something like it is unlikely to yield anything useful.

Let me repeat again, the purpose of this group is to assist API users in their endeavours to make good use of it.

I'm going to lock this thread.

Richard King Group Owner and Moderator

Re: Have you ever noticed that markets match your price but your LMT order doesn't execute?

Joel Gross

#46603

What was your order size? I believe order sizes below the minimum ($25,000 I think) won't appear in the bid ask and end up with worse fills. I had the same reaction as you and was upset when I first saw this until I found out about the minimum order size issue.

Show quoted text

On Sat, Feb 20, 2021, 2:04 PM Bruce B <bruceb444@...> wrote:

The other night, when all server farms were connected (I checked status in TWS and reset data connections multiple times), I have noticed that my LMT order was not executing while market price was matching it or even exceeding it.?The order was placed manually on TWS for a Forex pair. I am pretty certain results would have been the same if it was done through API. I could see that the order had a different color sign beside it (that was unfamiliar to me) and tooltip said IBKR did not acknowledge the order yet.

Given there was no connection issue between my TWS and IBKR, it seems that IBKR can at their own discretion - without any penalty to them - do one or all of below:
A) Delay some users order executions (by putting certain servers on maintenance and not other servers) while other users executions are processing (be it intentionally or not). This in effect gives some users access to working servers and others not which creates an unfair advantage.

B) Delay some or all users order executions while their own orders might be executed (i.e. if they are loaning user shares to other brokers or exchanges for short selling). Or they could be participating in Forex market as a corporation while user orders are not executed because they do no eat their own dog food and their own order route through different servers.

OR another scenario that might be happening is that:
C) IBKR's internal execution system is down for that particular security (or all types of securities) and sending process to external or internal exchange system does not work BUT they are showing data feed from exchanges which basically excludes their own exchange trading which is not happening at that time. This scenario is a maintenance issue of course and no trickery since no one is advantaged over another. It is the data that is placed through other brokers to their internal exchange or external exchanges but IBKR reflects them to their users while their order placement system is down.

My questions:
1) If C above happens, do they ever show an indicator for this issue in either TWS *charts (not order window) or through API that you have seen? Something like, "Oops your LMT is matching wide market price now but our systems are down to place it and execute it - we are sorry for our horrible service).

2) If A or B happens, isn't there a regulatory body that checks their tapes and goes well at this 5 minutes you as a corporation sent millions of orders while none of your regular user orders were sent to exchanges? Or is this too in-depth for the feds still and they don't have the technical knowledge to monitor these sort of trickeries or mistakes?

3) I have seen this happening with Forex (I think it was around reset time - 12:00 am; not that reset time gives them an excuse). Have you seen this happen with regular stock (or other securities) or Forex at any other time or in another scenario?

4) Have you noticed any other serious - out of whack - data streams or order issues that you have experienced? and how do you try to remedy them?

Thanks,

Re: question about server support and login best practice

#46602

开云体育

No no no!

?

IBController is a defunct project that is no longer maintained.

?

It was superseded three ?years ago by IBC, which you can find at

?

IBC is fully maintained (by me as it happens) and works with every version of TWS/Gateway (a new release is imminent to cope with a change by IBKR in the installation location for the macOS version of TWS/Gateway 981.2r).

?

I don't know why the current owner of the old IBController project persists in not marking it as defunct: it is in no-one's interest to leave it in its current state.

?

Richard

?

Show quoted text

From: [email protected] <[email protected]> On Behalf Of Alex Gorbachev
Sent: 20 February 2021 18:37
To: [email protected]
Subject: Re: [TWS API] question about server support and login best practice

?

I think you should look at IBController -?

Official from IBRK.

?

Or check a docker image like this -?

?

I took the latter and adapted for me.

I also managed to hack my way into running IB gateway on Google Colab to experiment.

?

Have you ever noticed that markets match your price but your LMT order doesn't execute?

#46601

The other night, when all server farms were connected (I checked status in TWS and reset data connections multiple times), I have noticed that my LMT order was not executing while market price was matching it or even exceeding it.?The order was placed manually on TWS for a Forex pair. I am pretty certain results would have been the same if it was done through API. I could see that the order had a different color sign beside it (that was unfamiliar to me) and tooltip said IBKR did not acknowledge the order yet.

Given there was no connection issue between my TWS and IBKR, it seems that IBKR can at their own discretion - without any penalty to them - do one or all of below:
A) Delay some users order executions (by putting certain servers on maintenance and not other servers) while other users executions are processing (be it intentionally or not). This in effect gives some users access to working servers and others not which creates an unfair advantage.

B) Delay some or all users order executions while their own orders might be executed (i.e. if they are loaning user shares to other brokers or exchanges for short selling). Or they could be participating in Forex market as a corporation while user orders are not executed because they do no eat their own dog food and their own order route through different servers.

OR another scenario that might be happening is that:
C) IBKR's internal execution system is down for that particular security (or all types of securities) and sending process to external or internal exchange system does not work BUT they are showing data feed from exchanges which basically excludes their own exchange trading which is not happening at that time. This scenario is a maintenance issue of course and no trickery since no one is advantaged over another. It is the data that is placed through other brokers to their internal exchange or external exchanges but IBKR reflects them to their users while their order placement system is down.

My questions:
1) If C above happens, do they ever show an indicator for this issue in either TWS *charts (not order window) or through API that you have seen? Something like, "Oops your LMT is matching wide market price now but our systems are down to place it and execute it - we are sorry for our horrible service).

2) If A or B happens, isn't there a regulatory body that checks their tapes and goes well at this 5 minutes you as a corporation sent millions of orders while none of your regular user orders were sent to exchanges? Or is this too in-depth for the feds still and they don't have the technical knowledge to monitor these sort of trickeries or mistakes?

3) I have seen this happening with Forex (I think it was around reset time - 12:00 am; not that reset time gives them an excuse). Have you seen this happen with regular stock (or other securities) or Forex at any other time or in another scenario?

4) Have you noticed any other serious - out of whack - data streams or order issues that you have experienced? and how do you try to remedy them?

Thanks,

Re: CI-testing anyone?

Joel Gross

#46600

I feel guilty about my approach, but it has worked and made for fast development so far without any serious production problems:

I basically build and test everything on a paper account against live data and debug from there. Whenever I run it in production so far I have monitored it and occasionally found minor bugs, but nothing that caused me to lose money. This is enabled me to get to production more quickly I think, and over time I have found a lot of corner case weird production bugs that I would never have caught probably with some sort of test suite that emulates IB.?

Show quoted text

On Sat, Feb 20, 2021, 12:27 PM Jonathan S <jsheena@...> wrote:

My CI testing follows what Mark outlines.??
My code is in Java.? I wrote a broker interface which limits the IB API pieces I use.? I then use a mock class framework (Mockito) to simulate responses.? This of course doesn't simulate TWS, but at least allows for testing of my code.

But sometimes weird things happen in production and TWS doesn't do what you expect, so an idea I had was to implement a TWS API "recorder" so I can record/replay IB API events to diagnose problems that come up in production.? Then use those event records to generate test cases.

Re: question about server support and login best practice

#46599

I looked bit couldn't find docker for newer version of IB Gateway / TWS and IBC (IBController).

Dmitry has a great blog on this but it's not a quick deploy docker image but is nicely detailed.

Show quoted text

On Sat, Feb 20, 2021, 1:37 PM Alex Gorbachev <ag@...> wrote:

I think you should look at IBController -?
Official from IBRK.

Or check a docker image like this -?

I took the latter and adapted for me.
I also managed to hack my way into running IB gateway on Google Colab to experiment.

Alex

On Fri, 19 Feb 2021 at 07:37 Stuart Cracraft via <smcracraft=[email protected]> wrote:
My sens is that the current nature of IB is to provide two tools which either can be run.

One is a headful client cumbersmoe and poor UI called TWI and the other is a thinner client.

But that NEITHER of these can be programmed to?

?a) autostart at system boot (I run my clients on AWS and need absolutely no U/I.
? ? ?It is all done in CLI, with NO DESKTOP, to AUTOMATE it.

?b) NEVER require human interaction for the server to authenticate and login to the
? ? ?IB sservers to make paper and cash trades

?c) avoid a “Sunday required U/I human login” - what is that? It went out in the 1970’s…

My sense is that IB is backwards on the above, does not provide a mechanism for
serverless auto-login password WITHOUT mouse, keyboard, etc. but potentially
with the account + password sitting in a file, run automatically by a Unix/Linux (I use
Ubuntu) daemon, and doing the automation needful with HANDS-OFF.

Please correct me if my above perception is correct or incorrect and then the next question
is why possibly IB would not correct the above and take the final step to automate it all
the way out. This is supposed to be a platform to automate both paper and cash trading
to fully remove emotion and human activity from most market decisions about what to buy
and sell, when to do so, for improved performance.

The best analogy is , , and other platforms
which do ALL the selection and analysis robotically, the rebanacing robotically, and minimize all
human error, as the markets are too error-prone for humans to be in may confer extra security
in situations requiring robotic, boring, precise analysis and actions taken.

Pardon for the length of this question but I never got what think is a bottom line answer from
this list for the above

Stuart Cracraft

P.S. When replying to this email, please ensure you cc my personal email at smcracraft@...

And thanks for taking your time.

Re: Unable to fetch fundamental data

#46598

Hi Mark,

Message code 430 clearly means "No fundamental data available". This means there is absolutely nothing wrong with the code.?

However, it is intriguing how GOOGLE's fundamental data can be unavailable. I also received the same error for GOOG. While AMZN and other few stocks return data with same script.?

Thanks,
Pratik

Re: CI-testing anyone?

#46597

My CI testing follows what Mark outlines.??
My code is in Java.? I wrote a broker interface which limits the IB API pieces I use.? I then use a mock class framework (Mockito) to simulate responses.? This of course doesn't simulate TWS, but at least allows for testing of my code.

But sometimes weird things happen in production and TWS doesn't do what you expect, so an idea I had was to implement a TWS API "recorder" so I can record/replay IB API events to diagnose problems that come up in production.? Then use those event records to generate test cases.

Re: question about server support and login best practice

#46596

开云体育

how is your docker image working out for you Alex? GREAT intel by the way. I was wondering about this and the github is EXCELLENT!! thanks so much

On 2/20/21 12:36 PM, Alex Gorbachev wrote:

Show quoted text

I think you should look at IBController -?

Official from IBRK.

Or check a docker image like this -?

I took the latter and adapted for me.

I also managed to hack my way into running IB gateway on Google Colab to experiment.

Alex

On Fri, 19 Feb 2021 at 07:37 Stuart Cracraft via <smcracraft=[email protected]> wrote:

My sens is that the current nature of IB is to provide two tools which either can be run.

One is a headful client cumbersmoe and poor UI called TWI and the other is a thinner client.

But that NEITHER of these can be programmed to?

?a) autostart at system boot (I run my clients on AWS and need absolutely no U/I.

? ? ?It is all done in CLI, with NO DESKTOP, to AUTOMATE it.

?b) NEVER require human interaction for the server to authenticate and login to the

? ? ?IB sservers to make paper and cash trades

?c) avoid a “Sunday required U/I human login” - what is that? It went out in the 1970’s…

My sense is that IB is backwards on the above, does not provide a mechanism for

serverless auto-login password WITHOUT mouse, keyboard, etc. but potentially

with the account + password sitting in a file, run automatically by a Unix/Linux (I use

Ubuntu) daemon, and doing the automation needful with HANDS-OFF.

Please correct me if my above perception is correct or incorrect and then the next question

is why possibly IB would not correct the above and take the final step to automate it all

the way out. This is supposed to be a platform to automate both paper and cash trading

to fully remove emotion and human activity from most market decisions about what to buy

and sell, when to do so, for improved performance.

The best analogy is , , and other platforms

which do ALL the selection and analysis robotically, the rebanacing robotically, and minimize all

human error, as the markets are too error-prone for humans to be in may confer extra security

in situations requiring robotic, boring, precise analysis and actions taken.

Pardon for the length of this question but I never got what think is a bottom line answer from

this list for the above

Stuart Cracraft

P.S. When replying to this email, please ensure you cc my personal email at smcracraft@...

And thanks for taking your time.

For tickByTickBidAsk, does SMART exchange provide THE best rate available?

#46595

If fetching live market depth level 2 tick by tick data for a stock, does setting SMART for the exchange provide THE cheapest rate available in the market or should one consider querying all the exchanges at the same time and find the cheapest available price because SMART doesn't always provide the cheapest price available?

I have the following assumptions:
- For orders, SMART means either the best rate available (in case of market order) OR the fastest execution available (in case of Limit order) which may not be the cheapest available at that moment. So, IBKR algorithm for SMART may work it's best depending on the order type.

But for tickbytick request, does something like above apply or SMART is simply the cheapest available price available and there are no other options because this is just a data request and not an order. If there are other ways SMART works for providing tick by tick data, please explain.

Thanks,

Re: Getting invalid prices by following the relevant market rules.

#46594

Sorry for the double post but I wanted to let you know that I finally found the cause of this issue. It was related to the added by me in Configuration > Presets > Stock > WMT record that in some strange way interferes with the buy limit orders done by the API for WMT stock but not for the orders done by hand from the TWS. After removing this preset everything started working as expected. Unfortunately I have no time to figure out which parameter exactly was the issue but in case someone else have the same problem: you have to check also the presets in the configuration. Even though the logs in both sides show no problem with the requested price, the error message is misleading and must include information what exactly is wrong with the requested limit price.

Re: question about server support and login best practice

#46593

I think you should look at IBController -?

Official from IBRK.

Or check a docker image like this -?

I took the latter and adapted for me.

I also managed to hack my way into running IB gateway on Google Colab to experiment.

Alex

Show quoted text

On Fri, 19 Feb 2021 at 07:37 Stuart Cracraft via <smcracraft=[email protected]> wrote:

My sens is that the current nature of IB is to provide two tools which either can be run.

One is a headful client cumbersmoe and poor UI called TWI and the other is a thinner client.

But that NEITHER of these can be programmed to?

?a) autostart at system boot (I run my clients on AWS and need absolutely no U/I.
? ? ?It is all done in CLI, with NO DESKTOP, to AUTOMATE it.

?b) NEVER require human interaction for the server to authenticate and login to the
? ? ?IB sservers to make paper and cash trades

?c) avoid a “Sunday required U/I human login” - what is that? It went out in the 1970’s…

My sense is that IB is backwards on the above, does not provide a mechanism for
serverless auto-login password WITHOUT mouse, keyboard, etc. but potentially
with the account + password sitting in a file, run automatically by a Unix/Linux (I use
Ubuntu) daemon, and doing the automation needful with HANDS-OFF.

Please correct me if my above perception is correct or incorrect and then the next question
is why possibly IB would not correct the above and take the final step to automate it all
the way out. This is supposed to be a platform to automate both paper and cash trading
to fully remove emotion and human activity from most market decisions about what to buy
and sell, when to do so, for improved performance.

The best analogy is , , and other platforms
which do ALL the selection and analysis robotically, the rebanacing robotically, and minimize all
human error, as the markets are too error-prone for humans to be in may confer extra security
in situations requiring robotic, boring, precise analysis and actions taken.

Pardon for the length of this question but I never got what think is a bottom line answer from
this list for the above

Stuart Cracraft

P.S. When replying to this email, please ensure you cc my personal email at smcracraft@...

And thanks for taking your time.

Re: Unable to fetch fundamental data

#46592

Here is the error...

? ? ?0? 1 2? ? ? ? ? 3 4 5? ? 6? ? ? ? ? ? ? ? ? ? ?7

0? 555? 0? ? 208813720? ? ? 0.0? [ISIN=US02079K1079;]

208813720

were at the loop for saving data

press enter to continueError:? -1? ?2106? ?HMDS data farm connection is OK:fundfarm

Error: Error:? -1? ?2106? ? HMDS data farm connection is OK:fundfarm-1

?Error:? ?-1? 2106? ? 2106? HMDS data farm connection is OK:fundfarm

? HMDS data farm connection is OK:fundfarm

Error:? 999? ?430? ?The fundamentals data for the security specified is not available.failed to fetch

Unable to fetch fundamental data

#46591

Does anyone know why I'm getting an error unable to fetch fundamental data. This is the contract ID for google stock.

def getnasdaqOPThistdata():
    for index, row in DFcontractdata.iterrows():
        contract = Contract()
        contract.secType = 'STK'
        contract.exchange = "SMART"
        # contract.conId = row[3]
        contract.conId = 208813720
        print(contract.conId)
        # We are separating out the historical data requests by class to make it easier to identify the type of data coming back from IB.
        # app1.reqHistoricalData(row[3], contract, "", "1 D", "1 min", "BID", 1, 1, False, [])
        # app2.reqHistoricalData(row[3], contract, "", "1 D", "1 min", "ASK", 1, 1, False, [])
        # app3.reqHistoricalData(row[3], contract, "", "1 D", "1 min", "BID_ASK", 1, 1, False, [])
        # app4.reqHistoricalData(row[3], contract, "", "1 D", "1 min", "TRADES", 1, 1, False, [])
        app1.reqFundamentalData(999,contract,reportType='ReportSnapshot',fundamentalDataOptions='')

Re: CI-testing anyone?

#46590

Isolate the two parts, exchange connectivity is one part, and actually has a quite finite list of tests. The data processing and historical replication is a different problem that can be solved independent of the tws implementation.

Best wishes,

M

Show quoted text

On Fri, 19 Feb 2021, 22:00 Nick, <news1000@...> wrote:

I hate to say it but to do a robust job I think you would have to have a
whole separate project that simulates TWS and the exchanges. You then
connect to that for (most of) your testing.

Anything your framework does that depends on market actions (order
fills, account updates, recovery from outages, etc.) won't have a way to
be repeatable in a paper account.

Of course you'd also want to test against a paper account to verify that
basic request and response processing is working.

This could turn out to be one of the situations where the testing
framework is a much bigger project than the original project.

On 2/19/2021 4:33 PM, Matthias Frener wrote:
>
> Currently we look for a solution to run automated CI-testing.
>
> The setup is no issue, but against what do should we test?
>

Re: question about server support and login best practice

Stuart Cracraft

#46589

开云体育

Yes, fwogy.com is what I have on AWS already.?

It’s just so clumsy to have to login to an app manually.?

Awful restriction from interactive brokers.?

Grotesque.?

Show quoted text

On Feb 19, 2021, at 5:46 AM, ds-avatar <dimsal.public@...> wrote:

?Many IBKR API complications feel a bit frustrating and long in the tooth indeed. I see little reason to speculate why it's so and when it's about to change, even though I sure wish it did. If you need free solutions, I believe other participants could offer advice regarding third-party automation tools. If you are ready to pay, you can just set up a virtual desktop in AWS and run TWS or IBG permanently or through weekdays, and even accept outside client connections into it. That will still require logging in once a week, and not much else beyond that. I believe depending on your requirements this would cost about $1000-2000 per year with a modestly equipped AWS instance.

Re: CI-testing anyone?

Nick

#46588

I hate to say it but to do a robust job I think you would have to have a whole separate project that simulates TWS and the exchanges. You then connect to that for (most of) your testing.

Anything your framework does that depends on market actions (order fills, account updates, recovery from outages, etc.) won't have a way to be repeatable in a paper account.

Of course you'd also want to test against a paper account to verify that basic request and response processing is working.

This could turn out to be one of the situations where the testing framework is a much bigger project than the original project.

Show quoted text

On 2/19/2021 4:33 PM, Matthias Frener wrote:

Currently we look for a solution to run automated CI-testing.

The setup is no issue, but against what do should we test?

CI-testing anyone?

Matthias Frener

#46587

Hi all.

We are a group of developers working on an TWS API client in Typescript for node.js
(if you do node.js Apps, might be worth having a look??;)).

Currently we look for a solution to run automated CI-testing.

The setup is no issue, but against what do should we test?

Do you use your personal paper trading account on the CI? Maybe there is a way to get a second, no-expiring paper account? Or have a "dev" account type, that cannot trade live at all but insterad create 100 paper trade account for CI-servers?

How are you doing this?

Thanks for any hints!

?