开云体育

On Sat, Oct 28, 2023 at 08:05 PM, John wrote:

or maybe better use a semaphore or lock to do things properly, as copying the list won't prevent another potential conflict when resetting the list to list()

Yes, I'm not sure if a deep copy alone will suffice. But... this is what I meant by "make the smallest possible (but complete) standalone unit test". The test should focus on experimentally confirming your uncertain suspicions; it's a judgement call. Naturally there's no need to confirm what you already know.

So, I'd try to make the smallest possible (but complete)?standalone unit test?in order to really isolate the issue.

from time import time
import pandas as pd

a = list()
a.append((time(), '1', '2', '2'))
a.append((time(), '1', 0, '3'))
a.append((time(), '1', '1'))
a.append(( 0, 0, None))

df = pd.DataFrame(a, columns=['time', 'col1', 'col2', 'col3', ])
print(df)

anyway, in this case, i would even wonder what is the real value of market depth data on forex market.

BTW, since you mention this:

On Sat, Oct 28, 2023 at 08:20 AM, John wrote:

Essentially if price=0 and size=0 it implies operation=2, and with regards to insert or update, it doesn't make any difference to the analysis I am performing. Considering the amount of data to be stored on multiple FX pairs, I would rather not have a redundant operation column even if only uint8.

The other thing is, if it was related to the operation field, then we would not have any discrepancy between "bid" and "position", but we have bid/size 2151 825 and?position 2151 830 which is rather confusing.

I won't comment as to whether or not your deduction here is correct or not.

But, I will say that it seems you may be relying on too many assumptions. After all, it's been said that "premature optimization is the root of all evil (or at least most of it) in programming".

Therefore, I would suggest you try two things... 1) make your implementation as "correct" as possible according to the API specification & documentation without taking any shortcuts based on assumptions and 2) continue to use ib_insync as a comparison to see if you can reproduce the problem there.

Finally, since code interactions can get quite complicated, copying around snippets that are surrounded by ellipsis will only go so far. So, I'd try to make the smallest possible (but complete) standalone unit test in order to really isolate the issue.

Having that will also make it easy to present an argument to support people. They can then confirm the existence of bug(s) for themselves. Unlike the justice system, I think any application that exhibits "strange behavior" is presumed guilty until proven innocent (instead of the other way around).

Once this is done it should become very clear where the problem lies. And, you'll be able to go back and optimize more systematically.

1. Start the local Gateway API Java server
2. Open ?and login with DEMO account.
3. Check??to see the session is good.
4. Check ?and I get the following

You need to take a look at the discussion of pacing violations in the TWS API Guide section on , HHC. While IBKR has relaxed the pacing rules for bars that are wider than 30 secs, a few rules still exist.

• I received between 630 and 960 bars for each stock (I ran with rthOnly = false).
• Requesting and receiving data for 500 instruments took 26 seconds (an average of 52ms per instrument).
• It might take a little longer in the middle of a heavy trading day but you should be able to get data for 100 instruments in less than 10 seconds.

• Thread 1 is the main application thread that manages and makes all requests
• Thread 2 is the EReader thread that reads bytes from the socket connection with TWS/IBGW, assembles them into messages, and puts the messages into a queue. Each of these messages will eventually become an API callback. This thread is created automatically by TWS API.

• Many Python examples use only these two threads so that the Thread 1 executes the EClient code, reads the messages, and makes the required callbacks itself. In Java and some of the other languages, EClient code runs in a third thread so that API callbacks are made in parallel to any application code or API requests.

• After connecting with TWS/IBGW, the main thread works down the list of instruments it needs to request data for, creates a synchronization object (more below) and calls
• Once it reaches a certain number of outstanding requests (say 25) it waits (not sleep!) until one or more of the outstanding requests are complete. It immediately makes a request for an additional instrument for each completed one.

• Eventually, requests have been made for all instruments and the main thread waits (again not sleep!) for all still remaining requests to complete.

In Java we use make and return to the main thread one for each outstanding request. Each outstanding request proceeds at its own pace, collects the data it received through a series of callbacks in a list for that instrument. Once the callback arrives for an instruments, the for that instrument is marked "complete" and the list of bars is sent via the to the main thread.

In case the request for an instrument receives an error callback, the is marked completeExceptionally with the cause of the error.

The main thread uses the function to wait for outstanding requests to complete and instantaneously sends additional requests until all data has been received.

In Python you would use a synchronization feature from the asyncio package, such as (that are modeled after the Java Future class) to inform the main thread about completed requests.

Hope this helps,

闯ü谤驳别苍

Glad this helped and you are getting closer.

Pointing a browser to and logging in with your paper account credentials should work. At least it does for me Have you tried to log into the Client Portal or TWS/IBW with the paper account credentials?

When did you try this? IBKR is doing some major system maintenance this weekend and we had some outages already during the Friday night maintenance window. I have seen the error message "Invalid username password combination" for legitimate credentials in the past during times when IBKR maintains or resets the main servers.

Whatever works for your live account should work for the paper account/

闯ü谤驳别苍

Can you add 'operation' in your csv ?
I assume your 'bid' = IB 'side', correct ?

It is MANDATORY to use operation? to manage and ensure the integrity of your entries, otherwise you might very well end up receiving request to update a row you didn't inserted. Or insertion or a row you never removed (duplicate), doesn't explain your figures but area to consider
because what is surprising is to see entries with price =0 and volume = 0.

np.savetxt("series_output_1.csv", app.series[reqId], delimiter=",", fmt='%s')

I have only limited hands-on experience with the Client Portal API (it does not yet add anything that we can't do via TWS API already), but I think you have mixed up a couple things. If you have not done so yet, take a look at the and the explorer.

• REST requests through one of several API endpoints. All endpoint URLs start with "https://localhost:5000/v1/api/"

• Requests for WebApps that you develop and install in the folder "root/webapps". URLs for WebApps start with "https://localhost:5000" followed by the WebApp name. The gateway simply serves those files to your browser.

• root/webapps/demo/index.html

• root/webapps/demo/gateway.demo.js

The message you get indicates that the gateway is ready to server the WebApp "demo". You start it by pointing your browser to "https://localhost:5000/demo", but the gateway has to be logged into an IBKR account for it to work. In case you have a funded live account at IBKR, you can create a companion "paper" account to try out your strategies.

I do not think that it is possible to connect a WebApp through the Client Portal API gateway to what the TWS application calls a "demo" account. The kind off account you can reach through the TWS application login screen where it says "No username? Try the demo".

Hope that helps,

闯ü谤驳别苍

No intent to be pedantic but when you say "doesn't work" , you can guess that some people are able to make it working, so why don't tell us a little bit more ?
what did you do ? what did you expect ? what happens ?, etc ...

Did you read this one ?
and this one ?
A clue, a typical area of concern for the REST API that need a work/trial? investment: The authentication method.
It require great care for very legit reason, this is the REST API I know which are able to handle potentially the biggest amount of money I ever seen

def updateMktDepthL2(self, reqId, position, marketMaker, operation, side, price, size, isSmartDepth):
    self.series[reqId].append((time(), side, position, price, size))

...
if len(app.series[reqId]) >= number_of_rows_threshold:
    print('saving')
    try:
        df = pd.DataFrame(app.series[reqId], columns=['time', 'bid', 'position', 'price', 'quantity'])
        app.series[reqId] = list()
    except ValueError as e:
        print('Value Error', e)
...

df.to_parquet(parquet_file_name, compression='gzip')

• time 2151751
• bid 2151825
• position 2151830
• price 2151876
• quantity 2151876