I added some code so that my code calls reqCurrentTimeAsync every few seconds. I use it to make sure the network is properly connected, so this should be fine.

Yeah, my code was locking up periodically and it took me a while to realize that it was because sometimes reqCurrentTimeAsync never received a response.

Although I can't find it in the document, I believe IB may has restrictions no to have two calls within a second, similar to this:

I am having some very strange problems with reqCurrentime().

Look at the below script, which simply gets the current time and the remote time, compares them, waits for a little bit, and then tries again.

async def main2():
    N = 300
    times = np.zeros(N)
    ib = ibs.IB()
    await ib.connectAsync("gateway", 4004, clientId=20, timeout=60)

    for i in range(300):
        local_tm = datetime.datetime.now(datetime.UTC)
        try:
            server_tm = await asyncio.wait_for(ib.reqCurrentTimeAsync(), 10.0)
        except TimeoutError:
            print(f"{i} :Timeout!")
            times[i] = 1.0
        else:
            times[i] = (server_tm - local_tm).total_seconds()
        t2 = datetime.datetime.now(datetime.UTC)
        processing = (t2 - local_tm).total_seconds()
        await asyncio.sleep(0.5)
        print(f"{i} : diff = {times[i]:.2f} sec, processing = {processing:.2f} sec")

    print(f"Avg = {np.mean(times)}, std = {np.std(times)}")

asyncio.run(main2())

The strange thing here is that if I await asyncio.sleep(1) or longer, the program works more or less as expected. Below is an example of such normal output:

0 : diff = -0.88 sec, processing = 0.00 sec
1 : diff = -0.88 sec, processing = 0.00 sec
2 : diff = -0.88 sec, processing = 0.00 sec
3 : diff = -0.88 sec, processing = 0.00 sec
4 : diff = -0.89 sec, processing = 0.00 sec
5 : diff = -0.89 sec, processing = 0.00 sec
6 : diff = -0.89 sec, processing = 0.00 sec

However, once I await asyncio.sleep(0.5) or anything below 1, I see something as follows:

0 : diff = -0.94 sec, processing = 0.00 sec
1 :Timeout!
1 : diff = 1.00 sec, processing = 10.01 sec
2 : diff = -0.95 sec, processing = 0.00 sec
3 :Timeout!
3 : diff = 1.00 sec, processing = 10.01 sec
4 : diff = -0.97 sec, processing = 0.00 sec
5 :Timeout!
5 : diff = 1.00 sec, processing = 10.01 sec
6 : diff = -0.98 sec, processing = 0.00 sec
7 :Timeout!
7 : diff = 1.00 sec, processing = 10.01 sec
8 : diff = -1.00 sec, processing = 0.00 sec

In other words, reqCurrentTimeAsync() times out every other time, even though I have set the timeout to be 10 seconds! If I wait for 0.5 second between each call of reqCurrentTime(), I am only sending 2 messages per second.

(I was debugging some strange freezing up in production and finally produced this minimal example to reproduce the problem.)

Does anything know what might be going on?

Yeah, this does seem new. This thread seems to have a similar symptom: /g/twsapi/topic/88795216#msg48912 .

It doesn't feel that fields are missing randomly. In the most recent instance, all the bids of a bunch of stocks are missing, no matter how long I wait or retry. Statistically, this cannot be explained by the field not being updated because there is no new data.

Being a hassle aside, I am also worried that updates for some fields may suddenly stop and cause my program to silently consume wrong data. I have never tested this possibility, but it seems possible.

I would really appreciate if people could share about the problem/solution.

Sorry for the multipe email (as I can't edit my response). Just want to add that currently the output of the script is:

Retrying... 1
Retrying... 2
Retrying... 3
Failed to get data after 3 retries
PSA PRF nan 20.91 nan 800.0 20.98
PSA PRG nan 20.63 nan 1000.0 20.63

The sequence is simply: 1. Connect to TWS 2. reqMktData() 3. Give it some time to get data. 4. If some fields are NaN or None, cancelMktData(), and retry.

Below is a minimal example. When the problem happens, the bid is NaN. (Note the bid should usually be 20 some dollars. It's not a penny option where there can be no bid.) The problem can happen during market hours. After the gateway or TWS restarts, running the client the first time is always fine. But running it a second or more times can cause this. The problem is illusive. When it doesn't happen, it tends not to happen as long as I don't restart the gateway. If it happens, then it happens almost every subsequent time until I restart the gateway.

import asyncio
import math

import ib_async as ibs

tickers_list = ["PSA PRF", "PSA PRG"]

async def subscribe(ib, contracts):
    ib.reqMarketDataType(1)
    tickers = []
    for c in contracts:
        ticker = ib.reqMktData(c, "456,375")
        tickers.append(ticker)
        await asyncio.sleep(0.1)
    return tickers

def is_ok(ticker):
    attributes = ["bid", "ask", "bidSize", "askSize", "close"]
    def is_null(x):
        if isinstance(x, float):
            return math.isnan(x)
        return x is None

    return all(not is_null(getattr(ticker, attr, None)) for attr in attributes)


async def main():
    ib = ibs.IB()
    await ib.connectAsync("gateway", 4004, clientId=1, timeout=60)
    contracts = list(ibs.Stock(sym, "SMART", "USD") for sym in tickers_list)
    contracts2 = await ib.qualifyContractsAsync(*contracts)

    for i_retry in range(1, 4):
        tickers = await subscribe(ib, contracts2)
        await asyncio.sleep(5)
        if all(is_ok(ticker) for ticker in tickers):
            print("success")
            break
        print("Retrying...", i_retry)
        for c in contracts2:
            ib.cancelMktData(c)

    else:
        print("Failed to get data after 3 retries")
        for ticker in tickers:
            print(ticker.contract.symbol, ticker.bid, ticker.ask, ticker.bidSize,
                  ticker.askSize, ticker.close)

asyncio.run(main())

1. Connect TWS.
2. Connect client.
3. Subscribe reqMktData
4. ...

I tested it. It does seem that type 2 will give me the most recent quote and price when the market is open. However, even if I use type 2, the problem I described in the original email persisted, namely, some fields are missing if the client is not connected to the gateway the first time the gateway restarts.

Okay, MktDataType 2 may be the answer. Parts of the documentation say it is the quotes recorded at the last closing, which is not what I want, but part of it says it is the last quotes seen by their system, which is what I want. Since the market is closed, I have to find out on Monday. Thank you for your help.

There are still some pending questions that I need to investigate:

1. Does MkDataType 2 provide the last quotes recorded at closing or just the last recorded quotes?
2. Do I need to call reqMktData twice for each contract, once with type 2 and a second time with type 1, to get both the last recorded quotes and the live updates?
3. If I do this, it may mask the problem that a bid is in fact "stuck" and is never updated, which means I have to write extra monitoring code to see if quotes do update.

开云体育

开云体育

Type 2 is the last data recorded at market close, which is not what I want.

Also, "live" does not usually just mean that I will only get the data when there are new updates. It also means that I will get the most recent quote when there are no new updates.

If it were to only get the data when there is an update, then all the snapshot functions wouldn't work. Also, it is not really consistent with my experience. I find that sometimes it is missing a bid and sometimes the close. Note that a market maker rarely only send the ask and not the bid when updating their quotes. Missing the close is even harder to explain. The fact that I never experienced this on the initial connection to a freshly started gateway also means that this is perhaps not the reason.

I am having an odd problem. Everything works as expected when my client makes an initial connection to the gateway. However, if I restart the client process, sometimes reqMktData()won't get all the fields. Specifically, if I do the below, I find everything in the ticker object are filled except for bid and bidSize, which are nan. Sometimes, it is the close price that is nan.

ticker  = ib.reqMktData(contract)
asyncio.sleep(0.5)
# ticker.bid, ticker.bidSize, ticker.close may be nan

I have logic to retry the subscription as follows, but it rarely does anything:

ib.cancelMktData(contract)
asyncio.sleep(0.1)
ib.reqMktData(contract)

When this happens, the same few contracts out of 30 have this problem no matter how many times I retry (like the above). The other contracts are completely fine.

This does not happen every time. Occasionally, restarting the client to make a new connection the second time is okay, and when the second time is okay, it tends to be okay for subsequent connections. Over 50% of the time, however, making the connection the second time will have the above-described problem. If the second time fails, all subsequent connections will also fail.

The problem is not limited to the gateway. Connecting with TWS has exactly the same problem.

Restarting the gateway fixes this problem. I have never seen this happening to the initial connection to the gateway instance. This seems to say that the gateway has to be restarted every time I restart my client, which is tedious.

Has anyone encountered similar issues?

开云体育