Very nice, fordfrog, and it all makes a lot of sense to me. I think our approaches are identical on the "what" level, but there are minor implementation differences on the "how" level.
For example, as part of my general client connection logic, I already start a heart-beat thread (independent of MarketDepth feeds) that monitors the connection with TWS/IBGW by issuing a request once a second. That means I always have at least one response message from TWS/IBGW every second that can be used for MarketDepth update transaction boundary detection. That heart-beat thread:
- locks onto a specific offset from the top of the second (say 100ms) and monitors how precisely it gets activated. That is a good measure for how responsive (or possibly overloaded) the client is.
- makes sure every request gets an callback (e.g. TWS/IBGW are indeed responsive)
- makes sure the returned server time is identical to the one expected
- measures and monitors the response time to determine how responsive (or possibly overloaded) TWS/IBGW are
There can be only one outstanding request per connected client (as you found out) since it is one of the few requests that have no reqId. But I am digressing.
The approach you have outlined is based on only one (well informed) assumption: TWS/IBGW do not split market book update transactions. In other words, the sequence of MarketDepth messages that make up one update will not be interrupted by one or more other TWS API messages. There is a second (also well informed) assumption that could enable us to detect even more transaction boundaries (sub-second), but it may need some more thought:
- A market book update will often consist of changes for only a subset of all positions, and may sometimes even be limited to a single side (Buy or Sell).
- But? updates (operation==1) and inserts (operation==0) do take place in numerically increasing position order (first side 0, ..., n then possibly other side 0, ..., m) and no position is updated more than once between two transaction boundaries. Deletes (operation==2) are performed in reverse numeric position order.
With that in mind, it feels safe to declare additional transaction boundaries like this:
- When a new MarketDepth message arrives, check whether the new message and the last message in the queue of uncommitted MarketDepth messages have the same operation ("update") and operate on the same side (Buy or Sell).
- If not, check whether the new MarketDepth message and first message in that queue are both "update" operations for the same side.
- A new transaction boundary exists before the new MarketDepth message iff:
- one of those two checks passes
- and the position in the new message is equal to or smaller than the position in the queued message
This holds true at least for the CME and CBOT based futures I have been looking at for a long time. Maybe you can check whether that would work for FOREX, too.
Well done again,
´³¨¹°ù²µ±ð²Ô
On Thu, Nov 16, 2023 at 01:47 PM, fordfrog wrote:
@´³¨¹°ù²µ±ð²Ô
i mostly finished my implementation and i'm just reviewing the code, probably for the last time. in the text below i'll use word commit?to indicate the supposed end of transaction (end of market depth update items batch/transaction). by transaction?i mean supposed transaction boundary, end of the batch that creates a consistent state of the book. commit item is an instruction to issue a commit for market depth updates. market depth item is the update message received. market depth (message) processor is a class running in a separate thread and handling all market depth messages (updates and book reset). commit callback is a method defined on the listener interface and is used to pass the commit information to the listener. commit message generator is a class running in a separate thread and handling requests for commit messages, eventually sending a message to tws/gw to trigger some response.
i did it the same way as you did, that is an incoming message from a different topic commits the updates from the market depth request id that was processed the last.
i also implemented some additions:
- if a market depth update to a different request id comes, it commits previous request updates (you probably have this coded too as it seems a logical conclusion from your observations). (it is important to check the request id against the last request id added to the queue, not the last request id processed, and queue the commit item before the market depth item?just being added.)
- i did not implement triggering commit from book reset message as the state before the reset does not have to be at the end of a transaction.
- when processing the queue, market depth update sets uncommitted request id, and commit item commits the last uncommitted request?and resets the uncommitted request id.
- when commit occurs, it always commits the last market depth request if it is not committed yet. in the market depth processor i can have only one uncommitted request because (1) a different request id triggers a commit?and/or (2) a different message triggers a commit. so there can only be zero or one uncommitted request in the processor.
- before the commit callback is called, i also check whether there are more items in the queue with the same request id (i set parameter hasMoreItems?on the callback to true on first occurrence of the same request id in the queue, it's just an on/off flag, no need to report how many items are in the queue). this is to inform the listener that though the market depth data might be in a consistent state, more updates arrived in the meantime which makes the data obsolete.
- if market depth processor queue empties without issuing a commit (the last item retrieved from the queue before the queue becomes empty is not a commit item), a request for commit message is issued (more on that below). this i need in cases where there is no other traffic as i would otherwise get no commits. this would happen in my automated test of market depth as i don't issue any other request in parallel so i would get no commits which would cause the test to fail.
commit message request?is a message being sent to tws/gw to trigger a response message. here are the details:
- only one message per second can be triggered, even if more commit message requests are fired (in case of current time request / see below / it also happens that more frequent requests are simply ignored and not answered at all by tws/gw).
- the request is queued in the commit message generator. once an item is added to its queue, it is woken up (if waiting for a queue signal) and waits until the specified timeout (1000ms in my case) elapses. then the queue is inspected, the queued market depth message processor is asked whether it still needs a commit (for details see below), and if so, a request is sent to tws/gw.
- market depth message processor still wants a commit if the last request id was not committed and the queue is empty (the request came after the queue was emptied) or there was no commit yet (there are some items in the queue but there was so far no commit) or the last commit was triggered longer than a specified interval ago (in my tests 100ms - so there are still items in the queue, we already had at least one commit, but it's a long time since the last commit). this way relatively frequent commits should be triggered. the commit callback though still depends on messages incoming for other topics or a different request id.
- the commit message request?itself is issued only after the queue gets empty. it's just a request though, not sending of a message, that is handled by commit message generator.
- as the commit message request, i tried request for current time and request for managed accounts. both behaved the same and had pretty much the same response times. i'm really not sure that the request for current time is being sent to ib servers, i rather think that tws/gw syncs the time occasionally from ib servers and the response is generated directly from gw/tws (at least this is how i would probably code it myself). but the response times for managed accounts were pretty much the same. so either managed accounts are cached on login and never requested again (more probable), or both the requests are sent to ib servers (i doubt as the response times were pretty short, as short as 8ms what i've seen, and my gw is running on a server several hops away from my laptop where i ran the tests which adds some delay). anyway, i just tried to use a cheap request/response message. still, i saw a cases when i issued current time request, and then i received several market depth updates before i got the current time response (also occurs in the attached log). in most cases though there were no messages between the request and response. my conclusion is that at least the response is placed at the end of the tws/gw output buffer (towards the client). so the response (1) imo marks the end of buffered messages in tws/gw and (2) triggers commit in my code. also, if tws/gw would receive all updates as a single batch, it would also mark the end of the batch (as it would be placed at the output buffer?as a single unit), but this is an if.
i'm also attaching the log from my test case which shows the behavior. in the process, market depth request is issued, then the listener waits for 5 commits and cancels the request. there are no messages omitted (except two "error" messages at the beginning reporting market data farm statuses). logging times are not that important, more important are times in the square brackets just after the message type name, those indicate the timestamp when the message was sent/received. content of messages is displayed in standard ib format.
i hope it all makes some sense. i tried to be both as clear as possible and also as brief as possible, but i'm not sure i achieved that...
