To be clear the protocol I'm talking about is between TWS and the API client. I have not looked at the protocol between TWS and IB's servers.

The easiest way to see what's going on is to look at the source code that IB supplies for its libraries. Pick whatever language you like and go through the code (it's what I did).

In my view IB's libraries are unnecessarily complicated and convoluted but the most of the action is in what they call a reader which reads the socket and gathers the fields. You will see that it's just a bunch of null-terminated ascii fields going over the wire.

There is similar code that gathers the arguments from client requests and sends the fields over the socket.

The other part is the logon sequence. The last time I looked they had it scattered in a bunch of pieces but it's just sending a few fields back and forth to make the connection to TWS. You probably won't need a packet sniffer at all.

Hope this helps.

On Fri, Jul 30, 2021 at 06:15 AM, dent wrote:

ed about the IB wire protocol but never

Don't think it's documented somewhere.

For a Typescript port I have reverse engineered it form the Java code. The protocol level would be this:
?
socket.ts handling TCP connection and version handshake.
decoder.ts handling de-serialization from tokens.
encoder.ts handling serialization to token.
controller.ts doing queue handling (stuff starting/stopping queue on connect/disconnect or enforcing the rate limit and such).

But better to use the official Java codes a reference, Typescript might just be easier to read to get an idea.

?

I always wondered about the IB wire protocol but never took the time to look into it. Could you tell us where it's documented please? Also what do you use as a protocol analyzer?? Does wireshark have a plugin?

thanks for any help

It's not actually the API that has the architecture you describe - it's IB's libraries.

TWS itself exposes the API through a single socket. Several of us don't use IB's libraries at all and wrote our own direct socket connection to TWS. You can then make any architecture you want.

The wire protocol is simple so it's not a huge project but of course it is extra work and you will have to maintain the code from time to time as features change in the API.

Agree with Graham.

On UI it won't matter if it is 1ms late, you will not notice it. If you update tick values at 1ms rate (1000fps) this is like a Hollywood movie (24fps) at 41 times normal framerate. Your eye can't see that as number anymore, but it will be some blurry thing (I do 50ms refresh rate on UI which would be a 20fps and feels like way enough).

On App layer, queues actually are good. They make thing lot easier.
Like most of the UI frameworks are not multi-threaded. They want to you do modify UI from UI thread and not from some other (think even Qt ends up sending update messages over to UI thread), as you don't need resource locking on UI thread if you enforce that (brings a huge performance boost to UI thread).

Hey,

If your purpose is to just to display market/order data on a gui then I
wouldn't worry too much about straight line speed. GUIs don't have to
be that fast as the humans reading them aren't that fast. Rendering
the data will be the slowest thing. The queuing approach you've
described seems good to me. If you're only going to have a single GUI
client per TWS then great, although if you wanted to fan out a single
data feed to multiple consumers then you'd need a publisher of the data
as a separate process then manage the multiple subscriptions.

I think you'll be limited in performance only by TWS. It's not a high
speed solution that could support a lot of connections. You'd want to
use FIX I suspect.

If you don't need to scale, then connecting things with buffers and
blocking threads is good enough.

Not sure exactly what you're trying to do but that's my view on what
you've described.

HTH,
G.


On Thu, 29 Jul 2021 21:19:06 -0400
"Eduardo Esteva Kremer" <eekremer@...> wrote:

I am planning on adding a GUI like Qt to the TWS API in C++.

In a nutshell, I see this API has 2 threads: one for receiving
messages (the reader) and another for processing them (the client).
These 2 threads connect with each other through a queue by using
signal/wait.

Instinctively, I would tend to connect the GUI to the *client thread*
because it actually decodes messages and invokes the corresponding
callbacks according to the type of message received. However, because
messages must go first through the queue (signal/wait) I foresee some
delays which could impact the display of real-time data on the GUI.

Another option is to connect the GUI directly to the *receiving
thread*, so data goes straight to the GUI code (no queue, waits...in
between). But this option would force us to call TWS API's decode
functions and callbacks from this thread itself. And this seems a
little unnatural.

Any comments greatly appreciated !!

Ed Kremer

?

... but I am not sure that this group is the best place for you to publish your progress and code. The group has 6,000 members and many (most ?) have years of experience in automated trading and do not use Python.

?

But there is a large audience that would greatly benefit from your templates and they are not members of this group. GitHub or similar code repository would be a much better home for your contributions. That way, anyone who is interested can browse all your templates in one place, you can collaborate with others, the community can help you improve your work, and those artifacts that really make your code valuable (such as documentation, howto guides, unit tests, and version control since something will change and improve) can be stored in one place.

?

None of this can effectively take place within a group messaging environment such as [email protected].

?

Just a thought, and I might be wrong. Unlikely, but possible

?

JR

?

Here is a template for creating 2 indicators at the same time on tick-by-tick data, so that you can use the 2 indicators for crossover strategies, etc.? Feel free to contact me directly with any questions or if you need any customized templates.? Happy to walk you through the code.??

Regards,

Javed (ebtrader)

import pandas as pd
from ibapi.client import EClient
from ibapi.wrapper import EWrapper
# types
from ibapi.common import *  # @UnusedWildImport
from ibapi.contract import * # @UnusedWildImport
from ibapi.order import Order
import datetime
from finta import TA

MOVING_AVG_PERIOD_LENGTH = 3
MOVING_AVG_PERIOD_LENGTH_1 = 5
TICKS_PER_CANDLE = 4

class TestApp(EWrapper, EClient):
    def __init__(self):
        EWrapper.__init__(self)
        EClient.__init__(self, wrapper=self)
        self.contract = Contract()
        self.data = []
        self.data1 = []
        self.data_counter = 0
        self.data_counter1 = 0
        self.mov_avg_length = MOVING_AVG_PERIOD_LENGTH
        self.mov_avg_length1 = MOVING_AVG_PERIOD_LENGTH_1
        self.ticks_per_candle = TICKS_PER_CANDLE
        self.tick_count = 0
        self.indicator = 0
        self.indicator1 = 0

    def nextValidId(self, orderId: int):
        # we can start now
        self.start()

    def start(self):
        self.tickDataOperations_req()
        # self.accountOperations_req()
        print("Executing requests ... finished")

    def running_list(self, price: float):
        self.data.append(price)
        self.data_counter += 1
        if self.data_counter < self.mov_avg_length:
            return
        while len(self.data) > self.mov_avg_length:
            self.data.pop(0)

    def running_list1(self, price: float):
        self.data1.append(price)
        self.data_counter1 += 1
        if self.data_counter1 < self.mov_avg_length1:
            return
        while len(self.data1) > self.mov_avg_length1:
            self.data1.pop(0)

    def calc_indicator(self):
        df_indicator = pd.DataFrame(self.data, columns=['close'])
        df_indicator['open'] = df_indicator['close']
        df_indicator['high'] = df_indicator['close']
        df_indicator['low'] = df_indicator['close']
        df_indicator['indicator'] = TA.SMA(df_indicator, self.mov_avg_length) # choose indicator here
        self.indicator = df_indicator['indicator'].iloc[-1]

    def calc_indicator1(self):
        df_indicator1 = pd.DataFrame(self.data1, columns=['close'])
        df_indicator1['open'] = df_indicator1['close']
        df_indicator1['high'] = df_indicator1['close']
        df_indicator1['low'] = df_indicator1['close']
        df_indicator1['indicator1'] = TA.SMA(df_indicator1, self.mov_avg_length1) # choose indicator here
        self.indicator1 = df_indicator1['indicator1'].iloc[-1]

    def tickDataOperations_req(self):
        # Create contract object

        # futures contract
        self.contract.symbol = 'NQ'
        self.contract.secType = 'FUT'
        self.contract.exchange = 'GLOBEX'
        self.contract.currency = 'USD'
        self.contract.lastTradeDateOrContractMonth = "202109"

        # Request tick data
        self.reqTickByTickData(19002, self.contract, "AllLast", 0, False)

    # Receive tick data
    def tickByTickAllLast(self, reqId: int, tickType: int, time: int, price: float,
                          size: int, tickAttribLast: TickAttribLast, exchange: str,
                          specialConditions: str):
        print('Candle:', str(self.tick_count // self.ticks_per_candle+1).zfill(3),
              'Tick:', str(self.tick_count % self.ticks_per_candle + 1).zfill(3),
              'Time:', datetime.datetime.fromtimestamp(time),
              "Price:", "{:.2f}".format(price),
              'Size:', size,
              'Indicator:', "{:.2f}".format(self.indicator),
              'Indicator1:', "{:.2f}".format(self.indicator1))
              # 'Data', self.data)
        if self.tick_count % self.ticks_per_candle == self.ticks_per_candle - 1:
            self.running_list(price)
            self.running_list1(price)
            self.calc_indicator()
            self.calc_indicator1()
        self.tick_count += 1

def main():
    app = TestApp()
    app.connect("127.0.0.1", port=7497, clientId=102)
    print("serverVersion:%s connectionTime:%s" % (app.serverVersion(),app.twsConnectionTime()))
    app.run()

if __name__ == "__main__":
    main()

开云体育

When this happened to us, it was always related to our code. If you have not done so, consult Richard King's amazing ContractInspector? for a second opinion that does not use your code. I can attest the "This can be invaluable in resolving problems with the error 200 response to such calls."



JR

Hi,

Not really a technical question, but I've just been looking at
brokerage on CFDs for the UK main market in detail and the brokerage is
wider than spread in large cap stocks. Maybe that's normal, but to
someone who's worked on the institutional side, given the
internal crossing opportunities for a large broker, it seems
exploitatively high. And I'm some way off doing $10m/month in volume
(more like 5) to access the cheaper pricing tiers. Maybe the solution
is to look at mid-cap instead where the spreads are wider compared to
brokerage.

The big question though - are there viable alternatives to IB? It's
very time consuming to dig in and find the actual pricing of all brokers
operating in the UK market for large cap stocks who do DMA.

Has anyone done this? Would you be willing to share your research?

Cheers,
G.

开云体育

Just checked the recent API sources and it looks like C++ and Python libraries do set eTradeOnly and firmQuoteOnly, explicitly to "true". That explains why youy get the warnings. We use the Java library which does not do that and both are false by default.

Apparently both fields have been removed entirely from the "985.01 latest" build of the API that was released four weeks ago.

@JR

Thanks, i think the link you sent explains it. There is a box at right showing how it can be used and the way i understand it is that it does not support TIF, which i am using.?

Btw for the eTradeOnly and firmQuoteOnly, i was getting warnings about those and could not figure out why ( was not using it ), it stopped after i added these settings.

Did you check (referenced at the bottom of the document you mention) whether the exchange and routing you have asked for actually support AON orders? For example, AON is only supported for US products.

JR

PS. Not sure that it matters for your issue, but eTradeOnly and firmQuoteOnly are not supported since 983 any longer. You may want to leave them alone and use the default value the API library for your langue uses.

I am setting up Buy Order. For now, i would prefer not to deal with partially filled orders as it creates many issues. I considered cancelling partially filled order in OrderStatus, but it seems as clumsy solution, which can result in situations where next fill happens sooner than OrderStatus gets executed and cancels the order and thus it would break anyway.

I was looking for an order setting, which would either automaticly cancel the order after partial fill or allow only for filling the whole order. I found order setting allOrNone

This is code i am using for buy order

? ? buy_order = Order()

? ? buy_order.orderId = nextOrderId

? ? buy_order.action = "BUY"

? ? buy_order.orderType = buy_type

? ? buy_order.lmtPrice = buy_price

? ? buy_order.totalQuantity = units

? ? buy_order.tif = "GTD"

? ? buy_order.goodTillDate = wait_date

? ? buy_order.eTradeOnly = False

? ? buy_order.firmQuoteOnly = False

? ? buy_order.allOrNone = True