¿ªÔÆÌåÓý

ctrl + shift + ? for shortcuts
© 2025 Groups.io
Date

Limit orders, placing limit orders with a certain distance from market price

 

Hi! I'm developing using the TWS APIs an application to manage/input orders from Excel.

The problem I have now is that inputting orders manually via official TWS? dashboard seems to work "fine" for limit orders: I can, for instance, enter a limit price 10% lower than current best bid price.

Using the APIs it seems I cannot and I have to enter prices much closer to the actual market price? ( best current bid for instance)

Any help on what could be I am missing??
Thanks


MOO order "slippage"

 

I have a strategy that was backtested on daily data from Yahoo Finance, Tiingo, etc. It open/closes positions only at the market open and close.

However, this daily data represents the "consolidated" price across multiple exchanges and not that of the single primary exchange that a MOO order would be routed to. From my research, MOC orders do not have this issue--the closing auction price is almost always the same as the consolidated closing price.

1) In my backtests, I would like model the opening price "slippage", i.e. the difference in consolidated opening price vs. primary exchange opening price.?How should I go about doing this? Is 0.04% of "slippage" reasonable?? Or where can I get historical data for primary exchanges such as ARCA for backtesting purposes?

2) How should I go about reducing this "slippage" in practice? One idea I have is monitoring the ticks across multiple exchanges for the first 5 seconds at market open and then very shortly after 9:30, place a limit order for the average price of those ticks. But this has the the risk of adverse selection and the order never getting filled. Would this risk be decreased for highly liquid stocks?

Thank you.








Re: reqHistoricalData returns live data with date in the future

 

Timezones handling has been recently updated.

According to the , starting from 10.18, the time zone used by all timestamps sent by TWS can be one of:

  • UTC
  • Instrument timezone
  • User timezone (i.e. the one selected at login)

Which of the three is used is set globally by the (slightly cryptically named) setting:

TWS -> Configuration -> API -> Settings -> "Send instrument-specific attributes for dual-mode API client in"


Re: reqHistoricalData returns live data with date in the future

 

From in the TWS user guide:
The time zone of returned bars is the time zone chosen in TWS on the login screen.

One solution is to set your computer time to the required time zone. Works for me.

KH


Re: reqHistoricalData returns live data with date in the future

 

On Sun, Jul 3, 2022 at 10:16 PM, Dan L wrote:
Not sure if anyone else has had this issue, since it seems very niche, but since the recent daylight savings time adjustment in Australia, I've realized that my future date issue happens at 11am (AEST) instead of as previously at 10am (AEST), so its clearly a timezone issue and not something to do with the market opening as I previously assumed.

Does anyone know a way to get the daily bars in UTC/GMT Timezone or unix? The documentation () says you can use 'formatDate' for everything except Daily bars!

If no one has knows a way suppose I'll just have to request intraday bars and then aggregate them, thanks.


Re: TWS accepts new socket API connections but doesn't complete connection

 

¿ªÔÆÌåÓý

Actually, as it happens, someone has created an order server type of process¡­

?

My own trading platform has a command line program that can read orders from a file (or interactively by user input) and submit them to TWS via the API for execution. Typically you use it in conjunction with a file listener program that monitors a folder for new order files, and executes the orders immediately when it detects one (or it can send them to TWS for manual review, modification and onward transmission).

?

It will run for weeks on end if need be, coping with TWS restarts, connection losses, etc. If the program is shut down and restarted, it recovers the exact current situation from information stored in a recovery file and additional API requests.

?

As a simple example, an order file could contain this line:

?

BUY STK:MSFT@SMART 100 MKT

?

As you might guess, this would buy 100 shares of MSFT at market, using SMART routing.

?

A more advanced example:

?

BUY FUT:ES(0)@CME 1 LMT BID[-1T]

?

This would buy 1 contract of the current ES future at CME, using a limit order with the price set 1 tick below the current bid price.

?

An even more advanced example:

?

contract FUT:ES(0[2d])@CME

bracket buy 1

entry lmt /price:bid[-2t]

stoploss stp /trigger:entry[-10t]

target mit /trigger:entry[20t]

endbracket

?

This one buys 1 contract of the current ES future (unless we¡¯re within 2 days of its expiry, in which case use the next contract) at CME; enter with a limit order 2 ticks below bid; create a stoploss using a STOP order with the trigger price set 10 ticks below the price achieved by the entry order; create a target (ie take-profit) market-if-touched order with a trigger price 20 ticks above the entry price.

?

The syntax is quite straightforward and easy to generate programmatically, or even to type straight in.

?

If you¡¯re interested in pursuing this option, let me know (contact me directly via email). It¡¯s very easy to install the software, and there¡¯s no charge. I¡¯d need to give you a bit of guidance because the documentation hasn¡¯t caught up with the program¡¯s current functionality. By the way, I should point out that this particular command line program is part of a complete trading platform that includes manual trading clients, automated trading capabilities, historical data downloading, charting etc, so it¡¯s a fairly major piece of work developed over the past 18 years.

?

Richard

?

?

From: [email protected] <[email protected]> On Behalf Of 2graves via groups.io
Sent: 02 December 2022 18:05
To: [email protected]
Subject: Re: [TWS API] TWS accepts new socket API connections but doesn't complete connection

?

I start a new process for each order for ease of use and programming.? I don't do it for latency reasons.? I can pass in the order parameters on the command line and the process dumps everything to a file and when it is done exits and then I can get the order status from what is in the file.? What you propose is probably my next step although I don't look forward to it.? And I'm kind of surprised someone hasn't created an "order server" type of process like this.? Do you know of any project like this that already exists?? Also, my order process is somewhat complicated, it gets market data, submits orders, cancels, resubmits, hedges, etc.

Thanks for the suggestion,
2graves


Alternative to reqPositions() to obtain option combo information

 

Hello devs,

I'd like to request option combo positions (strangle, spreads, ...) in the current portfolio using the TWS API. Is that possible at all?
When I call reqPositions() and check what is being returned, then I can only see each position individually but not the BAG itself with all of its legs.

When my app runs continuously and I monitor the execDetails I can retrieve the BAG information and look it up using the complex position scanner. However, I should also be able to retrieve that information on exisiting portfolio positions manually opened by the user.

Is there any option to request that information? In TWS itself its displayed beautifully - but how to retrieve that information on the API?

Thanks,
Sven


Re: TWS accepts new socket API connections but doesn't complete connection

2graves
 

I start a new process for each order for ease of use and programming.? I don't do it for latency reasons.? I can pass in the order parameters on the command line and the process dumps everything to a file and when it is done exits and then I can get the order status from what is in the file.? What you propose is probably my next step although I don't look forward to it.? And I'm kind of surprised someone hasn't created an "order server" type of process like this.? Do you know of any project like this that already exists?? Also, my order process is somewhat complicated, it gets market data, submits orders, cancels, resubmits, hedges, etc.

Thanks for the suggestion,
2graves


Re: TWS accepts new socket API connections but doesn't complete connection

2graves
 

Good to know about the PACEAPI, I will remove that.

What you describe is exactly what I have seen.? The connection is made from my client at the TCP level and my client sends data and TWS even responds but it isn't "normal" and a few packets go back and forth, I think two from my client with data, two from TWS with data, and then the connection does nothing else, it just sits there.

Thanks for pointing out the data->connections window as I did not know about that and next time TWS is in this state I will look at that and see how it compares with what I see in netstat.

2graves


Re: TWS accepts new socket API connections but doesn't complete connection

2graves
 

I did not see this with version 9.x although I was using it much less then.? But perhaps that itself is a clue.? I was told by IBKR to upgrade to the latest when I was having a different set of problems.? I haven't seen the problem in two days but have no explanation for that as number of orders submitted per day has been roughly the same over the last two weeks.

2graves


Re: MO tick 9 showing as 46.25 instead of 46.58 today (2022-12-01)

 

You know, I still find this a bit strange since I can't find any mention of this 33 cent dividend on the investor relations site. Are there any Altria analysts out there that can fill me in? Is it buried in a press release somewhere? Is it a "secret" wildcard dividend or something, lol?


Re: TWS accepts new socket API connections but doesn't complete connection

 

¿ªÔÆÌåÓý

And a further thought¡­

?

2graves: perhaps you could just sidestep the issue by not starting a new process for each order?

?

It¡¯s not clear to me what benefit you think this provides. I doubt it gets your orders to the exchange any quicker: the overheads of starting a new process and setting up the API connection must surely outweigh the potential single-threading of orders through TWS if they¡¯re all sent via a single connection. (If you¡¯ve measured this and have shown a genuine improvement, I¡¯d be very interested to know.)

?

So I¡¯m suggesting a single order-placing process that runs continuously, connects to the API when it starts, and disconnects when it stops. You then have the issue of passing the order details to this process, but there are several ways to do this. And presumably you must already have some way of passing this each time you create a new process.

?

Richard

?


Re: How to define / call function to avoid code duplication

 

Hey David, just confirming it worked, thank you so much for your help! Have a great weekend


Re: TWS accepts new socket API connections but doesn't complete connection

 

A couple thoughts.

Sending +PACEAPI is not necessary any longer. For a while now, TWS/IBGW V10 throttle API requests by default and, in fact, you'd need to set option "Reject messages above maximum allowed message rate" in the Global API settings if you'd rather get error messages. +PACEAPI is now ignored (but should not hurt nor should it be the cause of your issue)

Is there any chance, one of your clients occasionally attempts a connection, establishes the TCP/IP socket, but does not attempt or complete the TWS API "welcome" phase of the protocol (the version negotiation)?

Take a look at Richard King's reply in post API client not able to connect to remote TWS instance . Richard nicely describes a condition that results in exactly the situation you experience.

We have also seen this in two different scenarios:

  • We attempted to write a TWS Monitor that would regularly test whether TWS was running, was properly connected with IBKR, and that requests would receive responses within an expected time frame. Similar to what you are describing, this monitor created thousands of short lived TWS API connections per day (one at a time) and TWS would get stuck occasionally. In that case, TWS would still accept additional TCP/IP connections, but the TWS API protocol would never be established for those connections. TWS would still properly handle API conversations that had been established earlier.
  • Our TWS/IBGW installation and most client programs are co-located in a remote server that is shielded from access via the internet (except for SSH). We occasionally connect some local tools to the remote TWS/IBGW through an ssh tunnel with port forwarding. Under certain circumstances (a combination of a Windows ssh implementation and interference by our ISP, a well known "Cable Company" playing games) TCP/IP connections could get into a zombie state where the remote TWS thought the connection was open (according to the DATA -> Connections window) but the local client was long gone. TWS would accept local and remote TCP/IP connections but the API welcome protocol phase did not take place so that the clients would be stuck. That is, until all zombie connections had expired after a long time.

If you still have tcpdumps handy (or next time you run network traces), look for a client TCP/IP connection that stays open for a while with no or incomplete TWS API welcome handshake (e.g. API version negotiation).

Hope that helps,

´³¨¹°ù²µ±ð²Ô

On Wed, Nov 30, 2022 at 10:52 PM, 2graves wrote:

My setup:? Ubuntu 22.04.1
TWS: latest
TWS api: 10.19
Trading: US equity market hours trading US equities, US equity options
Problem: At some point TWS accepts further socket connections but doesn't really respond or even finish setting up the connection.

In my system I have one socket connection that I use for market data.? When I want to submit an order I start up another process with a unique client id that connects to TWS and deals with one order and then that process shuts down and exits.? Generally, it works great.? I often do thousands of orders a day.? However, there are times where the process that sends in an order can't connect.? The socket connection is made, the client sends in +PACEAPI and TWS responds but not in the normal way.? At this time TWS is working fine and the socket connection for market data continues to work.? I can fix this problem by turning off "Enable ActiveX and Socket Clients" and turning it back on and then it works again.? It has never happened more than once during the day and some days it doesn't happen at all.? It happens roughly two or three times a week.? I make sure that I don't have more than 32 socket connections at once by limiting the number of trade processes that can run.? I don't have more than about 5 at once.? I check current connections with netstat and I do some file locking to make sure this isn't the case.? I've looked at the TWS logs, strace and tcpdump to verify all of this but nothing there sticks out.??

Has anyone else encountered this problem?? Or have suggestions on how to proceed?? I've opened up a ticket with IBKR but so far they haven't been helpful.

Thanks,
2graves


Re: IB Java API throws "Bad message length null"

 

I agree that the connection status should be checked, but not with just an else clause. The IBKR suggested code snipped is, IMHO, is the best approach since it assures the reader thread properly terminates in all cases (even during programmatic disconnects and reconnects0.

    from: 

...
m_client.eConnect( ... )
...

final EReader reader = new EReader(m_client, m_signal); reader.start();
new Thread(() -> { while (m_client.isConnected()) { m_signal.waitForSignal(); try { reader.processMsgs(); } catch (Exception e) { System.out.println("Exception: "+e.getMessage()); } } }).start();

The Java and C++ API implementations are different enough that C++ best practices and approaches cannot simply be applied to the Java implementation. In Java during startup,
  • EClientSocket::eConnect is only responsible for the socket, the connection request with TWS/IBGW, and the API version negotiation before actual API messages are exchanged.
  • The client architecture has the option for eConnect to operate in an asynchronous fashion, meaning it does not block. The signal, in that case, is needed for the client to know when the connection has been established. For synchronous connection establishment, signal is only really needed within EClientSocket.
  • Once the connection has been established, the separate reader thread makes a lot of sense but is, by no means, the only architectural option. But if it is used, it should check the connection status between processing messages.
  • Our implementation uses two separate signal instances, one for the eConnect connection establishment and a different one for the reader thread.? We feel that is warranted since the signals are used during two very different phases of the API, EClientSocket/EClient keep their copy beyond just the eConnect, and things could get really ugly were EClientSocket/EClient to issue a signal during the regular message processing phase. Having said that, in the current implementation, sharing the same signal instance between EClientSocket::eConnect and the reader thread should work just fine and is most likely not the cause of the issue reported in the OP.
I'd like to recommend once more the ApiController class as an example. It nicely separates the concerns:
  • ApiConnection --> The connection itself with EClientSocket and connect/disconnect operations
  • ApiController --> The use of the connection such as processing requests, messages, and EWrapper callbacks
  • It uses two different EReaderSignal instances one each and private to ApiConnection and ApiController respectively
To bring it back to the issue described in the OP:
  • It looks to me that in the failure case (that occurs intermittently right after system startup) the clientSocket.eConnect call actually fails and the connection is not established. EClientSocket would indicate such a failure with an EWrapper::error callback and not with an exception
  • Since the code that was presented starts the reader thread regardless of connection status, the immediate attempt to read from the closed socket will fail with an EOFException which leads the the Bad Message length error.

To get to the bottom of this, I'd intercept and log the exact exception in line 100 (aprox) of EClientSocket to understand why the creation of the socket or the connection sometimes fail.

´³¨¹°ù²µ±ð²Ô


On Thu, Dec 1, 2022 at 08:35 PM, <hymagik@...> wrote:

...
Suggested code from IBKR are suggestion, doesn't means it's the best way to write production code, but easy to understand and learn how do design your implementation based on your own architectural design constraints.

In particular, it clear for me that a else clause is missing. If eConnect failed it's dangerous to persist and launch a Ereader, or even worst: starting a thread, while assuming that a non observable throw mechanism will bypass the eConnect failure by triggering the catch,? This is an abusive usage of a code safety mechanism.


Re: IB Java API throws "Bad message length null"

 

You may speculate rightfully, but for reason I explained, as this could be eConnect launching it's own EReader using signal send to EClientSocket ctor
I guess your will need to step/step in eConnect to see what's happening after plain old? socket connect.
I don't believe that eConnect exits without connection up and ready.? eConnect contains very standard socket code. Oldies and Goldies connect.

Your attention should be raised by the fact that
EClientSocket require "signal", that's not typical for a connection ABI to require a 'signal',
unless some synchronous mechanism exists, like using the signal for
sync? recv decoding like maybe a? EReader sync that unwind in your eConnect. (as it does in mine)

Suggested code from IBKR are suggestion, doesn't means it's the best way to write production code, but easy to understand and learn how do design your implementation based on your own architectural design constraints.

In particular, it clear for me that a else clause is missing. If eConnect failed it's dangerous to persist and launch a Ereader, or even worst: starting a thread, while assuming that a non observable throw mechanism will bypass the eConnect failure by triggering the catch,? This is an abusive usage of a code safety mechanism.


Re: What exchanges are historical data and real time data pulled from?

 

When dealing with Bar, real time data are not real time, bcs you have to wait end of Bar anyway. (Slightly exaggerated as High and Low are updated even during Bar build time, as well as in Real time bar)

SMART seems to aggregate on "first come first registered" rules.
And Yes it seems that you get the aggregated volume from all supported exchanges (very many)
But I don't know any method to get where Bar data come from using SMART.
Conceptually the Bar API using SMART is not tailored for that as would need as many individual bars as there is exchanges involved,
This is feasible but have other drawbacks.
You can always call individual historical for specific exchange, even launching simultaneous request on a few of interest for you. But you'll quickly hit pacing violation.

Best solution is usage of "Ticks".
Better than Real time Bar. (which is not really as fast ans accurate as Ticks)

If you are ready to design code to listen to Ticks and to aggregate yourself the Bar, Ticks could fit your bill.
Because now you can know the exch from which each Ticks come from.
Dig Level1 doc (or even Level 2 market depth docs)? and setup proper Tick Kinds.
Ticks are more complex to handle but a very powerful data system from IBKR and very decently real time.

If only looking at SPY, and RT Bar are enough, and ready to look simultaneously at multiple exch, I suggest add ARCA to NYSE, that should cover enough
Or you will have to consider investing in Ticks handling


Re: Minimum Requirement future

 

¿ªÔÆÌåÓý

You also need to check out the minimum liquid asset requirement (ie are you wealthy enough!). I lost my futures trading permissions earlier this year because I didn¡¯t have enough liquid assets (due to a significant drop in the GBP/USD exchange rate). You can find this information on the IBCR website.

?

Richard

?

?

From: [email protected] <[email protected]> On Behalf Of Matthias Frener
Sent: 01 December 2022 22:48
To: [email protected]
Subject: Re: [TWS API] Minimum Requirement future

?

You need about 1800$ (MES margin) and futures trading permission which you can get after confirming you have some experience (not sure how many trades or years is the min). It's not tested, just a number on your account settings that can be changed to fit the needs - still it's a good idea to learn about the market and product before you trade it and don't just yolo into it ;)


Re: Minimum Requirement future

Matthias Frener
 

You need about 1800$ (MES margin) and futures trading permission which you can get after confirming you have some experience (not sure how many trades or years is the min). It's not tested, just a number on your account settings that can be changed to fit the needs - still it's a good idea to learn about the market and product before you trade it and don't just yolo into it ;)


Re: IB Java API throws "Bad message length null"

 

My sample code is basically the same but...

In my case the signal and clientSocket are created when the main class is created (basically static parameters). I can speculate a race condition where you new EClientSocket hasn't finished it's initialization and isn't ready to eConnect().