Re: I have a hard time doing Paper Trading
Thank you once again !
On Wed, Jul 26, 2023 at 9:20?PM ´³¨¹°ù²µ±ð²Ô Reinold via <TwsApiOnGroupsIo= [email protected]> wrote:
Glad you were able to fix your issue locally.
When I referred to TWS logs, I was thinking about the encrypted logs you can access via the Help -> Troubleshooting -> Diagnostics -> TWSLogs menu. These are quite cryptic and more useful for IBKR, but when you search for "ERROR" they sometimes give you really good clues as to what is going on.
TWS launcher logs cover only the very early TWS startup phase and do not receive any updates, once TWS is past the login screen.
´³¨¹°ù²µ±ð²Ô
|
Re: What API does Sterling Trade Tech use?
|
Re: I have a hard time doing Paper Trading
Glad you were able to fix your issue locally.
When I referred to TWS logs, I was thinking about the encrypted logs you can access via the Help -> Troubleshooting -> Diagnostics -> TWSLogs menu. These are quite cryptic and more useful for IBKR, but when you search for "ERROR" they sometimes give you really good clues as to what is going on.
TWS launcher logs cover only the very early TWS startup phase and do not receive any updates, once TWS is past the login screen.
´³¨¹°ù²µ±ð²Ô
|
Re: What API does Sterling Trade Tech use?
I can imagine there are more connectivity solutions once you generate "Sterling Trade Tech" levels of order volume. But it could be as simple as the FIX API the refers to.
One of the key features is "Available via internet, extranet, VPN, dedicated line or cross-connect". "Internet" most likely refers to an instance of IB Gateway you'd have to run yourselves, but I can envision that "dedicated line" and "cross-connect" integrations work without IBGW.
But you'd be in a completely different league, even if you run FIX via IBGW.
´³¨¹°ù²µ±ð²Ô
|
Re: Cancel order lag - sometimes when lag they can fill
tracking oder status...
so i have orders open:
the status:
OrderStatus. Id: ?7691 , Status: ?PreSubmitted , Filled: ?0.0 , Remaining: ?1.0 , LastFillPrice: ?0.0
OrderStatus. Id: ?7693 , Status: ?PreSubmitted , Filled: ?0.0 , Remaining: ?1.0 , LastFillPrice: ?0.0
OrderStatus. Id: ?7692 , Status: ?PreSubmitted , Filled: ?0.0 , Remaining: ?1.0 , LastFillPrice: ?0.0
makes sense
so i am now cancelling these
OrderStatus. Id: ?7691 , Status: ?Cancelled , Filled: ?0.0 , Remaining: ?1.0 , LastFillPrice: ?0.0
OrderStatus. Id: ?7692 , Status: ?Cancelled , Filled: ?0.0 , Remaining: ?1.0 , LastFillPrice: ?0.0
OrderStatus. Id: ?7693 , Status: ?Cancelled , Filled: ?0.0 , Remaining: ?1.0 , LastFillPrice: ?0.0
also there is the messages
ERROR 7691 202 Order Canceled - reason: ERROR 7692 202 Order Canceled - reason:
ERROR 7693 202 Order Canceled - reason:
this is where in the TWS GUI I can see 'the lag'
ok so now they are cancelled and i sent new orders
for state
OrderStatus. Id: ?7695 , Status: ?PreSubmitted , Filled: ?0.0 , Remaining: ?1.0 , LastFillPrice: ?0.0
OrderStatus. Id: ?7696 , Status: ?PreSubmitted , Filled: ?0.0 , Remaining: ?1.0 , LastFillPrice: ?0.0
OrderStatus. Id: ?7694 , Status: ?PreSubmitted , Filled: ?0.0 , Remaining: ?1.0 , LastFillPrice: ?0.0
makes sense
not sure how often it can sit in "PendingCancel" these ones so far are going straight to "Canceled".
So while I am monitoring this system
is it at all possible - if the state was "PendingCancel" that there could be a fill when price meets the parent fill price!?
OR
once an order is in "PendingCancel" fills are NOT possible.
Thanks, Andrew
On Tue, Jul 25, 2023 at 9:46?AM Andrew Bannerman via <bannerman1985= [email protected]> wrote: Buddy, Gordon and Jurgen - thanks. My thinking around holding the threads hostage was that 2 seconds 'should' be enough time to see an order canceled which it certainly is. However my specific case there are 'collisions' when order has been canceled and pushed to cancel pending. I see that the order can still get filled in this state or what I called earlier in the lag state. I'm dealing with bracket orders so parent canceled. In pending cancel price hits parent fill level filled and the tp and sl proceed to cancel. And yes because I'm not tracking the executions order states ahead of sending the next bracket order. So I back tested over 1 minute data and I execute this strategy at the even 1 min time stamp so I'm about checking signals around 57 sec mark it's reasonable enough for this but I can improve. But in my backtests I am not accounting for the fallaffalling around with these orders and I may have to just take the fills. On IB side if order is cancel pending and flagged as such can it be that it will also not be able to be a fillable order? I need to check the specific states using order call backs though to verify. But feels like if it'd pushed to cancel pending it can still be filled is that accurate?
On Mon, Jul 24, 2023, 11:27 PM buddy < spy@...> wrote: Gordon and ´³¨¹°ù²µ±ð²Ô are completely right, trying to use sleep to manage concurrency and synchronization will not go far. The correct(tm) approach needs to accept that there is no shared memory between your machine and IB machines.
Therefore, as Gordon alludes, the state of a TWS/IBGW application relies on message passing. And, when ´³¨¹°ù²µ±ð²Ô urges us to observe all callbacks it is because they handle the messages that transfer state updates back to us.
Given all this, it may be useful to review the basic ideas and common techniques used to manage concurrency in general (esp. via message passing). There are a lot of resources on the net and a quick search revealed a few helpful introductions for those who are interested...
Of course it's a huge topic and there are many details to consider. However, understanding the fundamentals is essential and will definitely ease the creation of robust trading software.
After reviewing the above, why and how to track the (with accuracy) should become a bit clearer.
Not that sleep isn't useful of course, but it's mostly so you can free resources of the local machine when nothing is going on. It is not meant to aid with the management of concurrent processes or synchronize state... although it can be badly abused to that end.
-Good luck
|
Asyncio clashes with IBKR threading, prevents some orders from going through.
I have the following code which pulls out prices for multiple stocks in parallel from a REST api. not from IB. Then it places my orders in ib.
results = asyncio.run(get_data(new_portfolio.keys()))
app = ib_class("127.0.0.1", 4002, 999)
api_thread = threading.Thread(target=run_loop, daemon=True)
api_thread.start()
app.cancel_all_orders()
open_positions = app.read_positions()
# some calculations to check what positions to add and remove in a for loop
new_portfolio = open_positions.update(results)
for ....
for o in bracket:
app.placeOrder(...
app.nextOrderId()
However, it takes a while to query the connect and query the openpositions.
I want to do it before I do my asyncio.run to query the prices from my api like this.
app = ib_class("127.0.0.1", 4002, 999)
api_thread = threading.Thread(target=run_loop, daemon=True)
api_thread.start()
app.cancel_all_orders()
open_positions = app.read_positions()
results = asyncio.run(get_data(new_portfolio.keys()))
# some calculations to check what positions to add and remove in a for loop
new_portfolio = open_positions.update(results)
for ....
for o in bracket:
app.placeOrder(...
app.nextOrderId()
But when I do that, some of the orders don't go through.
I think it's because there's a clash with the asyncio and IB's threading.
Is there a way to make sure that this does not happen?
|
link with symbol that open the symbol on tws client ?
is it possible to create a link that open specify symbol in TWS ?
|
What API does Sterling Trade Tech use?
For Interactive Brokers, placing stock buy orders using Java I need to have Trader Workstation running.
I believe there is a web based IB Gateway API I could use, but it still all runs locally.
So which API does Sterling Trade Tech use, are they talking to a web app server hosted by IB??
Does IB provide them with a commercial API that we don't have access to?
|
get symbols from ScannerSubscription and get data from reqHistoricalData , reqMktData
can someone give and example code to use also ScannerSubscription and reqHistoricalData and reqMktData
|
Re: (Paid) Finishing up a software
|
Re: Cancel order lag - sometimes when lag they can fill
Buddy, Gordon and Jurgen - thanks. My thinking around holding the threads hostage was that 2 seconds 'should' be enough time to see an order canceled which it certainly is. However my specific case there are 'collisions' when order has been canceled and pushed to cancel pending. I see that the order can still get filled in this state or what I called earlier in the lag state. I'm dealing with bracket orders so parent canceled. In pending cancel price hits parent fill level filled and the tp and sl proceed to cancel. And yes because I'm not tracking the executions order states ahead of sending the next bracket order. So I back tested over 1 minute data and I execute this strategy at the even 1 min time stamp so I'm about checking signals around 57 sec mark it's reasonable enough for this but I can improve. But in my backtests I am not accounting for the fallaffalling around with these orders and I may have to just take the fills. On IB side if order is cancel pending and flagged as such can it be that it will also not be able to be a fillable order? I need to check the specific states using order call backs though to verify. But feels like if it'd pushed to cancel pending it can still be filled is that accurate?
On Mon, Jul 24, 2023, 11:27 PM buddy < spy@...> wrote: Gordon and ´³¨¹°ù²µ±ð²Ô are completely right, trying to use sleep to manage concurrency and synchronization will not go far. The correct(tm) approach needs to accept that there is no shared memory between your machine and IB machines.
Therefore, as Gordon alludes, the state of a TWS/IBGW application relies on message passing. And, when ´³¨¹°ù²µ±ð²Ô urges us to observe all callbacks it is because they handle the messages that transfer state updates back to us.
Given all this, it may be useful to review the basic ideas and common techniques used to manage concurrency in general (esp. via message passing). There are a lot of resources on the net and a quick search revealed a few helpful introductions for those who are interested...
Of course it's a huge topic and there are many details to consider. However, understanding the fundamentals is essential and will definitely ease the creation of robust trading software.
After reviewing the above, why and how to track the (with accuracy) should become a bit clearer.
Not that sleep isn't useful of course, but it's mostly so you can free resources of the local machine when nothing is going on. It is not meant to aid with the management of concurrent processes or synchronize state... although it can be badly abused to that end.
-Good luck
|
Re: I have a hard time doing Paper Trading
I just uninstalled, deleted the entire JTS folder, reinstalled, and now everything works as expected.
I probably had an unintentional setting checked somewhere, sorry for taken your time with this and thank you for the help.
|
Re: I have a hard time doing Paper Trading
Hello, Thank you Jurgen for taking your time to test this. To answer your questions:
- ?yes is a paper trading account linked to a real brokerage account where I have NASDAQ subscription.
- ?account type is IBKR Pro.
- I have placed orders using ib-insync as well, seen them placed, but after that I encounter the same behavior as with the GUI.
- Related to logs, I have attached the audit, it seems a ping-pong between ISLAND and IBKRATS. Other than that I only discovered the 'launcher' logs from C:\JTS. Nothing that caught my eyes there.
- I also deselected adaptive setting
Thank you, Flaviu
|
Re: Cancel order lag - sometimes when lag they can fill
Gordon and ´³¨¹°ù²µ±ð²Ô are completely right, trying to use sleep to manage concurrency and synchronization will not go far. The correct(tm) approach needs to accept that there is no shared memory between your machine and IB machines.
Therefore, as Gordon alludes, the state of a TWS/IBGW application relies on message passing. And, when ´³¨¹°ù²µ±ð²Ô urges us to observe all callbacks it is because they handle the messages that transfer state updates back to us.
Given all this, it may be useful to review the basic ideas and common techniques used to manage concurrency in general (esp. via message passing). There are a lot of resources on the net and a quick search revealed a few helpful introductions for those who are interested...
Of course it's a huge topic and there are many details to consider. However, understanding the fundamentals is essential and will definitely ease the creation of robust trading software.
After reviewing the above, why and how to track the (with accuracy) should become a bit clearer.
Not that sleep isn't useful of course, but it's mostly so you can free resources of the local machine when nothing is going on. It is not meant to aid with the management of concurrent processes or synchronize state... although it can be badly abused to that end.
-Good luck
|
Re: I have a hard time doing Paper Trading
I am sure the delays you are experiencing are real, but I have the feeling that it is not just 'the paper account" that is at fault here. There must be some account attribute or setting that triggers or causes the issue especially since you seem to be able to cure it by restarting "the platform". Do you refer to TWS here?
I tried a few TSLA BUY/SELL cycles in a paper account today. All of them filled in less than a second (e.g. all audit trail entries for an order have the same timestamp second):
Just a few thoughts:
- You are using a "paper trading account" that is linked to a real brokerage account and shares the market data subscription, right?
- You are not using a "demo" account. Those are the fake accounts you create from the TWS login form through the "No username? Try the demo" link
- Is your real brokerage account an IBKR PRO, IBKR Lite, or some other account type?
- Have you tried placing the orders through the API instead of the GUI?
- Did you take a quick look at the TWS logs to see whether TWS was aware of any issues?
´³¨¹°ù²µ±ð²Ô
On Mon, Jul 24, 2023 at 02:49 AM, Flaviu Paul wrote:
Honestly I would love to have no issues here, but in my opinion there is clearly a bug somewhere.
50 shares is a small amount for TSLA in my opinion.
From logs I can see that the request is 'ping-pong' between ISLAND and IBKRATS, for minutes.
The entry order works every-time = the buy or sell order.
If I close the platform and reconnect, I can exit the trade without a problem.
|
Re: I have a hard time doing Paper Trading
Honestly I would love to have no issues here, but in my opinion there is clearly a bug somewhere.
50 shares is a small amount for TSLA in my opinion.
From logs I can see that the request is 'ping-pong' between ISLAND and IBKRATS, for minutes.
The entry order works every-time = the buy or sell order.
If I close the platform and reconnect, I can exit the trade without a problem.
|
Re: Cancel order lag - sometimes when lag they can fill
Hear! Hear!
Gordon is right on and the only pointer I can add is the TWS API Guide chapter . It starts with:
When placing orders via the API and building a robust trading system, it is important to monitor for callback notifications, specifically for , changes, warnings, and to ensure proper operation. If you experience issues with orders you place via the API, such as orders not filling, the first thing to check is what these callbacks returned. Your order may have been rejected or cancelled.
TWS API is an asynchronous API that connects client applications to an entirely asynchronous trading world. Events take place whenever they do, no sequence or timing of events can be assumed, and, consequently, using "sleep" to synchronize events is absolutely unacceptable. And so is ignoring callbacks.
I wish IBKR support would eliminate "sleep" from their vocabulary and IBKR API engineering would remove the last few uses of sleep from TWS API. Our platform is entirely event driven and asynchronous, our client apps never use sleep, observe all callbacks and everything runs rock solid and fast. We have experimented with strategies that place/change/cancel several orders per second very reliably.
Another tool for managing orders that works very reliably is . It triggers much faster than when client applications manage cancellations themselves (even on networks with very low latencies).
´³¨¹°ù²µ±ð²Ô
PS. We also "over cancel", except in cases where TWS API already sent messages that confirm proper cancellation of all orders.
On Sun, Jul 23, 2023 at 08:29 PM, Gordon Eldest wrote:
In case you wonder, by the asynchronous nature of the API (which is good) you should never assume instantaneous execution.
Sending the request is just queuing and intent. (more about callback below) and I like it that way.
I check explicitly order status before an assumption like a cancel being done, in particular if I have to re-submit a substitute order. (or you have to factor-in the risk of parallel execution, which could make sens if time is critical)
I am not sure it's a good practice to rely on any sleep, the API is a state machine and you should track the state of it either trough additional IB request or internally or both. (both is good as order are the crucial part of the usage of the API)
here "both" means:
Listening to orderStatus callback is generally enough. "generally" because:? When the callback fires, I always find it accurate,
but in IB doc I read that at least in some cases of immediate execution the callback may not be called. The sentence imply that there is a gray area there and better have a "plan B"
It's a difficult area for doing accurate tests,? it is very much market dependent, so that "belt and suspender" are required.
So doing an additional explicit detail request is safe and generally fast enough in view of execution time, so I never study that part of the API again. It work.
you surely read it but just in case:
also helpful
About cancel sl/tp:
I do cancel them but don't know if its needed, because I prefer to "over cancel" applying the rule that everything that have an individual orderid could and maybe should be cancelled.
So as I track childs ID, a "cancel session" is made of cancelling everything related to the bracket and thanks to the async API that have no impact I can notice.
Surely need some study.
|
Re: Cancel order lag - sometimes when lag they can fill
Main reason ordercall back making if full cancel confirmed.... this strategy time based thus I'd miss x trades which wouldn't work here.
On Sun, Jul 23, 2023, 10:15 PM Andrew Bannerman via <bannerman1985= [email protected]> wrote: Thanks - so yes sl tp I thought if cancel them directly vd parent only that they dont lag. The result of the order lag in canceling is that parent fills and the sl and tp are canceled. 2 sets later I fire off a new bracket order. It may or may not get filled..... again I cancel tge order... get lag get another parent fill. Tp and sl canceled so I end up building a massive unintended position. So this strategy is executed and relies on this method of sending orders and canceling and sending new.? Maybe I can edit the new bracket order that already exists but change the parent, sl and tp prices. Basically if I can't find an answer I will need to run thus strategy on another platform which is last thing I'd want to do.?
In case you wonder, by the asynchronous nature of the API (which is good) you should never assume instantaneous execution.
Sending the request is just queuing and intent. (more about callback below) and I like it that way.
I check explicitly order status before an assumption like a cancel being done, in particular if I have to re-submit a substitute order. (or you have to factor-in the risk of parallel execution, which could make sens if time is critical)
I am not sure it's a good practice to rely on any sleep, the API is a state machine and you should track the state of it either trough additional IB request or internally or both. (both is good as order are the crucial part of the usage of the API)
here "both" means:
Listening to orderStatus callback is generally enough. "generally" because:? When the callback fires, I always find it accurate,
but in IB doc I read that at least in some cases of immediate execution the callback may not be called. The sentence imply that there is a gray area there and better have a "plan B"
It's a difficult area for doing accurate tests,? it is very much market dependent, so that "belt and suspender" are required.
So doing an additional explicit detail request is safe and generally fast enough in view of execution time, so I never study that part of the API again. It work.
you surely read it but just in case:
also helpful
About cancel sl/tp:
I do cancel them but don't know if its needed, because I prefer to "over cancel" applying the rule that everything that have an individual orderid could and maybe should be cancelled.
So as I track childs ID, a "cancel session" is made of cancelling everything related to the bracket and thanks to the async API that have no impact I can notice.
Surely need some study.
|
Re: Cancel order lag - sometimes when lag they can fill
Thanks - so yes sl tp I thought if cancel them directly vd parent only that they dont lag. The result of the order lag in canceling is that parent fills and the sl and tp are canceled. 2 sets later I fire off a new bracket order. It may or may not get filled..... again I cancel tge order... get lag get another parent fill. Tp and sl canceled so I end up building a massive unintended position. So this strategy is executed and relies on this method of sending orders and canceling and sending new.? Maybe I can edit the new bracket order that already exists but change the parent, sl and tp prices. Basically if I can't find an answer I will need to run thus strategy on another platform which is last thing I'd want to do.?
In case you wonder, by the asynchronous nature of the API (which is good) you should never assume instantaneous execution.
Sending the request is just queuing and intent. (more about callback below) and I like it that way.
I check explicitly order status before an assumption like a cancel being done, in particular if I have to re-submit a substitute order. (or you have to factor-in the risk of parallel execution, which could make sens if time is critical)
I am not sure it's a good practice to rely on any sleep, the API is a state machine and you should track the state of it either trough additional IB request or internally or both. (both is good as order are the crucial part of the usage of the API)
here "both" means:
Listening to orderStatus callback is generally enough. "generally" because:? When the callback fires, I always find it accurate,
but in IB doc I read that at least in some cases of immediate execution the callback may not be called. The sentence imply that there is a gray area there and better have a "plan B"
It's a difficult area for doing accurate tests,? it is very much market dependent, so that "belt and suspender" are required.
So doing an additional explicit detail request is safe and generally fast enough in view of execution time, so I never study that part of the API again. It work.
you surely read it but just in case:
also helpful
About cancel sl/tp:
I do cancel them but don't know if its needed, because I prefer to "over cancel" applying the rule that everything that have an individual orderid could and maybe should be cancelled.
So as I track childs ID, a "cancel session" is made of cancelling everything related to the bracket and thanks to the async API that have no impact I can notice.
Surely need some study.
|
Re: I have a hard time doing Paper Trading
I concur with ´³¨¹°ù²µ±ð²Ô and Hilmar
Paper account is pretty accurate, I don't know how IB simulate it accurately on "big orders" as they may influence the book in real life, rather hard on AAPL or TSLA but easier on low liquidity stocks, but seems the only caveat I imagine about it.
|
buddy
