2024 NQ and ES futures historical data
Hi everyone,
?
I need to test something and I have been having a bit of trouble requesting data from IB. Does anyone have last year?s historical data for ES and NQ? doesn?t have to be tick data. Alternatively where can I find it cheap and reliable??
?
thanks a bunch.?
|
Hotkey for market data lines?
In the recent past I came across a hotkey that provided a detailed list of data lines (window, instrument, and exchange). It was updated dynamically. I link several windows together so clicking on a line in a watchlist changed several windows and the data lines associated with them.
?
This is different that the summary you get with <ctrl><alt>=.
?
I'm sure I wrote it down but that sticky note is probably behind my desk somewhere.
?
I'm using TWS 10.30
?
|
Re: How to identify futures orders placed by ATS
For those who trade futures. IB support told to populate one of the fields below with custom ID that differentiates manual and ATS orders.?
?
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
Interestingly the documentation on what reqMktData (both old and new). Have this statement
"A tickPrice value of -1 or 0 followed by a tickSize of 0 indicates there is no data for this field currently available". Which I read as: if there is no data I get -1, if there is data I get some number. I guess this is there IB could be much more explicit. If one subscribes to market data in the middle of the trading session what happens? Does one get a "snapshot" of data = the latest data IB has (some might be very old) and then updates only then changes happen (which I believe how 10.19 and older worked). OR the "snapshot" part never happens now (10.30 and later?).
?
However, a couple of observations from live environment which make me doubt the above explanation and make me lean on "some resource optimization went wrong". Is more likelly explanation. Here is a problem from one of live environments?
1. if the above is true, then high/low would be really rarelly updated. And this is not the case. Once asked for data those (high/low) get streamed, most of the time. Ok, it could be that for high/low there is an exception
2. In real env I often see?lastTimestamp missing (timestamp of the last trade) yet the price is present. For me it points to that "timestamp" stream got dropped by IB somehow in this case.?
Of course, the way API is setup we should not expect a perfect synchronicity here. but not getting within 20 seconds a timestamp is really a bug and probably of the same nature as we discuss here.?
3. Another fact that points to optimization problem. In this test app one needs to run several loops to consitently see the problem. With the first loop is likelly to be fine, only in later loops the issues will pop up.?
?
No updates from IB on the issue yet (I reported end of Nov). I keep asking for progress and they keep replying that "they cannot repeat the issue". I of course sent all the test code and tips and tricks...
?
I keep wondering, since IB is silint of the issue I have no other choice but to continue guessing that this is an "optimization went wrong". Makes me wonder (untill IB proves otherwise): what els is not stable in 10.30? Like if you "push API too much you might only get a fraction of you account reported or you order will not be posted"-
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
jy.yngbld@..., 1. first loop of the test not always gives the errors. If you run several loops, then it is more likelly to repeat the issue
Program_get_bid.py -f symbols.csv -m REALTIME --data-lines 75 --loops 5 --port 4001
2. "I noticed that I don't have any output files from this script." if you scroll in this discussions, the very first message has the app which outputs much more?
3. "I am on TWS 10.30.1, did you recently update to 10.30 or have you been fighting with this for a while?" I had this issue for more than 2 months now. Ever since 10.30 was promoted to "stable". I updated and immidiatelly run into this.
|
Problems with " error advancedOrderRejectJson" Using version API_Version=10.33.01 from December 2024.
Using version ?API_Version=10.33.01 ?from December 2024.
1. ? ? ? decoder.py uses ?" self.wrapper.error(reqId, errorTime, errorCode, errorString, advancedOrderRejectJson)"
?
Instead, it does not use the argument ""advancedOrderRejectJson":
2. ? ? ? client.py in 238 lines ""self.wrapper.error(reqId, currentTimeMillis(), NOT_CONNECTED.code(), NOT_CONNECTED.msg())""
?
3. ? ? ? connection.py in 2 lines does not use "" advancedOrderRejectJson"""
?
This topic is confusing and several problems emerge in error scenarios.
|
How to identify futures orders placed by ATS
TWS sent a notification to complete a task saying that US futures exchanges require brokers to identify orders placed by automated systems.?
How do you mark order as ATS??
Is there a field for this, like tag??
?
Please create an operator ID for each group of persons who operate/monitor the system at the same time.
Whenever an automated trading system submits an order message (incl. order revisions and cancellations), the message must include the ID of the team or individual who is operating/monitoring the system at that time.
|
On Fri, Jan 3, 2025 at 09:03 PM, ´³¨¹°ù²µ±ð²Ô Reinold wrote:
parameter keepUpToDate
They really should be redirecting people away from the old docs (see attached).
|
for example i want create one client, and request two or more contract history data, this is wrong way?
```
app = Client()
app.connect('127.0.0.1', 7497, 0)
threading.Thread(target=app.run).start()
time.sleep(1)
for row in ['AAPL','GOOG']:
app.reqHistoricalData(....)
app.disconnect()
```
the app can not disconnect
?
so i write like this:
```
for row in ['AAPL','GOOG']:
app = Client()
app.connect('127.0.0.1', 7497, 0)
threading.Thread(target=app.run).start()
app.reqHistoricalData(args)
```
then i disconnect on historicalDataEnd function, then it is work fine.
?
so my question is, one client only can do one thing?
?
|
Not sure whether it works for FEE_RATE (you could easily try it), but has parameter keepUpToDate that, if set to TRUE, continues sending you real-time-like updates after the initial historical bars have been delivered.
There are a few caveats (such as you may have to resubscribe under certain conditions and you might get multiple partial updates in some periods), but you should be able to create a reliable regular update service (say once a minute) assuming updates are available for FEE_RATE.
´³¨¹°ù²µ±ð²Ô
?
On Fri, Jan 3, 2025 at 01:04 PM, Scott Kister wrote:
Thanks for the pointer, I was able to get that to work. I was hoping I could just use a market data subscription to keep it up-to-date as with shares available to short. It seems very odd that they only provide historical data for fee rate and not the current value, but at least the data is available via the API, I can request a 1-minute bar from the last minute.
|
Thanks for the pointer, I was able to get that to work. I was hoping I could just use a market data subscription to keep it up-to-date as with shares available to short. It seems very odd that they only provide historical data for fee rate and not the current value, but at least the data is available via the API, I can request a 1-minute bar from the last minute.
|
|
|
Is the hard to borrow fee available via the API? It is vieable in TWS under Short Selling -> Fee Rate
ib.reqMktData(contract, tick_list=236) returns the number of shares available to short, but doesn't return the fee rate. I know it is available via FTP, but sometimes it is missing there and I would prefer to keep it up-to-date via the API. I tried searching the docs, but was unable to find it.
|
Re: reqMktData subscription is not continuous
Good to know that after the Holidays, the liver is still functioning fine!
Mel
toggle quoted message
Show quoted text
-------- Original message -------- From: "Andy Sanders via groups.io" <arteinvolo@...> Date: 2024-12-28 9:29 a.m. (GMT-08:00) Subject: Re: [TWS API] reqMktData subscription is not continuous
Appeared to be that was some glitch with paper account. Liver data feed works fine.?
|
Re: reqMktData subscription is not continuous
Appeared to be that was some glitch with paper account. Liver data feed works fine.?
|
Re: Question on nextValidId function
thanks again ´³¨¹°ù²µ±ð²Ô, everything is clearer now, complicated as you said, but clearer :)
BR, Marco On Sat, Dec 28, 2024 at 4:48?AM ´³¨¹°ù²µ±ð²Ô Reinold via <TwsApiOnGroupsIo= [email protected]> wrote:
You are correct, Marco. Each clientId has its own sequence of orderIds and, since nextValidId() has a value of 1 when a clientId is used for the first time, the numeric values for orderIds for different clientIds can (and will be) be identical at times.
The technical requirement your client has to be aware of is that each placeOrder() call shall present an orderId that is higher than all orderIds the calling clientId has ever received in openOrder() or orderStatus() callbacks.
As long as a client receives only openOrder() callbacks for orders it placed itself, keeping a counter that is initialized from nextValidId() at connection time is sufficient. When a client terminates, TWS/IBGW will remember the highest orderId used in placeOrder() calls and assures that nextValidId() returns a value higher than that when it connects again later.
But there are cases where a client can receive openOrder() callbacks for orders placed by other clients (or TWS). In these cases, the client may need to bump the internal nextValidid counter accordingly. Let me quote directly from the TWS API documentation since it nicely describes this:
However if there are multiple client applications connected to one account, it is necessary to use an order ID with new orders which is greater than all previous order IDs returned to the client application in openOrder or orderStatus callbacks. For instance, if the client is set as the Master client, it will automatically receive order status and trade callbacks from orders placed from other clients. In such a case, any orderID used in placeOrder must be greater than the orderIDs returned in these status callbacks.
Alternatively if the function reqAllOpenOrders is used by a client, subsequent orders placed by that client must have order IDs greater than the order IDs of all orders returned because of that function call.?
The second part of that statement is what you need to be aware of. Say your client needs to be know about orders for the instrument is manages that are open when it connects, a call to reqAllOpenOrders() might generate openOrder() callbacks with orderIds that are higher than the nextValidId(). In these cases, simply bump the internal counter to that orderId + 1.
And to make this very simple and safe for all scenarios, just perform that test unconditionally for all openOrder() and orderStatus() callbacks regardless of why your client receives them.
Sounds complicated, but is really not.
There is also an API Global Configuration option called "Use negative numbers to bind automatic orders." Take a look at the tool-tip.
´³¨¹°ù²µ±ð²Ô
?
?
On Fri, Dec 27, 2024 at 09:36 AM, Marco wrote:
Hello ´³¨¹°ù²µ±ð²Ô,
thanks for your detailed reply. Yes I?confirm my counter(s) get initialized with the value from the automatic nextValidId() callback. It is not completely clear to me when you mention "in contact". Does it mean in case different clients can manage same order?
if the clients are independently managing different orders I can have independent?counters so in principle the same numerical orderId can be reused for different clients, am I correct?
thanks
Marco
On Fri, Dec 27, 2024 at 3:04?PM ´³¨¹°ù²µ±ð²Ô Reinold via <TwsApiOnGroupsIo= [email protected]> wrote:
There is nothing wrong with your approach, as long as your internal counter gets initialized with the value of the automatic nextValidId() callback you receive when your client connects to TWS API.
If your client gets "in contact" with orders from other clients, you need to adjust your interal counter so that it is higher than the orderId of all orders your client sees. That is mostly important for client 0 and the master client since they receive openOrder() callback for orders placed by other clients or TWS in real time, but all clients might need to do that in case they request all open orders, for example, and the list contains order from other clients.
The simplest approach is that your openOrder() callback makes sure that your internal counter is always higher than all orderIds it sees.
´³¨¹°ù²µ±ð²Ô
?
On Fri, Dec 27, 2024 at 08:03 AM, Marco wrote:
Hi All,
one question on the orderId and the necessity of nextValidId function. From the documentation () it is stated that ¡°if there are multiple client applications connected to one account, it is necessary to use an order ID with new orders which is greater than all previous order IDs returned to the client application in openOrder or orderStatus callbacks?¡±. so I expect that the nextValidId function is needed in case of multiple clients for one account. However i have created a program in python that handles orderIds internally without relying on the nextvalidid function. By launching multiple instances of it (each with different client id) I see no problems in handling orders (each client autonomously updating its orderId). What am I missing? Is nextValidId really needed?
thanks,
Marco
?
|
Re: IB's New automated trading system questionnaire
I send push notifications to my mobile using an app called pushover. This works great and IB was fine with this solution.
|
Re: Question on nextValidId function
You are correct, Marco. Each clientId has its own sequence of orderIds and, since nextValidId() has a value of 1 when a clientId is used for the first time, the numeric values for orderIds for different clientIds can (and will be) be identical at times.
The technical requirement your client has to be aware of is that each placeOrder() call shall present an orderId that is higher than all orderIds the calling clientId has ever received in openOrder() or orderStatus() callbacks.
As long as a client receives only openOrder() callbacks for orders it placed itself, keeping a counter that is initialized from nextValidId() at connection time is sufficient. When a client terminates, TWS/IBGW will remember the highest orderId used in placeOrder() calls and assures that nextValidId() returns a value higher than that when it connects again later.
But there are cases where a client can receive openOrder() callbacks for orders placed by other clients (or TWS). In these cases, the client may need to bump the internal nextValidid counter accordingly. Let me quote directly from the TWS API documentation since it nicely describes this:
However if there are multiple client applications connected to one account, it is necessary to use an order ID with new orders which is greater than all previous order IDs returned to the client application in openOrder or orderStatus callbacks. For instance, if the client is set as the Master client, it will automatically receive order status and trade callbacks from orders placed from other clients. In such a case, any orderID used in placeOrder must be greater than the orderIDs returned in these status callbacks.
Alternatively if the function reqAllOpenOrders is used by a client, subsequent orders placed by that client must have order IDs greater than the order IDs of all orders returned because of that function call.?
The second part of that statement is what you need to be aware of. Say your client needs to be know about orders for the instrument is manages that are open when it connects, a call to reqAllOpenOrders() might generate openOrder() callbacks with orderIds that are higher than the nextValidId(). In these cases, simply bump the internal counter to that orderId + 1.
And to make this very simple and safe for all scenarios, just perform that test unconditionally for all openOrder() and orderStatus() callbacks regardless of why your client receives them.
Sounds complicated, but is really not.
There is also an API Global Configuration option called "Use negative numbers to bind automatic orders." Take a look at the tool-tip.
´³¨¹°ù²µ±ð²Ô
?
?
On Fri, Dec 27, 2024 at 09:36 AM, Marco wrote:
Hello ´³¨¹°ù²µ±ð²Ô,
thanks for your detailed reply. Yes I?confirm my counter(s) get initialized with the value from the automatic nextValidId() callback. It is not completely clear to me when you mention "in contact". Does it mean in case different clients can manage same order?
if the clients are independently managing different orders I can have independent?counters so in principle the same numerical orderId can be reused for different clients, am I correct?
thanks
Marco
On Fri, Dec 27, 2024 at 3:04?PM ´³¨¹°ù²µ±ð²Ô Reinold via <TwsApiOnGroupsIo= [email protected]> wrote:
There is nothing wrong with your approach, as long as your internal counter gets initialized with the value of the automatic nextValidId() callback you receive when your client connects to TWS API.
If your client gets "in contact" with orders from other clients, you need to adjust your interal counter so that it is higher than the orderId of all orders your client sees. That is mostly important for client 0 and the master client since they receive openOrder() callback for orders placed by other clients or TWS in real time, but all clients might need to do that in case they request all open orders, for example, and the list contains order from other clients.
The simplest approach is that your openOrder() callback makes sure that your internal counter is always higher than all orderIds it sees.
´³¨¹°ù²µ±ð²Ô
?
On Fri, Dec 27, 2024 at 08:03 AM, Marco wrote:
Hi All,
one question on the orderId and the necessity of nextValidId function. From the documentation () it is stated that ¡°if there are multiple client applications connected to one account, it is necessary to use an order ID with new orders which is greater than all previous order IDs returned to the client application in openOrder or orderStatus callbacks?¡±. so I expect that the nextValidId function is needed in case of multiple clients for one account. However i have created a program in python that handles orderIds internally without relying on the nextvalidid function. By launching multiple instances of it (each with different client id) I see no problems in handling orders (each client autonomously updating its orderId). What am I missing? Is nextValidId really needed?
thanks,
Marco
?
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
?I have executed the short file you posted multiple times through IB_Gateway.
Final result in prompt: ?
done symbols: 1283
Number of not done symbols: 0
In about 52 seconds.
Also, gateway is sending with each ticker
2024-12-27 13:09:32.976 [OQ] INFO ?[JTS-usfarmDispatcherS3-52S3-53] - Sending for ticker=1001373 market data info: bboExchangeAndSecType=a60001, snapshot perms=REALTIME_TOP
?
I noticed that I don't have any output files from this script. Could you send the code you used to write the bid data to file.
I am on TWS 10.30.1, did you recently update to 10.30 or have you been fighting with this for a while?
|
Re: Question on nextValidId function
Hello ´³¨¹°ù²µ±ð²Ô, thanks for your detailed reply. Yes I?confirm my counter(s) get initialized with the value from the automatic nextValidId() callback. It is not completely clear to me when you mention "in contact". Does it mean in case different clients can manage same order? if the clients are independently managing different orders I can have independent?counters so in principle the same numerical orderId can be reused for different clients, am I correct? thanks Marco On Fri, Dec 27, 2024 at 3:04?PM ´³¨¹°ù²µ±ð²Ô Reinold via <TwsApiOnGroupsIo= [email protected]> wrote:
There is nothing wrong with your approach, as long as your internal counter gets initialized with the value of the automatic nextValidId() callback you receive when your client connects to TWS API.
If your client gets "in contact" with orders from other clients, you need to adjust your interal counter so that it is higher than the orderId of all orders your client sees. That is mostly important for client 0 and the master client since they receive openOrder() callback for orders placed by other clients or TWS in real time, but all clients might need to do that in case they request all open orders, for example, and the list contains order from other clients.
The simplest approach is that your openOrder() callback makes sure that your internal counter is always higher than all orderIds it sees.
´³¨¹°ù²µ±ð²Ô
?
On Fri, Dec 27, 2024 at 08:03 AM, Marco wrote:
Hi All,
one question on the orderId and the necessity of nextValidId function. From the documentation () it is stated that ¡°if there are multiple client applications connected to one account, it is necessary to use an order ID with new orders which is greater than all previous order IDs returned to the client application in openOrder or orderStatus callbacks?¡±. so I expect that the nextValidId function is needed in case of multiple clients for one account. However i have created a program in python that handles orderIds internally without relying on the nextvalidid function. By launching multiple instances of it (each with different client id) I see no problems in handling orders (each client autonomously updating its orderId). What am I missing? Is nextValidId really needed?
thanks,
Marco
?
|
buddy
Despair