Re: IB's New automated trading system questionnaire
Thanks to everyone for their suggestions.? I flailed around for longer than I would like to admit before finding a send email solution that works for me.
1) I found out gmail offers something called an "app password" that requires you to use 2-factor authentication and then you can create a password that works with SMTP over port 465. 2) I installed MSVC 2010, changed my email program to use the new password and rebuilt.? (It was easier to use MSVC 2010 than to build with MSVC 2019 which had various complaints.)
Voila, I'm once again able to send emails.? The only ongoing cost will be dealing with gmail's two factor log ins.
Frank
|
Re: No definition for SPX option using LocalSymbol
For the future readers.
I was using data coming from Schwab and each SPX contract had two dates, "LastTradingDate" and "ExpirationDate". I was trying to use option contract expiring the same day to place order in IB but it was rejected because even though contract is expiring today, the last trading day for that contract was yesterday.?
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
Uh, my sanity is restored :)
?
As for your comment about the network, yes my dev setup is in EU (yet its 250Mbit up/down fiber, typically no issues here with Internet otherwise, but of course anything can happen :) and again switching back and forth between 10.19 and 10.30 all the time, where 10.19 does't have issues) However, I can reproduce it from my test site in US just as well. Test site is in AWS with pretty biffy setup on the East Coast? (us-east-1, i..e supposed to be closest to IB servers, ?t3.2xlarge instance aws claims 5gbit network, ok it is probably less in reality but itis not likelly a bottleneck even it is "just" 500Mbit to the Internet ).?
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
On Thu, Dec 19, 2024 at 10:23 PM, ajn wrote:
It is often happening on the second loop or later.
Ok, I managed to reproduce the problem.
|
Re: IB's New automated trading system questionnaire
?
Once you've set up the appropriate Telegram bot on your Telegram account, sending a message programmatically is pretty easy. ?Here's my (redacted) code for that:
?
#!/usr/bin/env python3
import requests
def send_message(text, parse_mode=None):
bf_token = "YOUR_TOKEN_HERE" # replace this your Bot's token
user_id = 1234567890 # replace this with your user id
assert parse_mode in (None, "HTML", "MarkdownV2")
# from BotFather
url = f"https://api.telegram.org/bot{bf_token}/sendMessage"
params = {"chat_id": user_id, "text": text}
if parse_mode:
params["parse_mode"] = parse_mode
resp = requests.get(url, params=params, timeout=3)
# Throw exception if Telegram API fails
resp.raise_for_status()
if __name__ == "__main__":
from sys import argv
if len(argv) <= 1:
i = 50 // 3
MESSAGE = "```\n" + (f"{3 * i}." * i) + "\n```"
else:
MESSAGE = " ".join(argv[1:])
send_message(MESSAGE, parse_mode="MarkdownV2")
|
Re: IB's New automated trading system questionnaire
Another option is Telegram. I wouldn't trust too much on email as
it's very difficult these days to have your mail accepted by the
receiving end as non-spam or not get blocked.
For instant notification, Telegram has been a great option for me.
It has? good api documentation. Start here:
Greetings,
Raoul
On 18-12-2024 18:10, 闯ü谤驳别苍 Reinold via
groups.io wrote:
toggle quoted message
Show quoted text
There are many ways you can handle this. @wordd already
mentioned sending messages to slack channels, for example. All
large cloud providers have services that are called something
like "Simple Message Service" or so, that allow you to send
messages through simple API calls.
If you want to stick with email, check how your email provider
supports "SMTP for outgoing messages" and look for an SMTP
library for your preferred environment. Your service provider
may place various restrictions on email delivered to them by
SMTP (such as SSH/TLS encryption or certificates), but generally
I would expect that those do not need two-factor authentication.
For Java, for example, you'd use "JavaMail" directly or through
higher level libraries such as "Apache Commons Email".
闯ü谤驳别苍
?
On Tue, Dec 17, 2024 at 11:31 PM, Frank Bell wrote:
I logged on to my account and found a must complete
questionnaire about how I'm monitoring my automated api.? It
wanted to know
?
1) Did I write it or is it commercial and if so the
vendor?
2) Is the API completely automated?? If not to what
extent it is manual?
3) How are critical errors handled (audible alert, email,
text)?
4) Is it constantly monitored?
5) and more
?
Apparently it is now being required by regulators.
?
Given that, can anyone point to code that will allow me
to send email/texts now that Google and Yahoo are requiring
two factor authentication?? I used to send emails and texts
when single factor was good enough.
?
Frank
|
Re: -1 negative VOLUME in Futures BID_ASK historical request
I see the same. It looks like you only get VOLUME or BARCOUNT if you ask for TRADES. That doesn’t seem unreasonable to me. ? Richard
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
On Thu, Dec 19, 2024 at 10:23 PM, ajn wrote:
However I did notice 2 things
I just ran your sample program again, after hours w/ realtime, data_lines=75, loops=5 and everything worked OK using IBGW 10.30.1s. IDK what to say... I may try it again tomorrow since you indicate there's more likelihood of a problem during RTH.
However, I'm also wondering if your network connection may have slightly degraded in the recent past? Is someone streaming HDTV while you test, lol? Are you sure you can't reproduce the problem w/ 10.19? I suppose it's possible that 10.30 has just enough extra network demand than 10.19 to make a difference but...
I mention this mostly because my server has an exceptionally good connection (and I'm located close to the USA NE data center). When I look at your email headers I notice you're in Sweden, so if you're using a local computer... you might have more variation in connection quality. Just a guess since the code itself looks OK and the test results look OK.
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
Thank you very much!
yes, I can confirm it is there, at least it was all day for me on that very code and right now. (yet it is funky atm, see end of post)
can you run it with several loops? It is often happening on the second loop or later. Does account you run has real time subscription? (see end of post, why maybe one really need subscription not 100% sure yet)
?
Using args Namespace(port=4002, global_cancel=False, file='symbols.csv', market_data_type='REALTIME', data_lines=75, loops=5)
serverVersion:187 connectionTime:b'20241219 16:56:08 EST'
1713296 Waiting... Symbols in work : 75, done symbols: 0
1714476 Waiting... Symbols in work : 75, done symbols: 1180
TIMEOUT ? ? ?reqId: 1714133, symbol: MU CANCEL ? ? ?TIMEOUT!!!!
1714504 Waiting... Symbols in work : 18, done symbols: 1264
? ? ? ? ? ? ?Number of not done symbols: 1
reqId ? ? symbol done ?bid ? ? ask ? ? last ? ?volume ?high ? ?low ? ? last_timestamp
1714133 ? MU ? ? 0 ? ? N/A ? ? N/A ? ? 87.03 ? 904214 ?91.0 ? ?84.61 ? 2024-12-19 22:56:09 ? ? ? ?
However I did notice 2 things
- I have run the test after hours and the frequency of errors is significantly less. I get 1 failure per 5 loops. whereas during trading hours I was getting maybe 5-20 per same test. So maybe it is related somehow on how IB infra is loaded? i.e. the infra is probably less loaded now (just a guess)
- I can repeat the issue for REALTIME and DELAYED options on paper trading account which HAS market data subscriptions. However, at least now, after hours trying it on the account which DOESN'T have subscription doesn't generate issues with DELAYED. Maybe it has to do with after hours or maybe DELAYED means different things for account with subscription and without. I don't know.? So I can confirm the issue for account with subscription, yet after hours it is less rare one really need to run 5 loops to be sure.?
?
? ?
?
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
On Wed, Dec 18, 2024 at 12:08 AM, ajn wrote:
Actually, running against a paper account is perfectly fine (this is how I run
it all day). I believe running it against a paper account with no subscription
will also work (need to set DELAYED data).
I presume you mean the problem will still exhibit itself. Can you confirm? Because I just ran your last sample program (w/ inconsequential changes for convenience; see attached) against a demo acct using DELAYED and it finished successfully.
- TWGW ver: 10.30.1s
- IBAPI ver: 1030.01
- Platform: Linux 4.4.0-260-generic #294-Ubuntu SMP Fri Sep 27 16:14:57 UTC 2024 x86_64 GNU/Linux
Output:
Using args Namespace(data_lines=65, file='symbols.csv', global_cancel=False, loops=1, market_data_type='DELAYED', port=4002)
serverVersion:187 connectionTime:b'20241219 15:00:41 EST'
NextValidId received: 1
NextValidId set : 1000009
Executing requests
LOOP 0 START
1000074 Waiting... Symbols in work : 65, done symbols: 0
1000209 Waiting... Symbols in work : 65, done symbols: 135
1000334 Waiting... Symbols in work : 65, done symbols: 260
1000459 Waiting... Symbols in work : 65, done symbols: 385
1000584 Waiting... Symbols in work : 65, done symbols: 510
1000709 Waiting... Symbols in work : 65, done symbols: 635
1000834 Waiting... Symbols in work : 65, done symbols: 760
1000959 Waiting... Symbols in work : 65, done symbols: 885
1001080 Waiting... Symbols in work : 65, done symbols: 1006
1001207 Waiting... Symbols in work : 65, done symbols: 1133
1001292 Waiting... Symbols in work : 24, done symbols: 1259
Number of not done symbols: 0
reqId symbol done bid ask last volume high low last_timestamp
DONE reqMktData_test 51.7 seconds
ALL Tests DONE
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
Just run a couple of more tests (using the app I posted, just changed False to True and commented cancel request). Snapshot works 100% reliable as Daniel experiences i.e. snapshot delivers all that is expected and no errors printed
self.reqMktData(orderId,?contract,?"", True,?False, [])
live data (4th argument is False) oth is the case which has the problems
self.reqMktData(orderId,?contract,?"",?False,?False, [])
It is live data is bit broken in gateway 10.30+ . We are making some progress :)
?
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
E.g. in C#: client.reqMktData(reqId, contractObject, string.Empty, true, false, null);
yes it is the same call I am making, except I am not asking for snapshot (4th argument is False) but whole set of data to be streamed to me
self.reqMktData(orderId, contract, "", False, False, [])
?
Last time I checked the difference between asking for a snapshot and streaming, snapshot was much more delayed (but that was 4-5 years ago, maybe things improved since). Still streaming or snapshot, api is promising the data, we pay for it. We should be getting it :)?
?
And even in this test. last was there, In this case bid/ask was missing. Sometimes "last_timestamp" (time of last trade) is missing, sometime high/low/volume. bit random. I believe last is present most of the time
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
Hi AJ,?
I connect via TWS and my TWS version is 10.30.1s, whereas the version of my CSharpAPI.dll is?10.15.2.0.?
"I didn't receive all of ["bid", "ask", "last", "volume", "last_timestamp", "high", "low"]"
?I think this might be the difference. I only request snapshots every few seconds.?
E.g. in C#: client.reqMktData(reqId, contractObject, string.Empty, true, false, null);
As said, 50.000+ requests per day and working fine. Of?course, I only get 'the last Last' snapshot, for my app that's fine.
Hope it helps, Daniel.
In the attached zip I have the app. It is now only 250 lines including all the IB connection and argument parsing. The working function are all the same as before ( described earlier) I just deleted everything else. I do agree, as a standalone app it gets much more readable ?
Program_get_bid.py -f symbols.csv -m REALTIME --data-lines 75 --loops 5 --port 4001
?
As an added bonus this app can be run from anywhere, not just from python samples folder. so long your python environment has 10.30 installed
?
AJ
-- Daniel
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
In the attached zip I have the app. It is now only 250 lines including all the IB connection and argument parsing. The working function are all the same as before ( described earlier) I just deleted everything else. I do agree, as a standalone app it gets much more readable ?
Program_get_bid.py -f symbols.csv -m REALTIME --data-lines 75 --loops 5 --port 4001
?
As an added bonus this app can be run from anywhere, not just from python samples folder. so long your python environment has 10.30 installed
?
AJ
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
Thanks Daniel,
?
Just to be clear the problem is not that I am not receiving the callbacks at all. The problem is that I am receiving not entire set of? the data I expect for some of the contracts (always random which contracts will be missing some data). Sometime it is bid which is missing, sometimes ask, sometimes last etc. bit random misses. And everytime I check gateway log for the request I had issues with It is clear that that piece of data is not there (say I see that for req 123456 I get bid, ask, volume but not a last value etc.).?
?
First off: note that `Contract.primaryExch` is not the same as `Contract.Exchange`. You might be constructing invalid Contract objects.?
I am 99.99% it is not the contract issue, as I am getting most of data,?
?
Moreover, today I was experimenting with unsubscribing/subscribing for data for a contract which at this point was missing data (my definition of "missing" is that after subscription I didn't receive all of ["bid", "ask", "last", "volume", "last_timestamp", "high", "low"] within 20 seconds at least once, typically one or two would be missing), and 100% of time I was receiving the data after a second subscription. So contract part is very likely fine.?
?
Possible issues:
1: make sure you comply with the max 50 requests per second pacing violation. I don't know if you'll receive an error message if that's your case.
2: error code 10197 "No market data during competing session": if you're pulling reqMktData and at the same time visualizing the data in TWS, you might get this error and hence not receive an eWrapper callback.?
Yes, I monitor error codes. no such errors. As for pacing, it used to be needed but SDK I believe build it in and there is an option ?setConnectOptions("+PACEAPI"), which I use for several years now on top of own pacer anyway (just ot be safe, remember first versions were glitchy). And typically pacing errors result that you don't get any data at all. which is not current case.?
?
About reqMktData: just for fun I checked my code and out and yesterday I made 54.000 calls to it, all received a callback except few
Great stats, :) it used to be like this for me up and including gateway 10.19.?
- what is your definition of "received"? are you looking for say "last" only or a bit wider scope like I mentioned above?
- What version of gateway/TWS are you using (not API) ?? Gateway 10.19 and older work fine for me (and have been for 5 years), yet once I switch gateway to 10.30 or 10.31 I immediately start seeing issues.?
AJ
|
Re: No definition for SPX option using LocalSymbol
Thanks.?
I've already tried using US format like SPXW ?241218C05945000 but for some reason it still couldn't find the contract.?
I just tried again and it seems to work now.?
|
Re: No definition for SPX option using LocalSymbol
Yes, I mentioned that I can use TradingClass to separate monthly and weekly options. The problem is that it is supposed to be generic method that I could use to form a contract for any security type, SPX index option, QQQ stock option, ES futures option. In this case, it is not clear what I should provide in the TradingClass property.?
?
If there is a strict rule that defines how TradingClass needs to be set, e.g. symbol name + expiration type = SPXW, then I could use TradingClass instead of LocalSymbol but I haven't seen any docs describing how to set TradingClass depending on SecType. LocalSymbol does define specific format for all type of options, it is more generic, so I decided to go with it.?
|
Re: No definition for SPX option using LocalSymbol
The Contract Inspector shows you the actual local symbol to use. US options and European options have different local symbol formats: you’ve tried to use the European format for a US option. ? The local symbol you need is: ? SPXW? 241218C05945000 ? which is probably in your screenshot, except it’s too small and fuzzy to read. ? Richard ? ?
toggle quoted message
Show quoted text
From: [email protected] <[email protected]> On Behalf Of Andy Sanders via groups.io
Sent: 19 December 2024 05:16
To: [email protected]
Subject: [TWS API] No definition for SPX option using LocalSymbol? Could somebody advise why I'm getting "No security definition" error for a contract with this local symbol : C SPX 20241218 5945 W Trying to send order to buy and sell SPX index options. When I set separate contract properties such as strike, symbol, expiration, currency, and exchange, place order request returns error "ambiguous definition" because API returns the same expiration date for monthly and weekly options.? There are several ways to make contract definition for option unique, e.g. using TradingClass or LocalSymbol properties. When I try to use LocalSymbol I get error "No security definition found" Example of local symbol that I used : C SPX 20241218 5945 W Format mentioned in the docs : ?
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
Hi AJN.?
First off: note that `Contract.primaryExch` is not the same as `Contract.Exchange`. You might be constructing invalid Contract objects.?
About reqMktData: just for fun I checked my code and out and yesterday I made 54.000 calls to it, all received a callback except few, like ~50, because of the reason #2 mentioned below.?
Possible issues: 1: make sure you comply with the max 50 requests per second pacing violation. I don't know if you'll receive an error message if that's your case. 2: error code 10197 "No market data during competing session": if you're pulling reqMktData and at the same time visualizing the data in TWS, you might get this error and hence not receive an eWrapper callback.?
Note I'm in C# API 10.15.2.0 - no capacity to test on another version at?the moment.
Hope it helps.?
I created a second version where I removed all code which is not absolutely necessary It is roughly 150 line of code which one easily sees comparing Program.py from samples to Program_get_bid.py. As per first post: The file is part of \samples\Python\Testbed?extract provided python.zip in that folder Then, run (or even better run from debugger and step)
Program_get_bid.py -f symbols.csv -m REALTIME --data-lines 75 --loops 5 --port 4001
?
There is 3 new functions left. Main entry , runs the test in the loop  ?
Actual code subscribing to data. making sure the datalines limit is not exceeded. And knowing that Subscription is expensive I typically run 65% of line limit. I.e. for test accounts where I have only 100 lines I would run 65 max. So we never run into "max data lines limit reached" error.?
?
monitor_not_done is called after each subscription and stops new subscribtions once we reach the limit or (finalize = True) simply waits until all subscriptions returned the data or Timed out.
?
The wrappers are easier described by comparison to Program.py the original test app. Really all wrappers do is saving incoming data
?
and there is unsubscribing bit which is driven by wrappers (here on example of error), if all data is present the cancel is called and sets "sentCancelMktData" which gives signal to test loop that one received all the data
?
That is pretty much it.?
?
|
Re: reqMktData callbacks are missing data with gateway 10.30 and 10.31 sporadically, works fine with 10.19 and older gateways/TWS
On Wed, Dec 18, 2024 at 08:09 PM, ajn wrote:
where I removed all code which is not absolutely necessary It is roughly 150
line of code which one easily sees comparing Program.py from samples to
Program_get_bid.py.
Pull it out and make it stand alone. Get rid of ALL SUPERFLUOUS things like colorization constants... or no one will want to help you.
|
buddy