Keyboard Shortcuts
Likes
- Twsapi
- Messages
Search
Re: Tips for fastest performance with IBG?
Ben Alex
On Mon, Dec 7, 2015 at 11:31 AM, fantasyfootball672@... [TWSAPI] <TWSAPI@...> wrote: the traditional "command line" memory parameters are becoming deprecated, meaning garbage collection is essentially out of my hands I should clarify there are still hundreds of JVM garbage collector parameters you can tune. The permanent generation removal has been offset by the . My key point was JVM tuning options aren't likely to help very much in the case of IB Gateway due to its probable network IO bound workload, small heap and reasonable JIT hotspot defaulting. The non-JVM options are way more likely to give you value. But each have trade-offs (usually greater economic costs) for the improved latency outcomes. Even order replacement has costs with IB, so you should always check the fees involved. For example many folks aren't aware API placed orders (at least for US equities on the fixed pricing structure) cost 50% more than orders placed via TWS. So always . :-) |
||||||||||||||||||||
Re: Tips for fastest performance with IBG?
Nick
I have seen this suggested by several different people but I have always been skeptical.
toggle quoted message
Show quoted text
If I submit an order at 9am that passes margin checks and then proceed to loose 90% of my account by noon, I can't bring myself to believe that IB will allow the now non-compliant order to fill when I change it. In other words I believe there has to be some validation at execution time, not just submit time. It may be that modifying an order has less overhead than submitting a new order but I can't believe that IB would expose itself to changes in a client's risk profile that happened after the order was submitted. On 12/6/2015 7:07 PM, Ben Alex ben.alex@... [TWSAPI] wrote:
* Place orders away from the market price so that risk management checks have already been completed, then modify the order when ready to invoke. |
||||||||||||||||||||
Re: Tips for fastest performance with IBG?
Ben, this is great. I don't understand Java so I appreciate the "answer" that there isn't much that can be done since the source is closed and the traditional "command line" memory parameters are becoming deprecated, meaning garbage collection is essentially out of my hands. I also appreciate the "modify order" tip- that is very, very cool. My own trading tool is compiled C++ and is very fast, and the IB market data "feed" such as it is is almost certainly the biggest source of latency, especially with your tip for the credit workaround.
|
||||||||||||||||||||
Re: Tips for fastest performance with IBG?
Ben Alex
What type of speed improvements are you looking for? When most people talk about speed and trading they are trying to achieve a specific latency outcome, ie completion of a given operation within a bounded amount of time. That's different from say the 99th percentile completion time, where outliers are of minimal consequence (eg batch applications, web applications etc). IB Gateway, TWS and IB Controller are JVM hosted applications that cannot be embedded within your own application (ie you can't run them in the same process). In addition, IB Gateway and TWS are closed source applications where you cannot look at (let alone improve) their threading, network use, object allocations/deallocations, retry/timeout mechanisms and so on. You're limited to using the provided wire level APIs (ie IB API or FIX). Assuming you need to use IB Gateway, the most you can do is (a) adjust JVM properties and (b) consider non-JVM issues. First, the good news. In general the easiest way to obtain the best performance for almost any Java application is to upgrade to the latest released JVM. As an aside the new Install4J-based distribution of TWS 954 bundles Java 1.8.0_65, which is pretty close to the latest (1.8.0_66 is ). Actual JVM configuration is also quite easy, although unlikely to help. That's because most JVM configuration latency wins come from adjusting memory management and JIT compilation settings. IB Gateway does not use a large amount of heap relative to normal Java applications. If you needed an 8+ GB of heap to run IB Gateway, tuning the garbage collector may prove useful. However for a few hundred megabytes (like IB Gateway uses) it isn't going to make much difference (and in many cases it's likely to actually be worse than using the defaults that Oracle have selected based on CPU class, memory size and operating system). The hotspot options can also control exactly when the JVM inlines certain methods. Like a (PGO) build for languages like C++, there's a requirement to wait for enough workload to complete so the routine code paths can be profiled (ie "warm up" the JVM). Doing it too soon (and there are JVM arguments that force it to happen immediately etc) and you will actually end up with worse performance than the default bytecode execution. But given IB Gateway is mostly a converter between local wire formats (IB API / FIX) and a proprietary wire format (to the IB backend), you are almost certainly going to be blocking on network IO as opposed to CPU or memory use. If you are wanting to play around with JVM options anyway, I assume your IBController install is based off the batch file. In that batch file is the JAVAOPTS setting. There's room for improvement, for example -XX:MaxPermSize option is now deprecated in Java 8, as the permanent generation has been removed (see in Java 8). What's an appropriate value? Well as of TWS 954 the officially-provided Install4J launch doesn't set any memory options for Java. So if you do anything to your JAVAOPTS, I'd be removing them unless benchmarking shows they're useful. A nice introductory book if you want to dig into JVM performance tuning is Scott Oak's . It covers you to use included JVM tools (eg visualvm, jcmd, jinfo, jstack etc) to get a better idea what's happening in your JVM and start tuning away. But again, you're extremely unlikely to gain very much improvement for IB Gateway. The biggest opportunity for meaningful latency improvement is likely to come from other techniques that have been mentioned on this list at various times. They basically boil down to order tuning, connection tuning and machine tuning: * Place orders away from the market price so that risk management checks have already been completed, then modify the order when ready to invoke. * Profile and control order routing (SMART vs directed orders). Note the execution price, liquidity rebate/charge and commissions though. * Colocate nearer IB. Commercial data centre hosting will have lower average latencies than SOHO connections. * Buy a commercial-grade, dedicated network connection to IB. Check out . * Reduce load on IB Gateway by not using it for market data. Specialist data vendors offer hyper-compressed, UDP-delivered feeds (eg Nanex). * Typical OS-level issues: latest versions, patches, malware/virus/privacy scanner settings, unnecessary services, network settings, swap config etc. The above isn't a complete list by any stretch. I've personally found minimal latency issues during realtime trading. To be using IB in the first place effectively rules out most low latency strategies. If you can expand on the specific speed improvements you're looking for, others might be able to chime in with more suggestions. On Mon, Dec 7, 2015 at 8:23 AM, fantasyfootball672@... [TWSAPI] <TWSAPI@...> wrote:
|
||||||||||||||||||||
Re: Download page for IB Gateway...
Ben Alex
I'm on Linux so I can't be sure what the OSX installer does. However I assume they're similar. The following was tested with TWS release 954.2a. On Linux the Install4J-based installer mainly installs TWS into $HOME/Jts/954). In there is a .desktop file which runs $HOME/Jts/954/tws, the latter of which is a lengthy shell script that runs Java and specifies jclient.LoginFrame as the main method. What you need is to use ibgateway.GWClient as the main method.So if OSX is similar, what I'd do is copy the file to a new file (the paths may differ on your machine, depending on where Install4J wrote the files): Then change the main method from "jclient.LoginFrame" to "ibgateway.GWClient" (any text editor can do this, but here's the sed command for clarity): sed -i $HOME/Jts/954/ibgw 's/jclient\.LoginFrame/ibgateway\.GWClient/' ibgw $HOME/Jts/954/ibgw The above launched IB Gateway just fine for me. Maybe OSX gives you an IB Gateway launcher and you can save yourself the inconvenience of this. However the Linux Install4J only created TWS launchers. The other option is to use IBController, which specialises in loading IB Gateway (or TWS) and dealing with its assorted authentication needs, popups and similar. |
||||||||||||||||||||
Tips for fastest performance with IBG?
Are there any tips to eek out any speed improvements when implementing IB Gateway (IBG) for the access to IB. My OS is Win7 Pro. Using IB Controller to invoke IBG. I'm wondering about the Setting in IBG, and perhaps about system-level settings that promote IBG(Java) efficiency. I have lots of RAM and have a RAM Disk available if that might help. I'm using an SSD of course. Gigabit ethernet of course. |
||||||||||||||||||||
Strange end of Historical Data?
alvinxw
Hi, I am trying to get 1 year of daily historical data for stock "APA". ?Below is the last a few days' high prices my program printed: 20151125: 50.94 20151127: 50.19 20151130: 49.99 20151201: 49.73 20151202: 49.28 20151203: 48.1 20151204: 46.04 finished-20141213 ?00:00:00-20151205 ?00:00:00: -1.0 public synchronized void historicalData(int reqId, String date, double open, double high, double low, double close, As you can see the last date (String) I received is not a "date" but a strange string "finished-20141213 ?00:00:00-20151205 ?00:00:00". Is it a bug a designed signal for end of historical data? thanks |
||||||||||||||||||||
Re: Download page for IB Gateway...
Ben, thanks for pointing out. The comments you referred to seem to interchangeably refer to TWS and IBGateway. Is it possible to somehow run the same download "as" IBGateway, or are you just talking about both at the same time? Or do you actually mean IBGateway, most of the time, when you refer to TWS?
|
||||||||||||||||||||
Re: Download page for IB Gateway...
Ben Alex
Last week I encountered what is probably a related issue, being that IB significantly changed the TWS release format between 952 and 953. They refer to this in the version 953 release notes as the . Version 952 and earlier were available as smallish (~53 MB) JARs, whereas now they release large (~123 MB) operating system specific Install4J installers. These installers embed a specific JRE release. Unfortunately IB still do not version the standalone install URLs, nor even provide a way to determine the version number without running the installer. I can see the merit of the change for most of their users, but this new launcher approach imposes inconvenience if you run automated solutions to monitor version numbers, package TWS or unattended install TWS/IB Gateway. If anyone reading this would like scripts that have been updated to address the new installer mechanism, please see the comments at the end of . |
||||||||||||||||||||
Re: Historic data req is rejected: SP500 with TRADE type (not MIDPOINT). Why?
It could be spelling.? This is from the API
docs:
toggle quoted message
Show quoted text
whatToShow String Determines the nature of data being extracted. Valid values include:
At 07:37 AM 12/5/2015, you wrote: I'm trying to get Historic data from the ES ticker on GLOBEX and with type TRADE, and the request is rejected. If I ask for MIDPOINT it works. |
||||||||||||||||||||
Download page for IB Gateway...
...seems to be gone. I kept checking the download page for a standalone version of IB Gateway (Mac OS X) that would correspond to TWS "latest" version 954.2 (or newer), but there was only an older one from March 2015.
Now the download section for IB Gateway seems to be gone completely. Is this lately a normal thing before the arrival of new versions, or something like that? |
||||||||||||||||||||
Re: OT: An app I've been working on
FWIW:
? I have enjoyed this sort of thing "forever" -- even back to The Journal Of Mathematical Psychology -- which was the vehicle by which Kahneman, Tversky, et.al. got their stuff published in the early/mid 80s. Finding such an official-sounding word like "Quorum sensing" re-assures me that, if I won The Lottery and funded my own fun-for-thinking Institution, "Quorum sensing" might be in the title. (Or at least on somebody's door, right?) In the meantime, you can bet your bippy that those who sample word-flow from Twitter, StockTwits, etc. have just these sorts of frameworks in place. (I want my own! Ha-HA!) |
||||||||||||||||||||
Re: ECBOT historical data
No, you don¡¯t need to use the expiry field.
It works just fine with secType=¡±FUT¡±, exchange=¡±ECBOT¡±, and localSymbol=¡±ZB DEC 15¡±. The only difficulty is that different exchanges have different local symbol formats. For Globex, Nymex and many other futures exchanges the ¡®short¡¯ format that you¡¯ve always used (eg CLF6, ESZ5, 6EH6) is correct, but for ECBOT, DTB (and possibly others) the SSSS MMM YY format is required, eg ZC DEC 16, FDAX DEC 15. The trick with this ¡®long¡¯ format is to ensure that there are 5 characters before the first month character, padding with spaces if need be. So for a two character symbol like ZC, you need three spaces after it, for a four character symbol like FDAX you need one space after it. What¡¯s even trickier is that what you put in the SSSS part is not necessarily the plain symbol. For example, the Euro STOXX50 contracts at DTB have a symbol of ESTX50, but in the local symbol you have to use FESX, thus ESTX50 with an expiry date of 201512 has a local symbol of FESX DEC 15. Simlarly for DAX you use FDAX in the local symbol (eg FDAC DEC 15), and for GBL you use FGBL (eg FGBL DEC 15). Simple, huh?... By the way, the long format is arguably much more logical, because it¡¯s immune to changes of decade. For example I have historical data stored back to 2003, so I need contract definitions in my database for the ESZ5 in both 2005 and 2015. But I use the local name as a key to access the data, so I¡¯ve had to change all the old ones to put the decade in as part of the local symbol: thus the ESZ5 for Dec 2005 is now called ESZ05 in my database. Whereas FDAX DEC 05 and FDAX DEC 15 are perfectly clear (I probably won¡¯t care when we wrap round to the next century!). Richard From: TWSAPI@... [mailto:TWSAPI@...] Sent: 03 December 2015 02:02 To: TWSAPI@... Subject: Re: [TWS API] Re: ECBOT historical data Thanks to those who responded. I have it working now. To summarize, the trick for the ECBOT contracts is to specify the expiry in the Contracts object. For stocks, forex and all other futures, I have never used this field. I have left it "" and I have always used just the localSymbol field, e.g. CLF6. For some reason IB chose to make the localSymbol difficult for this exchange as one of the responders pointed out. I have watched this forum for a long time and am amazed that I am the only one who has had this problem. I guess when everyone else started programming this interface that they started using the expiry field at the outset. Phil On Wednesday, December 2, 2015 12:48 PM, "fantasy football fantasyfootball672@... [TWSAPI]" <TWSAPI@...> wrote: Phil, It's a very annoying system for the "Local Symbol" with 3 embedded spaces: For instance, Corn for December is: "ZC DEC 15" [Non-text portions of this message have been removed] |
||||||||||||||||||||
Re: ECBOT historical data
Thanks to those who responded. I have it working now. To summarize, the trick for the ECBOT contracts is to specify the expiry in the Contracts object. For stocks, forex and all other futures, I have never used this field. I have left it "" and I have always used just the localSymbol field, e.g. CLF6. For some reason IB chose to make the localSymbol difficult for this exchange as one of the responders pointed out. I have watched this forum for a long time and am amazed that I am the only one who has had this problem. I guess when everyone else started programming this interface that they started using the expiry field at the outset. Phil On Wednesday, December 2, 2015 12:48 PM, "fantasy football fantasyfootball672@... [TWSAPI]" wrote:
?
Phil, It's a very annoying system for the "Local Symbol" with 3 embedded spaces: For instance, Corn for December is: "ZC ? DEC 15" |
||||||||||||||||||||