Re: Latency of reqTickByTickData
Thanks for the links Mario. ? The first one in particular gives very useful information about the requirements for achieving very high accuracy with Windows 10 and Windows Server 2016. ? But it needs to be emphasised that these are the requirements to be able to guarantee the relevant level of accuracy, in such a way that you will be supported by Microsoft. But if you don’t meet the requirements it doesn’t meant that you won’t get a good result, just that you’re on your own as regards Microsoft support. ? As I indicated in my earlier post, I get pretty good results with Windows Server 2012 R2, and I’m nowhere near meeting any of those sets of requirements. I’ve now also done the same configuration on Windows 10 Professional on my rather underpowered Surface Pro 2, and that’s pretty good too. If I run a heavy workload on the Surface, the offsets do become a bit erratic. But my server is much more stable, probably because even though it’s running TWS and Gateway, two 40 GByte SQL Server databases, a mail server, and three API data collection programs, it’s actually very lightly loaded as it has 16 virtual processors and those sort of workloads just aren’t very processor intensive (essentially no significant graphics). ? Michael’s posts are interesting, though I must admit to being a bit baffled as to what exactly they’re showing me – I’d need to spend more time working through them to get a clearer understanding. But given that they’re Windows 7, I can’t say I’m surprised that the synchronisation isn’t very good (especially if it’s using the default W32Time settings)? - heck, Windows 7 is 11 years old now, and I certainly have no intention of ever using it again: in my opinion it’s positively out-of-date in practically every way in comparison to Windows 10. ? So my contention is that with an up-to-date version of Windows, with the Windows Time Service correctly configured (which means changing it from the default settings), you’ll get pretty good time syncing even if you’re not close to a stratum 1 time server (mine are about 30ms and 10-15 hops away). ? Richard ? ?
toggle quoted message
Show quoted text
From: [email protected] <[email protected]> On Behalf Of Mario Pisa
Sent: 08 September 2018 09:17
To: [email protected]
Subject: Re: [TWS API] Latency of reqTickByTickData?
Best regards
El 7 sept 2018, a las 22:46, michaelIC via Groups.Io <michael.i.c@...> escribió:
My bad.
I had a nagging thought, so I checked the log files from those runs.
The units across the bottom are not seconds, but the poll number.
The polls were done every three seconds.
So 1 is the first.
2 is the second, three seconds after the first.
3 is the third, three seconds after the second, etc..
So #20 is one minute into the run.
#40, two minutes.
|
Re: Install TWS or IB gateway on Ubuntu and ARM64 bit processor
Thanks for that Scott. Looks like there’s no need for me to do anything. ? Derek, just download and install the ARM 64 Java SE Development Kit from here: ?
? The script that Scott offered should locate the Java and do everything to launch Gateway without further ado, once you’ve copied the Jts folder over as Scott describes. ? Good luck. ? Richard ? ? ?
toggle quoted message
Show quoted text
From: [email protected] <[email protected]> On Behalf Of MochaSatin
Sent: 08 September 2018 21:25
To: [email protected]
Subject: Re: [TWS API] Install TWS or IB gateway on Ubuntu and ARM64 bit processor? Derek, I remembered a few other things so here is a follow up on running gateway on an ARM.? The installation will fail when installing a fresh IB or TWS download on an Arm Linux machine.? So the workaround is to install on an x86-X64 desktop, then compress and copy the 'Jts' folder onto the Arm filesystem.? Then used a modified script like the one I sent in the last post.? Make sure you install the Jts on the Arm in an identical file structure as your x86-x64 or you will get path errors.? For example, install in "/home/johndoe/Jts" in both the x86-64 and the Arm. As Richard indicated, the jar files will run fine on the Arm once started correctly, it is the install and startup scripts that are messing things up.?? ? On Sat, Sep 8, 2018 at 3:59 PM, MochaSatin <scott@...> wrote: Derek, You have to hack the ibgateway script to run correctly on Arm processor.? Do a diff on the attached script to see the changes I made to run the gateway.? I have run it on both a pi3 and and xu4. ? On Sat, Sep 8, 2018 at 5:20 AM, Derek Fung <ibmderekfung@...> wrote: Hi Richard, Apologize for the late response.? Last week was engaged with other urgent things. Yes, I am interested to try, please share with me.?? And which version of Orace Java for ARM that I should install? ? On Tue, Sep 4, 2018, 01:22 Richard L King <rlking@...> wrote: I think the problem here is that the TWS and Gateway installers now include their own ‘hidden’ copy of Java, this being an Oracle Java that I pretty much guaranteed to work well with that particular version of TWS/Gateway. The rationale for this is that a) it means the user doesn’t have to take the trouble to install and maintain a suitable Java version; and b) this should reduce support problems for IB because in theory the user will not accidentally use an incompatible Java version. ? Now, the actual Java compiled class files, being pure Java, are I believe identical on all platforms for TWS and Gateway, and should run with any Java VM. So in theory you should be able to install TWS/Gateway on another machine (Ubuntu perhaps or even Windows) and then just copy the relevant jar files over to the ARM machine. You’d then need to install a suitable Java on the ARM machine: I don’t know whether Oracle produce a Java version for the ARM architecture, but if they don’t you might be able to find another implementation. Ah, I’ve just discovered that indeed Oracle do support ARM. ? Once you’ve got the JAR files and the Java runtime in place, it should just be a matter of using an appropriate Java command with the correct arguments to start the thing off. I can help with that because that’s exactly what IBC/IBController do under the covers. ? So if you think this might be an avenue worth pursuing, then please let me know and get Oracle Java installed . I’ll then post the details of which jar files to copy, and the appropriate Java command (I can’t be bothered to go and look this up unless you’re actually going to use it). ? Needless to say, there’s no guarantee this will work, but there’s also no obvious reason why it won’t. ? Richard ? ? ? ? This is the error screen I have.? Seems it is related to Java.? Hey Derek,
I use ubuntu all the time for my gateway and tws machines, the ARM64 thing might be a bit odd though, I've no experience with that.
Does it state that your java version is wrong or is that a guess? If its as statement then it's a fairly good thing to look at updating, if it's a guess, well... ummm... maybe?! Best wishes, M ? On 3 September 2018 at 17:06, Derek Fung <ibmderekfung@...> wrote: Hi, Has anyone installed TWS or IB gateway successfully on ARM64 bit processor and Ubuntu?? I have problem in installation. Guess it is because the JVM is not supported.? Any suggestion?
--
(T) +44 (0) 131 2202413 ?? (M) +44 (0) 7528 551604 ? (W)
Gulfstream Software - Winner Risk Management Awards 2010
This message is subject to :
?
?
|
Re: Install TWS or IB gateway on Ubuntu and ARM64 bit processor
Derek,
I remembered a few other things so here is a follow up on running gateway on an ARM.? The installation will fail when installing a fresh IB or TWS download on an Arm Linux machine.? So the workaround is to install on an x86-X64 desktop, then compress and copy the 'Jts' folder onto the Arm filesystem.? Then used a modified script like the one I sent in the last post.? Make sure you install the Jts on the Arm in an identical file structure as your x86-x64 or you will get path errors.? For example, install in "/home/johndoe/Jts" in both the x86-64 and the Arm. As Richard indicated, the jar files will run fine on the Arm once started correctly, it is the install and startup scripts that are messing things up.??
Scott
On Sat, Sep 8, 2018 at 3:59 PM, MochaSatin <scott@...> wrote: Derek,
You have to hack the ibgateway script to run correctly on Arm processor.? Do a diff on the attached script to see the changes I made to run the gateway.? I have run it on both a pi3 and and xu4.
Scott
|
Re: Install TWS or IB gateway on Ubuntu and ARM64 bit processor
Derek,
You have to hack the ibgateway script to run correctly on Arm processor.? Do a diff on the attached script to see the changes I made to run the gateway.? I have run it on both a pi3 and and xu4.
Scott
On Sat, Sep 8, 2018 at 5:20 AM, Derek Fung <ibmderekfung@...> wrote: Hi Richard,
Apologize for the late response.? Last week was engaged with other urgent things.
Yes, I am interested to try, please share with me.??
And which version of Orace Java for ARM that I should install?
Thanks!
Derek On Tue, Sep 4, 2018, 01:22 Richard L King < rlking@...> wrote: I think the problem here is that the TWS and Gateway installers now include their own ‘hidden’ copy of Java, this being an Oracle Java that I pretty much guaranteed to work well with that particular version of TWS/Gateway. The rationale for this is that a) it means the user doesn’t have to take the trouble to install and maintain a suitable Java version; and b) this should reduce support problems for IB because in theory the user will not accidentally use an incompatible Java version. ? Now, the actual Java compiled class files, being pure Java, are I believe identical on all platforms for TWS and Gateway, and should run with any Java VM. So in theory you should be able to install TWS/Gateway on another machine (Ubuntu perhaps or even Windows) and then just copy the relevant jar files over to the ARM machine. You’d then need to install a suitable Java on the ARM machine: I don’t know whether Oracle produce a Java version for the ARM architecture, but if they don’t you might be able to find another implementation. Ah, I’ve just discovered that indeed Oracle do support ARM. ? Once you’ve got the JAR files and the Java runtime in place, it should just be a matter of using an appropriate Java command with the correct arguments to start the thing off. I can help with that because that’s exactly what IBC/IBController do under the covers. ? So if you think this might be an avenue worth pursuing, then please let me know and get Oracle Java installed . I’ll then post the details of which jar files to copy, and the appropriate Java command (I can’t be bothered to go and look this up unless you’re actually going to use it). ? Needless to say, there’s no guarantee this will work, but there’s also no obvious reason why it won’t. ? Richard ? ? ? ? This is the error screen I have.? Seems it is related to Java.? Hey Derek,
I use ubuntu all the time for my gateway and tws machines, the ARM64 thing might be a bit odd though, I've no experience with that.
Does it state that your java version is wrong or is that a guess? If its as statement then it's a fairly good thing to look at updating, if it's a guess, well... ummm... maybe?! Best wishes, M ? On 3 September 2018 at 17:06, Derek Fung <ibmderekfung@...> wrote: Hi, Has anyone installed TWS or IB gateway successfully on ARM64 bit processor and Ubuntu?? I have problem in installation. Guess it is because the JVM is not supported.? Any suggestion?
--
(T) +44 (0) 131 2202413 ?? (M) +44 (0) 7528 551604 ? (W)
Gulfstream Software - Winner Risk Management Awards 2010
This message is subject to :
|
Re: Install TWS or IB gateway on Ubuntu and ARM64 bit processor
Hi Richard,
Apologize for the late response.? Last week was engaged with other urgent things.
Yes, I am interested to try, please share with me.??
And which version of Orace Java for ARM that I should install?
Thanks!
Derek
On Tue, Sep 4, 2018, 01:22 Richard L King < rlking@...> wrote: I think the problem here is that the TWS and Gateway installers now include their own ‘hidden’ copy of Java, this being an Oracle Java that I pretty much guaranteed to work well with that particular version of TWS/Gateway. The rationale for this is that a) it means the user doesn’t have to take the trouble to install and maintain a suitable Java version; and b) this should reduce support problems for IB because in theory the user will not accidentally use an incompatible Java version. ? Now, the actual Java compiled class files, being pure Java, are I believe identical on all platforms for TWS and Gateway, and should run with any Java VM. So in theory you should be able to install TWS/Gateway on another machine (Ubuntu perhaps or even Windows) and then just copy the relevant jar files over to the ARM machine. You’d then need to install a suitable Java on the ARM machine: I don’t know whether Oracle produce a Java version for the ARM architecture, but if they don’t you might be able to find another implementation. Ah, I’ve just discovered that indeed Oracle do support ARM. ? Once you’ve got the JAR files and the Java runtime in place, it should just be a matter of using an appropriate Java command with the correct arguments to start the thing off. I can help with that because that’s exactly what IBC/IBController do under the covers. ? So if you think this might be an avenue worth pursuing, then please let me know and get Oracle Java installed . I’ll then post the details of which jar files to copy, and the appropriate Java command (I can’t be bothered to go and look this up unless you’re actually going to use it). ? Needless to say, there’s no guarantee this will work, but there’s also no obvious reason why it won’t. ? Richard ? ? ? ? This is the error screen I have.? Seems it is related to Java.? Hey Derek,
I use ubuntu all the time for my gateway and tws machines, the ARM64 thing might be a bit odd though, I've no experience with that.
Does it state that your java version is wrong or is that a guess? If its as statement then it's a fairly good thing to look at updating, if it's a guess, well... ummm... maybe?! Best wishes, M ? On 3 September 2018 at 17:06, Derek Fung <ibmderekfung@...> wrote: Hi, Has anyone installed TWS or IB gateway successfully on ARM64 bit processor and Ubuntu?? I have problem in installation. Guess it is because the JVM is not supported.? Any suggestion?
--
(T) +44 (0) 131 2202413 ?? (M) +44 (0) 7528 551604 ? (W)
Gulfstream Software - Winner Risk Management Awards 2010
This message is subject to :
|
Re: close all positions and cancel all open orders
It is in the sample java application, if you run it you'll see buttons to do exactly that.
Best wishes,
M
Hi,
I would like to cancel all open orders and close all positions.
Does anyone has an example code? (JAVA API)
Thanks.
?
|
close all positions and cancel all open orders
Hi,
I would like to cancel all open orders and close all positions.
Does anyone has an example code? (JAVA API)
Thanks.
?
|
Re: Latency of reqTickByTickData
toggle quoted message
Show quoted text
El 7 sept 2018, a las 22:46, michaelIC via Groups.Io < michael.i.c@...> escribió: My bad.
I had a nagging thought, so I checked the log files from those runs.
The units across the bottom are not seconds, but the poll number.
The polls were done every three seconds.
So 1 is the first.
2 is the second, three seconds after the first.
3 is the third, three seconds after the second, etc..
So #20 is one minute into the run.
#40, two minutes.
|
Re: Allow OCA functionality on conditional cancellations
I sometime submit OCA groups with, for example, multiple profit targets (each profit target trying to close the entire position); for a long position, a higher profit target at first, and if that first profit target is not met, then the order expires, and another, lower profit target order becomes active (subsequent profit targets have a good after equal to a previous order's good till). I wouldn't want the entire OCA group to be cancelled when the first, higher profit target's good till time is reached.
Jimmy
On Fri, Sep 7, 2018 at 6:55 PM Orionn via Groups.Io <psmiranda= [email protected]> wrote: Currently, OCA orders are only canceled if an order in the OCA group is either manually canceled or filled. If an OCA order is canceled by a good-til-date condition or by a conditional cancel price condition then the other orders in the same OCA group are not canceled. I find this logic surprising and, in my opinion, flawed. I would like to see orders in the same OCA group be canceled if another order in the same OCA group is canceled due to a good-til-date condition or a conditional cancel price condition. This issue is not API specific as orders generated in TWS also show this behavior.
In case anyone else would like the OCA functionality improved, you may vote and comment at the following New Features Poll entry:
Thank you.
--
|
Allow OCA functionality on conditional cancellations
Currently, OCA orders are only canceled if an order in the OCA group is either manually canceled or filled. If an OCA order is canceled by a good-til-date condition or by a conditional cancel price condition then the other orders in the same OCA group are not canceled. I find this logic surprising and, in my opinion, flawed. I would like to see orders in the same OCA group be canceled if another order in the same OCA group is canceled due to a good-til-date condition or a conditional cancel price condition. This issue is not API specific as orders generated in TWS also show this behavior.
In case anyone else would like the OCA functionality improved, you may vote and comment at the following New Features Poll entry:
Thank you.
--
www.tradingsoftwarelab.com
|
Re: Latency of reqTickByTickData
My bad.
I had a nagging thought, so I checked the log files from those runs.
The units across the bottom are not seconds, but the poll number.
The polls were done every three seconds.
So 1 is the first.
2 is the second, three seconds after the first.
3 is the third, three seconds after the second, etc..
So #20 is one minute into the run.
#40, two minutes.
|
Re: Latency of reqTickByTickData
Fourth.
With NTP turned back on, here's a graph of when some transactions are arriving from IB.
The width represents a second. Each axis tick across the bottom is 50 ms.
I find that the data transactions come in clustered around a time from TOS, and with system time increasing its lag, that cluster drifts to the right. This is seen in the four groups on the graph. Each group has a window typically around 150 ms wide, but this varies with load on the box, as does the elapsed number of seconds for the cluster of transaction times to drift from the left side of the 150 ms window over towards the right side of the window. Once at the right side, the cluster jumps back to the left side of the 150 ms window, and begins drifting back to the right.
So once I find the time, to get timestamps that are better than within a 150 ms drift, I'll be frequently adjusting system time off of my own local GPS disciplined time server.
|
Re: Latency of reqTickByTickData
Third.
Improved reset/align code, and resetting when the lag exceeds 20 ms. The raw NTP is being filtered by cascading outlier filters before averaging to get the correction. Looking at the NTP polls, it amazing how fast system time is 20 ms off of UTC. Or in the prior chart, 250 ms off of UTC.
So if one is using system time to create timestamps, if your load is nice and steady, then they might be close. But if the actual time of the timestamp matters, rather than just the order, you may wish to take a closer look between the time you're using to timestamp from vs. UTC.
When I compared the time adjustments I saw in the IB logs, there were very very close to the NTP UTC.
|
Re: Latency of reqTickByTickData
Second.
As code can set system time, to see how it would fare, I decided to have it realign system time to UTC TOS every time system time lagged UTC by more than 250 ms. In this graph, ms lag on the left, seconds on the bottom. In the first 20 seconds or so, system is under a heavy load, so you see the steepest lag rate. Then it goes to a medium load, then a light load after the correction. There is no meaningful difference in rate of lag between medium and light loads. Due to the lag in setting system time, you can see that it doesn't get fully aligned, but it is a lot closer than the 250 ms off of TOS.
|
Re: Latency of reqTickByTickData
I’m interested in your rather dismissive assertion about timekeeping in Windows. Your experience seems to differ considerably from mine.
So was I, once I actually measured what was going on when I observed timestamps and saw odd shifts in system time and looked at the NTP logs and then did my own polling. (I'm running windows 7) Windows calculates its time by counting clock pulses. As crystals vary from MB to MB, this varies from UTC, some are fast, some lag. Unfortunately, counting the clock pulses is a lazy process, and if the cpu is too busy, it doesn't get around to counting all of them, so some get missed/skipped. So how much the system time varies from UTC varies depending on load. With a nice steady load, with a long run of time for sampling, and a reasonably clean internet connection for quality poll times, NTP can compensate for this very very well, making slight adjustments over time to bring things into alignment. With the processing time for trading strategies being a surprisingly light CPU load, many high-speed multi-core CPUs end up using power savings and cut CPU speed and shut down cores. But I wanted the processing results ASAP, so I set my bios to not shut down cores or dial back speed. Eight times a second, my code does different tasks. And it can receive incoming data from IB at anytime. So eight times a second I get a burst of heavy load, and each time the load can be different, and each second can be very different. This results in widely varying missed clock counts for system time. NTP does not adjust time frequently enough to track, let along adjust for this. It's a longer time frame aligner. So by polling NTP sites I can maintain an offset of system time from UTC that is used to create a timestamp that is reasonably close to UTC. Unfortunately, the rate that windows system clock lag increases, and variably, with varying loads, means that the offset quickly becomes surprisingly large. So I captured and graphed it. This required polling NPT sites, and quality sites close to me, very frequently. Which means short test runs, as polling every second is not a nice thing to do to a NTP site. The results showed that my widely varying heavy loads meant that I needed a very frequent update of the UTC offset, which meant I needed my own local time server that I could poll every second. The results also showed that my cable internet's inconsistent times resulted in lower quality NTP results, so this would required a GPS disciplined time server. I'll attempt to attach the charts that demonstrate the issue of windows system time missed clock count when there are heavy loads. I don't know how this will work, so I'll do one graph per post. When an 18 hour run at idle with 10 minute polls is graphed, the result is very smooth straight line. Easily adjusted for. It's when you look at system time at the second level that it gets ugly, and a problem if you want to timestamp something to know when inside of a second that it arrived (i.e., at 228 ms from TOS). First, is tracking six NPT sites offset against the increasing system time lag. The left axis is ms. The bottom axis is seconds. Outliers are not removed; this is the raw poll results.
|
Re: How to get historical snapshots of fundamental ratios
Unfortunately no the fundamental ratios are not available historically.
|
How to get historical snapshots of fundamental ratios
Hi,
Is there a way to get historical snapshot for fundamental ratios ?
I see that?reqMktData can be used with tick id of 258 to request specific ratios or use?reqFundamentalData to get a snapshot report. Howerver, I am not able to figure out if its possible to get historical data - like a time series of EPS or DPS for example. Is there a way to do it on IB API ?
|
Re: C# reqHistoricalData, returns odd time spans
Matt,
The Day unit is special in historical data requests, when used either as a bar size or duration. Instead of a calendar day, it returns data for a trading session. So for the EUR/USD pair for instance, this will be 17:15-17:00 EST. A query with duration of '1 Day' will return data for the current trading session, corresponding to the time period starting at 5:15pm EST. To receive data for a calendar day the options would be to specify '2 D' as the duration, which would return more than a calendar day, or to specify the duration in a different unit, such as 86400 S (seconds).?
Josh (IBKR)
|
Re: Latency of reqTickByTickData
Michael ? I’m interested in your rather dismissive assertion about timekeeping in Windows. Your experience seems to differ considerably from mine. ? The first thing to say is that Windows, out of the box, comes with a fairly lax configuration for its NTP. This is because it hasn’t historically been a requirement for Windows to give high precision timestamping, just enough for the purposes of maintaining reasonable synchronization between the computers in an Active Directory domain. Microsoft have traditionally left it to third parties to provide precise mechanisms. I suspect at least a part of Microsoft’s motivation here is the sheer number of Windows machines: if every Windows computer were syncing to a time server every 20 seconds, the world’s time servers would collapse in a heap, and the vast majority of Windows computers don’t need to be synchronised to anything less than a second or so. With modern hardware, clock drift is relatively slow, so there’s no big issue here. ? So for example a workstation edition of Windows (eg Windows 10 Home or Windows 10 Professional) which is not configured as part of a domain,? synchronises infrequently with the default time server (which is time.windows.com). And a workstation that IS part of a domain uses the domain controller as a time server, and again synchronises infrequently. In both cases, in my experience, the offset between the client machine and the time server tends to be less than around 100 millisecs. ? By the way, for those who don’t know, there is a handy command which enables you to see directly the offset between your machine’s clock and a specified time server (note however that this command will only run if the Windows Time service is running, and for Windows 10 Home edition this is not the case by default) . From an administrative command prompt type a command like this: ? w32tm /stripchart /computer:pool.ntp.org /period 5 ? The /computer argument specifies which computer to compared your local clock against: pool.ntp.org is a large and growing set of worldwide time servers. ? (If you know of an equivalent command on Ubuntu 16.04 or later that gives similar information, please let me know.) ? For Windows Server, I’m not sure what the out-of-the box settings are, because I have changed mine, but it’s certainly possible to configure it so that the synchronisation is quite good. My Windows Server 2012 box typically has offsets hovering around 2 milliseconds, which I’m not unhappy with. I don’t see any evidence of amazing ugly jumps: maybe you’re talking about older versions? But hey, 2012 isn’t exactly recent! ? My understanding is that there were improvements in Windows Server 2016 to get millisecond accuracy. ? And the big news is that Windows Server 2019 will implement the new Precision Time Protocol, which will enable vastly more accurate time syncing. I suggest you read this: ? By the way, the Windows Time service in the workstation Windows editions is the same as in the server editions, so it may well be the case that a Windows 10 PC could be configured to be just as accurate as my Server 2012: I just haven’t tried to do it (because I don’t need to). Perhaps I should check it out… ? Richard ? ? ?
toggle quoted message
Show quoted text
From: [email protected] <[email protected]> On Behalf Of michaelIC via Groups.Io
Sent: 06 September 2018 06:51
To: [email protected]
Subject: Re: [TWS API] Latency of reqTickByTickData? Dmitry,
PPS on Pi has significant lag, variable lag, isn't a RT OS, is multi-core, jittery CPU, and then you're taking it as a reference delivered through variable Ethernet speeds as its under a USB controller. You may find it interesting to turn off NTP on your boxes, turn off any running load on your boxes, then bring the Pi reference time across and compare it to system time.
Then compare again with your trading load running.
And if you're comparing to Windows system time, you're going to be amazed at the variability and ugly jumps that makes. Windows system time lags variably and horribly depending on load.
|
With stocks, you would use the fields contract.secId, contract.secIdType, as in the example below.?
|
abbanerjee