|
When I used the IBKR API to stream the level 2 data into a MySQL database, the data seems to fall behind the actual level 2 quotes that are in the market.? Is there any way to avoid this eventual lag that takes place?
ebtrader
|
How do you determine that level 2 data falls behind the market? And how large is the lag you experience?
My immediate reaction would be to check whether MySQL database updates cause the lag. For liquid instruments, L2 data feeds easily generate hundreds if not thousands of callbacks per second and SQL database updates are generally very expensive and time consuming operations. In case your client performs any DB update functions within the TWS API callback thread, TWS API may be unable to report L2 updates to you in a timely fashion although your client already has received those messages from TWS/IBGW through the network socket.
´³¨¹°ù²µ±ð²Ô
|
When I check the latest updated rows in the DB show values different than the current market price.? The instrument I get quotes for is NQ futures.
|
"show" ? as in a "SELECT in a tool like MySQL WorkBench" ? because L2 run so fast that extremely unlikely you can catch it "visually"
L2 show B/A book, which are different beast than last trade. So what do you call "current market price" ?
Also: be careful that depending upon what you look for (and anyway in every cases as a default method)? you need to listen to both updateMktDepth and updateMktDepthL2 to get a complete picture.
Opinion:
MySQL is not a real time tool but should be fast enough, with proper indexing method (simplest one), proper DB engine, MyISAM is better for that than Innodb that may have "relaxation" period. If under windows for things like L2 prefer the MemoryMapped connection rather than the TCP/IP. (or even use the MEMORY engine, risky but theses L2 data are inherently rarely used aside of real time)
As per my own experience with it using L2 from IB and also from other provider, IB is generally faster (10ms to 1s) than others.
So IMHO IB deliver faster data, the only draw back is the limitation IB give you on simultaneous L2 you can listen to, probably the price to pay to get data faster.
|
Current market price meaning the top of the limit order book - what Updatemktdepth would display as the ¡°0¡± position.
"show" ? as in a "SELECT in a tool like MySQL WorkBench" ? because L2 run so fast that extremely unlikely you can catch it "visually"
L2 show B/A book, which are different beast than last trade. So what do you call "current market price" ?
Also: be careful that depending upon what you look for (and anyway in every cases as a default method)? you need to listen to both updateMktDepth and updateMktDepthL2 to get a complete picture.
Opinion:
MySQL is not a real time tool but should be fast enough, with proper indexing method (simplest one), proper DB engine, MyISAM is better for that than Innodb that may have "relaxation" period. If under windows for things like L2 prefer the MemoryMapped connection rather than the TCP/IP. (or even use the MEMORY engine, risky but theses L2 data are inherently rarely used aside of real time)
As per my own experience with it using L2 from IB and also from other provider, IB is generally faster (10ms to 1s) than others.
So IMHO IB deliver faster data, the only draw back is the limitation IB give you on simultaneous L2 you can listen to, probably the price to pay to get data faster.
|
Look to me like the Highest Bid and Lowest Ask. They are unrealized transaction.
They are reported in the callbacks, "I am sure of it."
How you compare it is the issue, if you display TWS in // to a SQL SELECT, definitively it's move too fast to be insync, you may even have TWS late to the API, but your DB is most probably the reference, or you should investigate your INSERT timing. (you better bulk it in a separate thread)
If you want to make a visual inspection, don't use the [0] but someplace deeper, like around the 20th row or with 0.25% <> to top, they change less frequently as they are more "fishing" position and any better bid or more aggressive ask may insert above but faster.
Or take a stock that not aggressively traded, low daily volume typically.
Or take a look at it outside RTH, market are way slower.
Or don't use a DB but a memory Map and dump it,
etc ....
Trust the API and dig elsewhere.
|
Outside RTH it stays in sync no issues.? Only during RTH is the issue.
Look to me like the Highest Bid and Lowest Ask. They are unrealized transaction.
They are reported in the callbacks, "I am sure of it."
How you compare it is the issue, if you display TWS in // to a SQL SELECT, definitively it's move too fast to be insync, you may even have TWS late to the API, but your DB is most probably the reference, or you should investigate your INSERT timing. (you better bulk it in a separate thread)
If you want to make a visual inspection, don't use the [0] but someplace deeper, like around the 20th row or with 0.25% <> to top, they change less frequently as they are more "fishing" position and any better bid or more aggressive ask may insert above but faster.
Or take a stock that not aggressively traded, low daily volume typically.
Or take a look at it outside RTH, market are way slower.
Or don't use a DB but a memory Map and dump it,
etc ....
Trust the API and dig elsewhere.
|
My buddy ran it with a laptop that has an Nvidia 2 series GeForce and it worked fine without falling behind.? I will upgrade my machine and then it should work.
Outside RTH it stays in sync no issues.? Only during RTH is the issue. Look to me like the Highest Bid and Lowest Ask. They are unrealized transaction.
They are reported in the callbacks, "I am sure of it."
How you compare it is the issue, if you display TWS in // to a SQL SELECT, definitively it's move too fast to be insync, you may even have TWS late to the API, but your DB is most probably the reference, or you should investigate your INSERT timing. (you better bulk it in a separate thread)
If you want to make a visual inspection, don't use the [0] but someplace deeper, like around the 20th row or with 0.25% <> to top, they change less frequently as they are more "fishing" position and any better bid or more aggressive ask may insert above but faster.
Or take a stock that not aggressively traded, low daily volume typically.
Or take a look at it outside RTH, market are way slower.
Or don't use a DB but a memory Map and dump it,
etc ....
Trust the API and dig elsewhere.
|
On Sat, Aug 12, 2023 at 02:01 PM, ebtrader wrote:
My buddy ran it with a laptop that has an Nvidia 2 series GeForce and it
worked fine without falling behind.? I will upgrade my machine and then it
should work.
Not THIS Buddy!
Please, I'm begging you, at least try dumping the data to a file first and watching it with . If you don't have tail, find it... in the WSL or install cygwin.
The test should be easy as pie, don't overthink it, just comment/ifdef out the SQL insert and replace it with a file append. Unless your computer has floppies with a 10Mbit network (i.e. ancient) everything should keep up. Then tune the db as previously suggested.
You haven't mentioned any kind of real-time computation happening on the CPU or GPU so the process is basically all network and disk I/O bound. If you need to upgrade anything it will be the network and/or disk first, and only then the [CG]PU; assuming you want to keep the data.
Be reasonable! With interest rates this high who can afford unnecessary capex?
|
Yes
No need to upgrade, especially this have nothing to do (or not enough) with GPU. It's only CPU based computation. Better invest in SSD than GPU
I guess your testing method (Still unclear to me how you proceed)? is not suited to what you want to check, and as @buddy suggest it, write a dumb CSV and check it later to get confidence.
'rogers'
|
You're not using wifi by any chance, are you?
Anyway, if you're ever wondering which end of your system can't keep up I suggest playing with dd progress=status for a few minutes.
Good luck!
|
I will try this file append.? I have an SSD drive.? No computations are happening. ?
On Sat, Aug 12, 2023 at 1:25 PM buddy < spy@...> wrote: On Sat, Aug 12, 2023 at 02:01 PM, ebtrader wrote:
My buddy ran it with a laptop that has an Nvidia 2 series GeForce and it
worked fine without falling behind.? I will upgrade my machine and then it
should work.
Not THIS Buddy!
Please, I'm begging you, at least try dumping the data to a file first and watching it with . If you don't have tail, find it... in the WSL or install cygwin.
The test should be easy as pie, don't overthink it, just comment/ifdef out the SQL insert and replace it with a file append. Unless your computer has floppies with a 10Mbit network (i.e. ancient) everything should keep up. Then tune the db as previously suggested.
You haven't mentioned any kind of real-time computation happening on the CPU or GPU so the process is basically all network and disk I/O bound. If you need to upgrade anything it will be the network and/or disk first, and only then the [CG]PU; assuming you want to keep the data.
Be reasonable! With interest rates this high who can afford unnecessary capex?
|
Using tail it is matching up with current market pricing in the after hours.? I will let you know if it works during regular market hours.
I will try this file append.? I have an SSD drive.? No computations are happening. ? On Sat, Aug 12, 2023 at 1:25 PM buddy < spy@...> wrote: On Sat, Aug 12, 2023 at 02:01 PM, ebtrader wrote:
My buddy ran it with a laptop that has an Nvidia 2 series GeForce and it
worked fine without falling behind.? I will upgrade my machine and then it
should work.
Not THIS Buddy!
Please, I'm begging you, at least try dumping the data to a file first and watching it with . If you don't have tail, find it... in the WSL or install cygwin.
The test should be easy as pie, don't overthink it, just comment/ifdef out the SQL insert and replace it with a file append. Unless your computer has floppies with a 10Mbit network (i.e. ancient) everything should keep up. Then tune the db as previously suggested.
You haven't mentioned any kind of real-time computation happening on the CPU or GPU so the process is basically all network and disk I/O bound. If you need to upgrade anything it will be the network and/or disk first, and only then the [CG]PU; assuming you want to keep the data.
Be reasonable! With interest rates this high who can afford unnecessary capex?
|
csv solved the problem. You were absolutely correct and saved me a boat load of money.? Much appreciated.? You are my new buddy!
Using tail it is matching up with current market pricing in the after hours.? I will let you know if it works during regular market hours. I will try this file append.? I have an SSD drive.? No computations are happening. ? On Sat, Aug 12, 2023 at 1:25 PM buddy < spy@...> wrote: On Sat, Aug 12, 2023 at 02:01 PM, ebtrader wrote:
My buddy ran it with a laptop that has an Nvidia 2 series GeForce and it
worked fine without falling behind.? I will upgrade my machine and then it
should work.
Not THIS Buddy!
Please, I'm begging you, at least try dumping the data to a file first and watching it with . If you don't have tail, find it... in the WSL or install cygwin.
The test should be easy as pie, don't overthink it, just comment/ifdef out the SQL insert and replace it with a file append. Unless your computer has floppies with a 10Mbit network (i.e. ancient) everything should keep up. Then tune the db as previously suggested.
You haven't mentioned any kind of real-time computation happening on the CPU or GPU so the process is basically all network and disk I/O bound. If you need to upgrade anything it will be the network and/or disk first, and only then the [CG]PU; assuming you want to keep the data.
Be reasonable! With interest rates this high who can afford unnecessary capex?
|
buddy