¿ªÔÆÌåÓý

ctrl + shift + ? for shortcuts
© 2025 Groups.io
  1. Twsapi
  2. Messages
Expanded Topics Messages Polls
Date
AW: twsapi: Re: Event driven garbage

 

¿ªÔÆÌåÓý

I do it similar:
?
1.? Manually check/correct open positions at startup (they a stored in a database and should be the same as yesterday evening)
2. Just track the status of orders while ignoring position messages
3. Using the status of the orders, update your own internal list of
positions
4.? I compare my internat list of positions all the time with TWS position list. When it is different for more than 10 seconds, my software creates an error message.
?
Marcus

-----Urspr¨¹ngliche Nachricht-----
Von: CarlErikson [mailto:carlerikson@...]
Gesendet: Monday, July 22, 2002 1:35 AM
An: twsapi@...
Betreff: twsapi: Re: Event driven garbage

Can you do the following:

1. Request open positions at startup
2. Just track the status of orders while ignoring position messages
3. Using the status of the orders, update your own internal list of
positions

This seems to work for me even with complicated partial fills.? Note
that sometimes when I am manually trading with the TWS I can see a
lag between execution and position updates.? Several months ago the
position entries lagged for several seconds at a time.? So yes, the
position and order status events are definitely independent of each
other.

If the two types of events cannot be correlated in time, then just
use one to recreate the other.? It seems like knowing the positions
at startup, and then tracking all order status events thereafter is
good enough to know the correct state at all times.? Please correct
me if my logic is wrong.

Thanks,
Carl

--- In twsapi@y..., "grozzie2" wrote:
> TWS is an event driven beast.? No functions actually have any kind
of
> return to validate they actually did something, everything waits
for
> callbacks, to say events occurred.
>
> Event driven models are great, provided, the event source reports
the
> events in the actual order they occur.? TWS does not.
>
> If you are basing code on the events from the TWS activeX
component,
> and building up logic for handling orders etc on it, be warned, I
> have a captured log with the following sequence of events, and I'm
> pondering what to do with it.
>
> The state begins with a short position of 20 contracts in the es
> minis.? The program has a target cover buy in place, and a stop,
for
> 20 contracts each, and joined as a one cancels all group.
>
> Event 1 - TWS reports partial fill for 11 from target order
> program modifies internal structures to show -9 left in the
> position, with 11 bought.
> Event 2 - TWS reports current position on ES as -9
> program verifies this as correct.
> Event 3 - TWS reports partial fill for 9 contracts, order filled
> completely.
> program modifies internal structures to show target order filled,
and
> removed from the operating state.
> Event 4 - TWS reports current position on ES as -9 contracts
> program goes HUH ?????????
> program realizes, all buy orders are filled, and it still has a
short
> position of 9 contracts, places an order to buy 9 to cover.
> Event 5 - TWS reports order to buy filled.
> program modifies internal structures to show buy now filled, and
> position back to 0
> Event 6 - TWS reports position of 0
> program verifies this is the correct position
> Event 7 - TWS reports current position on ES as 9 contracts
> program emits expletives into the debug log.
>
>
> Computers are fast, and can react much faster than humans.? What's
> important to note in the above sequence, TWS reported a WRONG
> position, and the program reacted.? The position reported was
correct
> for the time BEFORE the second fill report, and this is
significant,
> because TWS reported the -9 position TWICE, once before, and once
> after the fill for 9.? It looks to me like the TWS internal state
on
> actual position size got delayed, and would have 'caught up' had my
> program waited longer before doing order/position reconciliation.?
A
> human cant react that fast, so if it displayed -9, and updated that
> display to 0 a fraction of a second later, or even a couple seconds
> later, a human would go 'great, filled'.
>
> 2 weeks ago, I realized, TWS events are not necessarily in
sequence,
> and had settled on the position report event as the 'atomic' point
of
> action, on the assumption, it would be the one event we could count
> on to report things, and if we postponed all position management to
> that event, everything would stay in synch.? Up until now, it's
> appeared that would work, but now i see it wont.
>
> I now have logs of various orders of events during the process to
> prove, without any possibility of it being otherwise, there is no
TWS
> event that you can rely on as an atomic thing, ie, it will be
> correct.? I have fills updated before positions, positions updated
> before fills, and now i add to it, position reported late, or in
> error.? For now, i'll assume it was late, because if it was
actually
> in error, and TWS was not going to follow the second -9 report with
a
> 0 report, then it will not be possible to build a 'safe' program
> using this.
>
> I can say the following with a certainty now.? TWS is 2 or more
> independant state machines internally, reporting events with no
> relationships in time or order of events.? If you are trying to
> reconcile the events, and take actions based on the reports from
tws
> in real time, at the time of events, then the following holds
true.?
> If you are trading more than quantity 1, at some point in time, you
> will recieve event notifications that will come in an order that
> breaks your logic.? I have drawn out carefully the timing diagrams
> from a number of event sequences in the area of partial fills, both
> on entry and on exit.? The last one provides a proof, there is no
> event from TWS that is atomic, on which you can base decisions to
> manage existing positions, and reconcile with fill reports.? The
TWS
> position reports are independant of fill reports, and there will
> ALWAYS be potential for discrepancies.? The only way to do a
> reconciliation, is to postpone it until a point in time that you
can
> verify there are no potential outstanding events.
>
> It's time to re-design here.



To unsubscribe from this group, send an email to:
twsapi-unsubscribe@...



Your use of Yahoo! Groups is subject to the .

Re: Event driven garbage

 

--- In twsapi@y..., "CarlErikson" <carlerikson@y...> wrote:
Can you do the following:

1. Request open positions at startup
2. Just track the status of orders while ignoring position messages
3. Using the status of the orders, update your own internal list of
positions
In theory, this will work, provided there is never a communications
hiccup. This is why i must reconcile positions and orders as a cross
check. It's quite one thing to have an order entry front end, where
somebody is watching, and can 'deal with' errors, it's quite another
to create something that can run without human supervision, and 'get
it right', even when 'shit happens' and something goes wrong. For
that scenario, you need cross checks.
Re: twsapi: Event driven garbage

marinindextrader
 

Kind of depressing Grozzie....Programming is not difficult at all,
just complex. And it is this sort of nonesense that makes it even
more complex...

What a bummer...

They should just turn over all the source to us and then pay us to re-
write the whole bloody mess...



Scott

--- In twsapi@y..., "grozzie2" <grozzie2@y...> wrote:
--- In twsapi@y..., "Kent Rollins" <kentr@m...> wrote:
grozzie

How much time elapsed during this series of events?

Kent
Total elapsed time of that sequence was just inder 1.5
seconds. I dont use the 'poll the socket on a timer'
methodology that the sample code uses, I have a dedicated
thread reading the socket from TWS, so my events arrive
in the rest of the application 'as they come in', with
no delays. The thread reading the socket is running as
the highest priority thread in the system. Since all the
events come from a single thread, that's reading the socket
serially, so there is no doubt i'm getting events from tws
in the order it's sending them.

I'm less interested in the time of the sequence, than the
order. The order there proves, position reports are not
sychronized with fill reports. That introduces a lot of
'issues'.
Re: Event driven garbage

CarlErikson
 

Can you do the following:

1. Request open positions at startup
2. Just track the status of orders while ignoring position messages
3. Using the status of the orders, update your own internal list of
positions

This seems to work for me even with complicated partial fills. Note
that sometimes when I am manually trading with the TWS I can see a
lag between execution and position updates. Several months ago the
position entries lagged for several seconds at a time. So yes, the
position and order status events are definitely independent of each
other.

If the two types of events cannot be correlated in time, then just
use one to recreate the other. It seems like knowing the positions
at startup, and then tracking all order status events thereafter is
good enough to know the correct state at all times. Please correct
me if my logic is wrong.

Thanks,
Carl

--- In twsapi@y..., "grozzie2" <grozzie2@y...> wrote:
TWS is an event driven beast. No functions actually have any kind
of
return to validate they actually did something, everything waits
for
callbacks, to say events occurred.

Event driven models are great, provided, the event source reports
the
events in the actual order they occur. TWS does not.

If you are basing code on the events from the TWS activeX
component,
and building up logic for handling orders etc on it, be warned, I
have a captured log with the following sequence of events, and I'm
pondering what to do with it.

The state begins with a short position of 20 contracts in the es
minis. The program has a target cover buy in place, and a stop,
for
20 contracts each, and joined as a one cancels all group.

Event 1 - TWS reports partial fill for 11 from target order
program modifies internal structures to show -9 left in the
position, with 11 bought.
Event 2 - TWS reports current position on ES as -9
program verifies this as correct.
Event 3 - TWS reports partial fill for 9 contracts, order filled
completely.
program modifies internal structures to show target order filled,
and
removed from the operating state.
Event 4 - TWS reports current position on ES as -9 contracts
program goes HUH ?????????
program realizes, all buy orders are filled, and it still has a
short
position of 9 contracts, places an order to buy 9 to cover.
Event 5 - TWS reports order to buy filled.
program modifies internal structures to show buy now filled, and
position back to 0
Event 6 - TWS reports position of 0
program verifies this is the correct position
Event 7 - TWS reports current position on ES as 9 contracts
program emits expletives into the debug log.


Computers are fast, and can react much faster than humans. What's
important to note in the above sequence, TWS reported a WRONG
position, and the program reacted. The position reported was
correct
for the time BEFORE the second fill report, and this is
significant,
because TWS reported the -9 position TWICE, once before, and once
after the fill for 9. It looks to me like the TWS internal state
on
actual position size got delayed, and would have 'caught up' had my
program waited longer before doing order/position reconciliation.
A
human cant react that fast, so if it displayed -9, and updated that
display to 0 a fraction of a second later, or even a couple seconds
later, a human would go 'great, filled'.

2 weeks ago, I realized, TWS events are not necessarily in
sequence,
and had settled on the position report event as the 'atomic' point
of
action, on the assumption, it would be the one event we could count
on to report things, and if we postponed all position management to
that event, everything would stay in synch. Up until now, it's
appeared that would work, but now i see it wont.

I now have logs of various orders of events during the process to
prove, without any possibility of it being otherwise, there is no
TWS
event that you can rely on as an atomic thing, ie, it will be
correct. I have fills updated before positions, positions updated
before fills, and now i add to it, position reported late, or in
error. For now, i'll assume it was late, because if it was
actually
in error, and TWS was not going to follow the second -9 report with
a
0 report, then it will not be possible to build a 'safe' program
using this.

I can say the following with a certainty now. TWS is 2 or more
independant state machines internally, reporting events with no
relationships in time or order of events. If you are trying to
reconcile the events, and take actions based on the reports from
tws
in real time, at the time of events, then the following holds
true.
If you are trading more than quantity 1, at some point in time, you
will recieve event notifications that will come in an order that
breaks your logic. I have drawn out carefully the timing diagrams
from a number of event sequences in the area of partial fills, both
on entry and on exit. The last one provides a proof, there is no
event from TWS that is atomic, on which you can base decisions to
manage existing positions, and reconcile with fill reports. The
TWS
position reports are independant of fill reports, and there will
ALWAYS be potential for discrepancies. The only way to do a
reconciliation, is to postpone it until a point in time that you
can
verify there are no potential outstanding events.

It's time to re-design here.
Re: twsapi: Event driven garbage

 

My first thought as I was reading your original post was to introduce a
small delay in processing responses. I agree completely that the message
order problem you have found is serious and needs fixin'.

IB has taken a difficult route in making this one app into a User interface,
an ActiveX interface, and a socket interface. Separate apps/libraries would
have been a better way to go.

Kent

----- Original Message -----
From: "grozzie2" <grozzie2@...>
To: <twsapi@...>
Sent: Sunday, July 21, 2002 10:34 PM
Subject: Re: twsapi: Event driven garbage


--- In twsapi@y..., "Kent Rollins" <kentr@m...> wrote:
grozzie

How much time elapsed during this series of events?

Kent
Total elapsed time of that sequence was just inder 1.5
seconds. I dont use the 'poll the socket on a timer'
methodology that the sample code uses, I have a dedicated
thread reading the socket from TWS, so my events arrive
in the rest of the application 'as they come in', with
no delays. The thread reading the socket is running as
the highest priority thread in the system. Since all the
events come from a single thread, that's reading the socket
serially, so there is no doubt i'm getting events from tws
in the order it's sending them.

I'm less interested in the time of the sequence, than the
order. The order there proves, position reports are not
sychronized with fill reports. That introduces a lot of
'issues'.





To unsubscribe from this group, send an email to:
twsapi-unsubscribe@...



Your use of Yahoo! Groups is subject to

Re: twsapi: Event driven garbage

 

--- In twsapi@y..., "Kent Rollins" <kentr@m...> wrote:
grozzie

How much time elapsed during this series of events?

Kent
Total elapsed time of that sequence was just inder 1.5
seconds. I dont use the 'poll the socket on a timer'
methodology that the sample code uses, I have a dedicated
thread reading the socket from TWS, so my events arrive
in the rest of the application 'as they come in', with
no delays. The thread reading the socket is running as
the highest priority thread in the system. Since all the
events come from a single thread, that's reading the socket
serially, so there is no doubt i'm getting events from tws
in the order it's sending them.

I'm less interested in the time of the sequence, than the
order. The order there proves, position reports are not
sychronized with fill reports. That introduces a lot of
'issues'.
Re: twsapi: Event driven garbage

 

grozzie

How much time elapsed during this series of events?

Kent

----- Original Message -----
From: "grozzie2" <grozzie2@...>
To: <twsapi@...>
Sent: Sunday, July 21, 2002 3:43 PM
Subject: twsapi: Event driven garbage


TWS is an event driven beast. No functions actually have any kind of
return to validate they actually did something, everything waits for
callbacks, to say events occurred.

Event driven models are great, provided, the event source reports the
events in the actual order they occur. TWS does not.

If you are basing code on the events from the TWS activeX component,
and building up logic for handling orders etc on it, be warned, I
have a captured log with the following sequence of events, and I'm
pondering what to do with it.

The state begins with a short position of 20 contracts in the es
minis. The program has a target cover buy in place, and a stop, for
20 contracts each, and joined as a one cancels all group.

Event 1 - TWS reports partial fill for 11 from target order
program modifies internal structures to show -9 left in the
position, with 11 bought.
Event 2 - TWS reports current position on ES as -9
program verifies this as correct.
Event 3 - TWS reports partial fill for 9 contracts, order filled
completely.
program modifies internal structures to show target order filled, and
removed from the operating state.
Event 4 - TWS reports current position on ES as -9 contracts
program goes HUH ?????????
program realizes, all buy orders are filled, and it still has a short
position of 9 contracts, places an order to buy 9 to cover.
Event 5 - TWS reports order to buy filled.
program modifies internal structures to show buy now filled, and
position back to 0
Event 6 - TWS reports position of 0
program verifies this is the correct position
Event 7 - TWS reports current position on ES as 9 contracts
program emits expletives into the debug log.


Computers are fast, and can react much faster than humans. What's
important to note in the above sequence, TWS reported a WRONG
position, and the program reacted. The position reported was correct
for the time BEFORE the second fill report, and this is significant,
because TWS reported the -9 position TWICE, once before, and once
after the fill for 9. It looks to me like the TWS internal state on
actual position size got delayed, and would have 'caught up' had my
program waited longer before doing order/position reconciliation. A
human cant react that fast, so if it displayed -9, and updated that
display to 0 a fraction of a second later, or even a couple seconds
later, a human would go 'great, filled'.

2 weeks ago, I realized, TWS events are not necessarily in sequence,
and had settled on the position report event as the 'atomic' point of
action, on the assumption, it would be the one event we could count
on to report things, and if we postponed all position management to
that event, everything would stay in synch. Up until now, it's
appeared that would work, but now i see it wont.

I now have logs of various orders of events during the process to
prove, without any possibility of it being otherwise, there is no TWS
event that you can rely on as an atomic thing, ie, it will be
correct. I have fills updated before positions, positions updated
before fills, and now i add to it, position reported late, or in
error. For now, i'll assume it was late, because if it was actually
in error, and TWS was not going to follow the second -9 report with a
0 report, then it will not be possible to build a 'safe' program
using this.

I can say the following with a certainty now. TWS is 2 or more
independant state machines internally, reporting events with no
relationships in time or order of events. If you are trying to
reconcile the events, and take actions based on the reports from tws
in real time, at the time of events, then the following holds true.
If you are trading more than quantity 1, at some point in time, you
will recieve event notifications that will come in an order that
breaks your logic. I have drawn out carefully the timing diagrams
from a number of event sequences in the area of partial fills, both
on entry and on exit. The last one provides a proof, there is no
event from TWS that is atomic, on which you can base decisions to
manage existing positions, and reconcile with fill reports. The TWS
position reports are independant of fill reports, and there will
ALWAYS be potential for discrepancies. The only way to do a
reconciliation, is to postpone it until a point in time that you can
verify there are no potential outstanding events.

It's time to re-design here.





To unsubscribe from this group, send an email to:
twsapi-unsubscribe@...



Your use of Yahoo! Groups is subject to

Re: twsapi: Event driven garbage

 

TWS's API orderStatus event "twitches" too frequently. Frequently
there are 2 reports (at least 50% of trades) sometimes even 3 (10% of
trades). They contain the same info.

I see this all the time in the demo. Lots of things reported twice,
and once in a while, 3 times.

Logic failure here is that if TWS's API really did place an order
bigger that I planned, I am long or short more shares than I know
about. Very exciting with my money!

Well, I am in the process of a rather nasty re-design, and this time
I have removed all but two items from my 'assume correct' list. The
first, TWS will get the size right on my orders, and eventually, it
will report correct positions, but there may be significant time lag
between executions and actual reporting of correct position size,
with incorrect positions reported in the interim.

On my 'cannot trust' list I have

- open orders
- order status
- safe id's

My num #2 Son, who works with me on this project, tunneled through
IB's system by phone on 7/19, and actually talked to someone in
programming. They said they are very aware of this problem, it
is "numerous Uno", and are working on it "full court press".

If we cant trust they will get size right on submitted orders, then
it becomes more than difficult, it becomes hopeless.

"Man" this is just like writing a program in 1983 on my Apple
computer in Basic using edlin as a text editor.

well, as we say around here:-

'developers' use an ide.
'programmers' start with 'copy con test.cpp'
'real programmers' skip a step, and just go 'copy con test.exe'
'a real guru' doesn't bother with all the intermediate steps just
starts directly with 'copy con test.zip'


Be careful. Beta programs are dangerous! Well I know!
Beta programs have issues, and normally have failure modes.
Junk is junk, and lets call a spade a spade. The ib interface is
junk. It had zero testing before it was released, and the c++
wrapper code wasn't even reviewed by somebody that knows the
difference between (&object) (object) and (*object). For proof,
look at all the various hoops and jumps it uses to try find an
indirection to CArchive that'll work in various places. Code by
trial and error by somebody that couldn't pass the entrance exam
for 'beginners c++', never mind the first quiz after the first class.

Gonna be interesting to see if it's even possible to come up with a
methodology that will deal with all these problems, i have far to
much time invested in this to give up now.
Re: TWS API market data to Create a 1 minutes stock chart ?

marinindextrader
 

Waste of time...IMHO..Its not realtime event driven...and there is no
server sided database to call and construct historical bars..

So your stuck with snagging your own data (which is fake)...

....if you insist...

Pretty simple... Set a timer on your form and start snagging
tickEvents. Create your own timestamp and display it in a picture
box.

Scott

--- In twsapi@y..., "icm63" <icm63@y...> wrote:
Any one seen vb code to get market data from API to create a
barchart
in 1 minute time frame.

Are there any examples of getting tick data to a excel sheet to
even
form a line chart ?

Any info will help .

thx

Ian
Re: Event driven garbage

marinindextrader
 

Grozzie....your right...maybe the lawyers told them to do it this
way...never can prove one way or the other wether an event fired...

Functionless functions would be a liability...Sub's that trigger (one
would hope) Events, seem like a a real slick and subtle way to avoid
lawsuits..

Then I am a cynical Monster

Scott






--- In twsapi@y..., "grozzie2" <grozzie2@y...> wrote:
TWS is an event driven beast. No functions actually have any kind
of
return to validate they actually did something, everything waits
for
callbacks, to say events occurred.

Event driven models are great, provided, the event source reports
the
events in the actual order they occur. TWS does not.

If you are basing code on the events from the TWS activeX
component,
and building up logic for handling orders etc on it, be warned, I
have a captured log with the following sequence of events, and I'm
pondering what to do with it.

The state begins with a short position of 20 contracts in the es
minis. The program has a target cover buy in place, and a stop,
for
20 contracts each, and joined as a one cancels all group.

Event 1 - TWS reports partial fill for 11 from target order
program modifies internal structures to show -9 left in the
position, with 11 bought.
Event 2 - TWS reports current position on ES as -9
program verifies this as correct.
Event 3 - TWS reports partial fill for 9 contracts, order filled
completely.
program modifies internal structures to show target order filled,
and
removed from the operating state.
Event 4 - TWS reports current position on ES as -9 contracts
program goes HUH ?????????
program realizes, all buy orders are filled, and it still has a
short
position of 9 contracts, places an order to buy 9 to cover.
Event 5 - TWS reports order to buy filled.
program modifies internal structures to show buy now filled, and
position back to 0
Event 6 - TWS reports position of 0
program verifies this is the correct position
Event 7 - TWS reports current position on ES as 9 contracts
program emits expletives into the debug log.


Computers are fast, and can react much faster than humans. What's
important to note in the above sequence, TWS reported a WRONG
position, and the program reacted. The position reported was
correct
for the time BEFORE the second fill report, and this is
significant,
because TWS reported the -9 position TWICE, once before, and once
after the fill for 9. It looks to me like the TWS internal state
on
actual position size got delayed, and would have 'caught up' had my
program waited longer before doing order/position reconciliation.
A
human cant react that fast, so if it displayed -9, and updated that
display to 0 a fraction of a second later, or even a couple seconds
later, a human would go 'great, filled'.

2 weeks ago, I realized, TWS events are not necessarily in
sequence,
and had settled on the position report event as the 'atomic' point
of
action, on the assumption, it would be the one event we could count
on to report things, and if we postponed all position management to
that event, everything would stay in synch. Up until now, it's
appeared that would work, but now i see it wont.

I now have logs of various orders of events during the process to
prove, without any possibility of it being otherwise, there is no
TWS
event that you can rely on as an atomic thing, ie, it will be
correct. I have fills updated before positions, positions updated
before fills, and now i add to it, position reported late, or in
error. For now, i'll assume it was late, because if it was
actually
in error, and TWS was not going to follow the second -9 report with
a
0 report, then it will not be possible to build a 'safe' program
using this.

I can say the following with a certainty now. TWS is 2 or more
independant state machines internally, reporting events with no
relationships in time or order of events. If you are trying to
reconcile the events, and take actions based on the reports from
tws
in real time, at the time of events, then the following holds
true.
If you are trading more than quantity 1, at some point in time, you
will recieve event notifications that will come in an order that
breaks your logic. I have drawn out carefully the timing diagrams
from a number of event sequences in the area of partial fills, both
on entry and on exit. The last one provides a proof, there is no
event from TWS that is atomic, on which you can base decisions to
manage existing positions, and reconcile with fill reports. The
TWS
position reports are independant of fill reports, and there will
ALWAYS be potential for discrepancies. The only way to do a
reconciliation, is to postpone it until a point in time that you
can
verify there are no potential outstanding events.

It's time to re-design here.
Re: twsapi: Event driven garbage

Robert Carey
 

¿ªÔÆÌåÓý

Same problem here.
My world consists of a local area network of 8 computers. I trade US equities.?One computer has a direct internet connection and runs TWS and its API and my VB6 application that interfaces with that API. Another?other computer can track one stock, another can track 4. Both of these can initiate open and close orders. They report them to the IB_Interface application by UDP packets across the network.
I created a "trail of crumbs" system. Every procedure reports the variables coming in and coming out to a database.
TWS's?API orderStatus event?"twitches" too frequently. Frequently there are 2 reports (at least 50% of trades)?sometimes even 3 (10% of trades). They contain the same info.
To get around this bug, I had to create a database of orders and their filled status. If TWS twitches beyond the ordered size, I block reporting this event to the trading computers. It took a few days and several really screwed up trades to get that logic straight. Based on one day's experience this is now working.
None of this occurs when running TWS in demo mode.
Logic failure here is that if TWS's API really did place an order bigger that I planned, I am long or short more shares than I know about. Very exciting with my money!
I reported this to IB by eMail and phone two weeks ago. I got some late replies.
My num #2 Son, who works with me on this project, tunneled through IB's system by phone on 7/19, and actually talked to someone in programming. They?said they are very aware of this problem, it is "numerous Uno",?and are working on it "full court press".
"Man" this is just like writing a program in 1983 on?my Apple computer in Basic using edlin as a text editor.
Be careful. Beta programs are dangerous! Well I know!
RCarey
?
?
?
?

----- Original Message -----
From: grozzie2
Sent: Sunday, July 21, 2002 3:43 PM
Subject: twsapi: Event driven garbage

TWS is an event driven beast.? No functions actually have any kind of
return to validate they actually did something, everything waits for
callbacks, to say events occurred.

Event driven models are great, provided, the event source reports the
events in the actual order they occur.? TWS does not.

If you are basing code on the events from the TWS activeX component,
and building up logic for handling orders etc on it, be warned, I
have a captured log with the following sequence of events, and I'm
pondering what to do with it.

The state begins with a short position of 20 contracts in the es
minis.? The program has a target cover buy in place, and a stop, for
20 contracts each, and joined as a one cancels all group.

Event 1 - TWS reports partial fill for 11 from target order
program modifies internal structures to show -9 left in the
position, with 11 bought.
Event 2 - TWS reports current position on ES as -9
program verifies this as correct.
Event 3 - TWS reports partial fill for 9 contracts, order filled
completely.
program modifies internal structures to show target order filled, and
removed from the operating state.
Event 4 - TWS reports current position on ES as -9 contracts
program goes HUH ?????????
program realizes, all buy orders are filled, and it still has a short
position of 9 contracts, places an order to buy 9 to cover.
Event 5 - TWS reports order to buy filled.
program modifies internal structures to show buy now filled, and
position back to 0
Event 6 - TWS reports position of 0
program verifies this is the correct position
Event 7 - TWS reports current position on ES as 9 contracts
program emits expletives into the debug log.


Computers are fast, and can react much faster than humans.? What's
important to note in the above sequence, TWS reported a WRONG
position, and the program reacted.? The position reported was correct
for the time BEFORE the second fill report, and this is significant,
because TWS reported the -9 position TWICE, once before, and once
after the fill for 9.? It looks to me like the TWS internal state on
actual position size got delayed, and would have 'caught up' had my
program waited longer before doing order/position reconciliation.? A
human cant react that fast, so if it displayed -9, and updated that
display to 0 a fraction of a second later, or even a couple seconds
later, a human would go 'great, filled'.

2 weeks ago, I realized, TWS events are not necessarily in sequence,
and had settled on the position report event as the 'atomic' point of
action, on the assumption, it would be the one event we could count
on to report things, and if we postponed all position management to
that event, everything would stay in synch.? Up until now, it's
appeared that would work, but now i see it wont.

I now have logs of various orders of events during the process to
prove, without any possibility of it being otherwise, there is no TWS
event that you can rely on as an atomic thing, ie, it will be
correct.? I have fills updated before positions, positions updated
before fills, and now i add to it, position reported late, or in
error.? For now, i'll assume it was late, because if it was actually
in error, and TWS was not going to follow the second -9 report with a
0 report, then it will not be possible to build a 'safe' program
using this.

I can say the following with a certainty now.? TWS is 2 or more
independant state machines internally, reporting events with no
relationships in time or order of events.? If you are trying to
reconcile the events, and take actions based on the reports from tws
in real time, at the time of events, then the following holds true.?
If you are trading more than quantity 1, at some point in time, you
will recieve event notifications that will come in an order that
breaks your logic.? I have drawn out carefully the timing diagrams
from a number of event sequences in the area of partial fills, both
on entry and on exit.? The last one provides a proof, there is no
event from TWS that is atomic, on which you can base decisions to
manage existing positions, and reconcile with fill reports.? The TWS
position reports are independant of fill reports, and there will
ALWAYS be potential for discrepancies.? The only way to do a
reconciliation, is to postpone it until a point in time that you can
verify there are no potential outstanding events.

It's time to re-design here.





To unsubscribe from this group, send an email to:
twsapi-unsubscribe@...



Your use of Yahoo! Groups is subject to the .

Event driven garbage

 

TWS is an event driven beast. No functions actually have any kind of
return to validate they actually did something, everything waits for
callbacks, to say events occurred.

Event driven models are great, provided, the event source reports the
events in the actual order they occur. TWS does not.

If you are basing code on the events from the TWS activeX component,
and building up logic for handling orders etc on it, be warned, I
have a captured log with the following sequence of events, and I'm
pondering what to do with it.

The state begins with a short position of 20 contracts in the es
minis. The program has a target cover buy in place, and a stop, for
20 contracts each, and joined as a one cancels all group.

Event 1 - TWS reports partial fill for 11 from target order
program modifies internal structures to show -9 left in the
position, with 11 bought.
Event 2 - TWS reports current position on ES as -9
program verifies this as correct.
Event 3 - TWS reports partial fill for 9 contracts, order filled
completely.
program modifies internal structures to show target order filled, and
removed from the operating state.
Event 4 - TWS reports current position on ES as -9 contracts
program goes HUH ?????????
program realizes, all buy orders are filled, and it still has a short
position of 9 contracts, places an order to buy 9 to cover.
Event 5 - TWS reports order to buy filled.
program modifies internal structures to show buy now filled, and
position back to 0
Event 6 - TWS reports position of 0
program verifies this is the correct position
Event 7 - TWS reports current position on ES as 9 contracts
program emits expletives into the debug log.


Computers are fast, and can react much faster than humans. What's
important to note in the above sequence, TWS reported a WRONG
position, and the program reacted. The position reported was correct
for the time BEFORE the second fill report, and this is significant,
because TWS reported the -9 position TWICE, once before, and once
after the fill for 9. It looks to me like the TWS internal state on
actual position size got delayed, and would have 'caught up' had my
program waited longer before doing order/position reconciliation. A
human cant react that fast, so if it displayed -9, and updated that
display to 0 a fraction of a second later, or even a couple seconds
later, a human would go 'great, filled'.

2 weeks ago, I realized, TWS events are not necessarily in sequence,
and had settled on the position report event as the 'atomic' point of
action, on the assumption, it would be the one event we could count
on to report things, and if we postponed all position management to
that event, everything would stay in synch. Up until now, it's
appeared that would work, but now i see it wont.

I now have logs of various orders of events during the process to
prove, without any possibility of it being otherwise, there is no TWS
event that you can rely on as an atomic thing, ie, it will be
correct. I have fills updated before positions, positions updated
before fills, and now i add to it, position reported late, or in
error. For now, i'll assume it was late, because if it was actually
in error, and TWS was not going to follow the second -9 report with a
0 report, then it will not be possible to build a 'safe' program
using this.

I can say the following with a certainty now. TWS is 2 or more
independant state machines internally, reporting events with no
relationships in time or order of events. If you are trying to
reconcile the events, and take actions based on the reports from tws
in real time, at the time of events, then the following holds true.
If you are trading more than quantity 1, at some point in time, you
will recieve event notifications that will come in an order that
breaks your logic. I have drawn out carefully the timing diagrams
from a number of event sequences in the area of partial fills, both
on entry and on exit. The last one provides a proof, there is no
event from TWS that is atomic, on which you can base decisions to
manage existing positions, and reconcile with fill reports. The TWS
position reports are independant of fill reports, and there will
ALWAYS be potential for discrepancies. The only way to do a
reconciliation, is to postpone it until a point in time that you can
verify there are no potential outstanding events.

It's time to re-design here.
TWS API market data to Create a 1 minutes stock chart ?

icm63
 

Any one seen vb code to get market data from API to create a barchart
in 1 minute time frame.

Are there any examples of getting tick data to a excel sheet to even
form a line chart ?

Any info will help .

thx

Ian
Globex problem last week

 

There was a problem on Globex last week on Wednesday or Thursday that lasted
for several hours. The message I got said something to the affect of
"Globex is experiencing difficulties. All orders for ES and NQ are being
processed manually. Globex will only accept Market orders during this
time."

Did anyone have a TWS API application running during this period? What
error messages does TWS return in this circumstance?

Kent
twsapi: Re: To grozzie2:

 

Hmm, not to split hairs, but aren't we talking about the same thing
really? The gist of my post was to create a central place where
things
would get coordinated. It looks like you needed to do more than
Yah, we settled on the final solution, one program deals totally with
tws, other programs talk to it. An extra layer, but, isolates issues
cleanly.


Thanks for the info about the re-assignment of id's. I can see this
is
going to be one of those "10% effort on the real project, 90%
effort on
spelunking TWS" deals.
Well, it was befuddling intially, we got a lot of issues that
appeared at some times, and not others, and it took a while going
thru detailed execution logs, before we nailed down the exact causes,
but I think we have it now.

Get a methodology for safe order ids in place first. Then firmly
enforce this rule, everywhere. NEVER NEVER NEVER touch an order
between the call to place it, and the callback on order status. Dont
change it, dont cancel it, dont touch it. This is especially
important on something like a trail stop, where the stop condition
can be changing fast. This of course introduces a side effect, what
if the callback never comes ? Add timeouts, and more validation.

I will say this, the demo account is a godsend, and makes this whole
platform 'wonderful' from a programming perspective. It's got
quirks, but we can validate at zero cost. I've done this for another
client, using a different broker, where no test accounts were
available. A test run to validate a specific condition handler, was
a planned item, with an associated cost, because it entailed actually
taking a position in the market, knowing in advance, the goal was to
create a loss, to validate program execution. With IB, we can be
fully validated in advance.
Newbie question

 

¿ªÔÆÌåÓý

I'm using the sample TwsDde.xls sheet that IB supplies.? This sheet is really all one needs to automate their trading.? However, I would like to be able to set up a program that will automatically send orders once they appear in the Orders page.? That is, with the sample program you have to hit the Place/Modify Order button to actually transmit?each order.? Is there an easy way to set something up that will automatically send orders without having to hit that Place order macro.? As you can tell my programming knowledge is limited.
?
?
Any help would be appreciated.
?
?
thanks,
?
Mike
?
twsapi: Re: To grozzie2:

 

Don't tell me
about the "low possibility" of a collision. This is supposed to be
a game
of skill, not a game of chance.
There's more to it than that. Different setups have different goals
for the automation of trades. I think the majority of folks here are
individuals, that have the dream of a 'system' that can work on it's
own, and achieve some trading success. It is a good dream to have,
but a very difficult point to achieve.

My goals for trading applications are much more specific, and clearly
defined by a customer. The primary purpose of the application I'm
working on now, is stated as 'reduce/eliminate execution risks'.
Execution risk comes in various forms:-

a) The time taken to enter and send an order
b) The probability of keystroke errors in the process
c) The reaction time for placing target/stop exit orders once an
entry order is filled.
d) Operator reaction time when a market entry condition is detected.

Item d is the single highest source of 'missed entries', operator was
on the telephone, or in the bathroom, at the time an analysis
detected a suitable market entry condition. Keystroke error is the
single highest source of bad exits.

It's all about risk management. For some implementations, a given
methodology may be 'acceptable risk', and for others, it's
unacceptable. For our implementation, there can be no automation
introduced 'extra risk'.

That's the same issue as folks discussing problems with the api, and
wether or not one should work around them, or wait for it to be
fixed. If you are building a personal toy, to dabble with, then
waiting for IB to correct errors is probably prudent. If you are
building a tool, that needs to operate now, there is no point
complaining about api problems, you just have to find solutions to
them, and implement the solutions. We can re-visit a revised api
if/when the time comes.
twsapi: Re: To grozzie2:

bee_jay_61
 


So why don?t you use the QueryPerformanceCounter?
Because the order id is only 32 bits.
I know it is only Int32. Don?t worry about that. And I did not mean
to use the Result of QueryPerformanceCounter directly.

At the moment I am using GetTickCount and this works without
problems. If I will place one day more than one order in a
millisecond, than I will use QueryPerformanceCounter and
QueryPerformanceFrequency to calculate the order id.

Besides, if I understand it correctly he has multiple machines
placing
orders so there could be duplicates even when using a clock. Don't
tell me
about the "low possibility" of a collision. This is supposed to be
a game
of skill, not a game of chance.
I don?t have multiple machines placing orders. If I would to this,
that I would write a server and only this server would communicate
with the TWS.
Re: twsapi: Re: To grozzie2:

Nick
 

Well, it's probably less painful to set up your own id server. Have
one > app that requests id's from tws and have your trading software request id's > from the id server instead of tws.

Yes, it's another layer of stuff but I suspect it's the only reliable
way > to get around the problem you are facing.

That was the first approach we took. It doesn't work, here's why.
Assume you have 10 orders outstanding, each with an id. Disconnect
from tws, and reconnect, the call the open orders request. You get
10 reports, and they have all been assigned NEW id's, starting at the
first id you got from the nextid callback.

I had to take your approach a whole layer farther. I now have 'the
trading application' running on the TWS machine, and it exports a
network accessible api. All of the other machines talk to that
application, not TWS itself. I took one look at the issues of having
multiple programs trying to keep order lists strait, when they have
different id's, and said 'no thanks'.
Hmm, not to split hairs, but aren't we talking about the same thing really? The gist of my post was to create a central place where things would get coordinated. It looks like you needed to do more than just hand out unique id's but you did wind up with a central place to keep things together.

Thanks for the info about the re-assignment of id's. I can see this is going to be one of those "10% effort on the real project, 90% effort on spelunking TWS" deals.

- Nick
twsapi: Re: To grozzie2:

 

Well, it's probably less painful to set up your own id server.
Have one
app that requests id's from tws and have your trading software
request id's
from the id server instead of tws.

Yes, it's another layer of stuff but I suspect it's the only
reliable way
to get around the problem you are facing.
That was the first approach we took. It doesn't work, here's why.
Assume you have 10 orders outstanding, each with an id. Disconnect
from tws, and reconnect, the call the open orders request. You get
10 reports, and they have all been assigned NEW id's, starting at the
first id you got from the nextid callback.

I had to take your approach a whole layer farther. I now have 'the
trading application' running on the TWS machine, and it exports a
network accessible api. All of the other machines talk to that
application, not TWS itself. I took one look at the issues of having
multiple programs trying to keep order lists strait, when they have
different id's, and said 'no thanks'.
53201 - 53220 of 53803