I am curious if anybody has come up with ways of doing real time
validation of the connection between TWS and IB. When using the
demo, our experience in this area was 'not good'. If I pull the plug
on the TWS connection at the firewall, it goes into a cycle of trying
to log back in, but never gives any kind of error indication to the
client application. I've kludged together a bit of an indication by
monitoring the bid/ask on the nq contract, basically we work on the
assumption that if there have been no updates for 30 seconds on the
nq, then it's likely the connection to IB is dead. The other option
we looked at, was a pinger thread, but that only verifies the network
connection, doesn't verify that TWS is indeed connected and live.

I am developing a socket API to TWS in C as I don't like C++ or MFC. I'm using the source they provide to the C++ lib as a "reference". Despite Richard Foulk's urging to "make the world a better place" the raw socket protocol as it stands is not well thought out and probably needs to be completely re-done. The C++ lib they provide is a thin wrapper over the socket i/o. Providing feedback on a beta product is one thing. Providing free consulting to do a proper design is another.

As for your specific situation of monitoring the connection to TWS, there is no heartbeat as far as I can tell. I know if I don't send anything to TWS the connection will be idle for over a minute. In essence TWS only talks the client in response to client requests. If you don't ask for anything you won't receive anything from the server (TWS). For now you would have to do your own fake heartbeat by making a request for something harmless and looking for a reply.

I is also don't think it's possible to recover from any communication problems on the socket due to the way data is sent from the server. They send a bunch of null-terminated strings for each logical record. If there are10 fields in a record then you read 10 "lines" from the socket. There is no header info with the number of lines or total length of the packet. All you can do is time out if you don't get all 10 lines, but then if any more data does arrive you don't know if it's the remaining part of the previous packet or a new packet altogether. At this point a timeout is fatal and you would have to close the connection and try to re-connect.

I'm just starting my investigation and this is what I have found out so far. If you are interested I can keep you posted on what I find as I go along. I'm not sure I will stay subscribed to this group as the dominance of messages appear to be VB programming questions.

Anyway, I hope this little bit of info is helpful.

- Nick

I is also don't think it's possible to recover from any

communication

problems on the socket due to the way data is sent from the

server. They

My expereince with communications problems is similar, any problem is
fatal. The c++ wrapper tends to end up in a state that's
unrecoverable, and to make it worse, if you call disconnect or delete
the object to try restart things, you get an exception thrown from
deep within windows somewhere, and it's a non recoverable event.
edisconnect always seems to end up referenecing a null pointer within
the windows kernel when called, and we have not found a solution to
this yet. Creation of a new connect object doesn't help at all, once
it's gone out to lunch, it's gone, and we have not figured out how to
successfully re-initialize the connection once it throws an error.
The only successful method we've got is really really ugly, but it
does work. When we catch a windows exception, spawn a new copy of
our application, and terminate the exception instance rudely. This
will work thru a few iterations of the process, and ultimately, the
windows 2000 TCP stack ends up confused eventually, and it's BSOD
time.

I'm not sure I will stay subscribed to this group as the dominance
of messages appear to be VB programming questions.

That's exactly why I piped up and asked if anybody is actually doing
anything other than VB. Wondering if I've found the right place to
discuss the issues of handling a windows exception thrown by the
edisconnect() function of the TWS api, vs how to handle various
various VB beginners issues that are totally unrelated to TWS.

Anyway, I hope this little bit of info is helpful.

Most helpful, knowing a bit about what's happening under the covers
kind of confirms my original take on the whole api. It's something
kludged together very quickly by a junior programmer that doesn't
understand real time, error recovery, or all of the 'little things'
that differentiate serious business software from casually developed
hobby stuff where no considerations are made for error identification
and recovery. This is kinda proven by all the functions
having 'void' return types, rather than some kind of value to at
least let you know the function did something, is trying to do
something, or failed completely.

I have forwarded detailed commentary to IB about specific problem
areas we have. I haven't had a response yet, but that's not my
concern. I'm not here to whine about shortcomings, just to
understand what it can/cant do, and how to work with it.

I'm VERY interested in hearing your success/problem stories of
rolling a new wrapper yourself. Our eventual goal here is to get all
of this 'off windows' onto a reliable host that we can run remotely
on a colo site. I dont dare consider running remotely until such
time as we have a methodology that includes full recovery from all
error situations. Like you, we'll probably have to roll our own for
the whole thing before it's done, and abandon the wrapper code that
IB has provided, just to get error handling into the system.

I'm VERY interested in hearing your success/problem stories of
rolling a new wrapper yourself.

I plan on releasing my library and source code for non-commercial purposes. The first shot will be essentially a replacement for the IB C++ lib without any warts. It has platform-specific code separated out so ports to *nix are intended to be easy.

The lib is primarily for my own use so it may not have features or functionality that others need, and it won't be a formal "supported product". I'm making it available as a service to the community.

Email me privately if you want to be on the distribution list.

- Nick

I thought the TCP socket will take care of the reliability part
between TWS and your client. Besides they are on the same local
host, so the only chance is buffer overflow, and that should be
taken care of by the TCP protocal.

Now regarding the internect connection with IB, do you actually
experienced such disconnection causing the client blocked in the
middle of a message? Would be very interested to learn the patterns
of "always where" it is blocked.

In my app I modified IB's code a bit - still using MFC though - but
does it in the burst mode, so that it doesn't have to call those
readInt or ReadFloat all the time. I also recongnize the possibility
that the available data in the socket may not be null terminated. So
by pulling them to a much larger buffer locally and having a macro
parsing through the buffer for readInt and readFloat I greatly
reduce the chance of blocking in the middle of a message. But I
admit I don't have a mechanism in place to deal with when such an
event actually happens, which could be caused by the internet
problem for example.

Anyway, I am very interested in your investigations. And if enough
of us C++ users find this group too irrelevent to our project, then
we should form our own group.


-weidong

--- In twsapi@y..., Nick <nickrbox@o...> wrote:

As for your specific situation of monitoring the connection to

TWS, there

is no heartbeat as far as I can tell. I know if I don't send

anything to

TWS the connection will be idle for over a minute. In essence TWS

only

talks the client in response to client requests. If you don't ask

for

anything you won't receive anything from the server (TWS). For

now you

would have to do your own fake heartbeat by making a request for

something

harmless and looking for a reply.

I is also don't think it's possible to recover from any

communication

problems on the socket due to the way data is sent from the

server. They

send a bunch of null-terminated strings for each logical record.

If there

are10 fields in a record then you read 10 "lines" from the

socket. There

is no header info with the number of lines or total length of the
packet. All you can do is time out if you don't get all 10 lines,

but then

if any more data does arrive you don't know if it's the remaining

part of

the previous packet or a new packet altogether. At this point a

timeout is

fatal and you would have to close the connection and try to re-

connect.

I'm just starting my investigation and this is what I have found

out so

far. If you are interested I can keep you posted on what I find

as I go

along. I'm not sure I will stay subscribed to this group as the

dominance

of messages appear to be VB programming questions.

Anyway, I hope this little bit of info is helpful.

- Nick

weidong wrote:

I thought the TCP socket will take care of the reliability part
between TWS and your client. Besides they are on the same local
host, so the only chance is buffer overflow, and that should be
taken care of by the TCP protocal.

They don't have to be on the same machine although I agree they probably would be most of the time.

Now regarding the internect connection with IB, do you actually
experienced such disconnection causing the client blocked in the
middle of a message? Would be very interested to learn the patterns
of "always where" it is blocked.

I haven't seen this yet. In my approach I never block anyway - the request will simply time out.

In my app I modified IB's code a bit - still using MFC though - but
does it in the burst mode, so that it doesn't have to call those
readInt or ReadFloat all the time. I also recongnize the possibility
that the available data in the socket may not be null terminated. So
by pulling them to a much larger buffer locally and having a macro
parsing through the buffer for readInt and readFloat I greatly
reduce the chance of blocking in the middle of a message. But I
admit I don't have a mechanism in place to deal with when such an
event actually happens, which could be caused by the internet
problem for example.

I have a generic "sockGets()" that reads lines from a socket and uses a large buffer to collect replies from the server. If there is an incomplete message the sockGets() will just time out. At that point the app can decide what to do but as far as I know a timeout is fatal.

- Nick

I thought the TCP socket will take care of the reliability part
between TWS and your client. Besides they are on the same local
host, so the only chance is buffer overflow, and that should be
taken care of by the TCP protocal.

Your are making the assumption here that they are on the same local
host. Our target end environment has analysis and trading on
separate computers, in separate locations.

Now regarding the internect connection with IB, do you actually
experienced such disconnection causing the client blocked in the
middle of a message? Would be very interested to learn the patterns
of "always where" it is blocked.

It's not so much as 'in the middle of a message', as 'is the
connection to IB still alive'. We currently process thru other
brokers, via different mechanisms, but, want IB in the loop. The
order volume generated by our setup can be significant, many
hundreds, and sometimes over a thousand, orders in a day. If the
system has 25 orders open in the market, and connectivity to the
broker dies, we need to know, so that manual procedures can be
initiated. Since the actual trading computer is sitting in a cubicle
on another site, where there is redundant net connection, redundant
power, etc, we need to do all of this programatically, there is no
human interaction at the computer actually executing trades in the
market.

Anyway, I am very interested in your investigations. And if enough
of us C++ users find this group too irrelevent to our project, then
we should form our own group.

I think i broke the ice, and the c/c++ folks are speaking up now :)

It looks to me like there's 2 main focus groups looking at the TWS
api. The first, is what i would term the 'hobby folks', toying with
automated trading for the first time, and learning VB in the
process. The second group, developers that have been doing trading
type stuff for a long time already, and the interface to IB is 'just
another tool to go in the toolbox'.

IB is an intriguing tool too. It will be interesting to see where it
all ends up.

I have a generic "sockGets()" that reads lines from a socket and

uses a

large buffer to collect replies from the server. If there is an

incomplete

message the sockGets() will just time out. At that point the app

can

decide what to do but as far as I know a timeout is fatal.

Now here is the real question, and if you have the answer, it'll save
a lot of digging and trial/error on my part.

Is the error truely fatal, ie application must totally close? Or,
will TWS 'be happy' if you close the connection cleanly, and then try
restart.

Up until now, I've only used the wrapper libraries they provided, and
have got all the rest of our application 'happy' now. It's the
connect/disconnect problem that is my current show stopper, and if
simply fixing the wrappers will cure it, that shouldn't be hard.

Now here is the real question, and if you have the answer, it'll save
a lot of digging and trial/error on my part.

Is the error truely fatal, ie application must totally close? Or,
will TWS 'be happy' if you close the connection cleanly, and then try
restart.

Up until now, I've only used the wrapper libraries they provided, and
have got all the rest of our application 'happy' now. It's the
connect/disconnect problem that is my current show stopper, and if
simply fixing the wrappers will cure it, that shouldn't be hard.

My answer right now is that closing the socket and opening a new one will work. I guess I should have said that the error is fatal *for the current connection*.

I have aborted many times while snooping the protocol and I was able to simply run another test without restarting TWS.

Also, as an experiment in response to your question I did the following:
1) Connect to TWS
2) Request some data (account info)
3) Deliberately not read the entire reply
4) Close the socket
5) Open a new socket
6) The newly opened socket appears to work fine

Of course, it's still early in my experiments and things could change, but for now it seems that it's not fatal for the app.

- Nick

--- In twsapi@y..., "grozzie2" <grozzie2@y...> wrote:

"I think i broke the ice, and the c/c++ folks are speaking up now :)

It looks to me like there's 2 main focus groups looking at the TWS
api. The first, is what i would term the 'hobby folks', toying with
automated trading for the first time, and learning VB in the
process. The second group, developers that have been doing trading
type stuff for a long time already, and the interface to IB is 'just
another tool to go in the toolbox'."

BREAK

I think that we all could benefit from professional C++ programmers
using this forum for discussion. I would hate to see a splinter,
rather I would like to see heightened level of discussion. I just
started my programming about 4 months ago. Eventually I will move to
C++. For now, I stay with VB because I have not reached the
limitations of this Psuedo OOL...when I do I would be greatful for
any past discussions here concerning C++. Let it Rip.

In fact you may hasten that moment of transformation as you make all
of us aware of the power of a true OOL...

Please feel free to take the discussion in the direction you please.

Scott

I thought the TCP socket will take care of the reliability part
between TWS and your client. Besides they are on the same local
host, so the only chance is buffer overflow, and that should be
taken care of by the TCP protocal.

Your are making the assumption here that they are on the same local
host. Our target end environment has analysis and trading on
separate computers, in separate locations.

Now regarding the internect connection with IB, do you actually
experienced such disconnection causing the client blocked in the
middle of a message? Would be very interested to learn the

patterns

of "always where" it is blocked.

It's not so much as 'in the middle of a message', as 'is the
connection to IB still alive'. We currently process thru other
brokers, via different mechanisms, but, want IB in the loop. The
order volume generated by our setup can be significant, many
hundreds, and sometimes over a thousand, orders in a day. If the
system has 25 orders open in the market, and connectivity to the
broker dies, we need to know, so that manual procedures can be
initiated. Since the actual trading computer is sitting in a

cubicle

on another site, where there is redundant net connection, redundant
power, etc, we need to do all of this programatically, there is no
human interaction at the computer actually executing trades in the
market.

Anyway, I am very interested in your investigations. And if

enough

of us C++ users find this group too irrelevent to our project,

then

we should form our own group.

I think i broke the ice, and the c/c++ folks are speaking up now :)

It looks to me like there's 2 main focus groups looking at the TWS
api. The first, is what i would term the 'hobby folks', toying

with

automated trading for the first time, and learning VB in the
process. The second group, developers that have been doing trading
type stuff for a long time already, and the interface to IB

is 'just

another tool to go in the toolbox'.

IB is an intriguing tool too. It will be interesting to see where

it

all ends up.

Also, as an experiment in response to your question I did the

following:

1) Connect to TWS
2) Request some data (account info)
3) Deliberately not read the entire reply
4) Close the socket
5) Open a new socket
6) The newly opened socket appears to work fine

Of course, it's still early in my experiments and things could

change, but

for now it seems that it's not fatal for the app.

Thank you VERY much. I did a quick recompile of the
c++ wrappers (found the code, didn't even realize it
was there), and have same problems as I did before,
but, it's pretty obvious why looking at the code.
There's a lot of exception handlers in there that
do nothing, absolutely nothing, and the connect/
disconnect code blindly closes sockets that were
never opened etc.

If the socket itself can be closed, and one can start
fresh with a new one, then I know, it's not so bad,
only the wrapper is causing the problems, and that I
can work around.

Your are making the assumption here that they are on the same

local

host. Our target end environment has analysis and trading on
separate computers, in separate locations.

I thought only the local host "127.0.0.1" is supported for now, from
IB's source code. Is this not the case any more?

-weidong

开云体育