Yes, that's right. I'm guessing you didn't change what you thought you changed. Here's a step by step if you want to try again:

• install the system package via sudo apt install libintelrdfpmath-dev
• cd to the sample directory... IBJts/samples/Cpp/TestCppClient

• in there edit the makefile so that it's using /usr/lib/x86_64-linux-gnu/libbidgcc000.a instead of libbid.a. That is, you should replace the option for libbid.a with -lbidgcc000. If you don't feel like taking chances with your typing use this patch:

  --- makefile.~1~	2022-10-03 11:50:16.000000000 +0000
  +++ makefile	2023-06-17 01:41:46.332638125 +0000
  @@ -8,7 +8,7 @@
  TARGET=TestCppClient
   
  $(TARGET)Static:
  -	$(CXX) $(CXXFLAGS) $(INCLUDES) $(BASE_SRC_DIR)/*.cpp ./*.cpp $(BASE_SRC_DIR)/lib/libbid.a -o$(TARGET)Static
  +	$(CXX) $(CXXFLAGS) $(INCLUDES) $(BASE_SRC_DIR)/*.cpp ./*.cpp -lbidgcc000 -o$(TARGET)Static
   
  $(TARGET):
  	$(CXX) $(CXXFLAGS) $(INCLUDES) ./*.cpp $(BASE_SRC_DIR)/lib/libbid.so $(SHARED_LIB_DIRS)/$(SHARD_LIBS) -o$(TARGET)
  

• then insure everything's clean with make clean

• finally build the static version of the sample with make

• view the docs? zcat /usr/share/doc/libintelrdfpmath-dev/README.gz

Or, as someone else suggested in the other thread, you can build the Intel sources directly (I won't outline that process).

That should get you started. Are you using Windows System for Linux or something? Not sure what to say then... I don't use it. Maybe someone else can help you troubleshoot if that's the case.

Good luck.

Thank you all for your help and information.

I tried the mods to compilation, packages, makefiles mentioned without any change in the outcome.

In fact it looks like the library is not a shared object at all

??? :: file !$
??? file lib/libbid.so
??? lib/libbid.so: PE32+ executable (DLL) (console) x86-64, for MS Windows, 18 sections


:: ldd lib/libbid.so
?? ?not a dynamic executable

Looking at the elf file header shows there is no header
??? :: elfedit --input-mach x86_64 --output-mach x86_64 --input-type dyn --output-type dyn lib/libbid.so
??? elfedit: Error: lib/libbid.so: Failed to read ELF header


I'm now at best confused because lib/libbid.so seems to be a windows dll according to the file command. Certainly it is not a linkable shared object.


So, if anyone has successfully compiled and run the cppclient would you please examine your lib/libbid.so and see if you see the same data?

On Fri, 16 Jun 2023 16:04:22 +0000 Lou Dudka loududka@... wrote:

The only other thing is I'm running TWS 981 and will have to upgrade. Maybe I'll try that.

BTW, I also just tried w/ 981 and the bracket went through. Which makes me think that you might try updating the API version you're using (not just TWS/IBGW). I'm sure you're aware that the API itself gets fixed and changes as time goes by.

Good luck.

Thanks, 闯ü谤驳别苍.

The API and gateway logs don't show any data or errors (although we had some gateway logs from the day it worked). We're raising a ticket with IBKR.

My best hypothesis at the moment is that we might have hit a limit on scanners even though the original ones were cancelled and the clients killed[^0] -- does that chime with anyone else's experience/expectations? If so, is there an easy fix?

[^0]: While I often felt like doing this in my freelancing days, I should stress that no humans were harmed in the production of this bug.

开云体育

my bet is that it's something internal to IB.? How about just setting up a reset loop with a count. So if you hit an error like this one log it, and retry the connection n times. If you blow the n times then shut down the connection gracefully and search for your towel.

???

Hello everyone,

Upon launching my trading app right before the market opened, the connection was established without any issues. However, after a few hours (during active market hours), while our app was functioning smoothly, an unexpected occurrence took place. I encountered the following errors in my app log:
"2157 - Sec-def data farm connection is broken:secdefil". ?No further messages were recorded in the app log.

Any ideas about the type of this error? And, how to deal with it?

Regards

On Wed, Jun 14, 2023 at 11:39 PM, David Walker wrote:

Here’s the full updated code which works fine:

I couldn't get your new version to work until I moved the reader_thread.start() call to after the self.connect("127.0.0.1", self.port, self.client_id) call; like 闯ü谤驳别苍's version.

But yeah... I'm glad the problem is resolved.

On Wed, Jun 14, 2023 at 11:00 PM, 闯ü谤驳别苍 Reinold wrote:

Hope this helps. I learned something new.

Me too... that was fun.

开云体育

There we have it, Dave. Let me see whether I can put the pieces together for you:

• The typical TWS API Python client is a two-thread solution.
• EReader runs in its own thread, reads TWS/IBGW messages from the network socket connection, puts them into msg_queue, but does not perform any callbacks that relate TWS messages.
• EClient and EWrapper run in the second thread. That thread not only processes your application logic but must also read messages from msg_queue and process the API callbacks.

• When your code calls wait(120) in open_connection, all API callback processing stops, you don't get any of the error messages, nextValidId is never called, and your program is in dead lock

Your approach would work just fine in a three-thread approach where EWrapper and EClient run in two separate threads (plus a third thread for EReader). That is the typical TWS API Java setup and has the advantage that callback processing continues, even if the application thread is very busy or preoccupied with a long blocking operation (say spends time reading or writing a file, or god-forbid, does a sleep()).

Depending on your design preference you'd take one of two routes:

• If you want to stay with the two-thread approach, take a look at what the IBKR Testbed/Program.py does. You'd basically define a "start" function as an entry point for your application code and call it at the end of the nextValidId function. No further synchronization is required since there is really only one thing going on at a time.

• If you want to go with the three-thread approach, just start self.run() in a separate thread (see below). The consequence is, however, that there is more parallelism in your application (callbacks can now happen all the time) and you may need more synchronization amount the pieces.

I simply added this to your code and it worked right away:

??? def processTwsCallbacks(self):
??????? self.run()

??? def open_connection(self):
??????? self.connection_error = False
??????? self.event_nextValidId_callback.clear()
??????? self.connect("127.0.0.1", self.port, self.client_id)

??????? callbackThread = threading.Thread( target=self.processTwsCallbacks, daemon=True )
??????? callbackThread.start()


This is only a rough outline. Keep in mind that I am not a Python practitioner and that there may be more to the three-thread solution than just starting self.run() in a separate thread. But It will be the more performant solution and callbacks will be performed much more timely.

Hope this helps. I learned something new.
闯ü谤驳别苍

On Wed, Jun 14, 2023 at 07:24 PM, buddy wrote:

between the client and wrapper then I'd say yes

Sorry, this should be between the client and reader. But, the set and wait are called from the same thread of Dave's example and so all that's accomplished is deadlock.

On Wed, Jun 14, 2023 at 06:12 PM, 闯ü谤驳别苍 Reinold wrote:

So the connect() call needs to create some kind of event object that can be used to block the main thread until the NextValidID can release the wait block.

If you mean between the client and wrapper then I'd say yes.

On Wed, Jun 14, 2023 at 06:12 PM, 闯ü谤驳别苍 Reinold wrote:

Would Event objects provided bz the asyncio package work better here (IBKR suggests asyncio and ib_insync is built upon it) ?

Hard to say since I think it depends on the overall context and mostly on what the program is meant to accomplish. For example, is it meant to be long lived or just do some processing and exit? What are the real-time requirements?

I could also envision simply setting a member to None and changing it to the next valid id once it arrives... then checking that member in busy loop; assuming concurrency isn't of any interest at all. Are we trying to conserve electricity? Insure simplicity and readability?

Not sure there are simple answers in this respect.

Would Event objects provided bz the asyncio package work better here (IBKR suggests asyncio and ib_insync is built upon it) ?

The goal, as I understand it, is to fully comply with the IBKR TWS API (as described in the section) and to block the main thread from making any API requests until:

• The socket connection with TWS/IBGW is established
• The protocol is started (client)

• And TWS/IBGW has sent the NextValidID message

So the connect() call needs to create some kind of event object that can be used to block the main thread until the NextValidID can release the wait block.

As a matter of practice, our (Java) based connect logic waits for the nextValidId() and managedAccounts() callbacks, since our clients cannot do any meaningful work until both have arrived.

闯ü谤驳别苍

On Wed, Jun 14, 2023 at 12:52 PM, buddy wrote:

On Wed, Jun 14, 2023 at 04:42 PM, 闯ü谤驳别苍 Reinold wrote:

you have to start the message reader in a separate thread

That's already happening via the base classes. Unfortunately, by calling the main thread itself (which should receive the callback is blocked) and then immediately throws a ConnectionError as soon as possible.

Dave, it's not clear what you're trying to accomplish by using the threading event. Threading events are a mechanism for IPC (inter-process communication) and meant to communicate between threads, but you don't start any more threads of your own.

Why not try getting rid of that stuff and just call app.run(), or in your case api.run(), as your final step in main like the example provided w/ the API?

Unfortunately I am not Python literate enough to give you concrete help, Dave. But on the surface, your approach looks fine.

I am wondering, though, whether there is some adverse interaction between the call to? self.event_nextValidId_callback.wait() and the ability of your client to process incoming TWS API messages. You say "Market data connection callbacks (2104, 2106, 2158) are shown in the IBG API log (but not received by the client)" which makes me wonder whether messages from TWS/IBGW get processed.

Not sure whether that happens behind the scene, but wouldn't you have to start the message reader in a separate thread as it is described in the chapter?

闯ü谤驳别苍

On Wed, Jun 14, 2023 at 09:59 AM, David Walker wrote:

Along your questions, I'd make sure that TWS/IBGW does indeed receive the requests and check what, if anything, is returned. So enable API logging in TWS/IBGW, run your program, decrypt the the API log in TWS/IBGW and check what the request looks like from the TWS/IBGW side.

The log entry for a successful request/response looks like this:

15:15:16:088 <- 22-10000001-20-STK-STK.US.MAJOR-MOST_ACTIVE_AVG_USD--------------0------

15:15:16:568 -> --..20-3-10000001-20-0-756733-SPY-STK--0--SMART-USD-SPY-SPY-SPY----- ... truncated ...

Assuming you see the incoming request but no response, I'd look at the TWS/IBGW log. It is verbose, cryptic, and complex, but you might see an internal error message in case you are triggering some kind of issue in TWS/IBGW. In any case, this would probably be a case for a trouble ticket with IBKR.

In case there is some kind of response, you'd obviously need to check in your own program. One area to check is error reporting. You seem to be getting the "regular" error callbacks, but EWrapper actually has three different versions of the error() callback method.

闯ü谤驳别苍

Hi 闯ü谤驳别苍 (and others),

Thanks for your comments on this, as helpful as always.? I'm experiencing a similar absence of the??callback, and have not been able to successfully troubleshoot it.

On calling connect(),

• the client immediately appears in the TWS/IBG API client list
• Market data connection callbacks (2104, 2106, 2158) are shown in the IBG API log (but not received by the client)
• the IBG API logs show the version handshake:
  
• tracing through connect() shows conn_time and server_version correctly obtained, and IBApi.EClient.connState set to 2 (Connected)
• API logs show no other errors, and the client receives no errors
• but....no nextValidId callback is recieved, at least within the 120 seconds I wait for it

I've copied the (immediately) relevant sections of the code below - this code reproduces the error with both TWS and IBG, so contains whatever is faulty.? The threading.Event() object (self.event_nextValidId_callback) is a standard async notification object in python.? But in any case, while debugging, the nextValidId function is never called, so it does not appear that the event notification itself is at fault.

I would really appreciate any help on this and hope I'm not missing something really obvious!.? Unil recently I have used other connection verification procedures which have largely worked fine; it is only since updating to using nextValidId that I have encountered this.

Dave

import threading
from ibapi.client import EClient
from ibapi.wrapper import EWrapper

class IB_API_Client(EWrapper, EClient):
??? def __init__(self, port, client_id):
??????? EClient.__init__(self, self)
??????? EWrapper.__init__(self)
??????? self.port = port
??????? self.client_id = client_id
??????? self.event_nextValidId_callback = threading.Event()

??? def nextValidId(self, request_ID):
??????? self.event_nextValidId_callback.set()
??????? super().nextValidId(request_ID)

??? def open_connection(self):
??????? self.connection_error = False
??????? self.event_nextValidId_callback.clear()
??????? self.connect("127.0.0.1", self.port, self.client_id)
? ? ? ??# wait for nextValidId

? ? ? ??if not self.event_nextValidId_callback.wait(120) or self.connection_error:
? ? ? ? ? ? raise ConnectionError((1, f'Could not connect on IB account port {self.port}'))
??????? else:
??????????? print('successful connection')

??? def error(self, reqId, errorCode, errorString, advancedOrderRejectJson=''):
??????? if errorCode in [502, 504]:
??????????? self.connection_error = True
??????? else:
??????????? print(f'Error {errorCode} - {errorString}')

??? def winError(self, text, lastError):
??????? raise WindowsError(text)


if __name__ == '__main__':
??? print('Seeking connection')
??? api = IB_API_Client(4002, 0)
??? api.open_connection()

[apologies if the code doesn't come through properly formatted]

I couldn't get the java client working (which is my problem, obviously), but I did manage to get a python client using ibsync up and running.

As far as I can see, it behaves in exactly the same way:

* it's definitely connected, other queries go through fine;
* the request is definitely reaching IB, because if I send a scan code like "NONSENSE", it returns an error; cancelling the scanner also works;
* no scan data is returned for a correct request

My question is really, how do I even go about trouble-shooting this? As you say, 闯ü谤驳别苍, it's a pretty simple scanner and (if I've read your message correctly) there's nothing wrong with the message reaching IB -- which fits with it having worked for a while last week.

Has anyone come across situations where a request is apparently mothballed rather than processed? Could there be some sort of problem with the account that causes a silent fail rather than an error? What would you try?

On Wed, Jun 14, 2023 at 08:51 AM, buddy wrote:

with different binutils

Sorry... this should read different compilers, not binutils.

Btw, searching around revealed where the same failed binutils assertion yielded a dangerous relocation and linker core dump. It certainly seems the problem is due to linking intermediate object files created with different binutils. I agree with the github comment that says:

This is not worth anyone's time. Just compile both sides for the same target triple and avoid the whole thing.

Fwiw, IDK why the IB SDK is distributing a pre-built library (either .a or .so). The only reason I can fathom is that they've modified the reference implementation but this strikes me as unlikely. Anyway...

This is not best practice as far as I know, C/C++ is not "write once run anywhere". IMHO they should include the source (but this would complicate the makefile a bit). Or they should instruct the user to use either... their own system packages or build the stuff from Intel.

Maybe they'll do this in the future but beware for now.