开云体育

ctrl + shift + ? for shortcuts
© 2025 开云体育
  1. Twsapi
  2. Topics

Pulling reqMktData for multiple tickers

 

Hey guys, what I am trying to perform is a reqMktData pull (I'm pulling the bid,ask,last prices) for multiple tickers. Normally I would just change the ID number for each ticker.. but for this specific program I am building. The ticker amounts are going to change each day. One day it could be 3 stocks, next day it could be 7.?

So I am trying to come up with a fancy way to "spin off" the reqMktData into its own threads so that can change by the input.?

What is happening is, that the threads spin off fine, but the reqMktData just keeps getting "replaced" by the next ticker in line in the function.?

Really what I need it to do is this

1) For ticker in tickers (the loop through). After we loop the first one. Stop the function and loop. And seal that up as the first thread.
2) Run the function again, for ticker in tickers (the loop) but we pick back up where we left off. Creates another ReqMktData in the 2nd thread. Stop the function and loop from continuing.
3) Function runs again, for ticker in tickers (the loop), and we once again pick back up where we left off. Creates another reqMktData in the 3rd thread. Stop the function and loop.

Since there is only 3 "tickercount" its all finished. Now I had 3 threads all running a reqMktData but each thread has a unique "tickerId" corresponding with each stock in the tickers list.?

Does that make sense? Am I just over complicating this? Here is the code for you to study. Its obviously not the whole program. Everything else in the program works fine.?

tickerId = 0
tickercount = 0

tickers = ["JG", "AAPL", "FB"]

def dataReturn(app):

? ? global tickerId

? ? for ticker in tickers:

? ? ? ? tickerId += 1

? ? ? ? app.reqMktData(tickerId, usTechStk(ticker), "", False, False, [])

? ? ? ? time.sleep(5) #Add some breathing room for testing

? ? ? ??

threads = []

for ticker in tickers:

? ? tickercount += 1

? ? ? ??

for _ in range(tickercount):

? ? t = threading.Thread(target=dataReturn, args=(app,))

? ? t.start()

?

? ? threads.append(t)
 

Hi Patric,

I have one thread and keep sequential thread IDs in DB as I corelate them to data received. The TCP socket is not asynchronous so I am not sure if your threading separately for each request doest anything because you still go through same TCP socket. It probably saves you some microsecond which might be significant if doing HFT. Otherwise, sending requests one after other should be fast too.? Callbacks keep receiving data in no order (out of our control) so again not sure if threading helps. So yeah, maybe you are over complicating.

^^^ I am talking in context of C++ and not Python.

I would be interested to hear a senior person defend your method as better as it may be somehow future proofing and something I am missing. Aha, if you will have multiple account support in future ;) this may help.

- Bruce


On Wed, May 5, 2021, 1:18 PM Patrick C <zenfiretrading@...> wrote:

Hey guys, what I am trying to perform is a reqMktData pull (I'm pulling the bid,ask,last prices) for multiple tickers. Normally I would just change the ID number for each ticker.. but for this specific program I am building. The ticker amounts are going to change each day. One day it could be 3 stocks, next day it could be 7.?

So I am trying to come up with a fancy way to "spin off" the reqMktData into its own threads so that can change by the input.?

What is happening is, that the threads spin off fine, but the reqMktData just keeps getting "replaced" by the next ticker in line in the function.?

Really what I need it to do is this

1) For ticker in tickers (the loop through). After we loop the first one. Stop the function and loop. And seal that up as the first thread.
2) Run the function again, for ticker in tickers (the loop) but we pick back up where we left off. Creates another ReqMktData in the 2nd thread. Stop the function and loop from continuing.
3) Function runs again, for ticker in tickers (the loop), and we once again pick back up where we left off. Creates another reqMktData in the 3rd thread. Stop the function and loop.

Since there is only 3 "tickercount" its all finished. Now I had 3 threads all running a reqMktData but each thread has a unique "tickerId" corresponding with each stock in the tickers list.?

Does that make sense? Am I just over complicating this? Here is the code for you to study. Its obviously not the whole program. Everything else in the program works fine.?

tickerId = 0
tickercount = 0

tickers = ["JG", "AAPL", "FB"]

def dataReturn(app):

? ? global tickerId

? ? for ticker in tickers:

? ? ? ? tickerId += 1

? ? ? ? app.reqMktData(tickerId, usTechStk(ticker), "", False, False, [])

? ? ? ? time.sleep(5) #Add some breathing room for testing

? ? ? ??

threads = []

for ticker in tickers:

? ? tickercount += 1

? ? ? ??

for _ in range(tickercount):

? ? t = threading.Thread(target=dataReturn, args=(app,))

? ? t.start()

?

? ? threads.append(t)
 

I'm not picking this method for any reason other then my mind cant figure out any other way to do it. LOL?

I'm trying to request tickTypes from multiple stocks that will change amount every day. Otherwise the easy way to do what I am doing is just this. Or the only way I know how to do what I am trying to do.?

app.reqMktData(1, usTechStk("SPY"), "", False, False, [])
app.reqMktData(2, usTechStk("AAPL"), "", False, False, [])
app.reqMktData(3, usTechStk("AMD"), "", False, False, [])

Ya know? Just copy and paste as neccessary, but I'm trying to make this program a little more elegant then that LOL. I was hoping I can just update my tickers list each morning and the program does the rest. Only change one line of a code if I have too, instead of having to copy/paste and change 10+ lines of code every morning. I've never built a program to handle multiple stocks at the same time. I only have experience with Futures.. and I have a individual program for each individual futures contract.
 

So what you want to do is to have a threadpool so that when you get ticks from the TCP socket, you can launch a task on that threadpool (the size of the pool should be the core count - 1 for the TCP socket's thread).? This is quite easy in the JVM languages, but you are out in Python land.? So you want to do something where you generate a Task on each tick and that task contains the ticker and tick pricing information.? The threadpool schedules that task, it is executed and in a synchronized way would do whatever action is necessary (make an order, etc).? Make sure to synchronize your access to IB and the TCP socket to take those actions to avoid synch bugs.

As a side note, I doubt this scheme is worthwhile.? The overhead of making the tasks is likely more than the CPU usage by your task.? Unless you are doing something very computational intensive in your tasks, the CPU would likely completely the work before a task could be created and executed.? IB is very slow, CPUs are very fast.

Hunter


On Wednesday, May 5, 2021, 11:02:06 AM PDT, Patrick C <zenfiretrading@...> wrote:


I'm not picking this method for any reason other then my mind cant figure out any other way to do it. LOL?

I'm trying to request tickTypes from multiple stocks that will change amount every day. Otherwise the easy way to do what I am doing is just this. Or the only way I know how to do what I am trying to do.?

app.reqMktData(1, usTechStk("SPY"), "", False, False, [])
app.reqMktData(2, usTechStk("AAPL"), "", False, False, [])
app.reqMktData(3, usTechStk("AMD"), "", False, False, [])

Ya know? Just copy and paste as neccessary, but I'm trying to make this program a little more elegant then that LOL. I was hoping I can just update my tickers list each morning and the program does the rest. Only change one line of a code if I have too, instead of having to copy/paste and change 10+ lines of code every morning. I've never built a program to handle multiple stocks at the same time. I only have experience with Futures.. and I have a individual program for each individual futures contract.
 

Hunter, thank you so much. But I've got no idea what you just said to me haha. I am a prime example of a "shade tree" python mechanic. I got no idea how to build an alternator in a car, but I know where it goes haha. I'm trying to build this program for my buddy but I might just tell him that its over my head with multiple stocks that change amount each day. I can actually build this program with multiple stocks. IF.. they didnt change. OR.. IF the number amount didnt change. And I would just tell him... ok copy/paste the new stock name here, here, here, ect ect. But since one day it could be 1 stock, next day 7 stocks. I'm trying to make it so the program just reacts accordingly. I think it might be over my head right now.?
 

I'm not a Python programmer.? I would hope that Python has some stock threadpool library for you to use.? Perhaps some of the Python folks could recommend one to you.?

Concurrent programming is hard; most professional programmers can't do it.? You can still handle multiple tickers without concurrent programming though.? There is no reason you need multiple threads.? Just make a callback for each thread (it will run on the TCP socket's thread).? That's far easier and will work well enough.? Like I said, it isn't assured that this single threaded way won't be faster.? Unless you are doing lots of work in your callback (as in doing matrix multiplies or DB queries or something like that), then you are fine.

Hunter


On Wednesday, May 5, 2021, 11:24:56 AM PDT, Patrick C <zenfiretrading@...> wrote:


Hunter, thank you so much. But I've got no idea what you just said to me haha. I am a prime example of a "shade tree" python mechanic. I got no idea how to build an alternator in a car, but I know where it goes haha. I'm trying to build this program for my buddy but I might just tell him that its over my head with multiple stocks that change amount each day. I can actually build this program with multiple stocks. IF.. they didnt change. OR.. IF the number amount didnt change. And I would just tell him... ok copy/paste the new stock name here, here, here, ect ect. But since one day it could be 1 stock, next day 7 stocks. I'm trying to make it so the program just reacts accordingly. I think it might be over my head right now.?

 

Oops.? Sorry for the second post.

Make that 'a callback for each ticker' and register said callback using reqMktData (or whichever API call you are using).? Basically, don't use threads unless you know you need them.

Hunter

On Wednesday, May 5, 2021, 12:04:32 PM PDT, Hunter C Payne <hunterpayne2001@...> wrote:


I'm not a Python programmer.? I would hope that Python has some stock threadpool library for you to use.? Perhaps some of the Python folks could recommend one to you.?

Concurrent programming is hard; most professional programmers can't do it.? You can still handle multiple tickers without concurrent programming though.? There is no reason you need multiple threads.? Just make a callback for each thread (it will run on the TCP socket's thread).? That's far easier and will work well enough.? Like I said, it isn't assured that this single threaded way won't be faster.? Unless you are doing lots of work in your callback (as in doing matrix multiplies or DB queries or something like that), then you are fine.

Hunter


On Wednesday, May 5, 2021, 11:24:56 AM PDT, Patrick C <zenfiretrading@...> wrote:


Hunter, thank you so much. But I've got no idea what you just said to me haha. I am a prime example of a "shade tree" python mechanic. I got no idea how to build an alternator in a car, but I know where it goes haha. I'm trying to build this program for my buddy but I might just tell him that its over my head with multiple stocks that change amount each day. I can actually build this program with multiple stocks. IF.. they didnt change. OR.. IF the number amount didnt change. And I would just tell him... ok copy/paste the new stock name here, here, here, ect ect. But since one day it could be 1 stock, next day 7 stocks. I'm trying to make it so the program just reacts accordingly. I think it might be over my head right now.?

 

So lets step back a bit.? Conceptually any regular old program you wirte wiith the standard IBK client library (there are other ways to do it) has two (2) threads of execution.? The first is the main thread that starts running when you run your Python program.? The second is a looping "Inbound" thread created by the IBK client library on the socket connection that reads any incoming messages from IBK, parses them, and calls the appropriate "callback" function for that particular message.

So this is Asychronous in the sense that the first thread can and will freely fire off requests to IBK and meanwhile the second thread is happily reading in all the responses and invoking the callback functions.??? An analogy is if you are the first thread you can write a bunch of letters to IBK and drop them off in the mailbox then the Postal Service delivers them to IBK.? Eventually, at some point, at their convience, IBK will answer those letters and the Postal Service will evenutally deliver them to your mailbox where, lets say your Wife gets all the mail in your home and distributes all the letters to the family.? The Postal Service is the internet and your Wife is the Second Thread.

Notice, unless you the Thread 1, the sender,? and your Wife, Thread 2, somehow "meetup" you will never know IBK responsed to your letters. This is actually somewhat key, but I'll not get into it now.

OK, so lets talk threads:

Should you create N threads to fire off N requests?? Not really necessary.? To send a Request Msg to IBK is a few 10s of bytes pushed over the socket and over the internet which a single thread can do Bang, Bang, Bang in a few milliseconds.? Could you spawn a dedicated thread for each individual and every individual request?? Yes, but the overhead in creating a thread is probably on par with just sending a request alone.

So given

Thread Main
?- Send Req 1
- Send Req 2
- Send Req 3

versus

Thread Main
?- Spawn Thread to Send Req 1
?- Spawn Thread? to Send Req 2
- Spane Thread to Send Req 3

Is a lot of compexity for little to no gain.? In fact the second will be longer.? Sending a request is a near nothing in time so in this example the first approach is fine.

OK so what about in the Inbound side after you ripped through your requests bang, bang, bang.?? IBK will start sending responses for all three request which will arrive in some arbitrary and mixed order.? So for three reqMktData requests you are going to get a stream of incoming TickXXX msgs all mixed together.? Say TickPrice, TickSize, ... for? AAPL, SPY,? AAPL, AMD, SPY, SPY, AMD, ... ticks etc.

So these are all coming in and are handled by the Inbound Thread, Thread 2.? However, you know that you used ReqId 101, 102 and 103 respectively for each request. So if a TickSize msg shows up, you know that if the ReqId is 101 that that was the AAPL mktDataReq TickSize etc.

So what to do?? Well as each message comes in you do whatever you like with each one.? Only interest in the BID and ASK,? then throw all the others away.? You could write the Bid and Ask values into a DB.? Or you put them into a hashmap into a vector/list ...? When you get the Bid and Ask msgs for each stock you can happily send a cancel mktDataReq message for ReqId XXX.?

Notice you are donig N requests with N streamed reponses and we haven't created a bunch of threads. We certainly could. No need!! (so far) as we are just doing: Send mktDataReq, Wait on Bid and Ask Tick msgs, Write to a file,? Cancel Req.

So when we would ever need threads at all. ? Yea, this would start to get a little subtle.? You might create and use a Native Thread Pool if say you were to stream N reqMktData requests for N stocks and you were doing complex computational Time Series Analysis on each data stream.? You might create N Green Threads if say you structured your code into Agent Classes where each Agent did a series of work requests and processing.?

For example, say I have a OptionChainAgent where creating and running an Agent with a given symbol would call ReqMktData on the underlying stock, collect some data, cancel the request, request a ContractDetails to get the ConId, they use the ConId to get the Options with 1.5 StdDevs from a set number of Exchanges for a given date, then for each stirke do a reqMktData to some data.?? Here I might run _each_ Agent on its own thread and then each Agent may create even more threads to do some parts of the above work stream. Mine do.

So one final thought as ti whether it is ever dealing with threads at all.? Yes, but avoid if at all possible.? Consider a Watchlist you created in the IBK's TWS with 50 stocks.?? IBK could loop through each one, one-by-one, creating a reqMktData, geting some data, canceling the request, update the Watchlist, loop to the next symbol, forever.? Very inefficient.? Sloooow.? Insteak IBK probably fires off 50 reqMktData requests bang, bang, bang, .... and as a msg arrives for? a particlur symbol updates the Watchlist immediately.??? Basically there is a crossover point where threads become necessary.

Sorry I know this is fustrating because you'd be a lot happier if I just answered with a nice snippit of working code.? But I don't use Python.? And I did offer the opinion that for your (from what I know) very simple case you don't need to deal with Threads? at all.?

Final Thougt: IF YOU ARE NOT AN EXPERIENCED PROGRAMMER WITH CONCURRENT AND PARALLEL PROGRAMS THEN JUST DON"T USE THREADS UNLESS YOU REALLY REALLY REALLY HAVE TO.






On Wed, May 5, 2021 at 2:02 PM Patrick C <zenfiretrading@...> wrote:

I'm not picking this method for any reason other then my mind cant figure out any other way to do it. LOL?

I'm trying to request tickTypes from multiple stocks that will change amount every day. Otherwise the easy way to do what I am doing is just this. Or the only way I know how to do what I am trying to do.?

app.reqMktData(1, usTechStk("SPY"), "", False, False, [])
app.reqMktData(2, usTechStk("AAPL"), "", False, False, [])
app.reqMktData(3, usTechStk("AMD"), "", False, False, [])

Ya know? Just copy and paste as neccessary, but I'm trying to make this program a little more elegant then that LOL. I was hoping I can just update my tickers list each morning and the program does the rest. Only change one line of a code if I have too, instead of having to copy/paste and change 10+ lines of code every morning. I've never built a program to handle multiple stocks at the same time. I only have experience with Futures.. and I have a individual program for each individual futures contract.
 

Ray, that's a pretty good description of why you use a threadpool.

For reference, here is how the Java API works (different from what Ray described which I suspect is how it works in Python).? Note, this is more to reduce future questions than for this question.

There is an IB thread that loops over the socket waiting for either requests from your program and/or responses from IB.? This is internal to IB's API.? Then there is a thread created by IB's API that handles callbacks for that socket.? Finally, there is your main thread (probably created when the program started).? Your main thread makes the calls to the API to register handlers for ticks for specific tickers.? The internal thread gets those messages and uses the socket to write messages requesting specific data feeds to the network (to IB).? When IB responds, the internal thread packages up the response and gives it to the IB callback thread which calls your callback you registered with your main thread.? You do your work on this callback thread and make your orders from this callback thread (synchronously, IB's API isn't thread-safe).? Your main thread waits for you to tell it to quit the program or for the connection to IB breaks (at which time you can reconnect or relaunch your program).

Ray is right.? You can add the complexity of a threadpool and have the callback thread create task objects (containing the ticker and tick) which run on the threadpool but that probably isn't necessary unless you are doing something very computational intensive.

I hope this clarifies things for you.? Concurrency is hard and it is very easy to get lost in the weeds when working out a design.

Hunter

PS don't write to a DB, use a queue in those cases

On Wednesday, May 5, 2021, 12:07:37 PM PDT, Ray Racine <ray.racine@...> wrote:


So lets step back a bit.? Conceptually any regular old program you wirte wiith the standard IBK client library (there are other ways to do it) has two (2) threads of execution.? The first is the main thread that starts running when you run your Python program.? The second is a looping "Inbound" thread created by the IBK client library on the socket connection that reads any incoming messages from IBK, parses them, and calls the appropriate "callback" function for that particular message.

So this is Asychronous in the sense that the first thread can and will freely fire off requests to IBK and meanwhile the second thread is happily reading in all the responses and invoking the callback functions.??? An analogy is if you are the first thread you can write a bunch of letters to IBK and drop them off in the mailbox then the Postal Service delivers them to IBK.? Eventually, at some point, at their convience, IBK will answer those letters and the Postal Service will evenutally deliver them to your mailbox where, lets say your Wife gets all the mail in your home and distributes all the letters to the family.? The Postal Service is the internet and your Wife is the Second Thread.

Notice, unless you the Thread 1, the sender,? and your Wife, Thread 2, somehow "meetup" you will never know IBK responsed to your letters. This is actually somewhat key, but I'll not get into it now.

OK, so lets talk threads:

Should you create N threads to fire off N requests?? Not really necessary.? To send a Request Msg to IBK is a few 10s of bytes pushed over the socket and over the internet which a single thread can do Bang, Bang, Bang in a few milliseconds.? Could you spawn a dedicated thread for each individual and every individual request?? Yes, but the overhead in creating a thread is probably on par with just sending a request alone.

So given

Thread Main
?- Send Req 1
- Send Req 2
- Send Req 3

versus

Thread Main
?- Spawn Thread to Send Req 1
?- Spawn Thread? to Send Req 2
- Spane Thread to Send Req 3

Is a lot of compexity for little to no gain.? In fact the second will be longer.? Sending a request is a near nothing in time so in this example the first approach is fine.

OK so what about in the Inbound side after you ripped through your requests bang, bang, bang.?? IBK will start sending responses for all three request which will arrive in some arbitrary and mixed order.? So for three reqMktData requests you are going to get a stream of incoming TickXXX msgs all mixed together.? Say TickPrice, TickSize, ... for? AAPL, SPY,? AAPL, AMD, SPY, SPY, AMD, ... ticks etc.

So these are all coming in and are handled by the Inbound Thread, Thread 2.? However, you know that you used ReqId 101, 102 and 103 respectively for each request. So if a TickSize msg shows up, you know that if the ReqId is 101 that that was the AAPL mktDataReq TickSize etc.

So what to do?? Well as each message comes in you do whatever you like with each one.? Only interest in the BID and ASK,? then throw all the others away.? You could write the Bid and Ask values into a DB.? Or you put them into a hashmap into a vector/list ...? When you get the Bid and Ask msgs for each stock you can happily send a cancel mktDataReq message for ReqId XXX.?

Notice you are donig N requests with N streamed reponses and we haven't created a bunch of threads. We certainly could. No need!! (so far) as we are just doing: Send mktDataReq, Wait on Bid and Ask Tick msgs, Write to a file,? Cancel Req.

So when we would ever need threads at all. ? Yea, this would start to get a little subtle.? You might create and use a Native Thread Pool if say you were to stream N reqMktData requests for N stocks and you were doing complex computational Time Series Analysis on each data stream.? You might create N Green Threads if say you structured your code into Agent Classes where each Agent did a series of work requests and processing.?

For example, say I have a OptionChainAgent where creating and running an Agent with a given symbol would call ReqMktData on the underlying stock, collect some data, cancel the request, request a ContractDetails to get the ConId, they use the ConId to get the Options with 1.5 StdDevs from a set number of Exchanges for a given date, then for each stirke do a reqMktData to some data.?? Here I might run _each_ Agent on its own thread and then each Agent may create even more threads to do some parts of the above work stream. Mine do.

So one final thought as ti whether it is ever dealing with threads at all.? Yes, but avoid if at all possible.? Consider a Watchlist you created in the IBK's TWS with 50 stocks.?? IBK could loop through each one, one-by-one, creating a reqMktData, geting some data, canceling the request, update the Watchlist, loop to the next symbol, forever.? Very inefficient.? Sloooow.? Insteak IBK probably fires off 50 reqMktData requests bang, bang, bang, .... and as a msg arrives for? a particlur symbol updates the Watchlist immediately.??? Basically there is a crossover point where threads become necessary.

Sorry I know this is fustrating because you'd be a lot happier if I just answered with a nice snippit of working code.? But I don't use Python.? And I did offer the opinion that for your (from what I know) very simple case you don't need to deal with Threads? at all.?

Final Thougt: IF YOU ARE NOT AN EXPERIENCED PROGRAMMER WITH CONCURRENT AND PARALLEL PROGRAMS THEN JUST DON"T USE THREADS UNLESS YOU REALLY REALLY REALLY HAVE TO.






On Wed, May 5, 2021 at 2:02 PM Patrick C <zenfiretrading@...> wrote:

I'm not picking this method for any reason other then my mind cant figure out any other way to do it. LOL?

I'm trying to request tickTypes from multiple stocks that will change amount every day. Otherwise the easy way to do what I am doing is just this. Or the only way I know how to do what I am trying to do.?

app.reqMktData(1, usTechStk("SPY"), "", False, False, [])
app.reqMktData(2, usTechStk("AAPL"), "", False, False, [])
app.reqMktData(3, usTechStk("AMD"), "", False, False, [])

Ya know? Just copy and paste as neccessary, but I'm trying to make this program a little more elegant then that LOL. I was hoping I can just update my tickers list each morning and the program does the rest. Only change one line of a code if I have too, instead of having to copy/paste and change 10+ lines of code every morning. I've never built a program to handle multiple stocks at the same time. I only have experience with Futures.. and I have a individual program for each individual futures contract.
 

Victems of Python?
Regarding this discussion, I recalled, how `IB-Ruby` does it:

# define symbos
> symbols = %W( ANW? AEB? AED? AEG? AEH AER? HIVE AJRD AET? AMG? AFL? MITT? AGCO? A? AEM? ADC? AL? APD? AYR )
# put into a file-database
> IB::Symbols.allocate_collection :w20????????????????????????????????????????????
? => IB::Symbols::W20
> symbols.each_with_index{ |s,i| IB::Symbols::W20.add_contract i, IB::Stock.new( symbol: s? ) }
# call threaded market-price method and wait for all threads to finish
> prices = Symbols::W20.map{|x| x.market_price(thread: true )}.each( &:join);

Documentation:
https://ib-ruby.github.io/ib-doc/market_price.html
 

Hi Patrick,
Have you considered using input to get the symbols? That way, no need to change any code with cut and paste. And, you can loop for as many symbols as you need, each day having a different set of symbols.

Maybe something like:

sym = ''
req_id = 0
while not sym=='0':
??? sym = input("Please enter symbol or 0 to end")
??? print(f'symbol is: {sym}')
??? req_id += 1
??? app.reqMktData(req_id, usTechStk(sym), "", False, False, [])
? ? ?
 

Ouch.? So my reply was more damaging then helpful.?

To follow up.

As some pointed out there is the IBK 2-Thread and 3-Thread model that is explaned in the iBK doc really well.? I conflated the internal socket reading thread with the separate call back thread for ease of discussion but probably muddied things up.

The link to the Ruby Library is pretty illustrative.? It shows that that library supports a snapshot request for market data that one may use sequentially.? It shows an async version and then a threaded version, as the library offers the capabiility for a syncronous request/response? on a per thread basis.? Note however this is a "higher level" library where the author did a lot of work to support that.

The standard IBK libarry is (deliberately) pretty bare bones.? If you use it, over time, as you do more sophiisticated things,? you will end up building an additionals layer of code? on top of the library with various abstractions, such as a synchronous request / reponse per thread capability,etc. (Notice that is quite different from just sending the requests each on their own thread.)

More specificallyl to your original dilemna.?
  1. ?I am using the base IBK client library and need to do a handful of basic Market Data requests should I use threading??? Probably not.?? The sketch of code you were laying out was the start of a slippery slop of writing your own layer of code to support an abstraction of a synchronous request/response on a single thread.? True once you wrote that, yes then you could loop N times, create N req/resp threads, and join (sync) on thread completion. BUT that is a lot of work for your simple problem.
  2. I am using a higher level, third party? (Python, Ruby etc) library that offers threaded Market Data request capability out of the box, should I leverage this capability.? Probably, as someone has already done a lot of heavy lifting for you and made it "simple". There is a highler level. open source, Python library that gets taked about here on the list.? I'd suggest looking into that.


On Wed, May 5, 2021 at 1:18 PM Patrick C <zenfiretrading@...> wrote:

Hey guys, what I am trying to perform is a reqMktData pull (I'm pulling the bid,ask,last prices) for multiple tickers. Normally I would just change the ID number for each ticker.. but for this specific program I am building. The ticker amounts are going to change each day. One day it could be 3 stocks, next day it could be 7.?

So I am trying to come up with a fancy way to "spin off" the reqMktData into its own threads so that can change by the input.?

What is happening is, that the threads spin off fine, but the reqMktData just keeps getting "replaced" by the next ticker in line in the function.?

Really what I need it to do is this

1) For ticker in tickers (the loop through). After we loop the first one. Stop the function and loop. And seal that up as the first thread.
2) Run the function again, for ticker in tickers (the loop) but we pick back up where we left off. Creates another ReqMktData in the 2nd thread. Stop the function and loop from continuing.
3) Function runs again, for ticker in tickers (the loop), and we once again pick back up where we left off. Creates another reqMktData in the 3rd thread. Stop the function and loop.

Since there is only 3 "tickercount" its all finished. Now I had 3 threads all running a reqMktData but each thread has a unique "tickerId" corresponding with each stock in the tickers list.?

Does that make sense? Am I just over complicating this? Here is the code for you to study. Its obviously not the whole program. Everything else in the program works fine.?

tickerId = 0
tickercount = 0

tickers = ["JG", "AAPL", "FB"]

def dataReturn(app):

? ? global tickerId

? ? for ticker in tickers:

? ? ? ? tickerId += 1

? ? ? ? app.reqMktData(tickerId, usTechStk(ticker), "", False, False, [])

? ? ? ? time.sleep(5) #Add some breathing room for testing

? ? ? ??

threads = []

for ticker in tickers:

? ? tickercount += 1

? ? ? ??

for _ in range(tickercount):

? ? t = threading.Thread(target=dataReturn, args=(app,))

? ? t.start()

?

? ? threads.append(t)
 

Hi Ray,
No ouch from me I hope. I thought your reply was very excellent and explained a lot about threading. I also think your advice to not use threading is sound for this current case of getting quotes.
I think I also might have failed to understand the original problem, so some clarification from Patrick might help.

Hi Patrick,
When I look at the code you posted (and if I'm seeing correctly), I now see something that seems problematic. Near the bottom, you have a loop that is spawning 3 threads (i.e., apparently one for each ticker symbol based on the count of tickers).?
Each thread, however, is targeted to dataReturn which is making 3 reqMktData requests, one for each of the tickers. So, the end result is 9 reqMktData requests (i.e., 3 from each thread).
But, (and here's where it gets fun), since tickerId is global, and each thread is bumping it, and there are no locks around the bump, you have a classic race condition that will likely lead to chaos.
So, you could be seeing a case where one thread tries to bump the tickerId for the next reqMktData request with the next symbol, but another thread was in the process of bumping it for a different symbol, and the end result could be that the same tickerId value
is assigned to two different symbols. Processing from this point is unpredictable.

I agree with Ray that threads are probably not needed for your case, but just for fun, here's your code with some adjustments that might get this to work:


----------------------------------------------------------------------

tickers = ["JG", "AAPL", "FB"]

def dataReturn(app, tickerId):

??? app.reqMktData(tickerId, usTechStk(tickers[tickerId]), "", False, False, [])

? ? time.sleep(5) #Add some breathing room for testing


threads = []

for tickerId in range(1, len(tickers)+1):

? ? t = threading.Thread(target=dataReturn, args=(app, tickerId))

? ? t.start()

? ? threads.append(t)

-----------------------------------------------------------------------------

Cheers,
Scott


? ? ? ?
 

You guys are rock stars, all of you guys! I'm gunna sit down and read through everything here a second time, and also test the code snip from Scott. I really really appreciate everyone trying to help me out with this problem! Seriously, I am very grateful! I'll let you know how things go!?
 

So leaving the +1 in:

for tickerId in range(1, len(tickers)+1):?

was giving me an IndexError: list index out of range

So I removed the +1 and this is what I get. I get 3 threads running. But I'm still only getting prints from the last ticker FB

I attached an example screenshot of my Spyder screen. I tried to just attach the photo, but it kept making the photo which was 1500 x 900 way too small to read once it uploaded. And when I would adjust the already uploaded photo to 1500 x 900 it stretched the small photo and made it blurry... lol... and also here is a the code from my ticktype

------------------------

def tickPrice(self, reqId, tickType, price, attrib):
? ? ? ? if tickType == 1 and reqId == tickerId:
? ? ? ? ? ? self.bidprice = {"ReqId": reqId, "TickType": tickType, "Price": price, "Attrib": attrib}
? ? ? ? ? ? print("The current bid price is: ", price)
? ? ? ? ? ??
? ? ? ? elif tickType == 2 and reqId == tickerId:
? ? ? ? ? ? self.askprice = {"ReqId": reqId, "TickType": tickType, "Price": price, "Attrib": attrib}
? ? ? ? ? ? print("The current ask price is: ", price)
? ? ? ? ? ??
? ? ? ? elif tickType == 4 and reqId == tickerId:
? ? ? ? ? ? self.lastprice = {"ReqId": reqId, "TickType": tickType, "Price": price, "Attrib": attrib}
? ? ? ? ? ? print("The current last price is: ", price)? ? ? ? ? ??
? ? ? ? ? ??
? ? ? ? ? ? '''NOTES: BID Price = 1, ASK Price = 2, Last Price = 4'''

-----------------

example.jpg

 

Hi Patrick,
Ooops, my bad with the tickerId - I was trying to get the first tickerId to be 1 and to also use it as the index to the tickers list, but I failed to subtract 1 from the tickerId when using it to index the zero based tickers list.
In dataReturn, I should have had: app.reqMktData(tickerId, usTechStk(tickers[tickerId-1]), "", False, False, [])

What you have now will mean the first reqId will be zero - maybe that's OK, but I'm not sure.

But, here's the thing: In tickPrice, I see code that references tickerId. Which tickerId is that? It's not the local tickerId given to the thread. Seems it could be the tickerId used in the loop that spawns the threads.
This tickerId will now be 2 and will not change.
So, in tickPrice, you will be checking whether the reqId == 2 and printing the prices only for FB since that is the tickerId used when the reqMktDate request was made for FB.?

Just for fun, try the following code. I adjusted my original code to subtract 1 from the tickerId before using it as the index. I also modified the tickPrice method to NOT check the reqId against the tickerId.
Note that I also changed the print statement to an "f" style formatting string (a lower case f goes just before the string and braces contain the values) to print the symbol with the price to see whether we will now get all three symbols reporting.
I used reqId-1 as the index to tickers since the incoming reqId should be 1,2, or 3, so subtracting 1 will make it 0,1, or 2 to correctly index the tickers.?

--------------------------------------------------------------------------------

tickers = ["JG", "AAPL", "FB"]

def dataReturn(app, tickerId):

??? app.reqMktData(tickerId, usTechStk(tickers[tickerId-1]), "", False, False, [])?? # subtract 1 from tickerId to get correct tickers symbol

? ? time.sleep(5) #Add some breathing room for testing


threads = []

for tickerId in range(1, len(tickers)+1):? # start tickerId at 1

? ? t = threading.Thread(target=dataReturn, args=(app, tickerId))

? ? t.start()

? ? threads.append(t)

def tickPrice(self, reqId, tickType, price, attrib):
? ? ? ? if tickType == 1:
? ? ? ? ? ? self.bidprice = {"ReqId": reqId, "TickType": tickType, "Price": price, "Attrib": attrib}
? ? ? ? ? ? print(f"The current bid price for {tickers[reqId-1]} is: {price}")
? ? ? ? ? ??
? ? ? ? elif tickType == 2:
? ? ? ? ? ? self.askprice = {"ReqId": reqId, "TickType": tickType, "Price": price, "Attrib": attrib}
? ? ? ? ? ? print(f"The current ask price for {tickers[reqId-1]} is: {price}")
? ? ? ? ? ??
? ? ? ? elif tickType == 4:
? ? ? ? ? ? self.lastprice = {"ReqId": reqId, "TickType": tickType, "Price": price, "Attrib": attrib}
? ? ? ? ? ? print(f"The current last price for {tickers[reqId-1]} is: {price}")??

--------------------------------------------------------------------------------------------------

Cheers,
Scott
 

The code worked Scott!

Once I get a little breather from the markets today, I'll sit down and dig through the code so I can mentally understand how it all went together. This is a good lesson right here! This program could turn out pretty neat for my buddy, its going to be a fun project AND should be a good learning project for me!?


?
 

Excellent Patrick - I'm happy it's working for you now!?? ?

1 - 18 of 18