I, like many of you I am sure, have created a data downloader. I came across a problem recently with my code which has been running well for ages (years). The difference this time is I am trying to download quite old data. Let me explain.
1) I have no problem to download the last 10 years of daily S&P500 stock (503 stocks currently) price data. Works fine, no hang-ups except for a couple of stocks (VICI & LIN) which IB have admitted to a problem and need special handling. (Note: I always take the minimum period between 10 years or the head timestamp)
2) Recently I am trying to run some data analysis on some longer history, so I set my "END DATE" to 20141231-115959 US/EASTERN and started a similar download to #1 above asking for 10 years history. This time I get to AXP which happens to be my 98th API call and TWS returns nothing. No error in the API log, just nothing. I removed AXP and tried again but the same thing happened with the 98th stock (this time AZO). After a 2min time-out waiting for data or a response, the next API call to TWS gets no reply again (this time happens to be a reqHeadTimestamp() call). If I move AXP to the head of my list, the AXP data downloads fine so it is not a data issue with that stock.
My code is very conservative and I download data in serial so there are no concurrent calls to reqMktData() however I suspect this is a form of pacing, but is it normal to get no response? I literally have to disconnect and reconnect my app to get things moving again.
Has anyone seen a problem on older data vs newer data like this?
Perhaps rather than trying to do this in 2 steps I could just ask for 20 years daily history.
Can anyone offer me pointers to getting older data out of TWS?
