Hello,
Trying to source market depth data on FX pairs, USD JPY in the below case.
In the below code,?
self.series[reqId]?is just a python list that I append new ticks to.
def updateMktDepthL2(self, reqId, position, marketMaker, operation, side, price, size, isSmartDepth):
self.series[reqId].append((time(), side, position, price, size))
Then separately I have a running thread in charge of saving once in a while to a parquet file and resetting the list.
...
if len(app.series[reqId]) >= number_of_rows_threshold:
print('saving')
try:
df = pd.DataFrame(app.series[reqId], columns=['time', 'bid', 'position', 'price', 'quantity'])
app.series[reqId] = list()
except ValueError as e:
print('Value Error', e)
...
df.to_parquet(parquet_file_name, compression='gzip')
The above code works fine most of the time, however once in a while, I get an exception:
Length of values (112071) does not match length of index (112105)Trying to debug the Exception, and putting each column into a separate frame I get the below row count per column:
- time 2151751
- bid 2151825
- position 2151830
- price 2151876
- quantity 2151876
How is that even possible?
Note the numbers are different between the exception?112071 and investigation?2151751+,
I believe because data kept arriving between the exception catch and during the debugging I performed.
