¿ªÔÆÌåÓý

ctrl + shift + ? for shortcuts
© 2025 Groups.io

Performance optimization when exporting data to excel


 




? I have a stock scanner project where I am working with TWS C# API to get data based on certain price parameters.

the data gets callbacked in several methods which I then am using with EPPlus to open and update my excel file.

the data comes back in the following order

1st loop (request stock symbols from API) using?reqScannerSubscription


for each symbol we have another loop to get contract details from API, (2nd loop) using reqcontractdetails

for each contract we request market data from API (3rd loop) using reqMktdata

for each market data we store as variable (4th loop) The callbacks from tickprice ticksize etc...
each method is a public virtual void method

for loop 2-4 we are opening and editing an excel file with epplus


? ? using (var package = new ExcelPackage(pathtofile))
? ? {
? ? ? ? // edit excel file with the data callback from API
? ?
? ? ? ? package.Save();
? ? }
Instead of opening and saving the excel file in each method, which costs about 5 seconds? for each iteration?

1) I could instead pass the variables to the last method in a array and then do it there.
2) I could use `LoadFromCollection` of Epplus instead of calling each cell, currently I am calling each cell like so

? ? worksheet.Cells[row, 1].Value = mycelldata; where row is sheet.Dimension.End.Row + 1
3) I could use sql database instead of using excel, (keep in mind I don't plan on having large databases (less then 50,000 entries)

has anyone had any experience on my 3 previous methods or would like to suggest something new to speed up performance??


 

Well, weight in the fact that I am not a fan of Excel for handling fast pace data. (and here you deal with 'ticks' which is rated 2nd fastest data set of the API) Maybe other members have better experience.
Typically what seems an easy start may end up pretty painful, spending precious hours tweaking and 'workarounding' Excel limitations that will eventually fail to your expectations.

IMHO and without understanding if your purpose is to get an Excel spreasheet to look at, or to use Excel to compute and takje decision. Anyway you better organize "rendez-vous" in your code and get ride of "loop".
In short: you need a multi-thread method. yours seems single threaded. C# have good provision for it.

Assuming you never stop the 4th loop. Look to me that you need a 5th 'loop' (a thread in real), this is probably the most important for you within your current architecture, and it must be in a separate thread, this one only handle saving data.
Using SQL could be a simple solution as this will allow you to save faster (maybe) and more important collecting data from a central repository that by itself have all provision to handle concurrency of data access. This would preserve most of your existing code.

However depending upon how serious is your project you should use more of an appropriate language like Java or C# (in full) or C++ , it's not so hard to get some fundamental and it pay.
The API allows you to start any request you mention asynchronously and ASAP, Scanner answers are pretty slow, can be done at this moment. Be careful with contractdetails you may hit many pacing limitations.
IB have a pretty efficient and fast API, you can't keep up using Excel as the orchestrator.

Seems you use C#, keep the effort, if you don't know how to do multi-threading, start learning how to launch a thread, then either connect to a MySQL or learn how to share data between thread using a mutex, then ... many many details but later.