I am downloading 18 years of data from Tick History Raw using REST API but sometimes i unzip the zipGenius file, it is not able to extract the complete data into the csv file due to data overflow issue into csv.
Want to check if the zipped file contains the complete data and this is just the issue of csv capacity to have that much number of data rows? If yes, any other way to handle this situation?
Thanks for the response. I tried to use the suggested code to check the count of lines in the Gzip file. It is coming out to be more than 1.3 million but when i extract this file using ZipGenius, it extracts data till somewhere around 1 million rows in csv (precisely 1048576 rows in csv). I think it is due to the CSV limitation of having rows more than this. Isn't it?
If yes, any other way to get the complete data into excel?
CSV files have no limit of rows you can add to them.
Excel won't hold more that 1 million lines of data if you import a CSV file having more lines. The precise limit for Excel (versions >= 2007) is 2 to the power of 20 = 1'048'576 lines, exactly what you observe, so ZipGenius might have the same limit.
I suggest you try to extract the data using a different utility.
@pj4, I see you are using an extract of our TRTH_OnDemand_IntradayBars Python sample, which you have modified. That code has been tested on downloads of varying sizes (<100kB - > 100MB) without issues.
Considering you are downloading 18 years of data it might be a fairly large data set, though you do not mention how many instruments are in the request, nor the size of your gzip file.
Most Tick History reports deliver output as a gzip file. If a report is large, it delivers its output as several smaller gzip files concatenated into a single large gzip file. For more info on this topic, see this advisory.
I do not know the limits of zipGenius, but maybe it cannot handle concatenated Gzips correctly, or maybe it just has a limit on the file size ?
I suggest you try a different tool to open the Gzip and extract the CSV.
You could also use code to print out the first lines, and count all the lines as a sanity check, after downloading the file:
count = 0
maxPrintLines = 10
with gzip.open(fileName, 'rb') as fd:
for line in fd:
#Do something with the data:
count += 1
if count <= maxPrintLines:
dataLine = line.decode("utf-8")
print (dataLine)
fd.close()
Instead of counting them, you could also save them to a CSV file.
