@vaughan.rees, as Veerapath states, this looks like an issue with the retrieval of concatenated gzip files, which the TRTH servers generate for large data sets.

Depending on how you retrieve the data, how you code that retrieval, and which library you use, you can run into a situation where not all data is downloaded, without generating an error. The larger the data set, the higher the probability you will run into this issue. Decompressing data on the fly makes things worse, that is why we strongly recommend you first download and save the compressed data, and only open it up (from local hard disk) afterwards, for whatever treatment you want to apply to it.

Pages 4 and 5 of the advisory Veerapath mentioned give some information on how to avoid this issue when coding with C#. For further information, please refer to the .Net SDK Tutorial 5. Its downloadable C# code sample contains code that illustrates various ways of downloading and saving your extraction data from TRTH. It also shows how to download the data from AWS (the Amazon Web Service cloud), which is faster than the standard download. On the topic of AWS downloads, see this advisory.

In a nutshell, the recommended method is to download and save the compressed data, from AWS, without decompressing on the fly.

If this answer does not suffice, please post the code you use to retrieve and save the data, that could be useful for us to help you more.

Hi @vaughan.rees

It looks like the incomplete output issue, where some HTTP clients do not fully support concatenated gzip files. Please check this documentation ADVISORY: AVOID INCOMPLETE OUTPUT - DOWNLOAD THEN DECOMPRESS for more information.