question

Upvotes
Accepted
chinelo.okoli avatar image
1 ●0 ●1 ●3

Inconsistent result from DSS TRTH v2

Hi there,@manigandan.r

When i request the extraction result from a Time and Sales report schedule - i get different results using exactly the same request.

I see the file size is exactly the same but my call to response.getEntity().getContent() appears to return inconsistent results. E.g. the last 2 calls i made to the API, T&S returned 69937 vs 101031 lines.

This is how i set up my client:

private CloseableHttpClient httpclient = HttpClientBuilder.create().build();

.....

String urlGet = this.url + "/Extractions/ReportExtractions('"+reportExtractionId+"')/Files";

HttpGet request = new HttpGet(urlGet);

request.addHeader("Authorization", "Token "+this.sessionToken);

HttpResponse response = this.httpclient.execute(request);

BufferedReader rd = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));

StringBuffer result = new StringBuffer();

String line = "";

while ((line = rd.readLine()) != null) { result.append(line); }

JSONObject jsonGetResponse = new JSONObject(result.toString());

JSONArray valueJArray = jsonGetResponse.getJSONArray("value");

String ExtractedFileId = ""; String ExtractedFileName = ""; String FileType = ""; boolean success = false;

for (int i = 0; i < valueJArray.length(); i++)

{

ExtractedFileId = valueJArray.getJSONObject(i).getString("ExtractedFileId");

......

//FileType="Full"

urlGet = this.url + "/Extractions/ExtractedFiles('"+ExtractedFileId+"')/$value";

request = new HttpGet(urlGet);

request.addHeader("Authorization", "Token "+this.sessionToken);

response = this.httpclient.execute(request);

HttpEntity entity = response.getEntity();

rd = new BufferedReader(new InputStreamReader(entity.getContent()));

......

while ((line = rd.readLine()) != null) {...}

......

}

The same code works fine for Depth extraction which also has a gzipped file output.

Am i missing something here please?

Thanks

tick-history-rest-api
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

3 Answers

Upvotes
Accepted
Christiaan Meihsl avatar image
13.7k ●26 ●8 ●12

@chinelo.okoli

Sorry this took some time, but it was a complex nut to crack. The support team found a solution, kudos go to them.

The key problem is the merged GZip files cause issues for the default decompression methods. For Java, the Apache Commons Compress https://commons.apache.org/proper/commons-compress/index.html library seems like a safe option. However, it needs to be implemented correctly.

GZIPStream from the java.util.zip package will work if it is done on the local machine, but may fail if used while reading directly from the HTTP Stream.

Key factors for Apache Commons Compress implementation:

Disable Content Compression on the HTTP client

private CloseableHttpClienthttpclient = HttpClients.create().disableContentCompression().build();

Set decompressConcatenated = true when initializing the stream

GzipCompressorInputStream gis =new GzipCompressorInputStream(
myURLConnection.getInputStream(), true);

If these are set, you can use In-Stream Decompression and process the stream while downloading the file.

In a nutshell, here are the code details:

    Add the Apache Commons Compress library:

    import
org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream;

    Use this setting when declaring the httpclient:

    private CloseableHttpClient httpclient = HttpClientBuilder.create().disableContentCompression().build();

    Full code of a method to extract and display on the fly is attached.solutioncode.txt.

    I will update all our Java samples, in the next version they will use this method.


    solutioncode.txt (1.6 KiB)
    icon clock
    10 |1500

    Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

    Christiaan Meihsl avatar image Christiaan Meihsl

    All the TRTH REST API Java samples have been checked, and updated where required to implement the correction. The new package, which also includes new samples and enhanced functionality, is available for download.

    Upvotes
    Christiaan Meihsl avatar image
    13.7k ●26 ●8 ●12

    @chinelo.okoli, the code snippets you posted look ok.

    Could you share your entire code so I could test it ? Could you also share your report template and instrument list ? If you do not want to post these on the forum you can also send them to me directly.

    Added notes:

    Our sample code DSS2ImmediateScheduleTicksTRTHClient2 (available under the downloads tab of TRTH REST) was tested for extractions up to 14MB gziped (99MB unzipped) without issue. Could you try that one with your template and instrument list to see if it exhibits the same issue ?

    If you are using a pre-defined instrument list and report template on the server, I can send you a variant of DSS2ImmediateScheduleTicksTRTHClient2 that would use them which you could use for testing.

    When you get 2 files of different size, what does the last line of data in the file look like ? Is it truncated ?

    icon clock
    10 |1500

    Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

    Upvotes
    chinelo.okoli avatar image
    1 ●0 ●1 ●3

    Thanks a lot Christiaan

    icon clock
    10 |1500

    Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

    Click below to post an Idea Post Idea