Hello,
I made a multi RIC request (described below) :
{
"ExtractedFileId": "VjF8MHgwNWY2MmEyY2RhMGIyZjk2fA",
"ReportExtractionId": "2000000007224811",
"ScheduleId": "0x05f629f161ab2f76",
"FileType": "Full",
"ExtractedFileName": "_OnD_0x05f629f161ab2f76.csv.gz",
"LastWriteTimeUtc": "2017-11-24T15:05:28.000Z",
"ContentsExists": true,
"Size": 2329027
}
When downloading i get a file containing only one ric.
Could you advise please ?
@said.aitabaid-ext, can you please post the end point and body of your request, as well as the output data, and extraction notes if possible. They will help us understand the issue and deliver a response.
@said.aitabaid-ext, using AWS downloads might solve the issue, see this similar query.
@said.aitabaid-ext Have you tried AWS downloads yet? Can it solve this problem?
{
"ExtractionRequest" : {
"@odata.type" : "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.TickHistoryTimeAndSalesExtractionRequest",
"ContentFieldNames" : [ "Quote - Bid Price", "Quote - Bid Size", "Quote - Number of Buyers", "Quote - Ask Price", "Quote - Ask Size", "Quote - Number of Sellers", "Trade - Price", "Trade - Volume" ],
"IdentifierList" : {
"@odata.type" : "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",
"InstrumentIdentifiers" : [ {
"Identifier" : "EDF.PA",
"IdentifierType" : "Ric"
}, {
"Identifier" : "AIR.PA",
"IdentifierType" : "Ric"
}, {
"Identifier" : "ALCLS.PA",
"IdentifierType" : "Ric"
} ],
"ValidationOptions" : {
"AllowHistoricalInstruments" : true
},
"UseUserPreferencesForValidationOptions" : false
},
"Condition" : {
"MessageTimeStampIn" : "GmtUtc",
"ApplyCorrectionsAndCancellations" : true,
"ReportDateRangeType" : "Range",
"QueryStartDate" : "2017-11-15T23:00:00.000Z",
"QueryEndDate" : "2017-11-16T22:59:00.000Z",
"DisplaySourceRIC" : true,
"SortBy" : "SingleByRic",
"ExtractBy" : "Ric"
}
}
}
{"@odata.context":"https://hosted.datascopeapi.reuters.com/RestApi/v1/$metadata#RawExtractionResults/$entity","JobId":"0x05f7201dd5ab3036","Notes":["Extraction Services Version 11.2.37811 (51c11525fb7b), Built Nov 9 2017 14:30:01\nUser ID: 9013258\nExtraction ID: 2000000007804573\nSchedule: 0x05f7201dd5ab3036 (ID = 0x0000000000000000)\nInput List (3 items): (ID = 0x05f7201dd5ab3036) Created: 27-11-2017 15:45:34 Last Modified: 27-11-2017 15:45:34\nReport Template (8 fields): _OnD_0x05f7201dd5ab3036 (ID = 0x05f7201e2c6b3036) Created: 27-11-2017 15:43:40 Last Modified: 27-11-2017 15:43:40\nSchedule dispatched via message queue (0x05f7201dd5ab3036), Data source identifier (3912B9CF912C4B83AD83598C5E90B9D8)\nSchedule Time: 27-11-2017 15:43:43\nProcessing started at 27-11-2017 15:43:43\nProcessing completed successfully at 27-11-2017 15:45:34\nExtraction finished at 27-11-2017 14:45:34 UTC, with servers: tm04n01, TRTH (100.038 secs)\nInstrument <RIC,EDF.PA> expanded to 1 RIC: EDF.PA.\nInstrument <RIC,AIR.PA> expanded to 1 RIC: AIR.PA.\nInstrument <RIC,ALCLS.PA> expanded to 1 RIC: ALCLS.PA.\nQuota Message: INFO: Tick History Cash Quota Count Before Extraction: 4633; Instruments Approved for Extraction: 3; Tick History Cash Quota Count After Extraction: 4633, 61.6910785619174% of Limit; Tick History Cash Quota Limit: 7510\nManifest: #RIC,Domain,Start,End,Status,Count\nManifest: AIR.PA,Market Price,2017-11-16T03:07:08.554095130Z,2017-11-16T16:37:57.427417214Z,Active,198943\nManifest: ALCLS.PA,Market Price,2017-11-16T03:07:08.834758571Z,2017-11-16T16:50:55.413547815Z,Active,8609\nManifest: EDF.PA,Market Price,2017-11-16T06:15:00.311632845Z,2017-11-16T16:35:05.145556641Z,Active,63331\n"]}
This could be the problem mentioned in this advisory.
This advisory enables you to avoid a third-party decompression incompatibility that may be providing you with incomplete data, and may be doing so in a way that can be difficult to notice. This advisory applies to all Tick History REST API extractions that return compressed data using any of the following report template types: Tick History Time and Sales, Tick History Market Depth, Tick History Intraday Summaries, and Tick History Raw.
You need to disable decompression in the HTTP client while downloading. Therefore, the application will get a raw gzip file instead of CSV data. Then, the application needs to decompress the gzip file in order to get the CSV data.
With the above extraction, you should see 270884 rows of data including a header row.
Hello,
I followed the instruction in the advisory (below the code used for the call )
String urlGet = basePath + "/Extractions/RawExtractionResults('0x05f7201dd5ab3036')/$value";
CloseableHttpClient httpclient = HttpClients.custom()
.setRoutePlanner(routePlanner)
.setSSLSocketFactory(sslsf)
.disableContentCompression()
.build();
HttpGet getCall = new HttpGet(urlGet);
getCall.addHeader(HttpHeaders.AUTHORIZATION, "Token "+ sessionToken );
CloseableHttpResponse response = httpclient.execute(getCall);
try {
GzipCompressorInputStream gis = new GzipCompressorInputStream( response
.getEntity().getContent(), true);
/*InputStream entityInputStream = response
.getEntity().getContent();*/
IOUtils.copy(gis,
fileOutputStream);
gis.close();
fileOutputStream.close();
} finally {
response.close();
}
I'm getting the following :
java.io.IOException: Input is not in the .gz format
at org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream.init(GzipCompressorInputStream.java:162)
Could you advise please ?
Sincerly
Please try this code.
InputStream is = responseGetData.getEntity().getContent();
BufferedReader br = new BufferedReader(new InputStreamReader(new GzipCompressorInputStream(new BufferedInputStream(is), true)));
String line = "";
while ((line = br.readLine()) != null) {
System.out.println(line);
}
Hello,
I'm still getting the same issue : java.io.IOException: Input is not in the .gz format.
The contentEncoding attribute in the response wrapped entity is null.
Could you advise please ?
Regards
Due to an outage of the Thomson Reuters Developer Community email notification system from 2:30 PM CDT on November 29th until 9:00 AM CDT on November 30th we are posting to all questions updated during this timeframe in order to ensure all customers receive all relevant email updates. Please review this question, and its associated answers, in order to ensure you are aware of any possible updates that may have occurred during this outage. We apologize for the delay and any inconvenience this has caused.
Matthew Logterman, J.D.
Sr. Product Manager, Thomson Reuters Developer Community
@said.aitabaid-ex, are you using AWS downloads? I ask due to your comment stating: The contentEncoding attribute in the response wrapped entity is null. As explained in this article, Contrary to a non-AWS download, if you use AWS it does not return the Content-Encoding:gzip header in the response. Instead it sends this response header Content-Type:application/gzip.
That said, we have a Java sample, DSS2OnDemandTicksTRTH, available under the downloads tab, that downloads the result file and decompresses it. Although we recommend first saving the compressed data to file, and then reading and decompressing that, which the sample does, it also shows how to download and decompress on the fly (which seems to be what you want to do).
In the sample, see method awsRetrieveDataDecompressAndDisplayContents for an AWS download, or method retrieveDataDecompressAndDisplayContents for a non-AWS download. Note these 2 methods are different because AWS implies a redirection, which needs to be handled manually.
Hello,
We are not using aws downloads. Actually i used the
retrieveDataDecompressAndDisplayContents method from the
DSS2OnDemandTicksTRTH but still have the issue (
Input is not in the .gz format).
Could you please help ?
Regards
@said.aitabaid-ext, I shall try to reproduce your issue, and will come back to you soon.
I assume that your company's proxy may decompress the data and then send the csv data back to the application. It is similar to this answer.
You can verify this from a header in the HTTP response header that returns CSV data. If it contains the Via header, it means that the application is behind the proxy. The workaround is downloading a file from AWS.
@said.aitabaid-ext, I tried to reproduce your issue, but could not, I receive GZIP compressed data as expected, and can save it directly, or decompress it on the fly.
Attached (dss2ondemandtickstrth.zip) is the Java code I used, a modified version of DSS2OnDemandTicksTRTH using your instrument list, range and parameters). It generates the same request as what you posted. My extraction notes are very similar to yours. The result contains data for the 3 RICs.
If this sample does not help, then I think we'd need to look at your code. If you don't want to post it here we can take that action offline.
Investigation taken offline. Might be related to the code, libraries or proxy. Will post explanation / solution when found.
