question

Upvotes
Accepted
16 1 2 5

Multiple Rics On demand extraction returns only one Ric

Hello,

When running a multiple Rics On demand extraction, it returns only the first Ric.

The code I am using is the below code inspired from the DSS api example

What am I doing wrong?

Thank you for your help on this.

ExtractionsContext.Options.AutomaticDecompression = true; //Decompress gzip to plain text

// UTC start and endDate

var requestDate = new DateTime(2017, 7, 27);

var utcStartDate = requestDate.AddHours(8);

var utcEndDate = requestDate.AddHours(16);

var startDate = new DateTimeOffset(utcStartDate);

var endDate = new DateTimeOffset(utcEndDate);

var condition = new TickHistoryTimeAndSalesCondition { ReportDateRangeType = ReportDateRangeType.Range,

QueryStartDate = startDate,

QueryEndDate = endDate,

ApplyCorrectionsAndCancellations = false,

ExtractBy = TickHistoryExtractByMode.Ric,

MessageTimeStampIn = TickHistoryTimeOptions.GmtUtc,

SortBy = TickHistorySort.SingleByRic, DisplaySourceRIC = true };

var rics = new string[] { "AAL.L", "MMM.N" };

var instrumentIdentifiers = rics.Select(x => InstrumentIdentifier.Create(IdentifierType.Ric, x)).ToArray();

var indentifierList = new InstrumentIdentifierList { InstrumentIdentifiers = instrumentIdentifiers };

var request = new TickHistoryTimeAndSalesExtractionRequest { Condition = condition, ContentFieldNames = FIELDS, IdentifierList = indentifierList };

Task<RawExtractionResult> resultTask;

RawExtractionResult result;

try

{

resultTask = ExtractionsContext.ExtractRawAsync(request);

result = await resultTask;

}

catch (Exception ex) {

return; }

dss-rest-apitick-history-rest-api
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

Upvote
Accepted
13.7k 26 8 12

@sd, I just tried modifying our latest .Net sample 5 (it was updated on 28 June, along with the related tutorial), available from the downloads tab, to make the same request you make. It delivers data for several instruments (I tried with 5 RICs: ALVG.DE, AAL.L, IBM.N, MMM.N,0001.HK).

I attached the modified Program.cs file that requests the same data as you programcs.txt (modified the extension). Note that it reads the RICs from a file.

To install and run the code refer to .Net tutorial 1.

Hope this helps.


programcs.txt (22.8 KiB)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

Upvote
23k 22 9 14

Hello @sd,

Please try with automaticDecompression = false,

save on disk and unzip with 7zip?

Please refer to this question for the complete answer:

Multi file save on disk

Let us know if this helped?

Thanks,

-AHS

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

Upvotes
16 1 2 5

Thank you very much. It seems to work with this set up. I will keep you posted in case of issue

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

Ok, glad it works for you now.

Upvotes
1 0 1 4

Hello,

In my case, I tried with the RICs "FEXDZ8" and "FEXDZ9", on August 14 for example. When I submit a Time and Sales request (trades and quotes) with both RICs, I get all the data for the first instrument (FEXDZ8) and just a few lines from the second one (FEXDZ9). On the other hand, if I submit the same request with just FEXDZ9, I get the full response. Have you ever experienced this issue? I tried both of the proposed solutions and none of them seems to work.

Thank you for your help.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

You should try to retrieve the file as a raw gzip file instead of decompressed data (csv).

If you are using C#, please change ExtractionsContext.Options.AutomaticDecompression to false and then change the codes that your write the result to file. For more information, please refer to this question.

Hi,

Thank you for your reply. I already tried to follow this method, but the generated ".gzip" file is apparently not an archive (see the attached screenshot). Am I missing something?

capture.png (27.4 KiB)

Actually, it seems that the file I download is a CSV file regardless of the AutomaticDecompression settings.

using (var response = extractionsContext.RawExtractionResultOperations.GetReadStream(result))
            using (var fileStream = File.Create("savedExtraction.gzip"))
                response.Stream.CopyTo(fileStream);

In my case, the code above just writes the CSV data in a file names "savedExtraction.gzip". If I rename it as "savedExtraction.csv" then I am able to open it as a normal CSV file.

Please verify that AutomaticDecompression is set to false at the beginning before calling ExtractRawAsync.

ExtractionsContext.Options.AutomaticDecompression = false; 

If the application is still get the csv file, please share the full code. Therefore, we can test the code.

Show more comments
Upvotes
1 0 1 4

I am using 11.1.527 as well.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

Are you using proxy?

Please uncomment Status.Notify codes in RequestRawTransactionsDataAndSaveToGzipFile() function then set AutomaticDecompression to false.

ExtractionsContext.Options.AutomaticDecompression = false; 

Run the Tick History: Request Raw Transactions Data – Save to File example. After receiving the response, please verify the HTTP Response from STEP# 2. You should see the binary data with 1F 8B 08 xxx.

response.png (46.9 KiB)
Upvotes
1 0 1 4

I am not using a proxy. The response that I get is different from yours:


icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

In the response, I found Via header.

Via: veg14pxmzdd7

From this page:

The Via general header is added by proxies, both forward and reverse proxies, and can appear in the request headers and the response headers. It is used for tracking message forwards, avoiding request loops, and identifying the protocol capabilities of senders along the request/response chain

I assume that this proxy may decompress the data and send the csv back to the application. Can you verify this with your network team?

Hi,

Sorry, I checked quickly through my browser and did not identify any proxy, but it seems that you are right about the fact that I am behind a proxy. This could be in fact the reason for which the gzipped data is decompressed. I checked during the weekend on another network and I got the same reply of yours.

Hi,

Given that the problem is most likely related to the proxy, as you mentioned, are you aware of any other workaround for being able to get all the requested data? For example, if I have a large request to do (e.g. several securities on several days) then with the version 1 of the SOAP API I was able to split my request into smaller jobs containing one securities per single day. I tried the same method via the REST API but it seems quite slow to perform a single request.

Alternatively, in version 1 there was the option to split the request by RIC and by day. Is it possible in version 2 too?

Upvotes
3 1 1 2

I have also the same problem: csv file disguised as a gzip file. It's annoying because I need to store zipped files on my server.


untitled.png (108.8 KiB)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

Upvotes
38.1k 71 35 53

@mantonello

The workaround should be setting "Range: bytes=0-" in the request's header. Then the application will get 206 PartialContent with binary data. I assume that the proxy will not decompress this content.

The code looks like:

ExtractionsContext.SetRequestHeaders(() => { return new System.Collections.Generic.Dictionary<string, string> { { "Range", "bytes=0-" }}; });
   using (var response = ExtractionsContext.RawExtractionResultOperations.GetReadStream(result, 0))          

The code sets the "Range" header before calling GetReadStream.

If you verify the HTTP, you will see:

However, the output file may be corrupted. The workaround is removing Status.Notify and Status.EndNotify from the code to get the complete file, as mentioned in the above comment.


partial.png (27.4 KiB)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

Hi,

Thank you for your reply. Unfortunately, it doesn't work as expected: the file is still decompressed in a csv one.

Click below to post an Idea Post Idea