For a deeper look into our DataScope Select SOAP API, look into:

Overview |  Quickstart |  Documentation |  Downloads |  Tutorials

question

Upvotes
Accepted
rahul.deshmukh avatar image
41 ●3 ●4 ●9

full holdings for a fund and ETF DSS Rest API request generates Http error code : 408

Since 01.02.2021 i am getting below error:

java.io.IOException: Server returned HTTP response code: 408 for URL: https://hosted.datascopeapi.reuters.com/RestApi/v1/Extractions/ExtractWithNotes


I am using below DSS Rest API refinitiv request for full holdings for a fund and ETF in my java code:


"ExtractionRequest": {
            "@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.FundAllocationExtractionRequest",
            "ContentFieldNames": [
              "Security Description",
              "Allocation Asset Type",
              "Allocation Date",
              "Allocation Item",     
              "Allocation Percentage",      
              "Allocation ISIN",      
              "Market Value Currency",
              "Market Value Held"     
            ],
            "IdentifierList": {
              "@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",
              "InstrumentIdentifiers": [%s],
              "ValidationOptions": null,
              "UseUserPreferencesForValidationOptions": false
            },
            "Condition": {
              "FundAllocationTypes": [
                "Asset",
                "Currency",
                "FullHoldings",
                "IndustrySector",
                "InvestmentCountry",
                "TopTenHoldings"
              ]
            }
          }
        }


Below is connection request to url:

//Connection to the URL string https://hosted.datascopeapi.reuters.com/RestApi/v1/Extractions/ExtractWithNotes
    private static HttpURLConnection connection(String urlSpec) {
        HttpURLConnection connection = new URL(urlSpec).openConnection() as HttpURLConnection

        connection.setRequestProperty('Accept', 'application/json')

        connection.setRequestMethod("POST")
        connection.setRequestProperty("Content-Type", "application/json; utf-8")
        connection.setDoOutput(true)
        connection
    }


We are using the same API request since December 2020 but did not face any issue. We are running this process request everyday at 5am CET in the morning. Now if we try to run the process again at 1pm CET then it runs without any issue.

This is very critical issue for us.

dss-rest-apidatascope-selectdssfundsetferror-408
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

2 Answers

Upvotes
Accepted
zoya faberov avatar image
23k ●22 ●9 ●14

Hello @rahul.deshmukh,

My understanding from reading the question is that the code is running successfully at some times, but not at other. And this is a repeatable behavior, has happened more then once.

Therefore, we have to assume there is no issue with code.

A much likelier cause is connectivity or service availability issue. Especially, as 408 error is from timeout family of errors.

This forum is targeted at general questions and discussions targeted at Refintiiv API usage, the majority of the forum members are Refinitiv API developers, the moderators of the forums are Refinitiv API experts.

Datascope product support can help you, as a customer, to investigate this type of issue, and especially for a critical issue, in the future, would suggest raising this type of issue directly with Refinitiv Helpdesk Online -> Datascope.

At this time, have raised on your behalf, case # is 09586128 , please expect direct communication by email from DSS support team.

I would suggest investigating in parallel, with your local network admin/group, if any maintenance or network connectivity outages have coincided with your failed submission times- if you find out this is the case - please let us know.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

rahul.deshmukh avatar image rahul.deshmukh

I already confirmed from our side and we dont have any proxy or network connection issues. If you run the same request again different time it works then why it failed during sometimes ?

Upvotes
Gurpreet avatar image
13k ●32 ●12 ●18

Hi @rahul.deshmukh,

HTTP 408 means request timeout - and that is what you are experiencing when your request works some times and not other. This could be happening due to the additional load on the DSS server during that time, or due to network contention on your side. Either way, with a long running extraction like yours, it is never a good idea to use synchronous request.

I would recommend that you try asynchronous request and make your application more robust to handle the error conditions. You can see the example of async in the Java and C# examples here. Add a HTTP header:

"Prefer", "respond-async, wait=60"

and keep checking your response for HTTP 202, until you get a HTTP 200.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

rahul.deshmukh avatar image rahul.deshmukh

please check my previous question in forum https://community.developers.refinitiv.com/questions/67550/full-holdings-for-a-fund-and-etf-dss-rest-api-requ.html?childToView=68146#comment-68146 where it has been mentioned that i should remove this line from http request:

connection.setRequestProperty('Prefer', 'respond-async')
Gurpreet avatar image Gurpreet rahul.deshmukh

In that answer, I specifically mention:

Your application might have to wait a very long time for data, and intermediate network components like routers/proxies might drop your connection.

and this is what is happening now. I would strongly urge you to see how the DSS samples have implemented the extraction, and model your application around it.

rahul.deshmukh avatar image rahul.deshmukh

Will be really helpful here if you can help here to resolve this issue. What changes i have to do in http request ? I confirmed from our side and there is no proxy or network connection issues.

zoya faberov avatar image zoya faberov

Hello @rahul.deshmukh,

We can try and suggest, helping is our goal. However, many environment-specific issues do not yield themselves to resolution within the framework of the discussion forums.

I second @Gurpreet on the approach. I would try, in addition:

1. Include additional code, to resubmit the request on status 408 and document the resubmit, and consequently, review "the pattern". Once the status becomes 202 or 200, if it is 202, would keep checking status periodically (as shown in our examples), till it becomes 200, once the status is 200 would retrieve the result.

2. Consider, how large is your instrument list? How long does your request take, on average, at the time when it completes? Fund allocation request limit is 600 at this time, according to DSS Extraction Limits, if you are coming close to the limit, I would, as well as verifying to request async, as suggested by @Gurpreet, test dividing the request into several sub-lists, to see if this allows your request not to timeout at the times of high service load. Just a guess here, are any of the identifiers Chain RICs, and being expanded into many RICs as part of the request - then would count how many RICs in total, when expanded, and approach accordingly.

rahul.deshmukh avatar image rahul.deshmukh

Hello @zoya.farberov,

My instrument list is about 3000. Normally it takes around 50 minutes to complete all the request. I am processing 100 instrument at a time which takes around 2 minutes. The failure occur when it start to process request . For example since yesterday it waits for 20 minutes around and throws an error 408.


My concern is how do i test with async and sync method when i said the request does not fail during next run at different time ? Why it started failing for example after 2 months and only during specific time at 5am CET ? Why doesnt it fail till before yesterday ?

zoya faberov avatar image zoya faberov rahul.deshmukh

Hello @rahul.deshmukh,

If the error has first manifested on 01.02.2021 5AM CET, does it mean you have seen it twice? Or only once? From Feb 1st or from Jan 2nd?

Are you scheduling the job via cron/task scheduler? Would it make sense to schedule a small, no quota request call, such as search, at the same time, and see if it also fails with a timeout. That would allow to narrow down the possible causes a lot.

You can search the history of service alerts on My Refinitiv looking for your instrument set, on these dates, and to try to verify if there was any outage that was impacting:

I include resolved, otherwise you will not be able to discover what has happened unless it is still happening.

servicealerts.gif (105.5 KiB)
rahul.deshmukh avatar image rahul.deshmukh

@zoya.farberov Basically we are using prod and simu server and scheduling control-m job at 5am and 5.15am for both funds and etf. We are using the same API with different instrument list. Its difficult to verify actually with Histrory Service alert for which instrument there is outage and whether this is impacting. Today also we faced the same issue at 5am and 5.15am CET both the job failed with 408. And then we try to run the job at 7.30am again it works fine without any perfromance issue.

Is it possible for you to send me the ISIN list( we are using Identifier type as ISIN) which got affected from 01.02.2021 till 03.02.2021 due to outage ? Then i can verify easily if we are using those ISINs in our API request or not ?

zoya faberov avatar image zoya faberov rahul.deshmukh

Hello @rahul.deshmukh,

There is a working approach on this, if you wish. As a customer, you can open a case directly on Refinitiv Helpdesk Online -> DSS -> content, include your list of instruments, dates, and get a confirmation from Refintiiv content experts whether any of these instruments were affected on these dates.

However, as increasing the timeout as suggested by @Gurpreet was effective to avoid the timeout and allow the requests to be accepted for processing (status 202) you seem to have the solution.

Once 202 status is received, your request is accepted for processing successfully, this is different from 208. In this case the request will be completed by the service, resulting in status check = 200 and job being ready. Please refer to our examples (see above from @Gurpreet) on how to check status by ExtractionId and once it becomes 200, retrieve the job by extractionId.

Show more comments
Click below to post an Idea Post Idea