question

Upvotes
Accepted
sanchezTJUK avatar image
7 ●2 ●2 ●2

Incomplete data on historic download API Datascope

When downloading historic data for certain futures contracts I'm receiving a 200 status code immediately but the message is empty. This only seems to be happening for the 3rd from front month of the futures contract. The product is LS Gas Oil TAS, the date is 17/12/19 (uk date format) and the RIC is LGOTH0. Another example is the same product on the 02/01/20 with the RIC of LGOTJ0. This is also happening for the LS Gas Oil Singapore product, an example being 01/04/20 with the RIC of LGOmmN0.

Any help would be great,


Thanks!

tick-history-rest-api
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

6 Answers

Upvotes
Accepted
zoya faberov avatar image
23k ●22 ●9 ●14

Hi @sanchezTJUK,

The URL remains the same:

{
                {protocol}}{
                {host}}{
                {api}}Extractions/ExtractWithNotes 

Please find detailed info on HistoricalReference request in REST API Tutorial 10: On Demand Histo Reference extraction tutorial.

Alternatively, you can run Historical Search request as in REST API Tutorial 13: Historical Search. tutorial

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

Upvotes
sanchezTJUK avatar image
7 ●2 ●2 ●2

Thanks for getting back to me!

The extraction request is below. We don't actually get any extraction notes back because the script errors with: 

r3Json = json.loads(r3.text.encode('ascii', 'ignore'))
NameError: name 'r3' is not defined

The extraction notes seem to come out of the r3Json variable.

I've not tried this extraction on the GUI but I'll give that a go now.



requestUrl='https://hosted.datascopeapi.reuters.com/RestApi/v1/Extractions/ExtractRaw'

requestHeaders={
    "Prefer":"respond-async",
    "Content-Type":"application/json",
    "Authorization": "token " + token
}

requestBody={
  "ExtractionRequest": {
    "@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.TickHistoryIntradaySummariesExtractionRequest",
    "ContentFieldNames": ["Quote - Bid Price","Quote - Bid Size","Quote - Ask Price","Quote - Ask Size"],
    "IdentifierList": {
      "@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",  
      "InstrumentIdentifiers": [{
        "Identifier": "LGOTJ0",
        "IdentifierType": "Ric"
      }],
        "UseUserPreferencesForValidationOptions": "true"
    },      
    "Condition": {
      "MessageTimeStampIn": "GmtUtc",
      "ReportDateRangeType": "Range",
      "QueryStartDate": "2020-01-02T00:00:00.000Z",
      "QueryEndDate": "2020-01-03T00:00:00.000Z",
      "SummaryInterval": "OneMinute",
      "TimebarPersistence":"true",
      "DisplaySourceRIC":"true"
    }
  }
}
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

Upvotes
zoya faberov avatar image
23k ●22 ●9 ●14

Hi @sanchezTJUK,

There are two things that may be helpful.

"r3 is not defined" is likely related to a python module missing. Please check what is r3 imported as, and if that module is installed.

Another thing to check, I just tested this request on it's own, "naked" and submitting it from Postman tool. It did not go through, the fields do not appear to be valid.

I checked with the list of valid fields for TickHistoryIntradaySummaries, these fields do not appear on the list.

When I replace the fields with valid ones for the request type, the request gest accepted ( status code 202)

{
    "ExtractionRequest": {
    "@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.TickHistoryIntradaySummariesExtractionRequest",
    "ContentFieldNames": [
         "Close Ask",
         "Close Bid"
        ],
    "IdentifierList": {
      "@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",  
      "InstrumentIdentifiers": [{
        "Identifier": "LGOTJ0",
        "IdentifierType": "Ric"
      }],
        "UseUserPreferencesForValidationOptions": "true"
    },      
    "Condition": {
      "MessageTimeStampIn": "GmtUtc",
      "ReportDateRangeType": "Range",
      "QueryStartDate": "2020-01-02T00:00:00.000Z",
      "QueryEndDate": "2020-01-03T00:00:00.000Z",
      "SummaryInterval": "OneMinute",
      "TimebarPersistence":"true",
      "DisplaySourceRIC":"true"
    }
  }
}

Does the above run for you?

You may find useful tutorial REST API Tutorial 6: On Demand intraday bars extraction, it goes over the approach to Intraday in detail, using Postman, and if you use Python, the approach would be consistent. Tutorial Programming without SDK discusses Python, and includes download link to working python examples.

You may wish, as a quicker approach:

1. Use Postman tool to fully test and tune TickHistoryIntradaySummariesExtractionRequest that you require.

2. Download the Python examples ( they are linked from tutorial and also in RTH-> Downloads -> RTH Python Code Samples), and then swap you required request tuned in step 1 into working python code, instead of the included request.

Hope this helps

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

sanchezTJUK avatar image sanchezTJUK

I'm really sorry, i got that wrong, the extraction request is from: 


"@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.TickHistoryTimeAndSalesExtractionRequest"


A lot of my code is defined under variables so I needed to try and recreate from an original.

Thanks for the advice on using postman, but this is an issue because the same code and requests work with Brent Crude products and with LS Gas Oil months 1 and 2...

Regarding r3, this is not a module that is imported. It is a variable that is assigned to part of the json message that is not being delivered.

zoya faberov avatar image zoya faberov sanchezTJUK

@sanchezTJUK,

Replacing the request type, I get back

"

"error": {

"message": "Malformed request payload: For the property name \"SummaryInterval\" in the JSON request the value could not be parsed successfully. Please check the casing or spelling of the property."

Without this condition clause, I am able to get results back.

...
#RIC,Alias Underlying RIC,Domain,Date-Time,GMT Offset,Type,Bid Price,Bid Size,Ask Price,Ask Size
LGOTJ0,,Market Price,2020-01-02T07:24:56.596063285Z,+0,Quote,-0.25,10,,
LGOTJ0,,Market Price,2020-01-02T07:25:02.700408234Z,+0,Quote,-0.25,10,0.25,10
LGOTJ0,,Market Price,2020-01-02T10:47:24.335364201Z,+0,Quote,-0.25,90,0.25,10
LGOTJ0,,Market Price,2020-01-02T10:47:25.611314961Z,+0,Quote,-0.25,90,0.25,90
LGOTJ0,,Market Price,2020-01-02T15:44:43.453059247Z,+0,Quote,0,137,0.25,90
LGOTJ0,,Market Price,2020-01-02T15:44:57.936016337Z,+0,Quote,0,137,0.25,105
LGOTJ0,,Market Price,2020-01-02T15:46:25.500950349Z,+0,Quote,0,137,0.25,125
LGOTJ0,,Market Price,2020-01-02T16:08:52.450662974Z,+0,Quote,0,12,0.25,125
LGOTJ0,,Market Price,2020-01-02T16:11:23.754852370Z,+0,Quote,0,12,0.25,105
LGOTJ0,,Market Price,2020-01-02T16:18:44.751242199Z,+0,Quote,0,12,0.25,90
LGOTJ0,,Market Price,2020-01-02T16:18:48.603454951Z,+0,Quote,0,16,0.25,90
LGOTJ0,,Market Price,2020-01-02T16:22:10.061362751Z,+0,Quote,0,16,0.25,10
...


Please confirm the exact request - glad to help test on my side?

Upvotes
sanchezTJUK avatar image
7 ●2 ●2 ●2


Thanks for being so patient Zoya,

I got a couple of bits wrong on the original extraction request I sent you. Here is a revised copy that I've double checked:

requestUrl='https://hosted.datascopeapi.reuters.com/RestApi/v1/Extractions/ExtractRaw'

requestHeaders={
    "Prefer":"respond-async",
    "Content-Type":"application/json",
    "Authorization": "token " + token
}

requestBody={
  "ExtractionRequest": {
    "@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.TickHistoryTimeAndSalesExtractionRequest",
    "ContentFieldNames": ["Quote - Bid Price","Quote - Bid Size","Quote - Ask Price","Quote - Ask Size"],
    "IdentifierList": {
      "@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",    
      "InstrumentIdentifiers": [{
        "Identifier": "LGOTJ9",
        "IdentifierType": "Ric"
      }],
        "UseUserPreferencesForValidationOptions": "true"
    },      
    "Condition": {
      "MessageTimeStampIn": "GmtUtc",
      "ApplyCorrectionsAndCancellations": "false",
      "ReportDateRangeType": "Range",
      "QueryStartDate": "2019-01-02T00:00:00.000Z",
      "QueryEndDate": "2019-01-03T00:00:00.000Z",
      "DisplaySourceRIC":"false"
    }
  }
}

Could you maybe try it at your end with LGOTG9 and LGOTH9 because I know these two both work at my end.


Thanks again!

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

Upvotes
zoya faberov avatar image
23k ●22 ●9 ●14

Hello @sanchezTJUK,

On my side, this request has resulted in notes

"

"Notes": [

"Extraction Services Version 14.5.42174 (21810bfe19a8), Built Feb 4 2021 20:25:13\nUser ID: 19945\nExtraction ID: 2000000226250284\nSchedule: 0x0771fae45477448a (ID = 0x0000000000000000)\nInput List (1 items): (ID = 0x0771fae45477448a) Created: 02/16/2021 16:41:12 Last Modified: 02/16/2021 16:41:12\nReport Template (4 fields): _OnD_0x0771fae45477448a (ID = 0x0771fae45497448a) Created: 02/16/2021 16:41:07 Last Modified: 02/16/2021 16:41:07\nSchedule dispatched via message queue (0x0771fae45477448a)\nSchedule Time: 02/16/2021 16:41:08\nProcessing started at 02/16/2021 16:41:08\nProcessing completed successfully at 02/16/2021 16:41:12\nExtraction finished at 02/16/2021 16:41:12 UTC, with servers: x03q13\nHistorical Instrument <RIC,LGOTJ9> expanded to 0 RICS.\nReport suppressed because there are no instruments\n"

]"

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

Upvotes
zoya faberov avatar image
23k ●22 ●9 ●14

@sanchezTJUK,

Therefore, at that time interval, the instrument appears not to have been valid.

Next, I have checked when the instrument was valid (it is an expired future), by running HistoricalReference extraction request:

{
    "ExtractionRequest": {
        "@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.HistoricalReferenceExtractionRequest",
        "ContentFieldNames": [
            "RIC",
            "Start date",
            "Change Date",
            "Expiration Date"
        ],
        "IdentifierList": {
            "@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",
            "InstrumentIdentifiers": [
                {
                    "Identifier": "LGOTJ9",
                    "IdentifierType": "Ric"
                }
            ],
            "ValidationOptions": {
                "AllowHistoricalInstruments": true
            },
            "UseUserPreferencesForValidationOptions": false
        },
        "Condition": {
            "StartDate": "2007-01-01T00:00:00.000Z",
            "EndDate": "2020-08-01T23:59:59.000Z"
        }
    }
}

Resulting in

{
    "@odata.context": "https://hosted.datascopeapi.reuters.com/RestApi/v1/$metadata#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.ExtractionResult",
    "Contents": [
        {
            "IdentifierType": "Ric",
            "Identifier": "LGOTJ9",
            "RIC": "LGOTJ9",
            "Start date": null,
            "Change Date": "2019-01-10",
            "Expiration Date": "2019-04-11"
        },
        {
            "IdentifierType": "Ric",
            "Identifier": "LGOTJ9",
            "RIC": "LGOTJ9^1",
            "Start date": null,
            "Change Date": "2019-04-15",
            "Expiration Date": "2019-04-11"
        }
    ],

I am not a content expert, but having read this, I would expect valid data 2019-01-10 - 2019-04-11

So I have next run:

{
    "ExtractionRequest": {
    "@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.TickHistoryTimeAndSalesExtractionRequest",
    "ContentFieldNames": ["Quote - Bid Price","Quote - Bid Size","Quote - Ask Price","Quote - Ask Size"],
    "IdentifierList": {
      "@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",    
       "ValidationOptions": {
            "AllowHistoricalInstruments": true
            },
            "UseUserPreferencesForValidationOptions": false,
      "InstrumentIdentifiers": [{
        "Identifier": "LGOTJ9",
        "IdentifierType": "Ric"
      }]
      },
    "Condition": {
      "MessageTimeStampIn": "GmtUtc",
      "ApplyCorrectionsAndCancellations": "false",
      "ReportDateRangeType": "Range",
      "QueryStartDate": "2019-01-02T00:00:00.000Z",
      "QueryEndDate": "2019-04-11T00:00:00.000Z",
      "DisplaySourceRIC":"false"
    }
  }
}

And that have resulted in :

#RIC,Domain,Date-Time,GMT Offset,Type,Bid Price,Bid Size,Ask Price,Ask Size
LGOTJ9,Market Price,2019-01-10T02:37:16.198936629Z,+0,Quote,-0.25,500,,
LGOTJ9,Market Price,2019-01-10T02:37:20.331555914Z,+0,Quote,-0.25,500,0.25,500
LGOTJ9,Market Price,2019-01-10T08:14:59.704940277Z,+0,Quote,-0.25,525,0.25,500
LGOTJ9,Market Price,2019-01-10T08:15:04.505137097Z,+0,Quote,-0.25,525,0.25,525
LGOTJ9,Market Price,2019-01-10T16:14:12.123438568Z,+0,Quote,-0.25,525,0.25,500
LGOTJ9,Market Price,2019-01-10T16:21:34.416871597Z,+0,Quote,-0.25,325,0.25,500
...
LGOTJ9,Market Price,2019-04-10T15:29:58.146792544Z,+1,Quote,-0.5,32,0.25,25
LGOTJ9,Market Price,2019-04-10T15:29:59.102828245Z,+1,Quote,-0.5,32,0.25,15
LGOTJ9,Market Price,2019-04-10T15:29:59.126852382Z,+1,Quote,-0.25,10,0.25,15
LGOTJ9,Market Price,2019-04-10T15:29:59.226886984Z,+1,Quote,-0.5,37,0.25,15
LGOTJ9,Market Price,2019-04-10T15:29:59.554835560Z,+1,Quote,-0.5,42,0.25,15
LGOTJ9,Market Price,2019-04-10T15:29:59.686869731Z,+1,Quote,-0.25,10,0.25,15
LGOTJ9,Market Price,2019-04-10T15:30:00.114938765Z,+1,Quote,,,0.25,15

So this is the validity of the instrument and the content that is available for it, as a result.

If you would like a more in-depth consultation on content, why the future was only available at this time, for a customer, the best approach is to directly refer the question to RTH Content experts via Refinitiv Helpdesk Online -> Content -> RTH.

Hope this helps





icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

sanchezTJUK avatar image sanchezTJUK

Thanks a lot for your time, really appreciate it.

sanchezTJUK avatar image sanchezTJUK

Hi Zoya,


If I want to run a historical reference request like you've done above, does my request url have to change?

Click below to post an Idea Post Idea