question

Upvotes
Accepted
ayoub.serroukheidrissi avatar image
5 ●0 ●0 ●3

time and sales extraction incomplete

Hello,

I'm trying to extract some time and sales data via an on demand raw extraction but I'm only getting a small subset of the expected results, whereas I'm able to get the full results when using the GUI. Is there any limitation when using the on demand requests ?

An excerpt of my request and the notes returned below:

request =

{'ExtractionRequest': {'@odata.type': '#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.TickHistoryTimeAndSalesExtractionRequest',

'Condition': {'ApplyCorrectionsAndCancellations': True,

'DisplaySourceRIC': True,

'ExtractBy': 'Entity',

'MessageTimeStampIn': 'LocalExchangeTime',

'QueryEndDate': '2019-10-25T20:15:00',

'QueryStartDate': '2019-10-25T19:45:00',

'ReportDateRangeType': 'Range',

'SortBy': 'SingleByRic',

'TimeRangeMode': 'Inclusive'},

'ContentFieldNames': ['Quote - Bid Price',

'Quote - Bid Size',

'Quote - Ask Price',

'Quote - Ask Size',

'Trade - Bid Price',

'Trade - Bid Size',

'Trade - Ask Price',

'Trade - Ask Size'],

'IdentifierList': {'@odata.type': '#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.InstrumentIdentifierList',

'InstrumentIdentifiers': [{'Identifier': 'SPXWw011927250.U',

'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011927300.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011927350.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011927400.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011927450.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011927500.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011927550.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011927600.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011927650.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011927700.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011927750.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011927800.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011927850.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011927900.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011927950.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011928000.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011928050.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011928100.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011928150.U', 'IdentifierType': 'Ric'},

{'Identifier': 'SPXWw011928200.U', 'IdentifierType': 'Ric'},],

'UseUserPreferencesForValidationOptions': True}}}


notes =

'Extraction Services Version 13.2.41189 (85522a4c8b22), Built Nov 11 2019 22:20:39',

'User ID: ',

'Extraction ID: 2000000113205251',

'Schedule: 0x06dfaee189312bd0 (ID = 0x0000000000000000)',

'Input List (83 items): (ID = 0x06dfaee189312bd0) Created: 11/20/2019 08:31:46 Last Modified: 11/20/2019 08:31:46',

'Report Template (8 fields): _OnD_0x06dfaee189312bd0 (ID = 0x06dfaee189b12bd0) Created: 11/20/2019 08:29:34 Last Modified: 11/20/2019 08:29:34',

'Schedule dispatched via message queue (0x06dfaee189312bd0), Data source identifier (93ECB38A43154CAD8A1154D930EDB5C2)',

'Schedule Time: 11/20/2019 08:29:35',

'Processing started at 11/20/2019 08:29:36',

'Processing completed successfully at 11/20/2019 08:31:46',

'Extraction finished at 11/20/2019 08:31:46 UTC, with servers: tm02n02, TRTH (71.93 secs)',

'Historical Instrument <RIC,SPXWw011927250.U> expanded to 1 RIC: SPXWw011927250.U.',

'Historical Instrument <RIC,SPXWw011927300.U> expanded to 1 RIC: SPXWw011927300.U.',

'Historical Instrument <RIC,SPXWw011927350.U> expanded to 1 RIC: SPXWw011927350.U.',

'Historical Instrument <RIC,SPXWw011927400.U> expanded to 1 RIC: SPXWw011927400.U.',

'Historical Instrument <RIC,SPXWw011927450.U> expanded to 1 RIC: SPXWw011927450.U.',

'Historical Instrument <RIC,SPXWw011927500.U> expanded to 1 RIC: SPXWw011927500.U.',

'Historical Instrument <RIC,SPXWw011927550.U> expanded to 1 RIC: SPXWw011927550.U.',

'Historical Instrument <RIC,SPXWw011927600.U> expanded to 1 RIC: SPXWw011927600.U.',

'Historical Instrument <RIC,SPXWw011927650.U> expanded to 1 RIC: SPXWw011927650.U.',

'Historical Instrument <RIC,SPXWw011927700.U> expanded to 1 RIC: SPXWw011927700.U.',

'Historical Instrument <RIC,SPXWw011927750.U> expanded to 1 RIC: SPXWw011927750.U.',

'Total instruments after instrument expansion = 83',

'Range Query from 2019-10-25T19:45:00.000 to 2019-10-25T20:15:00.000 (UTC)',

'Quota Message: INFO: Tick History Options Quota Count Before Extraction: 47; Instruments Approved for Extraction: 1; Tick History Options Quota Count After Extraction: 47, 78.3333333333333% of Limit; Tick History Options Quota Limit: 60',

'Manifest: #RIC,Domain,Start,End,Status,Count',

'Manifest: SPXWk011930200.U,Market Price,2019-10-25T19:45:00.084016861Z,2019-10-25T20:14:59.197502072Z,Active,678',

'Manifest: SPXWk011930250.U,Market Price,2019-10-25T19:45:00.416646846Z,2019-10-25T20:14:59.528750063Z,Active,619',

'Manifest: SPXWk011930300.U,Market Price,2019-10-25T19:45:00.751666355Z,2019-10-25T20:14:59.860912512Z,Active,690',

'Manifest: SPXWk011930350.U,Market Price,2019-10-25T19:45:00.083406164Z,2019-10-25T20:14:59.201640345Z,Active,639',

'Manifest: SPXWk011930400.U,Market Price,2019-10-25T19:45:00.418477881Z,2019-10-25T20:14:59.529562257Z,Active,659',

'Manifest: SPXWk011930450.U,Market Price,2019-10-25T19:45:00.751679571Z,2019-10-25T20:14:59.861306049Z,Active,650',

'Manifest: SPXWk011930500.U,Market Price,2019-10-25T19:45:00.084726519Z,2019-10-25T20:14:59.200826455Z,Active,674',

'Manifest: SPXWk011930550.U,Market Price,2019-10-25T19:45:01.442438658Z,2019-10-25T20:14:58.534407536Z,Active,658',

'Manifest: SPXWk011930600.U,Market Price,2019-10-25T19:45:01.793153265Z,2019-10-25T20:14:58.863615075Z,Active,631',

'Manifest: SPXWk011930650.U,Market Price,2019-10-25T19:45:01.110501067Z,2019-10-25T20:14:58.201620846Z,Active,616',

'Manifest: SPXWk011930700.U,Market Price,2019-10-25T19:45:01.442281722Z,2019-10-25T20:14:58.534447688Z,Active,600',

'Manifest: SPXWk011930750.U,Market Price,2019-10-25T19:45:01.441433928Z,2019-10-25T20:14:58.531512914Z,Active,594',

'Manifest: SPXWk011930800.U,Market Price,2019-10-25T19:45:01.108691063Z,2019-10-25T20:14:58.200188611Z,Active,604',

'Manifest: SPXWk011930850.U,Market Price,2019-10-25T19:45:01.443025611Z,2019-10-25T20:14:58.533451126Z,Active,607',

'Manifest: SPXWk011930900.U,Market Price,2019-10-25T19:45:03.843663254Z,2019-10-25T20:14:58.865526983Z,Active,550',

'Manifest: SPXWk011930950.U,Market Price,2019-10-25T19:45:01.111667870Z,2019-10-25T20:14:58.200801172Z,Active,514',

'Manifest: SPXWk011931000.U,Market Price,2019-10-25T19:45:01.107683661Z,2019-10-25T20:14:58.196788052Z,Active,473',

'Manifest: SPXWk011931050.U,Market Price,2019-10-25T19:45:00.086552971Z,2019-10-25T20:14:31.029838257Z,Active,493',

'Manifest: SPXWk011931100.U,Market Price,2019-10-25T19:45:44.034855495Z,2019-10-25T20:14:56.525360521Z,Active,476',


tick-history-rest-api
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

3 Answers

Upvotes
Accepted
Christiaan Meihsl avatar image
13.7k ●26 ●8 ●12

@ayoub.serroukheidrissi,

We have Python samples described here that download compressed data, you might want to compare their code to yours. You might also find this article useful in this context.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

ayoub.serroukheidrissi avatar image ayoub.serroukheidrissi

It works with aws. Thanks a lot

Christiaan Meihsl avatar image Christiaan Meihsl ayoub.serroukheidrissi

@ayoub.serroukheidrissi, great, glad to know it is solved. And by using AWS you get faster downloads as well :)

Upvotes
Christiaan Meihsl avatar image
13.7k ●26 ●8 ●12

@ayoub.serroukheidrissi,

You do not mention exactly how you are retrieving the data, but if you are decompressing the extracted data on the fly you can end up with truncated results. On that topic see this advisory.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

Upvotes
ayoub.serroukheidrissi avatar image
5 ●0 ●0 ●3

Thanks Christiaan

I'm retrieving via http requests in python using the requests module. I'm unable to desactivate the decompression and I get the same truncated results in my browser with the request's url:

https://hosted.datascopeapi.reuters.com/RestApi/v1/Extractions/RawExtractionResults('0x06dfb21d04412bda')/$value

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

Click below to post an Idea Post Idea