question

Upvotes
Accepted
YK_deprecated_1 avatar image
9 ●5 ●3 ●8

TRTH: Python Extract market depth data

Hi,

I'm in trouble to extract data from http response object.

I can't figure out how to extract data using job ID and $value endpoint. Use case reference says users can download the report by folowing final step, but can't find any gzip file in my directory.

How can I retrieve or find data from report_gzip or anyother types of objjects?

Thanks,

report_gzip= requests.get( "https://hosted.datascopeapi.reuters.com/RestApi/v1/Extractions/RawExtractionResults('0x05c5d18a57bb2f96')/$value",
                            headers = header2, stream=True)
pythontick-history-rest-api
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

3 Answers

Upvotes
Accepted
veerapath.rungruengrayubkul avatar image
11.3k ●25 ●8 ●13

@YK

The raw extraction result will be streamed in gzip format, and then stored in the Response object. You can use this method of the Python Requests module to save raw gzip streamed response to file.

Below is the sample code. You can also see this question.

filename = "result.csv.gz"

with open(filename,'wb') as f:
   for data in report_gzip.raw.stream(decode_content=False):
       f.write(data)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

Upvotes
YK_deprecated_1 avatar image
9 ●5 ●3 ●8

Thank you, @veerapath.rungruengrayubkul

It works perfect on my environment. I was just wondering how to extract gzip file from response object, but it's now crystal clear.

Thanks again!

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

veerapath.rungruengrayubkul avatar image veerapath.rungruengrayubkul

@YK

I have just realized that the sample code I provided might lead to the issue that data will not be completely decompressed if the extracted result is huge.

I have edited the sample code in my answer. Please use the new sample code instead.

veerapath.rungruengrayubkul avatar image veerapath.rungruengrayubkul

@YK

If you want to process the extracted result instead of writing to a gzip file, you need to use gzip to decompress the data. Below is the sample code.

import io
import gzip

fd = io.BytesIO()
for data in report_gzip.raw.stream(decode_content=False):
    fd.write(data)


#decompress data
decompress_data = gzip.decompress(fd.getbuffer());
data = io.StringIO(decompress_data.decode('utf-8'))
    for line in data:
        print(line+'\n')

Hope this help.

Upvotes
YK_deprecated_1 avatar image
9 ●5 ●3 ●8
@ veerapath.rungruengrayubkul

ok will use that sample code if I need to process the result, thank you!

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

Click below to post an Idea Post Idea