question

Upvotes
Accepted
vaibhav.sahu avatar image
18 ●1 ●2 ●3

RDP Python API performance

I am doing POC of python Library with our first client .

Client requirement is pretty straight forward . They are subscribing around 9000 RIC & interested in snapshot data only with 2 fields (i.e. Dynamic View request for Image )

I tried 4 different models (for Delivery & Contents Layer functions in Sync/Async mode ) with 1000 Rics for POC . All are getting taking around 2 mins to process that much responses .

One of a Async non-streaming sample code is as such :-

def create_cache(name,msg ):

#outPut[msg['Key']['Name']]=msg

print( " Response Recevied for ",msg['Key']['Name'] )

.

.

.

ricList = getItemList() ## here geting 1000 rics for POC

tasks=[]

mySession = sh.get_session('deployed')

mySession.open()

display_stats( " Record processing started ") ## Start time for calculation

for ric in ricList:

order_book = rdp.ItemStream(

name = ric,

service = 'CANNED_DATA',

session = mySession,

domain = "MarketPrice",

streaming =False,

fields = ['BID', 'ASK'],

on_refresh = lambda name , msg : create_cache(name,msg), ## At the moment just printing

on_update = lambda s,msg : print (" Update Msg ", s,json.dumps(msg, indent=2))

)

tasks.append(order_book.open_async())

loop = asyncio.get_event_loop()

loop.run_until_complete(asyncio.wait(tasks))


display_stats(" process "+ str( len(outPut))+ " Records") ## End time for processing calc


I am wondering if there is something i can do to improve performance , expecting to process ~9000 Rics in 10 seconds

pythontreprdp-apirefinitiv-data-platformrefinitiv-data-platform-libraries
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

1 Answer

Upvote
Accepted
Jirapongse avatar image
38.1k ●71 ●35 ●53

@vaibhav.sahu

Typically, the WebSocket API supports the batch request. With the batch request, you can use one request message to request multiple items. This could improve the performance of the application.

However, from my checking, the RDP Python library may not support the batch request.

I think to get more performance you can directly use WebSocket API to consume the data. The example is available on GitHub.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

vaibhav.sahu avatar image vaibhav.sahu

Thanks Jirapongse , Actually we were using websocket API earlier for this client . We have written an Async wrapper on top of that . Anyhow Async wrapper is not feature rich . So thought of moving to RDP for two things

  1. Get more feature
  2. Better support

Anyhow we stuck with performance issue here . I have raised this to Olivier & waiting for some input from Refinitiv .

On Other note , i can see we can send Batch request in content layer i.e. StreamingPrices fucntion . It takes a list of instruments as universe parameters . Anyhow that's not giving decent performance .

Jirapongse avatar image Jirapongse vaibhav.sahu

I have checked the code. If I am correct, the API internally separates a batch request into individual requests. Therefore, it will get the same performance as individual requests.

vaibhav.sahu avatar image vaibhav.sahu

Hi Jirapongse,

Thanks for your reply. Actually our client was already using a in-house developed async wrapper written on top of websocket API . That wrapper has a decent performance but not feature rich (i.e connection recovery , item recovery ). Also that library was not easy to be used . That' why we thought of giving try to RDP python Library. I checked with Refinitiv support & look like you do have plan to add new features in RDP (i.e. connection recovery , item recovery etc.).

Anyhow we stuck with the performance issue with RDP here .


Click below to post an Idea Post Idea