question

Upvote
Accepted
mauryhill avatar image
18 ●0 ●1 ●2

Retrieving TS1 data encoded in FIDs ROW64_1-14 from Elektron (RTO)

There are two problems with the way that RTO is encoding TS1 data in UTF-8

  • One character is mapped to a different character in the same range
  • Several characters are expanded out into two characters and given the way it's done, I'm not sure it's possible to reverse the expansion 100% correctly in all cases

I've also captured the websocket traffic in Fiddler and it looks like this is coming from upstream and we're not distorting it any way in our "app". I've attached this rto-202105181200-ts1.zip showing this binary data in FIDs ROW64_1-14 before it reaches our app. Rename the filename from .zip -> .saz to open in Fiddler.

Also attached is "ts1-rto character mappings.txt" which shows the IDN characters and how they are being mapped to RTO. We arrived at this by making the same request through the LPC (Legacy Protocol Converter) 1.2 which connects to Elektron (RTO) and then comparing the binary data in the FIDs.

Can anyone shed an light on this decoding process? Or whom to have look at this?

Maybe our request in the Fiddler SAZ is just wrong? Its pretty basic.

{
  "ID":2,"Streaming":false
  "Key":{
    "Name":["dBOc1d"],
    "Service":"ELEKTRON_DD"
  }
}

Either the LPC is dealing with these different encodings or it is making a slightly different request that causes the data to come back encoded differently.

elektronrefinitiv-realtimeelektron-sdkrrtts1
rto-202105181200-ts1.zip (6.6 KiB)
ts1-rto-character-mappings.txt (619 B)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

mauryhill avatar image mauryhill

I created a MyRefinitiv ticket and was given the following response.

"We have received your query regarding problems with the way that RTO is encoding ts1realtime.txt in UTF-8 you saw.

We have tested on our side with ts1LPC.txt as well as with the real-time service but both cannot reproduce your mentioned issue. Please find our testing result in the attachment."

We are now comparing the above two file against our raw request with out .NET app. It is a bit more work to get what the LPC is doing since it communicates with https. We are investigating the usage of a proxy like Fiddler to capture the data for review.

ts1realtime.txt (31.4 KiB)
ts1lpc.txt (24.4 KiB)

1 Answer

Upvotes
Accepted
christopher.dugan1 avatar image
117 ●2 ●0 ●0

Hello Maury,

Per my colleague, there are UTF-8 issues when using websockets to retrieve data. I presume this applies to all data, not just TS1. Unfortunately, I don't have any ticket numbers available describing the issue of websockets handling UTF-8.

LPC comes with the testclient application for sample connections. It has the ability to output the data to a file or to STDOUT. Here is an example command line to use:


#> cd /opt/refinitiv/SOFTWARE/lpc1.2.0/linux7_x86_64/demo

#> export LD_LIBRARY_PATH=`pwd`

#> ./testclient -S ELEKTRON_DD -ct rssl -p 14002 -il dIBMd -o 512 -I 1 -O -u yourAppUserName -l stdout -dfile ../../etc/RDMFieldDictionary

The "stdout" option is using a lower case 'L'. The dictionary file is needed to decode the FIDs.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

mauryhill avatar image mauryhill
Thanks. I know it works through the LPC and am aware of the dictionary. I think the problem is as you have indicated, the handling of the data using websockets.


Is there somewhere I can raise a ticket?

Click below to post an Idea Post Idea