For a deeper look into our Elektron API, look into:

Overview |  Quickstart |  Documentation |  Downloads |  Tutorials |  Articles

question

Upvotes
Accepted
mauryhill avatar image
18 ●0 ●1 ●2

Data in ROW64_1-14 for ts1 using Elektron Cloud

**See below comment** RTO is returning strange character encodings.


If I request something like dIBMd, I'm expecting to get encoded data from ts1 via Elektron.

Does anyone know how what I get back in the FIDs ROW64_1 (through 14) is encoded? I have an is this in UCS-2, UTF-16, UTF-8, etc?

I already understand how to decode ts1 data once it is in its binary form, but I need to get this into it first.


RECEIVED on dIBMd:
{
  "PROD_PERM": 3063,
  "RDNDISPLAY": 0,
  "BOND_TYPE": "   ",
  "ROW64_1": " !????M$9\\+v???? <??????TC>????<??!??????7j m]??<J??1Th$????a6|(|????z(??C???? ??????\"??2g",
  "ROW64_2": "??Jz,$?? 0!????p#????(Vn9???????:??D)???^??^p?? 9z2 &??)n????j????n??E??2??)??          ",
  "ROW64_3": "                                                                ",
  "ROW64_4": "                                                                ",
  "ROW64_5": "                                                                ",
  "ROW64_6": "                                                                ",
  "ROW64_7": "                                                                ",
  "ROW64_8": "                                                                ",
  "ROW64_9": "                                                                ",
  "ROW64_10": "                                    M\"\"??+??   6?:aj   |??.,??   N]??",
  "ROW64_11": "??L     PZ_u_uv   +??_w?????F??}~9??@??$??,?????.??n??[????L??6??d???????????{!?? ????Ur9'????7??",
  "ROW64_12": "d????!??|3??6sP??5v???? ??I>%y_u??#q????v??_tQ3?????nA~!^;._tY????Ec??z??[????Z!K$?????????????",
  "ROW64_13": ")9??F>??3????{??H??ow??????6V]S???????????T????%Wi<??????+??7????g???_uv??~????_p????????_t????????LN|??",
  "ROW64_14": "?????yJ]??????????Bp????\"x??????E????????????%\"zVr??\"??Mx??l???????????1??????????)xD????i}B\"??&??|??????",
  "RECORDTYPE": 237,
  "REG_ID1": 0,
  "REG_FIELD1": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=",
  "ROW1_TIME": null,
  "ROW2_TIME": null,
  "ROW3_TIME": null,
  "ROW4_TIME": null,
  "ROW5_TIME": null,
  "ROW6_TIME": null,
  "ROW7_TIME": null,
  "ROW8_TIME": null,
  "ROW9_TIME": null,
  "ROW10_TIME": null,
  "ROW11_TIME": null,
  "ROW12_TIME": null,
  "ROW13_TIME": null,
  "ROW14_TIME": null,
  "BYTE_BMAP": 0,
  "PREF_DISP": 0,
  "DSO_ID": 0
}

I get the above using:

var readBuffer = new ArraySegment<byte>(new byte[Constants.BUFFER_SIZE]);
MemoryStream memoryStream = null;
byte[] dataBuffer;

while (true)
{
    try
    {
        var result = await WebSocket.ReceiveAsync(readBuffer, Cts.Token);
        if (!result.EndOfMessage)
        {
            if (memoryStream == null) memoryStream =
                new MemoryStream(Constants.BUFFER_SIZE * 5);

            memoryStream.Write(readBuffer.Array, readBuffer.Offset, readBuffer.Count);
            readBuffer = new ArraySegment<byte>(new byte[Constants.BUFFER_SIZE]);
        }
        else
        {
            if (memoryStream != null)
            {
                memoryStream.Write(readBuffer.Array, readBuffer.Offset, readBuffer.Count);
                dataBuffer = memoryStream.GetBuffer();
                memoryStream.Dispose();
            }
            else
            {
                dataBuffer = readBuffer.Array;
            }
            break;
        }
    }
    catch
    {
        cSpec.CloseSessions(this, null);
    }
}

JArray messages = JArray.Parse(Encoding.ASCII.GetString(dataBuffer));
elektronwebsocketsrrtotime-seriesrefinitiv-realtime-optimisedts1
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

4 Answers

Upvotes
Accepted
nick.zincone avatar image
9.7k ●49 ●38 ●60

Hi @mauryhill,

I believe you are correct. I have this in my extraction but have not attempted to parse the ROW64 fields:

ArraySegment<byte> buffer = WebSocket.CreateClientBuffer(ChunkSize, ChunkSize);
...
result = await _webSocket.ReceiveAsync(buffer, CancellationToken.None);
...
string str = Encoding.UTF8.GetString(buffer.Array, 0, result.Count);


Please confirm this works for you.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

Upvotes
mauryhill avatar image
18 ●0 ●1 ●2

Looks like this might work:

JArray messages = JArray.Parse(Encoding.UTF8.GetString(dataBuffer));

Will let y'all know (I'm sure the suspense is killing you).

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

Upvotes
mauryhill avatar image
18 ●0 ●1 ●2

Thanks.

Still investigating and had to write out "readBuffer" directly to a binary file for analysis because it appears that there are two problems from Elektron:

  1. 3 lines have 3 extra characters each (string length = 67 instead of 64; each of the 14 ROW64 FIDs should be 64 chars)
  2. There are codepoints outside the ASCII and Latin 1 supplemental blocks

But the day is still young.

For reference, the code for the "tap":

var readBuffer = new ArraySegment<byte>(new byte[Constants.BUFFER_SIZE]);
MemoryStream memoryStream = null;
byte[] dataBuffer;
var path = @"c:\temp\ts1.bin";
var fn = new FileInfo(path);if (fn.Exists) { fn.Delete(); }
using (FileStream fs = new(path, FileMode.OpenOrCreate))
using (BinaryWriter w = new(fs))
while (true)
{
    try
    {
        var result = await WebSocket.ReceiveAsync(readBuffer, Cts.Token);
        w.Write(readBuffer);
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

Upvotes
mauryhill avatar image
18 ●0 ●1 ●2

There are two problems with the way that RTO is encoding TS1 data in UTF-8

  • One character is mapped to a different character in the same range
  • Several characters are expanded out into two characters and given the way it's done, I'm not sure it's possible to reverse the expansion 100% correctly in all cases

I've also captured the websocket traffic in Fiddler and it looks like this is coming from upstream and we're not distorting it any way in our "app". I've attached this rto-202105181200-ts1.zipshowing this binary data in FIDs ROW64_1-14 before it reaches our app. Rename the filename from .zip -> .saz to open in Fiddler.

Also attached is "ts1-rto character mappings.txt" which shows the IDN characters and how they are being mapped to RTO. We arrived at this by making the same request through the LPC (Legacy Protocol Converter) 1.2 which connects to Elektron (RTO) and then comparing the binary data in the FIDs.

Can anyone shed an light on this decoding process? Or whom to have look at this?

Maybe our request in the Fiddler SAZ is just wrong? Its pretty basic.

{
  "ID":2,"Streaming":false
  "Key":{
    "Name":["dBOc1d"],
    "Service":"ELEKTRON_DD"
  }
}

Either the LPC is dealing with these different encodings or it is making a slightly different request that causes the data to come back encoded differently.


rto-202105181200-ts1.zip (6.6 KiB)
ts1-rto-character-mappings.txt (619 B)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

Click below to post an Idea Post Idea