question

Upvotes
Accepted
AHofer avatar image
3 ●0 ●0 ●0

TRCC redundent connections and reconnect behaviour

I have a couple of questions how to archive resilience when connecting to a TRCC via ESDK.
(I'm using EMA / Java)

As per the documentation on the developer site:
(see https://developers.refinitiv.com/article/contributing-your-data-thomson-reuters)
Quote :
    Establish an encrypted Tunnelstream connection to one of the RCC hosts. For production applications,
    it is recommended to establish two simultaneous connections, to two different servers.

Does that mean there should be two OMM consumers created - each with using a single channel configuration?

second quote (from the sampe page):
    An application should only contribute to a single stream at any given time.
    The second stream is for redundancy purposes, and should only be used if there are issues with the first one.

When talking about the stream: should both streams be established with the same or different user IDs?

Next point I have is the TRCC/EMA recovery from a network failure.
I tested a failure simply by unplugging and replugging the network cable.
What I see is a status message:
    Received status message for handle: domain = System(127), code = NoCode, name = TRCC_Tunnel, state = Closed,
        Recoverable / Suspect / None / 'channel down.',
        service id = 10(DDS_TRCE), stream = 5 (private):

But I never got a "recover" status once the connection was available.
It is also not possible to submit any post messages - this results in an OmmException.
Below is a log for this case (including an exception backtrace).

The question is: What is the recommended behaviour on a state "Closed, Recoverable".
Logfile

Lines starting with "####" are comments 
---------------------------------------
2020-09-09 11:32:45,360 INFO c.o.p.t.s.SlhTRCC - ##### Start Service Line Handler for TRCC
2020-09-09 11:32:45,362 INFO c.o.p.t.s.SlhTRCC -  ... have configuration
2020-09-09 11:32:45,544 INFO c.o.p.t.c.TRCCConfiguration - TRCC channel parameters: {TcpNodelay=1, Port=443, Host=chpw02-emea1.thomsonreuters.com, GuaranteedOutputBuffers=5000, ChannelType=ChannelType::RSSL_ENCRYPTED, CompressionType=CompressionType::None, ConnectionPingTimeout=30000}
2020-09-09 11:32:45,546 INFO c.o.p.t.c.TRCCConfiguration - TRCC dictionary parameters: {RdmFieldDictionaryFileName=config/trcc/RDMFieldDictionary, EnumTypeDefItemName=RWFEnum, EnumTypeDefFileName=config/trcc/enumtype.def, RdmFieldDictionaryItemName=RWFFld, DictionaryType=DictionaryType::FileDictionary}
2020-09-09 11:32:45,740 WARN c.t.e.a.OmmConsumerImpl - loggerMsg
    ClientName: EmaConfig
    Severity: Warning
    Text:    no configuration exists in the config file for consumer dictionary [DictionaryGroup|DictionaryList]. Will use dictionary defaults if not config programmatically.
loggerMsgEnd


2020-09-09 11:32:48,304 WARN c.t.e.a.OmmConsumerImpl - loggerMsg
    ClientName: ChannelCallbackClient
    Severity: Warning
    Text:    Received ChannelDownReconnecting event on channel TRCC_Channel1
   RsslReactor @15de0b3c
   RsslChannel @489115ef
   Error Id 0
   Internal sysError 0
   Error Location Reactor.processWorkerEvent
   Error text Error initializing channel: errorId=-1 text=Invalid IPC Mount Opcode (0)
loggerMsgEnd


2020-09-09 11:32:50,819 WARN c.t.e.a.OmmConsumerImpl - loggerMsg
    ClientName: ChannelCallbackClient
    Severity: Warning
    Text:    Received ChannelDownReconnecting event on channel TRCC_Channel1
   RsslReactor @15de0b3c
   RsslChannel @3857f613
   Error Id 0
   Internal sysError 0
   Error Location Reactor.processWorkerEvent
   Error text Error initializing channel: errorId=-1 text=Invalid IPC Mount Opcode (0)
loggerMsgEnd


2020-09-09 11:32:54,324 WARN c.t.e.a.OmmConsumerImpl - loggerMsg
    ClientName: ChannelCallbackClient
    Severity: Warning
    Text:    Received ChannelDownReconnecting event on channel TRCC_Channel1
   RsslReactor @15de0b3c
   RsslChannel @489115ef
   Error Id 0
   Internal sysError 0
   Error Location Reactor.processWorkerEvent
   Error text Error initializing channel: errorId=-1 text=Invalid IPC Mount Opcode (0)
loggerMsgEnd


2020-09-09 11:32:59,926 INFO c.t.e.a.OmmConsumerImpl - loggerMsg
    ClientName: ChannelCallbackClient
    Severity: Info
    Text:    Received ChannelUp event on channel TRCC_Channel1
   Instance Name TRCC_Contributor_1
   Component Version eta3.1.1.L1.linux.rrg 64-bit Static
loggerMsgEnd


2020-09-09 11:33:00,113 INFO c.o.p.t.c.TRCCService - All Message Log: Received message for handle 1: [domain = System(127), code = NoCode, name = TRCC_Tunnel]
2020-09-09 11:33:00,115 INFO c.o.p.t.c.TRCCService - Received status message for handle 1: [domain = System(127), code = NoCode, name = TRCC_Tunnel, state = Open / Ok / None / '', service id = 10(DDS_TRCE), stream = 5 (private)].
2020-09-09 11:33:00,116 INFO c.o.p.t.c.TRCCService - Submit login for user "GEM-DTC-113027"to private stream: handle '1', stream id = '5'
2020-09-09 11:33:01,235 INFO c.o.p.t.c.TRCCService - All Message Log: Received message for handle 2: [domain = System(127), code = NoCode, name = GEM-DTC-113027]
2020-09-09 11:33:01,238 INFO c.o.p.t.c.TRCCService - Received refresh message for handle 2: [domain = System(127), code = NoCode, name = GEM-DTC-113027, state = Open / Ok / None / 'Login accepted by host c638zaudpcb13', stream = 6 (private)]
2020-09-09 11:33:01,239 INFO c.o.p.t.c.TRCCService - Private stream is open / login accepted: contributionStreamId = 6
2020-09-09 11:33:01,336 INFO c.o.p.t.c.RdmDataDictionary - Decode dictionary - fieldlist; complete = false
2020-09-09 11:33:01,469 INFO c.o.p.t.c.RdmDataDictionary - Decode dictionary - enumeration tables; complete = true
2020-09-09 11:33:01,527 INFO c.o.p.t.c.RdmDataDictionary - Decode dictionary - fieldlist; complete = true
2020-09-09 11:33:08,157 INFO c.o.p.t.c.Contributor - Contribute with ID 1 to TRCCTEST01=RTRF @ DDS_TRCE using stream id 6
2020-09-09 11:33:08,246 INFO c.o.p.t.c.TRCCService - All Message Log: Received message for handle 2: [domain = MarketPrice(6), code = NoCode]
2020-09-09 11:33:08,247 INFO c.o.p.t.c.Contributor - Received ACK message: post id = 1, nack code = NONE(SUCCESS)
2020-09-09 11:33:18,159 INFO c.o.p.t.c.Contributor - Contribute with ID 2 to TRCCTEST02=RTRF @ DDS_TRCE using stream id 6
2020-09-09 11:33:18,279 INFO c.o.p.t.c.TRCCService - All Message Log: Received message for handle 2: [domain = MarketPrice(6), code = NoCode]
2020-09-09 11:33:18,281 INFO c.o.p.t.c.Contributor - Received ACK message: post id = 2, nack code = NONE(SUCCESS)
2020-09-09 11:33:28,199 INFO c.o.p.t.c.Contributor - Contribute with ID 3 to TRCCTEST03=RTRF @ DDS_TRCE using stream id 6
2020-09-09 11:33:28,326 INFO c.o.p.t.c.TRCCService - All Message Log: Received message for handle 2: [domain = MarketPrice(6), code = NoCode]
2020-09-09 11:33:28,328 INFO c.o.p.t.c.Contributor - Received ACK message: post id = 3, nack code = NONE(SUCCESS)
2020-09-09 11:33:38,229 INFO c.o.p.t.c.Contributor - Contribute with ID 4 to TRCCTEST04=RTRF @ DDS_TRCE using stream id 6
2020-09-09 11:33:38,331 INFO c.o.p.t.c.TRCCService - All Message Log: Received message for handle 2: [domain = MarketPrice(6), code = NoCode]
2020-09-09 11:33:38,332 INFO c.o.p.t.c.Contributor - Received ACK message: post id = 4, nack code = NONE(SUCCESS)

####
#### here the computer was unplugged from the network
####

2020-09-09 11:33:45,963 WARN c.t.e.a.OmmConsumerImpl - loggerMsg
    ClientName: ChannelCallbackClient
    Severity: Warning
    Text:    Received ChannelDownReconnecting event on channel TRCC_Channel1
   RsslReactor @15de0b3c
   RsslChannel @3857f613
   Error Id 0
   Internal sysError 0
   Error Location null
   Error text Eine vorhandene Verbindung wurde vom Remotehost geschlossen
loggerMsgEnd


2020-09-09 11:33:45,964 WARN c.t.e.a.OmmConsumerImpl - loggerMsg
    ClientName: LoginCallbackClient
    Severity: Warning
    Text:    RDMLogin stream state was changed to suspect with status message
   username <not set>
   usernameType <not set>
   
   State: Open/Suspect/None - text: ""
loggerMsgEnd


2020-09-09 11:33:45,965 INFO c.o.p.t.c.TRCCService - All Message Log: Received message for handle 1: [domain = System(127), code = NoCode, name = TRCC_Tunnel]
2020-09-09 11:33:45,965 INFO c.o.p.t.c.TRCCService - Received status message for handle 1: [domain = System(127), code = NoCode, name = TRCC_Tunnel, state = Closed, Recoverable / Suspect / None / 'channel down.', service id = 10(DDS_TRCE), stream = 5 (private)].
2020-09-09 11:33:45,965 WARN c.o.p.t.c.TRCCService - Tunnel stream status is closed/recover.
####
#### At this point, the software stops contribution - waiting for a state "Open"
####
2020-09-09 11:33:48,476 WARN c.t.e.a.OmmConsumerImpl - loggerMsg
    ClientName: ChannelCallbackClient
    Severity: Warning
    Text:    Received ChannelDownReconnecting event on channel TRCC_Channel1
   RsslReactor @15de0b3c
   RsslChannel @489115ef
   Error Id 0
   Internal sysError 0
   Error Location Reactor.processWorkerEvent
   Error text Error initializing channel: errorId=-1 text=Invalid IPC Mount Opcode (0)
loggerMsgEnd

###
### plugging the network cable ...
###

2020-09-09 11:33:51,993 WARN c.t.e.a.OmmConsumerImpl - loggerMsg
    ClientName: ChannelCallbackClient
    Severity: Warning
    Text:    Received ChannelDownReconnecting event on channel TRCC_Channel1
   RsslReactor @15de0b3c
   RsslChannel @3857f613
   Error Id 0
   Internal sysError 0
   Error Location Reactor.processWorkerEvent
   Error text Error initializing channel: errorId=-1 text=Invalid IPC Mount Opcode (0)
loggerMsgEnd

...
#### removed some "ChannelDownReconnecting" event logs ...
...

2020-09-09 11:34:56,654 WARN c.t.e.a.OmmConsumerImpl - loggerMsg
    ClientName: ChannelCallbackClient
    Severity: Warning
    Text:    Received ChannelDownReconnecting event on channel TRCC_Channel1
   RsslReactor @15de0b3c
   RsslChannel @3857f613
   Error Id 0
   Internal sysError 0
   Error Location Reactor.processWorkerEvent
   Error text Error initializing channel: errorId=-1 text=Invalid IPC Mount Opcode (0)
loggerMsgEnd


2020-09-09 11:35:03,223 INFO c.t.e.a.OmmConsumerImpl - loggerMsg
    ClientName: ChannelCallbackClient
    Severity: Info
    Text:    Received ChannelUp event on channel TRCC_Channel1
   Instance Name TRCC_Contributor_1
   Component Version eta3.1.1.L1.linux.rrg 64-bit Static
loggerMsgEnd

#### EMA log says "connection is up" but NO status message received for this new state!
#### Meaning: the application has no clue about the current state.
#### Any attempt to submit a post-message anyway results in the following OMM Exception:

com.thomsonreuters.ema.access.OmmInvalidHandleExceptionImpl: loggerMsg
    ClientName: ItemCallbackClient
    Severity: Error
    Text:    Attempt to use invalid Handle on submit(PostMsg). Instance name='TRCC_Contributor_1'.
loggerMsgEnd


   at com.thomsonreuters.ema.access.OmmBaseImpl.ommIHExcept(OmmBaseImpl.java:1388)
   at com.thomsonreuters.ema.access.OmmConsumerImpl.handleInvalidHandle(OmmConsumerImpl.java:472)
   at com.thomsonreuters.ema.access.ItemCallbackClient.submit(ItemCallbackClient.java:2380)
   at com.thomsonreuters.ema.access.OmmBaseImpl.submit(OmmBaseImpl.java:537)
   at com.thomsonreuters.ema.access.OmmConsumerImpl.submit(OmmConsumerImpl.java:179)
   at com.oraise.picos.trcc.contributor.TRCCService.submit(TRCCService.java:277)
   at com.oraise.picos.trcc.contributor.Contributor.contribute(Contributor.java:251)
   at com.oraise.picos.trcc.contributor.Contributor.update(Contributor.java:211)
   at com.oraise.picos.trcc.contributor.Contributor$ContributionThread.run(Contributor.java:334)
elektronrefinitiv-realtimeelektron-sdkrrtema-apielektron-message-apircc-api
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

4 Answers

Upvotes
Accepted
Gurpreet avatar image
13k ●32 ●12 ●18
Next point I have is the TRCC/EMA recovery from a network failure.

A while back I encountered this issue and it was traced back to the bug in ESDK. The development team expects to implement the fix in version 2.0.1 L1.

The underlying issue is that EMA is responsible for sending a refresh message after a successful exchange of login and directory messages (in that order).

It turns out that EMA sends a refresh message too early, even before directory message is exchanged on a network reconnection. This results in sending a tunnel stream request prematurely.

A workaround for this (until it is fixed in the SDK) is:

You can have the application make a Source Directory request, and instead of making the Tunnel Stream request in response to the Login Refresh, you can make the Tunnel stream request when you get the Directory Response.

I have attached the C++ code which shows this modification. You can port same changes over to Java. (Rename the file to .cpp)

Consumer.pdf



consumer.pdf (13.3 KiB)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

AHofer avatar image AHofer

Thanks for this - the workaround seems to do the trick.

AHofer avatar image AHofer

Hi Gurpreet,


meanwhile I'm on RTSDK-2.0.1.L1 java (EMA 3.6.1.0) but the reconnect issue still exists. Can you confirm, that it's not yet fixed?

Using the workaround you sent me I get the following messages on a ChannelUp event when subscribing to a dictionary:

received UPDATE message for handle 1, : [domain = DIRECTORY(4), code = NoCode]

received STATUS message for handle 6, : [domain = System(127), code = NoCode, name = TRCC_Tunnel, state = Open / Ok / None / '', service id = 10(DDS_TRCE), stream = 7 (private)]
-> STATUS message action: submit credentials for user "XXXXXX" to private stream: handle '6', stream id = '7'

received REFRESH message for handle 7: [domain = LOGIN(1), code = NoCode, name = GEM-DTC-113027, state = Open / Ok / None / 'Login accepted by host c566wtxdpca42', stream = 6 (private)]

When I don't subscribe to a dictionary I get nothing ...

Thanks and best Regards

Andreas


Gurpreet avatar image Gurpreet AHofer

Hi Andreas, I don't see automatic recovery with the latest SDK either. I would recommend that you raise an issue at Realtime SDK github, where you can directly interact with the developers of the SDK.

Upvotes
Gurpreet avatar image
13k ●32 ●12 ●18

Hello @AHofer, you have included multiple questions in a single one, and it will become difficult to answer and track them all. It is better to split a these into multiple atomic questions.

Does that mean there should be two OMM consumers created - each with using a single channel configuration?

Yes, the guidance from the RCC product management is that multiple OMMConsumer should be created and each one of these consumers should connect to a different endpoint. 

<Channel>
 <Name value="Channel_1"/>
 <ChannelType value="ChannelType::RSSL_ENCRYPTED"/>
 <Host value="****.com"/>
</Channel>
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

Upvotes
Gurpreet avatar image
13k ●32 ●12 ●18
When talking about the stream: should both streams be established with the same or different user IDs?

RCC only allows one concurrent connection from a single ID. You might accidently be able to connect more than once, when load manager shunts a new connection to a different server, but this will be an exception condition and should not be relied upon.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

AHofer avatar image AHofer

Hi Grupreet,

Our design approach (for resilience) was to have 2 contribution processes both maintaining two connections.

A single process creates two OMM consumers, but only one is used unless the "active" failes/disconnects.

In sum, these are 4 connections. According to your answer, this is not supported with two user ids. Am I correct here?

The alternatives are :

- One process with two connections using two user ids or

- two processes with one connection each using different user ids.

With these alternatives we would go for 2 processes but using both connections for contribution of disjunct instrument universes.

Is this second option a supported scenario?

Thanks Andreas

Gurpreet avatar image Gurpreet AHofer

Hi @AHofer, Both the approaches are technically feasible, but I suggest that you contact your Refinitiv account manager. They can present your use case to Contributions team and get the right recommendation for load balancing and redundancy.

Upvotes
Gurpreet avatar image
13k ●32 ●12 ●18

Please also see this Refinitiv Contributions Channel network and connectivity guide: https://my.refinitiv.com/content/dam/myrefinitiv/products/11372/en/Technical/616281.pdf

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

AHofer avatar image AHofer

Hi Gurpreet,

The given link does not work for me. Is the link still valid?

Thanks

Andreas

Gurpreet avatar image Gurpreet AHofer

Yes, the link is valid. You might need to be granted access - please talk to your Refinitiv account manager for that.

Click below to post an Idea Post Idea