question

Upvotes
Accepted
46 4 5 10

delay quote problem

Hello, everyone, my application encountered a strange problem on July 2nd, feed data is stopped at about 9:26 EST and re-continued at about 9:39 EST, which makes quote data of the starting 8 minutes of that market day lost.

Let me explain my application logic first. At 4:00 EST, my application will start request chain records starting with 0#UNIVERSE.NB and batch subscribe all rics from the results of the above chain records. The app will continue receive UpdateMsg until at 9:29, when it will force unsubscribe all rics and re-subscribe them again. (The app does so because of one day of last year, we subscribed .IXIC at pre-market open time but receives no data when market open. So add such force unsubscribe and re-subscribe logic). I contact the feed data team about the proble, but they found no reason.


The difference betweeen that strange day and normal day I can notice is:

On normal days, if I re-subscribe at 9:29, seen from stdout log of OnStatusMsg callback, I can found such logs:

onStatusMsg: [2020-07-09 21:29:20 240495][23779:139987978176256]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

onStatusMsg: [2020-07-09 21:29:22 258829][23779:139987978176256]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

onStatusMsg: [2020-07-09 21:29:22 685333][23779:139987978176256]
Item Name: KLR_t.NB
Service Name: ELEKTRON_DD
Item State: Closed / Suspect / Not found / 'The record could not be found'

onStatusMsg: [2020-07-09 21:29:24 277349][23779:139987978176256]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

onStatusMsg: [2020-07-09 21:29:26 289616][23779:139987978176256]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'


such 'Batch request acknowledged.' messages are printed in 21:29. But on that strange day, I found such logs continues from


onStatusMsg: [2020-07-02 21:29:18 676818][26493:140211614574336]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

onStatusMsg: [2020-07-02 21:29:20 677285][26493:140211614574336]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

onStatusMsg: [2020-07-02 21:29:22 677726][26493:140211614574336]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

onStatusMsg: [2020-07-02 21:29:24 677971][26493:140211614574336]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

onStatusMsg: [2020-07-02 21:29:26 678170][26493:140211614574336]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

onStatusMsg: [2020-07-02 21:29:28 678555][26493:140211614574336]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

...

onStatusMsg: [2020-07-02 21:33:28 735769][26493:140211614574336]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

onStatusMsg: [2020-07-02 21:33:30 736099][26493:140211614574336]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

onStatusMsg: [2020-07-02 21:33:32 736463][26493:140211614574336]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

...

onStatusMsg: [2020-07-02 21:40:41 880045][26493:140211614574336]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

onStatusMsg: [2020-07-02 21:40:43 898048][26493:140211614574336]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'

onStatusMsg: [2020-07-02 21:40:45 903138][26493:140211614574336]
Service Name: ELEKTRON_DD
Item State: Closed / Ok / None / 'Batch request acknowledged.'


Such acknowledge message is far more than normal days, 300 versus 5. So what's the possible reason for that quote delay ? From the acknowledge message, May I suspect the trep is too slow to respond to my subscription? Any suggestion is appreciated, thank you.

elektronrefinitiv-realtimeelektron-sdkrrtema-apielektron-message-api
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

1 Answer

Upvotes
Accepted
23k 22 9 14

Hello @wangfugen,

This informational log is not sufficient, in my opinion, to conclude what has happened. Was there any refresh messages coming in in parallel as the batches were requested? That could give additional insight into what was happening?

It looks like, there might have been an intermittent interruption in Elektron service, that did not allow requests to go through at that time; would expect in this suspected scenario, as soon as the suspected interruption was resolved, would expect the connection to be recovered by EMA fully, and item streams reestablished.

However, I would check with the operations stuff responsible for your infrastructure/connectivity, if there was perhaps a brief outage around that time.

Going forward, there may be a very long of time till any next interruption in service, however, if you would like to know more on the application side, may need to prepare ahead of time. To do so you may try:

  • increasing application's custom logging
  • adding EMA xml tracing, into your EmaConfig.xml, I seem to recall that you are in EMA C++, if so then XmlTraceToFile and XmlTraceFileName, see EMA C++ Config Guide for complete details (if you are in EMA Java, let me know and will share Java details)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 5.0 MiB each and 10.0 MiB total.

In fact, seen from the quote data file logged by my application, data stopped at 9:26 EST. At 9:29 I unsuscribe all rics and batch re-subscribe them again. Until 9:39 I got the first RefreshMsg callback. Between 9:27 and 9:38 I received not any data. I can only get many batch request acknowleged output from server. So as you suggested, there might have been an interruption in TREP service. I deployed my application in 4 environments, all has about the same problem.


I previously contacted Gracie Kuang & Lianna Liang for investigating the problem. Further detail about our trep infrastructure may consult them.


I shall adjust ema xml tracing config in EmaConfig.xml to log more detail from ema c++ sdk. That problem is hard to reappear, maybe months or year. I can Just to prepare for that problem. If it occurs again, I will continue with the question.


Thanks for your reply. @zoya.farberov

XmlTraceToFile and XmlTraceFileName settings will log all quote messages into specified file. It will grow very big each day, Is there's any parameter that I can use just log some important messages for exceptional cases ?

@wangfugen,

Do not think you can do that, because this option is intended for debugging and troubleshooting, and in this case, any message can contain relevant information that you would not want to miss.

You may wish to use XmlTraceMaxFileSize with XmlTraceToMultipleFiles and periodically clean up any files >=N.

OK, Thanks @zoya.farberov


Click below to post an Idea Post Idea