Hi,
We have data receive problem from Reuters WebSocket server after a while.
While we are getting prices from the server normally, after a time, it stops to send both the prices and PING message to our client side. So we are encountering idle connection case and also data loss case!
We have used 2 different WebSocket libraries and encountered with the same case. I have shared all the details below.
Dou you have any suggestions to solve this problem?
Best Regards.
#########################################################
Environment #1:
Uses Library: nv-websocket-client <=== This WebSocket library is used at Refinitiv github examples
In this test when we have detected that server did not send PING message for 30 seconds we disconnected from the server programmatically.
On login, we got pingTimeout is 30 seconds:
0531-110858.889-I-(WebSocketSession) - WSOCK: Handling login response on message:{"ID":1,"Type":"Refresh","Domain":"Login","Key":{"Name":"AQIC5wM2LY4SfczaaZd9CSZ7rS3ip37O4qx6l8Zg76TPSZY%3D%40AAJTSQACMTAAAlNLABMyMTU0MDE1NDg0MDk3NDc3MTc1AAJTMQACMjc%3D%23","Elements":{"AllowSuspectData":1,"ApplicationId":"256","ApplicationName":"ADS","AuthenticationErrorCode":0,"AuthenticationErrorText":{"Type":"AsciiString","Data":null},"AuthenticationTTReissue":1622459487,"Position":"10.57.5.20","ProvidePermissionExpressions":1,"ProvidePermissionProfile":0,"SingleOpen":1,"Su
pportEnhancedSymbolList":1,"SupportOMMPost":1,"SupportPauseResume":0,"SupportStandby":0,"SupportBatchRequests":7,"SupportViewRequests":1,"SupportOptimizedPauseResume":0}},"State":{"Stream":"Open","Data":"Ok","Text":"Login accepted by host ads-fanout-sm-az1-use1-prd."},"Elements":{ "PingTimeout":30,"MaxMsgSize":61426}}
We periodically got PING messages from server and answering with PONG message (generally getting PING messages in 20 seconds).
But after a while we cannot get either prices or PING messages from server:
0531-120818.206-I-(WebSocketSession) - WSOCK: Sending Pong message ß PING RECEIVED AND WE RESPONDED WITH PONG
// prices
0531-120838.206-I-(WebSocketSession) - WSOCK: Sending Pong message ß LATEST PING RECEIVED AND WE RESPONDED WITH PONG
// prices
0531-120852.227-I-(WebSocketSession) - WSOCK: {instr='TRY=', ASK=8.516, BID=8.474}
0531-120852.327-I-(WebSocketSession) - WSOCK: {instr='TRY=', ASK=8.516, BID=8.474}
0531-120852.987-I-(WebSocketSession) - WSOCK: {instr='XAU=', ASK=1905.7, BID=1904.7}
0531-120853.153-I-(WebSocketSession) - WSOCK: {instr='TRY=', ASK=8.5066, BID=8.4986}
0531-120853.427-I-(WebSocketSession) - WSOCK: {instr='TRY=', ASK=8.5065, BID=8.4996}
0531-120854.647-I-(WebSocketSession) - WSOCK: {instr='XAU=', ASK=1905.7, BID=1904.7}
0531-120854.727-I-(WebSocketSession) - WSOCK: {instr='TRY=', ASK=8.5063, BID=8.4963}
0531-120854.976-I-(WebSocketSession) - WSOCK: {instr='XAU=', ASK=1905.9, BID=1904.5}
0531-120855.527-I-(WebSocketSession) - WSOCK: {instr='TRY=', ASK=8.507, BID=8.4997}
// after then we do not get either any prices or PING message!
At 12:09:09 we detected that server did not send any PING message more than 30 seconds. So we disconnected connection and reconnected.
So here there is data loss due to server not sending prices until we reconnect.
0531-120909.001 -W-(WebSocketSession) - WSOCK: Server did not send PING for 31 seconds (>30). Closing WebSocket...
0531-120909.002-I-(WebSocketSession) - WSOCK: Close received. Disconnecting WebSocket...
0531-120919.013-W-(WebSocketSession) - WSOCK: Disconnected. closedByServer=false, closeFrame=WebSocketFrame(FIN=1,RSV1=0,RSV2=0,RSV3=0,Opcode=CLOSE,Length=47,CloseCode=1000,Reason="Server did not send PING for 31 seconds (>30)")
0531-120919.354-I-(WebSocketSession) - WSOCK: Connect received.
0531-120920.126-I-(WebSocketSession) - WSOCK: Successfully connected.
…. // login steps
Environment #1:
Uses Library: nv-websocket-client
+ Updated client code program: We did not close server connection after 30 seconds PING timeout. WebSocket library enforced to disconnect from the server after 2 minutes.
Websocket library reported this case with message: No more WebSocketframe from the server
Here is the details (I have openned our debug level messages also, so can see server side raw messages):
…
0531-143552.463-I-(WebSocketSession) - WSOCK: {instr='XAU=', ASK=1905.15, BID=1904.64}
0531-143552.597-I-(WebSocketSession) - WSOCK: Received message: [{"ID":2,"Type":"Update","UpdateType":"Unspecified","DoNotConflate":true,"Key":{"Service":"ELEKTRON_DD","Name":"TRY="},"SeqNumber":29838,"Fields":{"BID":8.5037,"ASK":8.5109}}]
0531-143552.597-I-(WebSocketSession) - WSOCK: {instr='TRY=', ASK=8.5109, BID=8.5037}
0531-143553.786-I-(WebSocketSession) - WSOCK: Received message: [{"ID":2,"Type":"Update","UpdateType":"Unspecified","DoNotConflate":true,"Key":{"Service":"ELEKTRON_DD","Name":"TRY="},"SeqNumber":29902,"Fields":{"BID":8.5038,"ASK":8.5108}}]
0531-143553.786-I-(WebSocketSession) - WSOCK: {instr='TRY=', ASK=8.5108, BID=8.5038}
0531-143554.492-I-(WebSocketSession) - WSOCK: Received message: [{"Type":"Ping"}]
0531-143554.492-I-(WebSocketSession) - WSOCK: Sending Pong message
0531-143554.492-I-(WebSocketSession) - WSOCK: Sending json message: {"Type":"Pong"}
0531-143555.712-I-(WebSocketSession) - WSOCK: Received message: [{"ID":3,"Type":"Update","UpdateType":"Unspecified","DoNotConflate":true,"Key":{"Service":"ELEKTRON_DD","Name":"XAU="},"SeqNumber":44366,"Fields":{"BID":1904.5035,"ASK":1905.4966}}]
0531-143555.712-I-(WebSocketSession) - WSOCK: {instr='XAU=', ASK=1905.4966, BID=1904.5035}
0531-143556.572-I-(WebSocketSession) - WSOCK: Received message: [{"ID":3,"Type":"Update","UpdateType":"Unspecified","DoNotConflate":true,"Key":{"Service":"ELEKTRON_DD","Name":"XAU="},"SeqNumber":44382,"Fields":{"BID":1904.6400,"ASK":1905.1500}}]
0531-143556.573-I-(WebSocketSession) - WSOCK: {instr='XAU=', ASK=1905.15, BID=1904.64}
0531-143557.372-I-(WebSocketSession) - WSOCK: Received message: [{"ID":3,"Type":"Update","UpdateType":"Unspecified","DoNotConflate":true,"Key":{"Service":"ELEKTRON_DD","Name":"XAU="},"SeqNumber":44398,"Fields":{"BID":1904.5000,"ASK":1905.5000}}]
0531-143557.372-I-(WebSocketSession) - WSOCK: {instr='XAU=', ASK=1905.5, BID=1904.5}
0531-143557.569-I-(WebSocketSession) - WSOCK: Received message: [{"ID":3,"Type":"Update","UpdateType":"Unspecified","DoNotConflate":true,"Key":{"Service":"ELEKTRON_DD","Name":"XAU="},"SeqNumber":44414,"Fields":{"BID":1904.6300,"ASK":1905.1400}}]
0531-143557.569-I-(WebSocketSession) - WSOCK: {instr='XAU=', ASK=1905.14, BID=1904.63}
0531-143558.396-I-(WebSocketSession) - WSOCK: Received message: [{"ID":2,"Type":"Update","UpdateType":"Unspecified","DoNotConflate":true,"Key":{"Service":"ELEKTRON_DD","Name":"TRY="},"SeqNumber":29966,"Fields":{"BID":8.5039,"ASK":8.5107}}]
0531-143558.396-I-(WebSocketSession) - WSOCK: {instr='TRY=', ASK=8.5107, BID=8.5039}
0531-143558.983-I-(WebSocketSession) - WSOCK: Received message: [{"ID":3,"Type":"Update","UpdateType":"Unspecified","DoNotConflate":true,"Key":{"Service":"ELEKTRON_DD","Name":"XAU="},"SeqNumber":44430,"Fields":{"BID":1904.6400,"ASK":1905.1500}}]
0531-143558.983-I-(WebSocketSession) - WSOCK: {instr='XAU=', ASK=1905.15, BID=1904.64}
0531-143559.402-I-(WebSocketSession) - WSOCK: Received message: [{"ID":3,"Type":"Update","UpdateType":"Unspecified","DoNotConflate":true,"Key":{"Service":"ELEKTRON_DD","Name":"XAU="},"SeqNumber":44446,"Fields":{"BID":1904.6300,"ASK":1905.1400}}]
0531-143559.402-I-(WebSocketSession) - WSOCK: {instr='XAU=', ASK=1905.14, BID=1904.63}
0531-143600.382-I-(WebSocketSession) - WSOCK: Received message: [{"ID":3,"Type":"Update","UpdateType":"Unspecified","DoNotConflate":true,"Key":{"Service":"ELEKTRON_DD","Name":"XAU="},"SeqNumber":44462,"Fields":{"BID":1904.6400,"ASK":1905.1500}}]
0531-143600.382-I-(WebSocketSession) - WSOCK: {instr='XAU=', ASK=1905.15, BID=1904.64}
0531-143600.583-I-(WebSocketSession) - WSOCK: Received message: [{"ID":3,"Type":"Update","UpdateType":"Unspecified","DoNotConflate":true,"Key":{"Service":"ELEKTRON_DD","Name":"XAU="},"SeqNumber":44478,"Fields":{"BID":1904.6400,"ASK":1905.1500}}]
0531-143600.583-I-(WebSocketSession) - WSOCK: {instr='XAU=', ASK=1905.15, BID=1904.64}
0531-143600.672-I-(WebSocketSession) - WSOCK: Received message: [{"ID":3,"Type":"Update","UpdateType":"Unspecified","DoNotConflate":true,"Key":{"Service":"ELEKTRON_DD","Name":"XAU="},"SeqNumber":44494,"Fields":{"BID":1904.4896,"ASK":1905.4858}}]
0531-143600.673-I-(WebSocketSession) - WSOCK: {instr='XAU=', ASK=1905.4858, BID=1904.4896}
0531-143601.122-I-(WebSocketSession) - WSOCK: Received message: [{"ID":3,"Type":"Update","UpdateType":"Unspecified","DoNotConflate":true,"Key":{"Service":"ELEKTRON_DD","Name":"XAU="},"SeqNumber":44510,"Fields":{"BID":1904.5600,"ASK":1905.3300}}]
0531-143601.122-I-(WebSocketSession) - WSOCK: {instr='XAU=', ASK=1905.33, BID=1904.56}
0531-143601.432-I-(WebSocketSession) - WSOCK: Received message: [{"ID":3,"Type":"Update","UpdateType":"Unspecified","DoNotConflate":true,"Key":{"Service":"ELEKTRON_DD","Name":"XAU="},"SeqNumber":44526,"Fields":{"BID":1904.6400,"ASK":1905.1500}}]
0531-143601.433-I-(WebSocketSession) - WSOCK: {instr='XAU=', ASK=1905.15, BID=1904.64}
0531-143602.662-I-(WebSocketSession) - WSOCK: Received message: [{"ID":3,"Type":"Update","UpdateType":"Unspecified","DoNotConflate":true,"Key":{"Service":"ELEKTRON_DD","Name":"XAU="},"SeqNumber":44542,"Fields":{"BID":1904.6500,"ASK":1905.1600}}]
0531-143602.662-I-(WebSocketSession) - WSOCK: {instr='XAU=', ASK=1905.16, BID=1904.65}
0531-143625.000-W-(WebSocketSession) - WSOCK: Server did not send PING for 31 seconds (>30).
0531-143626.001-W-(WebSocketSession) - WSOCK: Server did not send PING for 32 seconds (>30).
0531-143627.000-W-(WebSocketSession) - WSOCK: Server did not send PING for 33 seconds (>30).
0531-143628.001-W-(WebSocketSession) - WSOCK: Server did not send PING for 34 seconds (>30).
0531-143629.001-W-(WebSocketSession) - WSOCK: Server did not send PING for 35 seconds (>30).
… // deleted “Server did not send PING for XXX seconds (>30).” logs for simplicity
0531-143806.001-W-(WebSocketSession) - WSOCK: Server did not send PING for 132 seconds (>30).
0531-143807.000-W-(WebSocketSession) - WSOCK: Server did not send PING for 133 seconds (>30).
0531-143808.000-W-(WebSocketSession) - WSOCK: Server did not send PING for 134 seconds (>30).
0531-143808.694-W-(WebSocketSession) - WSOCK: Disconnected. closedByServer=false, closeFrame=WebSocketFrame(FIN=1,RSV1=0,RSV2=0,RSV3=0,Opcode=CLOSE,Length=42,CloseCode=1002,Reason="No more WebSocketframe from the server.")
0531-143809.057-I-(WebSocketSession) - WSOCK: Connect received. Started to close WebSocket...
0531-143809.057-I-(WebSocketSession) - WSOCK: Recreating WebSocket.
0531-143809.058-I-(WebSocketSession) - WSOCK: Connecting WebSocket...
…
Environment #1:
Uses Library: nv-websocket-client
+ Updated client code program: We did not close server connection after 30 seconds PING timeout. Instead we sent PONG messages secondly. But after ~11 minutes WebSocket library enforced to disconnect from the server.
Websocket library reported this case with message: An I/O error occurred while a frame was being read from the web socket: Connection reset
Here is the details (I have openned our debug level messages also, so can see server side raw messages):
531-150931.001-I-(WebSocketSession) - WSOCK: Sending json message: {"Type":"Pong"}
0531-150932.000-W-(WebSocketSession) - WSOCK: Server did not send PING for 374 seconds (>30). Sending PONG message...
0531-150932.001-I-(WebSocketSession) - WSOCK: Sending Pong message
0531-150932.001-I-(WebSocketSession) - WSOCK: Sending json message: {"Type":"Pong"}
0531-150933.001-W-(WebSocketSession) - WSOCK: Server did not send PING for 375 seconds (>30). Sending PONG message...
0531-150933.001-I-(WebSocketSession) - WSOCK: Sending Pong message
0531-150933.001-I-(WebSocketSession) - WSOCK: Sending json message: {"Type":"Pong"}
0531-150934.002-W-(WebSocketSession) - WSOCK: Server did not send PING for 376 seconds (>30). Sending PONG message...
0531-150934.002-I-(WebSocketSession) - WSOCK: Sending Pong message
0531-150934.002-I-(WebSocketSession) - WSOCK: Sending json message: {"Type":"Pong"}
0531-150935.001-W-(WebSocketSession) - WSOCK: Server did not send PING for 377 seconds (>30). Sending PONG message...
0531-150935.001-I-(WebSocketSession) - WSOCK: Sending Pong message
0531-150935.001-I-(WebSocketSession) - WSOCK: Sending json message: {"Type":"Pong"}
0531-150936.000-W-(WebSocketSession) - WSOCK: Server did not send PING for 378 seconds (>30). Sending PONG message...
0531-150936.000-I-(WebSocketSession) - WSOCK: Sending Pong message
0531-150936.001-I-(WebSocketSession) - WSOCK: Sending json message: {"Type":"Pong"}
0531-150937.001-W-(WebSocketSession) - WSOCK: Server did not send PING for 379 seconds (>30). Sending PONG message...
0531-150937.001-I-(WebSocketSession) - WSOCK: Sending Pong message
0531-150937.001-I-(WebSocketSession) - WSOCK: Sending json message: {"Type":"Pong"}
0531-150938.001-W-(WebSocketSession) - WSOCK: Server did not send PING for 380 seconds (>30). Sending PONG message...
0531-150938.001-I-(WebSocketSession) - WSOCK: Sending Pong message
0531-150938.002-I-(WebSocketSession) - WSOCK: Sending json message: {"Type":"Pong"}
0531-150939.001-W-(WebSocketSession) - WSOCK: Server did not send PING for 381 seconds (>30). Sending PONG message...
0531-150939.001-I-(WebSocketSession) - WSOCK: Sending Pong message
0531-150939.001-I-(WebSocketSession) - WSOCK: Sending json message: {"Type":"Pong"}
0531-150940.001-W-(WebSocketSession) - WSOCK: Server did not send PING for 382 seconds (>30). Sending PONG message...
0531-150940.002-I-(WebSocketSession) - WSOCK: Sending Pong message
0531-150940.002-I-(WebSocketSession) - WSOCK: Sending json message: {"Type":"Pong"}
0531-150941.001-W-(WebSocketSession) - WSOCK: Server did not send PING for 383 seconds (>30). Sending PONG message...
0531-150941.001-I-(WebSocketSession) - WSOCK: Sending Pong message
0531-150941.001-I-(WebSocketSession) - WSOCK: Sending json message: {"Type":"Pong"}
0531-150942.002-W-(WebSocketSession) - WSOCK: Server did not send PING for 384 seconds (>30). Sending PONG message...
0531-150942.002-I-(WebSocketSession) - WSOCK: Sending Pong message
0531-150942.002-I-(WebSocketSession) - WSOCK: Sending json message: {"Type":"Pong"}
0531-150943.001-W-(WebSocketSession) - WSOCK: Server did not send PING for 385 seconds (>30). Sending PONG message...
0531-150943.001-I-(WebSocketSession) - WSOCK: Sending Pong message
0531-150943.001-I-(WebSocketSession) - WSOCK: Sending json message: {"Type":"Pong"}
0531-150944.000-W-(WebSocketSession) - WSOCK: Server did not send PING for 386 seconds (>30). Sending PONG message...
0531-150944.001-I-(WebSocketSession) - WSOCK: Sending Pong message
0531-150944.001-I-(WebSocketSession) - WSOCK: Sending json message: {"Type":"Pong"}
0531-150945.000-W-(WebSocketSession) - WSOCK: Server did not send PING for 387 seconds (>30). Sending PONG message...
0531-150945.001-I-(WebSocketSession) - WSOCK: Sending Pong message
0531-150945.001-I-(WebSocketSession) - WSOCK: Sending json message: {"Type":"Pong"}
0531-150945.142-W-(WebSocketSession) - WSOCK: Disconnected. closedByServer=false, closeFrame=WebSocketFrame(FIN=1,RSV1=0,RSV2=0,RSV3=0,Opcode=CLOSE,Length=90,CloseCode=1008,Reason="An I/O error occurred while a frame was being read from the web socket: Connection reset")
0531-150945.142-W-(WebSocketSession) - WSOCK: State changing from LOGGEDIN to CLOSED
0531-150945.147-I-(WebSocketSession) - WSOCK: Connect received. Started to close WebSocket...
0531-150945.147-I-(WebSocketSession) - WSOCK: Recreating WebSocket.
0531-150945.148-W-(WebSocketSession) - WSOCK: State changing from CLOSED to CONNECTING
0531-150945.148-I-(WebSocketSession) - WSOCK: Connecting WebSocket...
0531-150945.282-I-(DsDbFeedThread) - sendAlarm 177950 Feed id (1) is down/disconnected
0531-150947.437-I-(WebSocketSession) - WSOCK: Finished closing WebSocket.
0531-150947.437-I-(WebSocketSession) - WSOCK: Successfully connected.
Environment #2:
Uses Library: Java-WebSocket <=== This WebSocket library is new for this test case
In this test when we have detected that server did not send PING message for 30 seconds and we disconnected from the server programmatically.
As seen below, latest PING message is received at 12:48:41 and prices are received until 12:48:58 from server.
At 12:49:12 we detected that server did not send any PING message more than 30 seconds. So we disconnected connection and reconnected.
So here there is data loss due to server not sending prices until we reconnect.
0531-124821.929-I-(WebSocketSession) - WSOCK: Sending Pong message ß PING RECEIVED AND WE RESPONDED WITH PONG
// prices flow normally
0531-124841.929-I-(WebSocketSession) - WSOCK: Sending Pong message ß LATEST PING RECEIVED AND WE RESPONDED WITH PONG
// prices flow until stopped at highlighted part
0531-124854.261-I-(WebSocketSession) - WSOCK: {instr='XAU=', ASK=1906.7, BID=1903.9}
0531-124854.301-I-(WebSocketSession) - WSOCK: {instr='XAU=', ASK=1905.61, BID=1905.1}
0531-124854.462-I-(WebSocketSession) - WSOCK: {instr='XAU=', ASK=1905.63, BID=1905.12}
0531-124855.286-I-(WebSocketSession) - WSOCK: {instr='TRY=', ASK=8.4893, BID=8.4829}
0531-124855.286-I-(WebSocketSession) - WSOCK: {instr='TRY=', ASK=8.4893, BID=8.4829}
0531-124855.712-I-(WebSocketSession) - WSOCK: {instr='XAU=', ASK=1905.8527, BID=1904.7554}
0531-124855.756-I-(WebSocketSession) - WSOCK: {instr='TRY=', ASK=8.4904, BID=8.4822}
0531-124856.261-I-(WebSocketSession) - WSOCK: {instr='XAU=', ASK=1905.8, BID=1904.9}
0531-124857.022-I-(WebSocketSession) - WSOCK: {instr='XAU=', ASK=1905.85, BID=1904.76}
0531-124857.275-I-(WebSocketSession) - WSOCK: {instr='TRY=', ASK=8.4892, BID=8.4832}
0531-124857.485-I-(WebSocketSession) - WSOCK: {instr='TRY=', ASK=8.4902, BID=8.4825}
0531-124858.182-I-(WebSocketSession) - WSOCK: {instr='XAU=', ASK=1905.73, BID=1904.96}
0531-124858.272-I-(WebSocketSession) - WSOCK: {instr='XAU=', ASK=1906.0, BID=1904.7}
// after then we do not get either any prices or PING message!
0531-124912.001-W-(WebSocketSession) - WSOCK: Server did not send PING for 31 seconds (>30). Closing WebSocket...
0531-124912.002-I-(WebSocketSession) - WSOCK: Close received. Disconnecting WebSocket...
0531-124931.137-W-(WebSocketSession) - WSOCK: Disconnected. closedByServer=false, code=1000, reason: Server did not send PING for 31 seconds (>30)
0531-124931.151-I-(WebSocketSession) - WSOCK: Connect received.
0531-124931.151-I-(WebSocketSession) - WSOCK: Reconnecting WebSocket...
0531-125046.228-I-(WebSocketSession) - WSOCK: Successfully connected.
…. // login steps