This section describes CQG recommended recovery processes. Based on the recovery and supplemental methods, client systems can choose to implement those methods as applicable. In general, two paths can be followed: processing while recovering and queuing while recovering.
Note: The recovering data process applies to affected channels only. Unaffected channels can continue normal processing.
The following sections provide a high level process to recover in various situations:
•Large Scale Outage Using Snapshots - Queuing
•Large Scale Outage Using Snapshots - Concurrent Processing
•Small Scale Data Recovery Using TCP Replay -Queuing
•Small Scale Data Recovery Using TCP Replay -Concurrent Processing
Note: TCP replay should be used for small-scale data recovery. The Snapshots feed should be used for large-scale data recovery (i.e. major outage or late joiners).
Large Scale Outage Using Snapshots - Queuing
This section describes at a high level the process to follow for a large scale outage in which the client system is out of sync. This process uses Snapshots - queuing of the Incremental Market Data feed and Snapshots feed until the client system is synchronized to the latest state advertised by CQG. In order to avoid an excessive number of queued messages, process snapshots and apply the applicable incremental feed as the snapshots arrive.
Note: If the Snapshots method is used, client systems also need to subscribe to the Instrument Definition feed to determine whether any new instruments were defined. Client systems will not recover any missed statistics on the Snapshots feed.
1. Identify channel(s) in which the client system is out of sync.
2. Listen to and queue the Incremental Market Data feed for the affected channel(s).
3. Listen to the Snapshots feed for the affected channel(s).
4. Verify that all snapshots have been received for a given Snapshots feed.
5. Apply all incremental data in sequence, where tag 34-MsgSeqNum is greater than the lowest value for tag 369-LastMsgSeqNumProcessed.
6. Using queued data, restart normal processing.
Large Scale Outage Using Snapshots - Concurrent Processing
This section describes at a high level the process to follow for a large scale outage using Snapshots while continuing to process the Incremental Market Data feed and obtaining snapshots from the Snapshots feed. Once books are recovered, they can resume normal processing even if other books are still being recovered.
Note: If the Snapshots method is used, client systems also need to subscribe to the Instrument Definition feed to determine whether any new instruments were defined. Client systems will not recover any missed statistics on the Snapshots feed.
1. Identify channel(s) in which the client system is out of sync.
2. Listen to the Incremental Market Data feed for the affected channel(s) and optionally attempt a natural refresh.
3. Listen to the Snapshots feed for the affected channel(s).
4. For each book, compare tag 369-LastMsgSeqNumProcessed on the Snapshots feed to tag 34-MsgSeqNum on the Incremental Market Data feed and verify that the value for tag34-MsgSeqNum is not lower.
5. Restart normal processing.
Small Scale Data Recovery Using TCP Replay - Queuing
This section describes at a high level the process to follow for small scale data recovery using TCP Replay and queuing the Incremental Market Data feed until all missed messages are recovered. Client systems can recover specific messages that were missed using the sequence number and the TCP historical replay component.
1. Identify which messages have been missed (tag 34-MsgSeqNum).
2. Stop normal processing and initiate data recovery for affected channel(s).
3. Listen to and queue the Incremental Market Data feed for the affected channel(s).
4. Login to the TCP Replay component and request a replay of missed messages.
5. Apply resent messages in proper sequence.
6. Apply queued messages once replayed messages have been processed.
7. Stop queuing messages and restart normal processing.
Small Scale Data Recovery Using TCP Replay - Concurrent Processing
This section describes at a high level the process to follow for small scale data recovery using TCP Replay while continuing to process the Incremental Market Data feed. Client systems can recover specific messages that were missed using the sequence number and the TCP historical replay component.
1. Identify which messages have been missed (tag 34-MsgSeqNum).
2. Listen to and process the Incremental Market Data feed.
3. Login to the TCP Replay component and request a replay of missed messages.
4. If using Natural Refresh or Instrument Level Sequencing on the Incremental Market Data feed, refer to the following flowchart and apply to process.
5. For queued messages (see diagram):
o Apply resent messages in proper sequence.
o Apply queued messages once replayed messages have been processed.
6. Process missed messages from the TCP Replay component.
7. Books will get resynchronized either due to message processing algorithm or data recovery.
8. Restart normal processing.
