I'm building a system that consumes live tick data over WebSocket, and I want to make sure I'm handling connection drops correctly before I run into issues in production. A few things I'm unsure about: 1. When the connection drops and reconnects, is there any way to recover the ticks I missed during the disconnect? Or is that data just gone once the socket closes? 2. What's a sensible reconnection strategy? Right now my code just tries to reconnect immediately in a loop if the connection drops. Is that a bad idea? 3. After reconnecting, should I trust my in-memory positions/orders as they were before the disconnect, or should I be re-fetching them from somewhere? 4. Is there a risk of the connection appearing "open" but not actually receiving any data? If so, how would I even detect that, since it wouldn't show up as a disconnect error? Would appreciate insights from anyone who has run this kind of setup in production, especially around expiry days or high-volatility periods when this stuff seems to matter most.
The thing to be clear about first: the tick stream is not a replayable log. There is no sequence number you can resume from, so anything you miss during a disconnect is gone rather than delayed. Any design that assumes otherwise will produce gaps that look like bugs elsewhere in your system. Which means reconnection handling has to do two jobs, not one. Re-establish the socket, and reconcile state from a source that is authoritative. For reconnection: exponential backoff with jitter, and a cap. Reconnecting instantly in a tight loop during an exchange-side issue is how you turn a five second outage into a rate limit. For reconciliation: on every successful reconnect, re-fetch positions and orders rather than assuming your in-memory view survived. It is a cheap call and it is the difference between a gap and a wrong position. Also worth handling separately: a socket that is open but silent. A connection can stay technically alive while delivering nothing. A watchdog that forces a reconnect after N seconds without a tick during market hours catches this, and it is the failure mode people usually discover on an expiry day.
Which means reconnection handling has to do two jobs, not one. Re-establish the socket, and reconcile state from a source that is authoritative.
For reconnection: exponential backoff with jitter, and a cap. Reconnecting instantly in a tight loop during an exchange-side issue is how you turn a five second outage into a rate limit.
For reconciliation: on every successful reconnect, re-fetch positions and orders rather than assuming your in-memory view survived. It is a cheap call and it is the difference between a gap and a wrong position.
Also worth handling separately: a socket that is open but silent. A connection can stay technically alive while delivering nothing. A watchdog that forces a reconnect after N seconds without a tick during market hours catches this, and it is the failure mode people usually discover on an expiry day.