I'm subscribing to 9000 instrument_tokens today and want to verify that no ticks were dropped. Would someone from the community or Kite team be able to share the expected total tick count for these instruments so I can cross-validate?
I can share a complete list of instrument_token via pastebin or DM.
My setup:
50 Nifty 50 EQ stocks
~8,900 option contracts (CEs and PEs) for the current expiry
50 Nifty 50 FUT
Total: 9000 instruments
Observed data:
By 9:25 AM IST (~10 minutes after market open), I received 498586 ticks in MODE_FULL
Does this tick count seem reasonable, or might some ticks have been dropped?
No of ticks(no. of rows after reducing it to 1 min ohlc) I received yesterday for 8326 instruments: 779953 My list of instruments is obviously different(all futures, all liquid equity, all nifty banknifty finnity options in NSE and only sensex options in BSE) Even I keep wondering if ticks are dropped etc
@vmx for me total raw ticks for the day: 56-62 million After 1 min ohlc sampling: 800k to 1 million (much less than your 2 million plus) total instruments: around 8400 on average
It depends I guess, few liquid instruments(especially indexes and their options) tend to get more updates per second than the other. I do not have use case for indexes at the moment, but if I get time will try to stream them or maybe to validate if things are comparable.
It isn’t possible to share an expected tick count for a trading day, as tick generation is event-driven and varies with liquidity and market activity.
Kite’s WebSocket provides snapshot-based market data, so there is no deterministic way to verify that every intermediate update was received or to validate completeness using a total tick count.
When subscribing to a large number of instruments, ensure the WebSocket callback is non-blocking. Avoid heavy processing or database writes in the streaming thread; instead, queue ticks and process them asynchronously to reduce the risk of dropped updates.
While lossless capture cannot be guaranteed, material data loss can be detected indirectly by aggregating ticks into candles and comparing volume and price ranges (high/low) with historical candle data by EOD. Significant deviations can indicate missed updates, even though exact tick-for-tick matching is not expected.
Noted. Having a ballpark number for a given day (even historical data) would help us rule out potential blocking issues in the code. While I've tried my best to decouple that (same approach you've suggested above), I just wanted to be sure.
I agree with your validation approach and I'll get started with that. For "depth" specifically, ensuring we actually received the data might be necessary, since a change in depth also constitutes a push from the exchange.
*btw, I do visualize the results and they look fine to me - I'm getting all data from 9:15 AM to 3:30 PM.
No of ticks(no. of rows after reducing it to 1 min ohlc) I received yesterday for 8326 instruments: 779953
My list of instruments is obviously different(all futures, all liquid equity, all nifty banknifty finnity options in NSE and only sensex options in BSE)
Even I keep wondering if ticks are dropped etc
Raw ticks
totalling 14263417 ticks today.OHLC 1 min aggregation result
totalling 2087297.I was also hoping maybe someone from the Kite team could chime in on this one.
After 1 min ohlc sampling: 800k to 1 million (much less than your 2 million plus)
total instruments: around 8400 on average
I do not have use case for indexes at the moment, but if I get time will try to stream them or maybe to validate if things are comparable.
Thank you for your update!
Kite’s WebSocket provides snapshot-based market data, so there is no deterministic way to verify that every intermediate update was received or to validate completeness using a total tick count.
When subscribing to a large number of instruments, ensure the WebSocket callback is non-blocking. Avoid heavy processing or database writes in the streaming thread; instead, queue ticks and process them asynchronously to reduce the risk of dropped updates.
While lossless capture cannot be guaranteed, material data loss can be detected indirectly by aggregating ticks into candles and comparing volume and price ranges (high/low) with historical candle data by EOD. Significant deviations can indicate missed updates, even though exact tick-for-tick matching is not expected.
Noted. Having a ballpark number for a given day (even historical data) would help us rule out potential blocking issues in the code. While I've tried my best to decouple that (same approach you've suggested above), I just wanted to be sure.
I agree with your validation approach and I'll get started with that. For "depth" specifically, ensuring we actually received the data might be necessary, since a change in depth also constitutes a push from the exchange.
*btw, I do visualize the results and they look fine to me - I'm getting all data from 9:15 AM to 3:30 PM.