This thread should be addressed by kiteconnect developers only.
While streaming data through WebSockets, I noticed data are being missed periodically! I calculated the delay(lag) in receiving the data with the local machine time. Even if the local machine clock is not synchronized with the the exchange clock, the gap would be a FIXED interval. However, as can be seen below, the delay(lag) is increasing with every consecutive data and when the delay goes over 1 sec, 1 data stream goes missing. Note: I calculated the delay in the 2nd data(with the same secondstamp) as 500 milliseconds apart from the first one.
BANKNIFTY22JUL37000CE; Date-2022Jul27
For BANKNIFTY option, 500 multiple strike (around the ATM spot) on Wednesday/Thursday; How many total number of data you receive at your server, is it 45000 or lower than that? What is the DELAY(lag) in receiving the data at your server? Do you see the same pattern in DELAY as in the image file, I attached? Have you matched the two data, one that you receive from the exchange at your server and the other by a local machine that is connected through WebSocket over the internet, using the code posted in https://pypi.org/project/kiteconnect/ ? How much is the increase in the 'delay'(lag) in this case (on an average, in terms of percentage)?
There are a couple of observations, I can make from your above-posted image : 1. Your calculation of delay i.e local time w.r.t exchange timestamp is flawed as the exchange timestamp does not have ms precision. 2. If you refer to your local time you can see a difference ranging from 300-600ms for successive ticks. Which is fine.
How many total number of data you receive at your server, is it 45000 or lower than that?
We don't log these tick data. It's streamed over Websocket.
What is the DELAY(lag) in receiving the data at your server? Do you see the same pattern in DELAY as in the image file, I attached?
I had written to kiteconnect previously, didn't get helpful response! Coming back to the question; It's a common practice, in such scenario, to match the receiving end (i.e. your server) data and the local machine (connected through websocket) data to check nothing going wrong in-between unless you have already done so. If you have already done it, then it's a problem with NSE. You may notify them or raise the issue with them. Total no of data should be 45000 based on the pattern, data is being received. Local machine is receiving 37000-38000 data. So if could count the data at your receiving server, then it would be a good check to figure out whether data are missing. Regarding the millisecond issue, I've assumed the 2nd data, with the same secondstamp, to be 500 millis apart from the 1st data. The 1st data of the same secondstamp is still at that second. Then how do you explain it's (the 1st data) increasing delay(lag)?
Total no of data should be 45000 based on the pattern, data is being received
What pattern? As explained before by @sujith in this thread, ticker follow no fixed timestamp pattern. Tick is generated only if a trade/market depth change happens. And it's not possible to predict this count over level 2 data. The successive tick timestamp difference of 300-600ms is fine.
Pattern is 2 data per sec. If you look carefully at the delay column, you'll see when the delay(lag) is below 1sec, there is 2data/sec and when the delay(lag) is about to overshoot 1sec that's when there is 1data/sec. (Even if you ignore the 2nd data of a particular timestamp, the 1st data is still at that sec. And still it's delay is increasing). This is happening periodically throughout the day. Besides have you ever matched the data between receiving end (i.e. your server) and the local machine (connected through websocket) ? See unless you do the matching check and tell me everything is fine at your end, I'll remain unconvinced. I've asked this to @sujith as well! As a developer you should always match the two data. This is the only way to ensure there is nothing wrong at your end!
tick data is thrown when there is a new transaction. a order got executed, a new order was placed, an order was cancelled. if there is no new transaction, what is the point of a tick?
now consider, a new transaction got complete WHILE you were receiving a tick. means, a tick was being transmitted, when a new transaction happened. to capture those data, tick by tick data you need a broker leased line connection directly to NSE and systems fast enough to capture multiple ticks in sub ms.
1. Your calculation of delay i.e local time w.r.t exchange timestamp is flawed as the exchange timestamp does not have ms precision.
2. If you refer to your local time you can see a difference ranging from 300-600ms for successive ticks. Which is fine. We don't log these tick data. It's streamed over Websocket. Answered above, point 2.
Coming back to the question;
It's a common practice, in such scenario, to match the receiving end (i.e. your server) data and the local machine (connected through websocket) data to check nothing going wrong in-between unless you have already done so. If you have already done it, then it's a problem with NSE. You may notify them or raise the issue with them.
Total no of data should be 45000 based on the pattern, data is being received. Local machine is receiving 37000-38000 data. So if could count the data at your receiving server, then it would be a good check to figure out whether data are missing.
Regarding the millisecond issue, I've assumed the 2nd data, with the same secondstamp, to be 500 millis apart from the 1st data. The 1st data of the same secondstamp is still at that second. Then how do you explain it's (the 1st data) increasing delay(lag)?
The successive tick timestamp difference of 300-600ms is fine.
If you look carefully at the delay column, you'll see when the delay(lag) is below 1sec, there is 2data/sec and when the delay(lag) is about to overshoot 1sec that's when there is 1data/sec. (Even if you ignore the 2nd data of a particular timestamp, the 1st data is still at that sec. And still it's delay is increasing). This is happening periodically throughout the day.
Besides have you ever matched the data between receiving end (i.e. your server) and the local machine (connected through websocket) ?
See unless you do the matching check and tell me everything is fine at your end, I'll remain unconvinced. I've asked this to @sujith as well! As a developer you should always match the two data. This is the only way to ensure there is nothing wrong at your end!
now consider, a new transaction got complete WHILE you were receiving a tick. means, a tick was being transmitted, when a new transaction happened. to capture those data, tick by tick data you need a broker leased line connection directly to NSE and systems fast enough to capture multiple ticks in sub ms.