While streaming data through WebSockets, I noticed data are being omitted periodically! I calculated the delay in receiving the data with the local machine time. Even if the local machine clock is not synchronized with the the exchange clock, the gap would be a FIXED interval. However, as can be seen below, the delay is increasing with every consecutive data and when the delay goes over 1 sec, 1 data stream goes missing. Does anyone faced this problem? How can this be fixed? On another note if you are receiving the millisecond in the timestamp, please provide it. Otherwise it seems two consecutive data is provided at the same time, which is not possible! Note: I calculated the delay in the 2nd data(with the same secondstamp) as 500 milliseconds apart from the first one.
Its not necessary that there will a fixed interval between 2 ticks. You receive a tick when there is a change in any of the fields present in the mode subscribed by you.
If nothing changes,you won't receive any tick.
Do you have any explanation for the delay, why it is PERIODICALLY increasing and then coming down when one (sometimes two) data goes missing! This is 500 multiple BANKNIFTY out of the money options. Total no of TICKS in a day is few crores. So it is unthinkable, nothing happens for 500 millisecond/1sec, that too PERIODICALLY for the whole day in the first 5 levels. I dealt with TBs of BANKNIFTY options data in the last 5 years. I recognize problem in data when I see one. I worked in a trading firm. The firm was running 10-12 years prior to I joined there. After some experience, when I recognize problem in data, the programmers refused to listened to me, same way you are doing. I had to go to the management to get the programmers to have a second look in the data. And the problem was found & fixed. Unfortunately I do not have access to the management here. The reason I'm streaming kite data is that, I left the firm and wanted to trade independently. Looks like, with such a stubborn infra support (who refuse to listen to an experienced researcher/trader, even though I showed where and how many data are missing) its not going to happen!
Also,you are thinking that data is missing because you are expecting a fixed pattern to be repeated which is not the case in real life scenarios.
You are right that it might be possible,that hundreds of trades might have happened in those 500 ms. But , Zerodha will get only 1 or 2 ticks per second. And the same is relayed to you by KiteConnect Websocket API.
There can't be a problem with KiteConnect.
Because, KiteConnect just streams what they get from the exchange without any modification from their side.
Let's assume for once, even if there is a problem with the ticks,then that problem should be from the exchange stream itself.
Still if you think that there is any problem with KiteConnect,then know this that the websocket data is used by the charts on Kite too. KiteConnect,as the name suggests, is just a way to provide the data on Kite , programmatically to the users so they can code their strategies.
We only deal with technical queries related to KiteConnect APIs on this forum.
But ,I don't believe there's any issue.
Everything seems fine to me.
Even after all of this,if you're not satisfied with KiteConnect data,you can go for colocation setup at the exchange.
Zerodha can help you setup lease lines.
It costs upwards 18 lakhs/annum. You can write to kiteconnect(at)zerodha.com, if you are interested.
Someone from the relevant team will get back to you.
Thank you for your response. I have some concerns;
>> It's just random. It's not random. Whenever the DELAY is about to overshoot 1 sec or it has overshot 1 sec by a little bit, that's when the data goes missing. Check the "delay" column properly, you'll see this pattern. This pattern occurs throughout the day. Towards the bottom of the image file, if you see, wherever I wrote missing data, the levels are changed by more that 1 rupee. So it's not possible that there were no tick. In your response you haven't mentioned anything about the DELAY.
>> You are right that it might be possible,that hundreds of trades might have happened in those 500 ms. But , Zerodha will get only 1 or 2 ticks per second.
Kite receives 1 tick (on rare occasion, no tick at all) when the DELAY is overshooting 1 sec. Regarding the 2 ticks per second; if you are receiving the millisecondstamp in the feed, please provide it to us. Two different data occurring at the same timestamp is not possible. If I have to either chop some data or fill missing feed with dummy data, without the millisecondstamp, it seems quite difficult.
>> And the same is relayed to you by KiteConnect Websocket API. There can't be a problem with KiteConnect.
This you need to check (assuming you are a developer with kiteconnect) at your end where you are receiving the feed from NSE. Check the DELAY at your server properly and verify if the same pattern, as in the image I attached, is emerging. If the same pattern emerges, then you need to bring it to the notice of NSE. If the same pattern doesn't emerge, i.e. you receive all the data from the exchange, please check with the code https://pypi.org/project/kiteconnect/. Because I copied the code from there.
Zerodha support is barely helpful. Some HR guy looks into it and tries to answer a highly technical question!
>> But ,I don't believe there's any issue. Everything seems fine to me.
Programmers rarely capture such issue with the data until some researcher/analyst bring it to there notice (in my experience). As a programmer, you said it's random. As a researcher, I noticed the pattern; whenever the delay is/about to overshoot 1 sec that's when the data goes missing.
>> Even after all of this,if you're not satisfied with KiteConnect data,you can go for colocation setup at the exchange. Zerodha can help you setup lease lines. It costs upwards 18 lakhs/annum. You can write to kiteconnect(at)zerodha.com, if you are interested. Someone from the relevant team will get back to you.
I did write to kiteconnect(at)zerodha.com couple of weeks back to rent co-located server. Sreenivasulu M responded to me saying no such service.
First,I am not actually a programmer ,I am a trader.
I just learnt some programming for algorithmic trading.
And I am just member of this forum like you,not officially related to Zerodha.
Second, you are making up things in your head.
You want to know the reason for delay??
There is NO DELAY. Delay means if there is a fixed timestamp/time period in which an event should happen,and it happens after that timestamp/period.
There is no fixed interval between two ticks ,so there is no question of delay.
Ok if you are not a developer with the kiteconnect then you should have never responded to this issue. Evidently you don't understand what DELAY is! I've 5 years professional experience handling TBs of BANKNIFTY option data. I've traded in 40 co-located servers simultaneously while in a trading firm. Just get this into your head, NO INFORMATION REACHES TO YOU INSTANTLY. Check the image file (difference between local time and exchange time).
First of all, this is a community driven forum.
Not a support forum.
Anyone can respond to a query or participate in a discussion if he/she wants to.
I understand that information doesn't reach instantly. It's just that there is no fixed interval in which the information will reach,or between 2 ticks.
Actually,your way of calculating the difference between the timestamps of two ticks is incorrect,which is resulting in this weird observation by you.
It is true it's a community driven forum, however if you do not have expertise in some issue or if you simply do not understand something, you should not unnecessarily increase the size of the thread and please do not make any more comment on this. I'm willing to wait for an expert to respond to this thread.
We only get level 2 data from the exchange which has timestamp precision till seconds and not milliseconds. Kite Connect doesn't guarantee 1 or 2 or 3 ticks in a second. We relay whatever is sent from the exchange, we don't do any kind of processing on the data or calculation on the data. It is not possible to capture all the ticks that happens on the exchange over the internet. You will have to contact any exchange registered data vendor or get a colocation setup at the exchange premises.
Kite Connect APIs are for retail users only. It is not a suitable product for time based or latency based strategies.
@sujith : Thank you for the response. Are you a developer with kiteconnect? Assuming you are, I'm asking to verify few checks and post the result here. For BANKNIFTY option, 500 multiple strike (around the ATM spot) on Wednesday/Thursday; How many total number of data you receive at your server, is it 45000 or lower than that? What is the DELAY in receiving the data at your server? Do you see the same pattern in DELAY as in the image file, I attached? Have you matched the two data, one that you receive from the exchange at your server and the other by a local machine that is connected through WebSocket over the internet, using the code posted in https://pypi.org/project/kiteconnect/ ? How much is the increase in the 'delay' in this case (on an average, in terms of percentage)?
@cskasm I have marked a portion on you image. Please have a look. The exchange time is missing milliseconds/microseconds. Your local time variable is having the fraction. The delay is the microseconds missing from the exchange time. I have no clue why the exchange time doesn't have milliseconds when it comes via kiteConnect. I have never worked in a firm so can't comment on that. You probably know it better.
@cskasm this might be a little too late but the problem you are facing is due to asynchronous programming thats basically multitasking on a single thread (read up more on it). when we are streaming data there is a lot that goes into the whole mechanism behind the scenes especially if you are using languages like python, javascript you trade latency with ease of use. your system tries to get the data packet and decode it for you on one single thread it'll halt the decoding while getting (this is async) and later you play with the data the way you want and there is a looooooot that goes under the hood for pandas, if you want truly low latency streaming try using C++ for it with multi thread setup for each part of your code this might just slightly improve it. in all fairness i think the 500 millisec is not a huge problem. end of the day HDFC and FII decide if we make money (jk)
If nothing changes,you won't receive any tick.
KiteConnect is just level 2 snapshot data,not TBT data.
You can read here for more information:
https://tradingqna.com/t/why-does-the-same-technical-charts-for-the-same-stock-from-nse-tame-google-finance-yahoo-finance-amibroker-not-match/1688
Also,you are thinking that data is missing because you are expecting a fixed pattern to be repeated which is not the case in real life scenarios.
You are right that it might be possible,that hundreds of trades might have happened in those 500 ms. But , Zerodha will get only 1 or 2 ticks per second. And the same is relayed to you by KiteConnect Websocket API.
There can't be a problem with KiteConnect.
Because, KiteConnect just streams what they get from the exchange without any modification from their side.
Let's assume for once, even if there is a problem with the ticks,then that problem should be from the exchange stream itself.
Still if you think that there is any problem with KiteConnect,then know this that the websocket data is used by the charts on Kite too. KiteConnect,as the name suggests, is just a way to provide the data on Kite , programmatically to the users so they can code their strategies.
So, it's better to write to support to ask your queries:
https://support.zerodha.com/
We only deal with technical queries related to KiteConnect APIs on this forum.
But ,I don't believe there's any issue.
Everything seems fine to me.
Even after all of this,if you're not satisfied with KiteConnect data,you can go for colocation setup at the exchange.
Zerodha can help you setup lease lines.
It costs upwards 18 lakhs/annum. You can write to kiteconnect(at)zerodha.com, if you are interested.
Someone from the relevant team will get back to you.
>> It's just random.
It's not random. Whenever the DELAY is about to overshoot 1 sec or it has overshot 1 sec by a little bit, that's when the data goes missing. Check the "delay" column properly, you'll see this pattern. This pattern occurs throughout the day. Towards the bottom of the image file, if you see, wherever I wrote missing data, the levels are changed by more that 1 rupee. So it's not possible that there were no tick. In your response you haven't mentioned anything about the DELAY.
>> You are right that it might be possible,that hundreds of trades might have happened in those 500 ms. But , Zerodha will get only 1 or 2 ticks per second.
Kite receives 1 tick (on rare occasion, no tick at all) when the DELAY is overshooting 1 sec. Regarding the 2 ticks per second; if you are receiving the millisecondstamp in the feed, please provide it to us. Two different data occurring at the same timestamp is not possible. If I have to either chop some data or fill missing feed with dummy data, without the millisecondstamp, it seems quite difficult.
>> And the same is relayed to you by KiteConnect Websocket API. There can't be a problem with KiteConnect.
This you need to check (assuming you are a developer with kiteconnect) at your end where you are receiving the feed from NSE. Check the DELAY at your server properly and verify if the same pattern, as in the image I attached, is emerging. If the same pattern emerges, then you need to bring it to the notice of NSE. If the same pattern doesn't emerge, i.e. you receive all the data from the exchange, please check with the code https://pypi.org/project/kiteconnect/. Because I copied the code from there.
>> So, it's better to write to support to ask your queries: https://support.zerodha.com/
Zerodha support is barely helpful. Some HR guy looks into it and tries to answer a highly technical question!
>> But ,I don't believe there's any issue. Everything seems fine to me.
Programmers rarely capture such issue with the data until some researcher/analyst bring it to there notice (in my experience). As a programmer, you said it's random. As a researcher, I noticed the pattern; whenever the delay is/about to overshoot 1 sec that's when the data goes missing.
>> Even after all of this,if you're not satisfied with KiteConnect data,you can go for colocation setup at the exchange. Zerodha can help you setup lease lines. It costs upwards 18 lakhs/annum. You can write to kiteconnect(at)zerodha.com, if you are interested. Someone from the relevant team will get back to you.
I did write to kiteconnect(at)zerodha.com couple of weeks back to rent co-located server. Sreenivasulu M responded to me saying no such service.
First,I am not actually a programmer ,I am a trader.
I just learnt some programming for algorithmic trading.
And I am just member of this forum like you,not officially related to Zerodha.
Second, you are making up things in your head.
You want to know the reason for delay??
There is NO DELAY. Delay means if there is a fixed timestamp/time period in which an event should happen,and it happens after that timestamp/period.
There is no fixed interval between two ticks ,so there is no question of delay.
Not a support forum.
Anyone can respond to a query or participate in a discussion if he/she wants to.
I understand that information doesn't reach instantly. It's just that there is no fixed interval in which the information will reach,or between 2 ticks.
Actually,your way of calculating the difference between the timestamps of two ticks is incorrect,which is resulting in this weird observation by you.
It is not possible to capture all the ticks that happens on the exchange over the internet. You will have to contact any exchange registered data vendor or get a colocation setup at the exchange premises.
Kite Connect APIs are for retail users only. It is not a suitable product for time based or latency based strategies.
Are you a developer with kiteconnect?
Assuming you are, I'm asking to verify few checks and post the result here.
For BANKNIFTY option, 500 multiple strike (around the ATM spot) on Wednesday/Thursday;
How many total number of data you receive at your server, is it 45000 or lower than that?
What is the DELAY in receiving the data at your server? Do you see the same pattern in DELAY as in the image file, I attached?
Have you matched the two data, one that you receive from the exchange at your server and the other by a local machine that is connected through WebSocket over the internet, using the code posted in https://pypi.org/project/kiteconnect/ ? How much is the increase in the 'delay' in this case (on an average, in terms of percentage)?
The exchange time is missing milliseconds/microseconds. Your local time variable is having the fraction.
The delay is the microseconds missing from the exchange time. I have no clue why the exchange time doesn't have milliseconds when it comes via kiteConnect. I have never worked in a firm so can't comment on that. You probably know it better.