I am seeing issue with historical candles (3 minutes timeframe) being generated very late after actually the timeframe is completed. The delay I am seeing is almost 50 seconds which is too huge. I understand that historical data is not supposed to be used for live ticks and websocket is better suited for that. But the problem with websocket is that if you are in middle of the development, and need to restart server often the data recorded with websocket will not be reliable.
e.g. If I query 3m candles at 9:18 AM, the response should contain the 9:18 AM candle.
I read somewhere else that there may be delay on Zerodha side after timeframe is elapsed due to candles being persisted in DB so 10 seconds delay can be added. I think that is just the wrong reason. Why can you not query all candles but latest one from DB and return the latest candle that is generated in-memory instead of waiting for all DB writes to happen and then returning correct data after whopping 1 minute delay?
The issue may be with other timeframes as well; I have tested only with 3m.
Hi @enliyo, The historical data is provided only for testing purposes. We don't recommend using it for live strategies. You need to generate candles at your end using Websocket data. If you have data loss then you may have to backfill it in your database using historical data or just use some other exchange registered data vendor.
Sure I understand but what I was hoping that data would be available soon, not after 1 minute delay. See the competition Upstox API which charge much less for historical data and are able to provide the historical candles as soon as those are formed. Just a constructive criticism
The objective of providing historical data is for backtesting purpose only. We don't recommend using it for live strategies. The way it works is, we listen to all the ticks for all the instruments that are being traded for the day and end of the minute it is used to generate candle and write to the database. Since there is a database operation involved, we can't guarantee any timeline for availability of the latest candle data. If one of the operation clogs then it will exponentially affect the other writes as well. Hence it is recommended to generate candles at your end for the instrument which you need only.
It was never built for polling during market hours.
Sure I understand how it works. I was proposing better implementation if Zerodha is willing to implement because what you have is not the solid solution in this aspect.
Probably no efforts are taken about this specific use case from your end because possible flooding of historical APIs if everybody starts polling which may bring down the quality of service to other fair users. So I now agree to what you are saying.
@sujith is the data coming over the websocket complete? That is, barring network disconnects, is the information from all trades guaranteed to be streamed?
I ask because I am coming here from Upstox, and in Upstox API, websockets don't stream the complete data. It is only a subset.
@manhat, There can be thousands or lakhs of ticks at the exchange, it is not possible for any of the platforms to capture all the ticks via the internet. We as a member of the exchange just get Level-2 data which means you can expect a maximum of one or two ticks per second. These are really like a SNAPSHOT published by the exchange and we just broadcast it to all the subscribed users.
You can find all the details about Kite Ticker (Websocket API) here.
To receive all ticks, you have to be on the exchange colocation. The infra + colo fees start at around Rs 18lk per annum minimum. If you are interested, send an email to [email protected], someone from our team will get back to you. If you are on the exchange colo, you can't use Kite connect API's, you will have to use our colo vendor TR (Reuters) API.
Thanks @sujith for the comprehensive reply. If I understand right, kite API will broadcast all the data that is available to it, and hence, the candles formed by API clients will be, in theory, same as the candles that Kite OHLC API provides.
The source for candles data and websockets API are different. It will not match the exactly but it will be near. The use-case is same as mentioned the above tradingqna thread.
Just saw the note and recommendation that for Live algo trades we need to build the candle from websocket calls rather than get history API....but this becomes a challenge when we have to apply indicators to make the trade logic work because it does not contain historical references. If your suggestion is to build a local DB and store live candle data over a period of time , it becomes very cumbersome ....can you provide some examples showing how this is implemented with indicators and Live market candle formation
@rji_rji The idea is to utilise both historical data and websockets. Fetch historical data according to your logic and store locally(and use it for further calls) and then add the websockets data on top of it.
The historical data is provided only for testing purposes. We don't recommend using it for live strategies.
You need to generate candles at your end using Websocket data. If you have data loss then you may have to backfill it in your database using historical data or just use some other exchange registered data vendor.
Sure I understand but what I was hoping that data would be available soon, not after 1 minute delay. See the competition Upstox API which charge much less for historical data and are able to provide the historical candles as soon as those are formed. Just a constructive criticism
https://upstox.com/forum/topic/2645/please-confirm-delay-of-historical-api-data
The way it works is, we listen to all the ticks for all the instruments that are being traded for the day and end of the minute it is used to generate candle and write to the database. Since there is a database operation involved, we can't guarantee any timeline for availability of the latest candle data. If one of the operation clogs then it will exponentially affect the other writes as well. Hence it is recommended to generate candles at your end for the instrument which you need only.
It was never built for polling during market hours.
Probably no efforts are taken about this specific use case from your end because possible flooding of historical APIs if everybody starts polling which may bring down the quality of service to other fair users. So I now agree to what you are saying.
I ask because I am coming here from Upstox, and in Upstox API, websockets don't stream the complete data. It is only a subset.
There can be thousands or lakhs of ticks at the exchange, it is not possible for any of the platforms to capture all the ticks via the internet. We as a member of the exchange just get Level-2 data which means you can expect a maximum of one or two ticks per second. These are really like a SNAPSHOT published by the exchange and we just broadcast it to all the subscribed users.
You can find all the details about Kite Ticker (Websocket API) here.
To receive all ticks, you have to be on the exchange colocation. The infra + colo fees start at around Rs 18lk per annum minimum. If you are interested, send an email to [email protected], someone from our team will get back to you. If you are on the exchange colo, you can't use Kite connect API's, you will have to use our colo vendor TR (Reuters) API.
You can take a look at a similar discussion here.
Thanks @sujith for the comprehensive reply. If I understand right, kite API will broadcast all the data that is available to it, and hence, the candles formed by API clients will be, in theory, same as the candles that Kite OHLC API provides.
Is that correct?
The use-case is same as mentioned the above tradingqna thread.
If your suggestion is to build a local DB and store live candle data over a period of time , it becomes very cumbersome ....can you provide some examples showing how this is implemented with indicators and Live market candle formation