I found some strange and interesting information/bug in our Historical data.
If you observe in below screenshot, for ACC NOV FUT and 10:05 Candle, only Open value matched and others are different. This is happening with Live market data and I need to verify Off-Market Historical data.
Further, I observed, if I am trying to pull 5 Min Candle data between 09:20 to 09:55 and found data for 1 min data frame also which look like a bug. As per my knowledge and understanding, at any time, if I selected 5 Min interval, the TimeStamp column should have only 0,5,10,15,20,25,30,35,40,45,50,55 as minutes and should not have 1,3,6,9.
Due to this, I am unable to apply my strategies.
Kindly let me know if you need any further information to resolve this issue.
Hi @vardhan36397, Can you let us know the source of the data for the excel sheet?
If the first tick is received at 9:18 AM then data is plotted from that time frame and hence it will have the time series somewhat like, 9:18, 9:23, 9:27 and so on.
The source for this excel is our KiteConnect API - Historical Data API and parsed the Json to Excel.
If I understood correctly, the 5 min will calculate based on the first tick of the day. Even in this case, whatever the data we have in our Kite Charts and the same should send to Historical API also which is not happening here.
Hi @vardhan36397, The way charting System at our end works is, Our system listens to every tick received and at end of the minute we generate a candle and store it in the database. This happens for all the instruments that are traded on the day. Since it involves a database write, one can't say that at the exact end of the minute a candle will be available there can be 10 to 15 seconds delay.
5min candles are generated using five 1min candles. Let's us say you did access 5min candle at end of the minute. You might get 5min data which is generated based on actually four 1min candles since the last candle is not available.
There are two solutions for this, 1. If your algorithm can afford some delay then you can add some delay and then fetch data. 2. You can generate your own candles using the tick data provided on Websockets API.
@Sujit, thanks for the listed options. 1) anyway depends on the strategy, so different users will have different view point. For 2) candle building from web sockets will also not be accurate as during polling of a volatile instrument, there are chances that we will miss some ticks within a minute. And if those ticks would have comprised of changing OHLC values, then definitely, the candle we create will not be the candle desired. Pls let me know if you agree. So net/net, I dont see historical API fetching us the correct data (even if we know how much delay can result in correct data, we should be good too). And websockets not helping too due to volatility and fast changing prices. Do you see any workaround?
I didn't find the solution to handle the Historical data from the Ticks. So, I am curious to know if there is any way to get this Historical data by passing an Array instead of a single instrument tokens.
Example: Now, I am doing this stuff using For Loop where it will loop all the instrument tokens individually within the array and it's taking a lot of time and sometimes showing "Too Many Requests".
To overcome this issue, I would like to know, if there is any way to get these data using Dictionary which we use to get OHLC or LTP for 200 Instruments. If this available, it will help us to get the candle OHLC and volume to build our strategies.
Let me know if you need any information regarding this.
Hi @vardhan36397 There is no way for multiple instruments historical data in one HTTP call. I am not sure if in 3.0 there will be, but currently no.
Per subscription, per second, its 3 HTTP requests throttling. If you have to increase this, you need to handle either the subscription part, by opening additional accounts (obviously if you need them), or put 0.3 seconds delay between each request.
So, if you have 200 instruments, and assuming total time to run the API is 0.2 seconds, thats a 0.5 seconds delay between two calls. So, 200 X 0.5 seconds = 100 seconds will be required to complete the array walk. If that kind of delay is acceptable as per your strategy you can try and see. Otherwise if other family members also trade, you can open their accounts and spread your instruments into their connections.
Bulk fetch for LTP is present in the version of API. You can check out this thread.
I am afraid we can't provide bulk fetch API as of now for historical data since data for each instrument has to be fetched from the database which will take up resources and block other requests.
@shortwire, I can only think of these two ways. If you are willing to spend more on infrastructure then you can try out colo wherein your setup will be at the exchange location which might cost you around 18L per annum.
I just want to clarify few things which stated below.
1. What information can we get from OHLC? As I worked on this, I didn't find matching values in the chart in any timeframe including 1M, 3M, 5M and 10M and I am not sure about the time of OHLC values. 2. If we don't have the bulk fetch option for Historical data, then what is the best way to get 1M candle information within 1 Minute. To explain this for more clarity, if a candle completed by 10:01 AM, for 200 instruments I should have data with me before 10:02:59.999 (HH:MM:ss:ms) to pick up next candle at 10:02.
But, this is not happening due to various reasons viz., the cap on requests i.e., 3 requests per second, if I want to use multiple instances, it will throw an exception as Too Many Requests and it will skip the instrument which leads to data miss and it will not fit in my strategy.
Due to these things, even I spent 4K per month, it's not serving my purpose. I would like to request you to work around these kinds of problem and provide the solution.
Suggestion: I am not sure about your database model, but I assume, if you are concerned about resources, you can keep the minute wise OHLC based on Exchange, Instrument Type, Expiry (FNO) etc in a normalized database, instead of hitting database for every request, so it will be easy for us to carry our day-to-day jobs without fail.
Kindly let me know if you need any further information.
@vardhan36397, Historical API is not feasible for the above use case. It is used by 60K users which includes people from all our platforms on a normal day. It will go up significantly during special days. Historical data API was never built for this reason. I think none of the data vendors also provide this. The only solution is to generate candles at your end.
Thank you Sujith for the response. But generating candles using tick data also not returning correct OHLC where returning false signals. If I use the same strategy with correct OHLC (as per Zerodha chart), my logic returning almost 90% to 95% accuracy when I checked with around 20 instruments. To work with more instruments, I need data.
Could you please check the feasibility to increase the 3 HTTP request cap in this case. If I get at least 1 Min OHLC, I will play further.
Kindly let me know the ways to get the Historical Data as early as possible once formation completed.
Live market data will have day's OHLC data. You need to generate your own candles which is something like this, Open = first LTP of the minute. High = highest LTP of the minute. Low = lowest LTP of the minute. Close = last LTP of the minute.
You can subscribe for up to 200 instruments in one ticker connection. I am afraid we can't increase the number of HTTP API calls of Kite Connect as of now.
Can you please suggest the ways we have to get 1 Min OHLC as soon as candle completes if not possible to increase the number of HTTP API calls? I don't have anything in my near sight.
Can you let us know the source of the data for the excel sheet?
If the first tick is received at 9:18 AM then data is plotted from that time frame and hence it will have the time series somewhat like, 9:18, 9:23, 9:27 and so on.
The source for this excel is our KiteConnect API - Historical Data API and parsed the Json to Excel.
If I understood correctly, the 5 min will calculate based on the first tick of the day. Even in this case, whatever the data we have in our Kite Charts and the same should send to Historical API also which is not happening here.
The way charting System at our end works is,
Our system listens to every tick received and at end of the minute we generate a candle and store it in the database. This happens for all the instruments that are traded on the day. Since it involves a database write, one can't say that at the exact end of the minute a candle will be available there can be 10 to 15 seconds delay.
5min candles are generated using five 1min candles.
Let's us say you did access 5min candle at end of the minute. You might get 5min data which is generated based on actually four 1min candles since the last candle is not available.
There are two solutions for this,
1. If your algorithm can afford some delay then you can add some delay and then fetch data.
2. You can generate your own candles using the tick data provided on Websockets API.
I didn't find the solution to handle the Historical data from the Ticks. So, I am curious to know if there is any way to get this Historical data by passing an Array instead of a single instrument tokens.
Example:
Now, I am doing this stuff using For Loop where it will loop all the instrument tokens individually within the array and it's taking a lot of time and sometimes showing "Too Many Requests".
To overcome this issue, I would like to know, if there is any way to get these data using Dictionary which we use to get OHLC or LTP for 200 Instruments. If this available, it will help us to get the candle OHLC and volume to build our strategies.
Let me know if you need any information regarding this.
Thanks & Regards,
R C Vamsi Vardhan
There is no way for multiple instruments historical data in one HTTP call. I am not sure if in 3.0 there will be, but currently no.
Per subscription, per second, its 3 HTTP requests throttling. If you have to increase this, you need to handle either the subscription part, by opening additional accounts (obviously if you need them), or put 0.3 seconds delay between each request.
So, if you have 200 instruments, and assuming total time to run the API is 0.2 seconds, thats a 0.5 seconds delay between two calls. So, 200 X 0.5 seconds = 100 seconds will be required to complete the array walk. If that kind of delay is acceptable as per your strategy you can try and see. Otherwise if other family members also trade, you can open their accounts and spread your instruments into their connections.
I am afraid we can't provide bulk fetch API as of now for historical data since data for each instrument has to be fetched from the database which will take up resources and block other requests.
@shortwire,
I can only think of these two ways.
If you are willing to spend more on infrastructure then you can try out colo wherein your setup will be at the exchange location which might cost you around 18L per annum.
I just want to clarify few things which stated below.
1. What information can we get from OHLC? As I worked on this, I didn't find matching values in the chart in any timeframe including 1M, 3M, 5M and 10M and I am not sure about the time of OHLC values.
2. If we don't have the bulk fetch option for Historical data, then what is the best way to get 1M candle information within 1 Minute. To explain this for more clarity, if a candle completed by 10:01 AM, for 200 instruments I should have data with me before 10:02:59.999 (HH:MM:ss:ms) to pick up next candle at 10:02.
But, this is not happening due to various reasons viz., the cap on requests i.e., 3 requests per second, if I want to use multiple instances, it will throw an exception as Too Many Requests and it will skip the instrument which leads to data miss and it will not fit in my strategy.
Due to these things, even I spent 4K per month, it's not serving my purpose. I would like to request you to work around these kinds of problem and provide the solution.
Suggestion:
I am not sure about your database model, but I assume, if you are concerned about resources, you can keep the minute wise OHLC based on Exchange, Instrument Type, Expiry (FNO) etc in a normalized database, instead of hitting database for every request, so it will be easy for us to carry our day-to-day jobs without fail.
Kindly let me know if you need any further information.
Thanks & Regards,
R C Vamsi Vardhan
Historical API is not feasible for the above use case. It is used by 60K users which includes people from all our platforms on a normal day. It will go up significantly during special days.
Historical data API was never built for this reason. I think none of the data vendors also provide this.
The only solution is to generate candles at your end.
Could you please check the feasibility to increase the 3 HTTP request cap in this case. If I get at least 1 Min OHLC, I will play further.
Kindly let me know the ways to get the Historical Data as early as possible once formation completed.
Awaiting your response.
Thanks & Regards,
R C Vamsi Vardhan
You need to generate your own candles which is something like this,
Open = first LTP of the minute.
High = highest LTP of the minute.
Low = lowest LTP of the minute.
Close = last LTP of the minute.
You can subscribe for up to 200 instruments in one ticker connection.
I am afraid we can't increase the number of HTTP API calls of Kite Connect as of now.
Please create a new thread for new queries, it might help others who are looking for the same.