I'm facing problems in backtesting my strategies due to gaps in historical data. I have checked the Zerodha charts also and the same mismatch exists. Let me give some examples here:
1) ACC - data missing for candle at 2020-11-02 15:12:00+05:30 (1min interval)
Historical download screenshot
Same missing on charts also 15:11:00 candle 15:13:00 candle
2.) PFC - found missing data (5 min candle) from 2015-12-22 10:20:00+05:30 to 2015-12-22 10:45:00+05:30 (highlighted potion) in all scripts downloaded (PFC shown here). In 1 min data is available upto 10:22 & then from 10:47 only. Same missing on charts also
5 min charts 22-12-15 10:20 candle 22-12-15 10:45 candle
Please see and help in resolving the problem. I'm having difficulty in backtesting my strategy.
@GJ3030 Thanks for letting us know about the exact missing candle. We are looking to back-fill these random missing data soon. We will update here once done.
@rakeshr Dear Sir, thanks for your response. However, please run an extensive internal check. The above mentioned dictionary is only those values where datetime of 1m and 3m candles doesnt match. hence this check is valid for 3m intervals only.
Since I have modified my algo to ignore such mismatches and only display the errors, please run an extensive minute by minute data integrity check as there are far too many missing candles.
@Devang_b, The historical data is provided for backtesting purposes only. We suggest using live market data via Websockets to generate candles at your end for live market strategies.
Although, we try to provide data that can be used reliably by users, there can be many instances wherein data might be missing because of many technical reasons. You may contact exchange registered data vendor, if Kite Connect data doesn't fit your requirements.
The missing data problem still persists. Completely messing up my backtesting. Is there a solution or workaround for this? There are gaps in historical data for banknifty 1 minute interval.
We are checking on this.
I have tested Nifty 50 stocks. There are other timeframes also where data is missing in 1 min timeframe.
Here is the dictionary with stockwise discontinuity points:-
{'ABBOTINDIA': '2020-06-30 13:45:00+05:30',
'ACC': '2020-11-02 15:12:00+05:30',
'ADANIGREEN': '2020-11-02 15:12:00+05:30',
'ADANITRANS': '2020-07-02 11:27:00+05:30',
'ALKEM': '2020-11-02 15:12:00+05:30',
'AMBUJACEM': '2020-11-02 15:12:00+05:30',
'AUROPHARMA': '2020-11-02 15:12:00+05:30',
'BAJAJHLDNG': '2020-07-24 11:48:00+05:30',
'BANDHANBNK': '2020-11-02 15:12:00+05:30',
'BANKBARODA': '2020-11-02 15:12:00+05:30',
'BERGEPAINT': '2020-11-02 15:12:00+05:30',
'BIOCON': '2020-11-02 15:12:00+05:30',
'BOSCHLTD': '2020-11-02 15:12:00+05:30',
'CADILAHC': '2020-11-02 15:12:00+05:30',
'COLPAL': '2020-11-02 15:12:00+05:30',
'CONCOR': '2020-11-02 15:12:00+05:30',
'DABUR': '2020-11-02 15:12:00+05:30',
'DLF': '2020-11-02 15:12:00+05:30',
'DMART': '2020-11-02 15:12:00+05:30',
'GICRE': '2020-06-02 11:45:00+05:30',
'GODREJCP': '2020-11-02 15:12:00+05:30',
'HAVELLS': '2020-11-02 15:12:00+05:30',
'HDFCAMC': '2020-11-02 15:12:00+05:30',
'HINDPETRO': '2020-11-02 15:12:00+05:30',
'HINDZINC': '2020-11-02 15:12:00+05:30',
'ICICIGI': '2020-11-02 15:12:00+05:30',
'ICICIPRULI': '2020-11-02 15:12:00+05:30',
'IGL': '2020-11-02 15:12:00+05:30',
'INDIGO': '2020-11-02 15:12:00+05:30',
'INFRATEL': '2020-11-02 15:12:00+05:30',
'LTI': '2020-06-18 11:12:00+05:30',
'LUPIN': '2020-11-02 15:12:00+05:30',
'MARICO': '2020-11-02 15:12:00+05:30',
'MCDOWELL-N': '2020-11-02 15:12:00+05:30',
'MOTHERSUMI': '2020-11-02 15:12:00+05:30',
'MUTHOOTFIN': '2020-11-02 15:12:00+05:30',
'NAUKRI': '2020-11-02 15:12:00+05:30',
'NMDC': '2020-11-02 15:12:00+05:30',
'OFSS': '2020-06-01 11:12:00+05:30',
'PEL': '2020-11-02 15:12:00+05:30',
'PETRONET': '2020-11-02 15:12:00+05:30',
'PFC': '2020-11-02 15:12:00+05:30',
'PGHH': '2020-06-01 12:00:00+05:30',
'PIDILITIND': '2020-11-02 15:12:00+05:30',
'PNB': '2020-11-02 15:12:00+05:30',
'SBICARD': '2020-11-02 15:12:00+05:30',
'SIEMENS': '2020-11-02 15:12:00+05:30',
'TATACONSUM': '2020-11-02 15:12:00+05:30',
'TORNTPHARM': '2020-11-02 15:12:00+05:30',
'UBL': '2020-11-02 15:12:00+05:30'}
Thanks for letting us know about the exact missing candle. We are looking to back-fill these random missing data soon. We will update here once done.
Dear Sir, thanks for your response. However, please run an extensive internal check. The above mentioned dictionary is only those values where datetime of 1m and 3m candles doesnt match. hence this check is valid for 3m intervals only.
Since I have modified my algo to ignore such mismatches and only display the errors, please run an extensive minute by minute data integrity check as there are far too many missing candles.
Hope you will do the needful very soon.
@sujith
The historical data is provided for backtesting purposes only. We suggest using live market data via Websockets to generate candles at your end for live market strategies.
Although, we try to provide data that can be used reliably by users, there can be many instances wherein data might be missing because of many technical reasons. You may contact exchange registered data vendor, if Kite Connect data doesn't fit your requirements.