Data missing (gap) in Historical Data Candles

GJ3030
Hello,

I'm facing problems in backtesting my strategies due to gaps in historical data. I have checked the Zerodha charts also and the same mismatch exists. Let me give some examples here:

1) ACC - data missing for candle at 2020-11-02 15:12:00+05:30 (1min interval)

Historical download screenshot


Same missing on charts also
15:11:00 candle

15:13:00 candle



2.) PFC - found missing data (5 min candle) from 2015-12-22 10:20:00+05:30 to 2015-12-22 10:45:00+05:30 (highlighted potion) in all scripts downloaded (PFC shown here). In 1 min data is available upto 10:22 & then from 10:47 only.

Same missing on charts also

5 min charts
22-12-15 10:20 candle

22-12-15 10:45 candle



Please see and help in resolving the problem. I'm having difficulty in backtesting my strategy.

Thanks!
  • rakeshr
    @GJ3030
    We are checking on this.
  • GJ3030
    @rakeshr Hello. Is there any update on the issue reported pls.
  • GJ3030
    @rakeshr
    I have tested Nifty 50 stocks. There are other timeframes also where data is missing in 1 min timeframe.

    Here is the dictionary with stockwise discontinuity points:-

    {'ABBOTINDIA': '2020-06-30 13:45:00+05:30',
    'ACC': '2020-11-02 15:12:00+05:30',
    'ADANIGREEN': '2020-11-02 15:12:00+05:30',
    'ADANITRANS': '2020-07-02 11:27:00+05:30',
    'ALKEM': '2020-11-02 15:12:00+05:30',
    'AMBUJACEM': '2020-11-02 15:12:00+05:30',
    'AUROPHARMA': '2020-11-02 15:12:00+05:30',
    'BAJAJHLDNG': '2020-07-24 11:48:00+05:30',
    'BANDHANBNK': '2020-11-02 15:12:00+05:30',
    'BANKBARODA': '2020-11-02 15:12:00+05:30',
    'BERGEPAINT': '2020-11-02 15:12:00+05:30',
    'BIOCON': '2020-11-02 15:12:00+05:30',
    'BOSCHLTD': '2020-11-02 15:12:00+05:30',
    'CADILAHC': '2020-11-02 15:12:00+05:30',
    'COLPAL': '2020-11-02 15:12:00+05:30',
    'CONCOR': '2020-11-02 15:12:00+05:30',
    'DABUR': '2020-11-02 15:12:00+05:30',
    'DLF': '2020-11-02 15:12:00+05:30',
    'DMART': '2020-11-02 15:12:00+05:30',
    'GICRE': '2020-06-02 11:45:00+05:30',
    'GODREJCP': '2020-11-02 15:12:00+05:30',
    'HAVELLS': '2020-11-02 15:12:00+05:30',
    'HDFCAMC': '2020-11-02 15:12:00+05:30',
    'HINDPETRO': '2020-11-02 15:12:00+05:30',
    'HINDZINC': '2020-11-02 15:12:00+05:30',
    'ICICIGI': '2020-11-02 15:12:00+05:30',
    'ICICIPRULI': '2020-11-02 15:12:00+05:30',
    'IGL': '2020-11-02 15:12:00+05:30',
    'INDIGO': '2020-11-02 15:12:00+05:30',
    'INFRATEL': '2020-11-02 15:12:00+05:30',
    'LTI': '2020-06-18 11:12:00+05:30',
    'LUPIN': '2020-11-02 15:12:00+05:30',
    'MARICO': '2020-11-02 15:12:00+05:30',
    'MCDOWELL-N': '2020-11-02 15:12:00+05:30',
    'MOTHERSUMI': '2020-11-02 15:12:00+05:30',
    'MUTHOOTFIN': '2020-11-02 15:12:00+05:30',
    'NAUKRI': '2020-11-02 15:12:00+05:30',
    'NMDC': '2020-11-02 15:12:00+05:30',
    'OFSS': '2020-06-01 11:12:00+05:30',
    'PEL': '2020-11-02 15:12:00+05:30',
    'PETRONET': '2020-11-02 15:12:00+05:30',
    'PFC': '2020-11-02 15:12:00+05:30',
    'PGHH': '2020-06-01 12:00:00+05:30',
    'PIDILITIND': '2020-11-02 15:12:00+05:30',
    'PNB': '2020-11-02 15:12:00+05:30',
    'SBICARD': '2020-11-02 15:12:00+05:30',
    'SIEMENS': '2020-11-02 15:12:00+05:30',
    'TATACONSUM': '2020-11-02 15:12:00+05:30',
    'TORNTPHARM': '2020-11-02 15:12:00+05:30',
    'UBL': '2020-11-02 15:12:00+05:30'}
  • rakeshr
    @GJ3030
    Thanks for letting us know about the exact missing candle. We are looking to back-fill these random missing data soon. We will update here once done.
  • GJ3030
    GJ3030 edited December 2020
    @rakeshr
    Dear Sir, thanks for your response. However, please run an extensive internal check. The above mentioned dictionary is only those values where datetime of 1m and 3m candles doesnt match. hence this check is valid for 3m intervals only.

    Since I have modified my algo to ignore such mismatches and only display the errors, please run an extensive minute by minute data integrity check as there are far too many missing candles.

    Hope you will do the needful very soon.
  • rakeshr
    @GJ3030
    However, please run an extensive internal check.
    Yes, we are in process of doing the same.
  • Devang_b
    Devang_b edited July 2022
    @rakeshr is this issue resolved now? Can we assume that the data is checked thoroughly from Zerodha's end and won't include missing data?

    @sujith

  • sujith
    sujith edited July 2022
    @Devang_b,
    The historical data is provided for backtesting purposes only. We suggest using live market data via Websockets to generate candles at your end for live market strategies.

    Although, we try to provide data that can be used reliably by users, there can be many instances wherein data might be missing because of many technical reasons. You may contact exchange registered data vendor, if Kite Connect data doesn't fit your requirements.
  • shivd
    shivd edited November 2023
    The missing data problem still persists. Completely messing up my backtesting. Is there a solution or workaround for this? There are gaps in historical data for banknifty 1 minute interval.
  • sujith
    Can you elaborate with complete debug logs? We will ask the data team to take a look at this?
Sign In or Register to comment.