Fastest and Best way to store ticks and retrieve candles in Dataframe

arulchellappa
Hi, Im bottle necked by the speed of storing ticks and retrieving as candles..I short i need to know how to store tickdata in memory and retrieve as candles in near realtime, im now researching in redis, timescale db, mysql, influx db, but confused ..
  • sujith
    You can take a look at the discussion here.
  • arulchellappa
    this thread is old and inconclusive..any one with fresh new ideas??
  • arulchellappa
    #some usefull info

    #Max number of instruments in instrument base is 64960
    # Max number of instruments that can be streamed is 3*4000 = 12000


    #get dump of all instruments
    instrument_base = kite.instruments()#options''-64960 ,'BCD'-2996,'BSE'-12888,'CDS'-6529,'MCX'-4340,'NFO'-35153,'NSE'-3324,
    instrument_base = pd.DataFrame(instrument_base)

    instrument_bcd = kite.instruments('BCD')
    instrument_df_bcd = pd.DataFrame(instrument_bcd)

    instrument_bse = kite.instruments('BSE')
    instrument_df_bse = pd.DataFrame(instrument_bse)

    instrument_cds = kite.instruments('CDS')
    instrument_df_cds = pd.DataFrame(instrument_cds)

    instrument_mcx = kite.instruments('MCX')
    instrument_df_mcx = pd.DataFrame(instrument_mcx)

    instrument_nfo = kite.instruments('NFO')
    instrument_df_nfo= pd.DataFrame(instrument_nfo)

    instrument_nse = kite.instruments('NSE')
    instrument_df_nse= pd.DataFrame(instrument_nse)


    #manipulating instrument dataframe

    #nse-futures list
    sec_fut_nse = instrument_df_nfo.loc[instrument_df_nfo.segment=='NFO-FUT','tradingsymbol']

    #mcx-futures list
    sec_fut_mcx =instrument_df_mcx.loc[instrument_df_mcx.segment=='MCX-FUT','tradingsymbol']
  • arulchellappa
    arulchellappa edited November 2020
    Best way to handle tick data is to injest all data in redis
    Ticks to Redis_timeseries to resample in pandas dataframe to timescale db or influx db
  • arulchellappa
    Then store store to relational Db like timescale db for historical data..

    This is the fastest method available for processing tick data
Sign In or Register to comment.