Fastest and Best way to store ticks and retrieve candles in Dataframe

arulchellappa

Hi, Im bottle necked by the speed of storing ticks and retrieving as candles..I short i need to know how to store tickdata in memory and retrieve as candles in near realtime, im now researching in redis, timescale db, mysql, influx db, but confused ..

sujith

You can take a look at the discussion here.

arulchellappa

this thread is old and inconclusive..any one with fresh new ideas??

arulchellappa

#some usefull info

#Max number of instruments in instrument base is 64960
# Max number of instruments that can be streamed is 3*4000 = 12000


#get dump of all instruments
instrument_base = kite.instruments()#options''-64960 ,'BCD'-2996,'BSE'-12888,'CDS'-6529,'MCX'-4340,'NFO'-35153,'NSE'-3324,
instrument_base = pd.DataFrame(instrument_base)

instrument_bcd = kite.instruments('BCD')
instrument_df_bcd = pd.DataFrame(instrument_bcd)

instrument_bse = kite.instruments('BSE')
instrument_df_bse = pd.DataFrame(instrument_bse)

instrument_cds = kite.instruments('CDS')
instrument_df_cds = pd.DataFrame(instrument_cds)

instrument_mcx = kite.instruments('MCX')
instrument_df_mcx = pd.DataFrame(instrument_mcx)

instrument_nfo = kite.instruments('NFO')
instrument_df_nfo= pd.DataFrame(instrument_nfo)

instrument_nse = kite.instruments('NSE')
instrument_df_nse= pd.DataFrame(instrument_nse)


#manipulating instrument dataframe

#nse-futures list
sec_fut_nse = instrument_df_nfo.loc[instrument_df_nfo.segment=='NFO-FUT','tradingsymbol']

#mcx-futures list
sec_fut_mcx =instrument_df_mcx.loc[instrument_df_mcx.segment=='MCX-FUT','tradingsymbol']

arulchellappa

Best way to handle tick data is to injest all data in redis
Ticks to Redis_timeseries to resample in pandas dataframe to timescale db or influx db

arulchellappa

Then store store to relational Db like timescale db for historical data..

This is the fastest method available for processing tick data