How to efficiently save and retrive tick data

pratiks

I am using redis and rq to process tick data

def process_data(tick):
opens csv file as dataframe
insert ltp data
alter some columns
save csv file

def on_ticks(ws, ticks):
q.enqueue(process_data,ticks)

this process usually takes longer time than processing in on_ticks directly.
I am accessing ticks of nearly 150 instruments (all stock futures)

processing this data takes long time, and i get delayed signals

so I want to know how other people store/use tick data?

@rakeshr
how does calling helper method this way is diferent than calling it inside on_ticks method directly?
i am unable to understand how this works
please explain

while True:
   #Perform required data operation using tick data
   def on_ticks(ws, ticks):
       ..........
       helper_method(ticks)
       .........

   def helper_method(ticks):
        .........
        Perform computation here
        ........
  #Assign callback
  kws.on_ticks=on_ticks

rakeshr

how does calling helper method this way is diferent than calling it inside on_ticks method directly?

As the thread explains, you are creating another ticker thread using threaded=True and let it handle tick data.

pratiks

you are creating another ticker thread using threaded=True
^
how this another thread gets created

pratiks

@rakeshr i didnt understand exactly when the new ticker thread gets created