I want to know from Kite members that considering the strategy might restart for whatever reasons(loss of network connectivity, glitches etc) , what is the best way to load candles quickly and run technical indicators over them. My initial idea is to use pandas data frame to load historical data from MySQL and then run technical indicators over them. For every new minute candle, add the new data to MySQL and update pandas data frame and re-run technical indicators. I am using MySQL because at present we have the limitation of not being able to send not more than 3 requests / sec and fetch historical data for only one instrument. I assume that MySQL will help me overcome this limitation. Any suggestions are welcome. Thanks!
Thanks @sujith . Well I have been storing data into my local database but it doesn't seem to be reliable solution...I either get empty candles or my connection gets some EOF errors. I was wondering how to tackle this problem
@sujith Single thread to fetch data for NSE 200 stocks in a loop and store it in DB. If i get empty candles i make 10 re-attempts with pauses of 2 sec before moving on to fetching data for next scrip. And this i I did for a year of data. Now i'm trying to find holes in my data and how to cover them up!
@cisk, The better approach would be to process and store data in the worker thread. If you block the main thread which receives data then you will lose ticks. We have minute candles from FEB 2015.
@sujith Thanks for the suggestion. Yes your point is well noted. As of now I am accumulating the previous historical data for stocks. But when the strategy would start running then what you suggested would be the right way ..to unblock the main thread for strategy and keep worker threads for fetching data.
My initial idea is to use pandas data frame to load historical data from MySQL and then run technical indicators over them. For every new minute candle, add the new data to MySQL and update pandas data frame and re-run technical indicators.
I am using MySQL because at present we have the limitation of not being able to send not more than 3 requests / sec and fetch historical data for only one instrument. I assume that MySQL will help me overcome this limitation.
Any suggestions are welcome. Thanks!
Fetching data and storing it in the local database is the solution for running strategies.
Are you guys using worker threads to write data? How many tokens are you subscribed to at any point of time?
The better approach would be to process and store data in the worker thread. If you block the main thread which receives data then you will lose ticks.
We have minute candles from FEB 2015.