Hey! I'm new to this forum, happy to be here. So my concern is regarding the efficient ways to store the ticks which are streamed from the websockets.
I used a simple WAMP server to setup an SQL database to store the tick data with each company being a different table and inserting the ticks I've subscribed to into their respective tables.
Initially the time to read these ticks from the database and form the OHLC candles using the simple resample function of pandas took about (0.1 to 0.3 seconds).
As the size of the data in these tables grew bigger, more than half the day's data, the time taken to fetch and form the candles grew to about 0.6 to 0.8 seconds.
My strategies are based on real time and I'm concerned with this type of delay. This is more a latency related issue. I'm hoping this community helps me with suggestions to make an informed decision about the database to choose?
If anyone has faced a similar issue, could you please help me out?
@Bromine Pandas OHLC sampling is for offline work not for real time. For real time OHLC over any time frame that you desire, you have to write your own code that does sampling over that time period and saves it into the database
So we do sampling while obtaining the ticks and then store the sampled timeframe data into the Database? So far I've been storing ticks in the database and used to sample it to my desired timeframe later.
So far I've been storing ticks in the database and used to sample it to my desired timeframe later.