Hi,
I have been trying to get live tick data for a while now and also form a candle chart for 1min.
Basically my problem is I have subscribed to around 50stocks at a time[As you have mentioned in other treads there is no problem in volatility or volume as most are from nifty50] but I get tick data every 3-5seconds and at every tick only 15-20 get updated that is for a stock to get updated takes around 10-15seconds delay.
I have tried LTP,quote and full but there is delay in all of modes.
I have compared the ticks of the websocket to ticks I get from live stream from kite android and there is a lot of delay which really makes my algorithm useless.
The python code written for websocket is same as the one give in https://github.com/rainmattertech/pykiteconnect/blob/master/README.md
Please help me out ASAP as I have spent over 6-8k only for kite subscribing but got no where cos unable to get quality data.
Hi @MohitGowda,
Source for ticker data is exactly same for Kite Android and Kite Connect. Make sure not to do any computations in onTick method and don't block it while writing to DB also. Websocket should be independent thread whose only job should be getting ticks. I think that will resolve the issue.
The only computation I do in onTick method is seperate the data of each stock and feed it into the DB.Do you think making a multithread process and opening 3 sockets may resolve it? And what exactly do you mean by don't block it while writing to DB
Yes, multithreading is the correct solution. Writing to DB is an IO operation, IO operations are always expensive. It will block available resources. I would recommend not to do that in Websockets streaming thread. In OnTick create a thread and pass ticks to that and do all your task in that thread.
@MohitGowda As Sujith suggested you should be delegating the DB write process to new thread instead of blocking current thread. If you are using Python then Celery is well known for these kind of applications. There are many other tools also for Job queuing such as AWS SQS, JQM etc.
I am a newbie in Node js and I have a doubt about multithreading in Node Js.
Let's say I want to store each tick data received for 20 symbols into MongoDB and do some operation. Could you please direct me to a better solution for this use case in Node JS? Should I care about multithreading in Node jS since it is by default single threaded?
Hi @svarunbe, It is better to use multithreading since you might miss ticks if you do everything on the main thread. Main thread must only receive ticks and do nothing. All your calculations and writing to DB must happen on the worker thread.
@MohitGowda Try something simpler. On the main thread, spawn a new permanent thread say DBThread and set up a mutithreading queue between main thread and DBThread. MainThread will receive tick and push it into Queue and do whatever you want to do with the tick. DBThread has pull tick from Queue and dump into Database.
So tick comes, MainThread processes it and pushes to queue. DBThread receives it and dumps to database.
That should be sufficient for you. The essential point is that don't spawn a new thread to write data every time you receive a tick.
I found the test_bulk_insert_mappings method best performing. Blindly implement it.
So now, you will receive a list of ticks (TickList) in MainThread. Push the list to Queue. DBThread will pull the TickList from queue and bulk insert into Database. That way, the lag in database will not be high considering the deluge of data it will be subjected to.
Source for ticker data is exactly same for Kite Android and Kite Connect. Make sure not to do any computations in onTick method and don't block it while writing to DB also. Websocket should be independent thread whose only job should be getting ticks. I think that will resolve the issue.
I am a newbie in Node js and I have a doubt about multithreading in Node Js.
Let's say I want to store each tick data received for 20 symbols into MongoDB and do some operation.
Could you please direct me to a better solution for this use case in Node JS?
Should I care about multithreading in Node jS since it is by default single threaded?
Thanks in advance.
It is better to use multithreading since you might miss ticks if you do everything on the main thread.
Main thread must only receive ticks and do nothing. All your calculations and writing to DB must happen on the worker thread.
So tick comes, MainThread processes it and pushes to queue. DBThread receives it and dumps to database.
That should be sufficient for you. The essential point is that don't spawn a new thread to write data every time you receive a tick.
Another point is that when you receive ticks, try and bulk dump into DB and not one insert per stock. That's very expensive. Google search Bulk Inserts for your database and you will know what I am talking. Use this link:
http://docs.sqlalchemy.org/en/latest/_modules/examples/performance/bulk_inserts.html
I found the test_bulk_insert_mappings method best performing. Blindly implement it.
So now, you will receive a list of ticks (TickList) in MainThread. Push the list to Queue. DBThread will pull the TickList from queue and bulk insert into Database. That way, the lag in database will not be high considering the deluge of data it will be subjected to.