We were facing problems with the python client dropping off very frequently. Upto 90 times a day with each reconnection taking upto 30 seconds, thus leaving large holes in our dataset.
So, we tried to use the java client and explore if it was more stable, however, it did not help. Although the disconnections were at different points, its hard to reconcile the two streams (python and java).
Can you please let us know which client is relatively the most stable and has the least number complaints about disconnections?
It would be useful if you can provide us the relative maturity of each clients on different counts - bug fixing, complaints, upgrades, documentation, support, etc.
Hi @arnavsaxena, Websockets that is available in Kite Connect is being used in Kite Web also. We didn't get any complaints regarding this. I am not sure why your connection keeps dropping. Maybe some stacktrace will be helpful. As per our ticker logs, we don't see any issues.
In kiteconnectjs, we have implemented reconnection you can use the same. We will implement the same mechanism on other clients also soon. If it is required immediately, you can go ahead and implement it.
@arnavsaxena : were you able to figure out a solution to this issue. Also, just to confirm, are you facing these errors while using multiple websocket clients (or single websocket client)?
I am facing this error while using multiple websocket client - "ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host" and thought of checking with you if you are facing the same issue.
@sujith if we are using websockets to get data and using that to run trading systems, then connection drops could lead to data holes. Does your websocket detect these dataholes and fill them. If not, then there is a big risk a connection error at your or our end could result in dataholes making it unreliable for live use. This mean that the data storing utility will have to detect dataholes and call those using historical data API. Sounds complicated. Data vendors usually have a backfill facility wherein they will detect missing data and fill them.
@naz We're facing disconnection issues when using a python client very consistently. However, since we've switched to a java client, such issues have reduced. With multiple websocket clients, the number of disconnections are definitely a lot more.
@skk afaik there's no backfill api with zerodha, unless you subscribe with the historical data api, which still only provides minute-by-minute data.
@arnavsaxena Are you able to use this data in your trading practically considering the data holes. In my understanding, these data holes are bound to come. At minimum, they are a big risk for intraday trading. While one can face data blackouts in any feeds, but here the issue is that once feed resumes you dont even have a backfill.
One workaround is to use multiple connections in parallel to download the same data at the same time and manage the duplication at db level. This will partially protect against spurious call failures. This is the strategy I am using to address the datahole problem.
@ramatius this is useful but I am facing a peculiar problem - while using multiple websocket connections (in python client), two of the websocket connections automatically die down after 15-20 min and the first websocket connection keeps on running. Did you also face the same challenge ? If yes, would be great if you could share the way you resolved this problem at your end.
@naz The LTP mode is quite reliable - the websocket connection lives through the day. There may be issues with QUOTE and FULL modes. I would suggest that you have multiple sockets, atleast one with LTP.
@ramatius : let me try this (I have been using full mode so far - so this may very well be the issue). Also, i am using multiple websockets by passing threaded=True in the ws.connect() method (basically inbuilt multi-threading of python websocket client ). Are you also using multiple sockets the same way or is there a better way.
I am not using threaded=true. I have one separate script for each websocket and run the scripts in parallel. That works fine. Haven't tried multi-threading yet.
@ramatius ok. But then How are you using all the realtime data into one single program - are you storing all the data from different scripts to some database (e.g. csv files ) on harddisk and then reading the database (e.g. .csv) back into your program for analyzing and execution. Asking this because, all the scripts won't be sharing the same memory space and hence wanted to understand the best way to do it.
Thanks a tonne for sharing this. Hopefully this will be helpful for others as well.
My backend is MySQL database. I have two streams of getting LTP data (for 300+ scripts in CM and 20+ symbols in FNO), of which python websocket is one. The algo scripts run asynchronously from data sourcing scripts. I am using RAM tables and optimized caching to minimize the latency & multi-user read/write access.
Probably we will get better throughput with postgreSQL or OracleDB. Is anyone using these?
Yes, there is slippage between the expected vs actual trades. However, it has drastically come down these days, thanks to improvements in MySQL, Zerodha data and my own scripts.
Websockets that is available in Kite Connect is being used in Kite Web also. We didn't get any complaints regarding this. I am not sure why your connection keeps dropping.
Maybe some stacktrace will be helpful. As per our ticker logs, we don't see any issues.
In kiteconnectjs, we have implemented reconnection you can use the same.
We will implement the same mechanism on other clients also soon.
If it is required immediately, you can go ahead and implement it.
No, we just log errors.
I am facing this error while using multiple websocket client - "ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host" and thought of checking with you if you are facing the same issue.
Any solutions?
@skk afaik there's no backfill api with zerodha, unless you subscribe with the historical data api, which still only provides minute-by-minute data.
If yes, would be great if you could share the way you resolved this problem at your end.
Are you also using multiple sockets the same way or is there a better way.
ok. But then How are you using all the realtime data into one single program - are you storing all the data from different scripts to some database (e.g. csv files ) on harddisk and then reading the database (e.g. .csv) back into your program for analyzing and execution.
Asking this because, all the scripts won't be sharing the same memory space and hence wanted to understand the best way to do it.
Thanks a tonne for sharing this. Hopefully this will be helpful for others as well.
Probably we will get better throughput with postgreSQL or OracleDB. Is anyone using these?
Yes, there is slippage between the expected vs actual trades. However, it has drastically come down these days, thanks to improvements in MySQL, Zerodha data and my own scripts.