Websockets in Kite3 python client: Not able to subscribe to >500 tickers/run multiple connections

sauravkedia

Problem Statement: I want to subscribe to 800 tickers.

I was able to do it in previous version. However, I am not able to do it on Kite3.

Steps tried:

Step 1: Run supplied example with 500 tickers

Using standard example, I tried subscribing to 500 tickers and it was successful. Code below.

import pandas as pd
from kiteconnect import KiteTicker, KiteConnect
import logging
logging.basicConfig(level=logging.DEBUG)

access_token = "my_access_token"
api_key="my_api_key"

instruments = KiteConnect(api_key=api_key, access_token=access_token).instruments(exchange="NSE")
instruments= pd.DataFrame(instruments)
my_tokens= list(instruments[:500].instrument_token.astype(int))

kws = KiteTicker(api_key=api_key, access_token=access_token)

def on_ticks(ws, ticks):
    logging.debug("1st row in Ticks: {}".format(ticks[0]))
    
def on_connect(ws, response):
    ws.subscribe(my_tokens)
    ws.set_mode(ws.MODE_FULL, my_tokens)

def on_close(ws, code, reason):
    logging.info("Connection closed: {code} - {reason}".format(code=code, reason=reason))

def on_error(ws, code, reason):
    logging.info("Connection error: {code} - {reason}".format(code=code, reason=reason))

def on_reconnect(ws, attempts_count):
    logging.info("Reconnecting: {}".format(attempts_count))

def on_noreconnect(ws):
    logging.info("Reconnect failed.")

kws.on_ticks = on_ticks
kws.on_close = on_close
kws.on_error = on_error
kws.on_connect = on_connect
kws.on_reconnect = on_reconnect
kws.on_noreconnect = on_noreconnect

kws.connect(threaded=True)

This step was successful.

Step 2 : I tried to increase the number of ticker to 600 by changing only one line in the previous code:

my_tokens= list(instruments[:600].instrument_token.astype(int))

It was unsuccessful and I got the following error:

ERROR:kiteconnect.ticker:WebSocket connection lost: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionDone'>: Connection was closed cleanly.
].
ERROR:kiteconnect.ticker:WebSocket connection lost: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionDone'>: Connection was closed cleanly.
].
ERROR:kiteconnect.ticker:WebSocket connection lost: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionDone'>: Connection was closed cleanly.
].

Step 3: I tried running two separate processes of KiteTicker.

In first process, instrument_tokens were selected with below line:

my_tokens= list(instruments[:500].instrument_token.astype(int))

In second process, everything was same but the instrument_tokens were selected with line below:

my_tokens= list(instruments[500:600].instrument_token.astype(int))

The first process was running fine and I was getting tickers, but I got the following error on the 2nd process:

INFO:root:Connection error: 1006 - connection was closed uncleanly (WebSocket connection upgrade failed (429 - TooManyRequests))
INFO:root:Connection closed: 1006 - connection was closed uncleanly (WebSocket connection upgrade failed (429 - TooManyRequests))
ERROR:kiteconnect.ticker:WebSocket connection lost: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionAborted'>: Connection was aborted locally, using.
].
INFO:root:Connection error: 1006 - connection was closed uncleanly (WebSocket connection upgrade failed (429 - TooManyRequests))
INFO:root:Connection closed: 1006 - connection was closed uncleanly (WebSocket connection upgrade failed (429 - TooManyRequests))
ERROR:kiteconnect.ticker:WebSocket connection lost: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionAborted'>: Connection was aborted locally, using.
].
ERROR:kiteconnect.ticker:Try reconnecting. Retry attempt count: 1
INFO:root:Reconnecting: 1
INFO:root:Connection error: 1006 - connection was closed uncleanly (WebSocket connection upgrade failed (429 - TooManyRequests))
INFO:root:Connection closed: 1006 - connection was closed uncleanly (WebSocket connection upgrade failed (429 - TooManyRequests))
ERROR:kiteconnect.ticker:WebSocket connection lost: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionAborted'>: Connection was aborted locally, using.
].

Inferences

1. Seems, the maximum number of tickers enabled for a thread is 500 rather than 1000. Otherwise, it seems to do with the maximum size of message related setting in Autobhan in maxOutgoingMessagePayloadSize setting. I had seen a 1009 error somewhere.
2. Seems, only one connection is working on python endpoint as of now instead of three. Why else we would see Error 429 in websocket where you have repeatedly mentioned that there is no cap on requests.

Requests

1. Please guide on how to stream 800 odd tickers.
2. If either of my inferences 1 or 2 are correct, please rectify it.
3. Please guide on how to run multiple threads of KiteTicker.

The Autobahn library is poorly documented without any good explanation of errors on their documentation /internet. Further, on your production software, I would expect you to catch these basic third party library errors.

Regards
Saurav

Vivek

This seems to be the same issue as others have reported in this thread. You can followup here - https://kite.trade/forum/discussion/3235/websocket-intial-tick-not-received-for-all-instruments#latest

If you are facing new issues apart from anything mentioned in other thread then please create a new thread.

The Autobahn library is poorly documented without any good explanation of errors on their documentation /internet. Further, on your production software, I would expect you to catch these basic third party library errors.

Agree that Autobahn library is poorly documented but we made sure to reciprocate the error messages properly to the user. For example when rate limit applied you could see 429 error code in logs and 1009 error code which means maximum message length exceeded. None of the issue above mentioned was related to Autobahn implementation rather kite ticker backend quirk.

sauravkedia

Thanks Vivek,

I would still say that from a user's perspective, your catching of errors in Autobahn is poor. Unfortunately it means we are having to delve into actual Autobahn codes.

The user shouldn't receive Autobahn error at all. Erorrs like Connection closed cleanly etc dont make sense.

All errors should be caught and wherever it is related to params set by you (say rate limit-1009 etc), user should get a simple error with the relevant details. Wherever it is originating out of Autobahn , you should still catch it and raise it as say GeneralException and then pass on the relevant autobahn error message. Why should user be handling Autobahn errors? It is not a common, popular library and websockets are complicated topics.

This approach gives practical, actionable errors to user while leaving them with flexibility to delve into Autobahn if they so desire.

In this case you compounded the problem by not testing it enough. Your server shouldn't have maxLength issues upto 1000 tickers under any scenario. How could it fail? Didn't you test and check you settings before rolling it out? Further, the 1009 error it threw was completely useless for a person who is coming through it for the first time. And then you switched off multiple connections abruptly not allowing users to have a workaround while they were coming to terms with 1009.

Short message is: Errors have to be graceful, and properly caught. Production systems have to be well tested before roll-out. Any changes in production systems have to be well publicized.

Else, you are setting up landmines for your users.

Regards
Saurav