Error: Can't subscribe to more than 3000 instruments.

naz

@sujith @rakeshr
Is there a place in documentation where limitations of subscription is mentioned. If yes, can you please put a reference here.
Is the limit 3000 per API key (1000 per socket with MAX 3 web sockets) OR 9000 per API KEY (3000 per socket with MAX 3 sockets). We have following questions:

1) What is the subscription limit per API KEY ? Will all websockets connection drop if we subscribe more than this ?
2) What is the subscription limit per WEB Socket ? Will the websocket be terminated if we subscribe more than this ?
3) what is the max no. of websockets per API KEY ? Will the API make a hard stop after the 3rd websocket ?

sujith

You can open up to 3 websocket connections per api key.
You can subscribe for up to 3000 instruments per websocket connection. If you subscribe for more than 3000 instruments then you will receive an error message but your connection is not affected by that. It is just that the subscription for new tokens will fail.
There is no hard limit for the number of websockets connections as of now. There is only a soft limit. But if it is flagged then your app might get suspended.

naz

@sujith
Thanks for the quick revert. From your response, it looks like we can subscribe for Max 9000 instruments.
I am getting the error even when I am subscribing for ~7700 instruments (2800 + 2800 + 2100). I am getting the below error:
"Can't subscribe to more than 3000 instruments."

Can't figure out why ? Is it possible for you to check at your end as to why the kite websocket server code is taking the requests as more than 3000 symbols per websockets ?
Happy to share my API key/client ID, if required.

rakeshr

@naz
We have checked at our end, we haven't found any error if you subscribe up to 3000 instruments in a single connection. Make sure, you are creating three different WebSocket connection and not merging them in the same thread.

naz

@rakeshr
Based on your response, i tried figuring out the websocket connections and current thread by running the below mentioned "on_ticks" function. Looks like what you mentioned was correct. While the 3 different websockets are being created and run, it is showing the current thread as Thread_1 always. On the other hand, it should have shown Thread_0, Thread_1, Thread_2 respectively (please correct me).
Also, while all the 3 websockets are dumping the data for ~8000 symbols, there are frequent disconnections/reconnections happening with the error signature "Can't subscribe to more than 3000 instruments."

I have tried following:
1) creating 3 websocket instances in the main thread and calling ws.connect(threaded=True) on each of those instances in a for loop (zerodha's implementation of threading)

2) creating 3 websocket instances in the main thread and creating our own "threading.Thread" objects which encapsulate "ws.connect()" for each of those instances and then doing thread.start() in a loop (basically our own implementation of threading)

3) create a threading.Thread object with a target function. Job of the target function is to both create a websocket instance and run we.connect(). Basically, this was done to create a ws object in a separate thread instead of in the main thread.

Nothing seemed to have worked. Could you share the test where you have implemented the 3 websockets and max 3000 symbols and which is working properly.


def on_ticks(self, ws, ticks):
logger.debug( "{} received ticks from websocket {}, thread: {}".format(dt.datetime.now(), ws, threading.current_thread().name))

Following is the output:
2018-12-19 15:06:00.075894 received ticks from websocket , thread: streamerThread_1
2018-12-19 15:06:00.357039 received ticks from websocket , thread: streamerThread_1
2018-12-19 15:06:09.103625 received ticks from websocket , thread: streamerThread_1


And below are the error signatures:
on_error: callback called for thread streamerThread_1, code: 0, reason: Can't subscribe to more than 3000 instruments.

zartimus

work fine here. It will only fail if you have more than 3000 tokens

import logging
from copy import copy
from kiteconnect import KiteConnect, KiteTicker

logging.basicConfig(level=logging.DEBUG)

api_key = "<API-KEY>"
access_token = "<ACCESS-TOKEN>"

client = KiteConnect(api_key=api_key, access_token=access_token)

_all_instruments_tokens = list(map(lambda s: s["instrument_token"], client.instruments()))

tokens1 = _all_instruments_tokens[0:3000]
tokens2 = _all_instruments_tokens[3000:6000]
tokens3 = _all_instruments_tokens[6000:9000]


def on_ticks1(ws, ticks):
    logging.debug("ticker 1: {}".format(len(ticks)))

def on_ticks2(ws, ticks):
    logging.debug("ticker 2: {}".format(len(ticks)))

def on_ticks3(ws, ticks):
    logging.debug("ticker 3: {}".format(len(ticks)))


def on_connect1(ws, response):
    ws.subscribe(tokens1)
    ws.set_mode(ws.MODE_FULL, tokens1)

def on_connect2(ws, response):
    ws.subscribe(tokens2)
    ws.set_mode(ws.MODE_FULL, tokens2)

def on_connect3(ws, response):
    ws.subscribe(tokens3)
    ws.set_mode(ws.MODE_FULL, tokens3)


if __name__ == "__main__":
    # Create three instances
    kws1 = KiteTicker(api_key, access_token, debug=True)
    kws2 = copy(kws1)
    kws3 = copy(kws1)

    kws1.on_ticks = on_ticks1
    kws1.on_connect = on_connect1

    kws2.on_ticks = on_ticks2
    kws2.on_connect = on_connect2

    kws3.on_ticks = on_ticks3
    kws3.on_connect = on_connect3

    kws1.connect(threaded=True)  
    kws2.connect(threaded=True)
    kws3.connect(threaded=True)

    # Block main thread
    while True:
        pass

naz

Thanks much @zartimus
while i will be able to test this tomorrow when the markets are open, I did a quick experiment and changed the piece of code to show current thread's name as well.

def on_ticks1(ws, ticks):
logging.debug ("ticker 1: {}".format(threading.current_thread().name))
logging.debug("ticker 1: {}".format(len(ticks)))

def on_ticks2(ws, ticks):
logging.debug("ticker 2: {}".format(threading.current_thread().name))
logging.debug("ticker 2: {}".format(len(ticks)))

def on_ticks3(ws, ticks):
logging.debug("ticker 3: {}".format(threading.current_thread().name))
logging.debug("ticker 3: {}".format(len(ticks)))

Following is the output with all threads being shown as Thread-6.

DEBUG:root:ticker 3: Thread-6
DEBUG:root:ticker 3: 3000
DEBUG:root:ticker 2: Thread-6
DEBUG:root:ticker 2: 3000
DEBUG:root:ticker 1: Thread-6
DEBUG:root:ticker 1: 3000

Is this the expected behavior ?
Shouldn't the 3 different threads be shown as output (probably Thread-6, Thread-7, Thread-8) because different daemon threads are being spawned internally as this is multi-threading scenario ?

naz

@zartimus

You can reproduce this error on the code provided by you in the following manner:
1) run the code as pasted by you above
2) after it has spawned all 3 threads, disconnect the internet
3) connect the internet again and you will face the error "Can't subscribe to more than 3000 instruments"

Looks like a false alarm and a misplaced error message in the zerodha's code. Please advise if it is safe to ignore (once you have reproduced it) and is a false alarm.

zartimus

@naz
Sorry, single process can only have one reactor
https://github.com/zerodhatech/pykiteconnect/blob/master/kiteconnect/ticker.py#L524
So, even if you try creating multiple threads, it wont. that explains the first issue

But you can easily circumvent using multiprocessing

    processes = []
    for i in range(1, 4):
        p = multiprocessing.Process(target=locals()["kws%s" % i].connect)
        processes.append(p)
        p.start()

    for p in processes:
        p.join()

Second issue,
Yes i can confirm that i m getting this error.
When i try adding unsubscribe before https://github.com/zerodhatech/pykiteconnect/blob/master/kiteconnect/ticker.py#L641. it works

But tokens are per connection. When you reconnect its a new connection. Basically you don't need to unsubscribe when resubscribing. Will check into this.

naz

Thanks @zartimus for the detailed response again - always helpful.
For the time being, I will ignore this error message (or put an try, exception) since we are getting correct ticks post reconnection as well.
Please do inform once you have checked into this.