Websocket Intial Tick not received for all instruments

__name__

I subscribed to 500 instruments at once on about 11 threads on WebSockets, but it is not working. I don't get an initial tick for every instrument:

Thread no - Ticks/Total number of instruments

0-186/550
1-205/550
3-207/550
2-195/550
5-184/550
4-178/550
6-193/550
7-193/550
8-208/550
9-182/550
10-195/549

How can I then get the entire live updates?

Also, multithreading gives me this error:

Exception in thread Thread-5:
Traceback (most recent call last):

  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.5/dist-packages/twisted/internet/base.py", line 1242, in run
    self.startRunning(installSignalHandlers=installSignalHandlers)
  File "/usr/local/lib/python3.5/dist-packages/twisted/internet/base.py", line 1222, in startRunning
    ReactorBase.startRunning(self)
  File "/usr/local/lib/python3.5/dist-packages/twisted/internet/base.py", line 728, in startRunning
    raise error.ReactorAlreadyRunning()
twisted.internet.error.ReactorAlreadyRunning

To reproduce this, create a new thread which creates a threaded KiteSocket. Make 2 such instances of Thread.

sauravkedia

Hi __name__,
This problem can be handled. In zerodha's kiteconnect.ticker module, there is connect mothod in class KiteTicker. At the end, there are the following lines:

        opts = {}
        if threaded:
            # Signals are not allowed in non main thread by twisted so supress it.
            opts["installSignalHandlers"] = False
            self.websocket_thread = threading.Thread(target=reactor.run, kwargs=opts)
            self.websocket_thread.daemon = True
            self.websocket_thread.start()
        else:
            reactor.run(**opts)

reactor should be run only once in the process. So create a subclass of KiteTicker, override connect method to remove this block . And, in you application, run this block only once after you have spawned all the threads listening to the websocket. The reactor object can be imported using the following (same as in kiteconenct.ticker)

from twisted.internet import reactor

Hopefully, this error will be taken care of, but you will likely get another error. One thread will run fine, but you will get following error in other threads:

INFO:root:Connection error: 1006 - connection was closed uncleanly (WebSocket connection upgrade failed (429 - TooManyRequests))
INFO:root:Connection closed: 1006 - connection was closed uncleanly (WebSocket connection upgrade failed (429 - TooManyRequests))
ERROR:kiteconnect.ticker:WebSocket connection lost: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionAborted'>: Connection was aborted locally, using.
].

I have started a new discussion with these issues, it is waiting to be approved by Zerodha before it appears on the forum.

Good luck and let us know if you are successful. Facing similar issues here.
Regards

__name__

As far as i know, there is no rate limiting for creating WebSockets. then, why we are getting 429.

Is the following error related to 429:

ERROR:kiteconnect.ticker:WebSocket connection lost: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionAborted'>: Connection was aborted locally, using.].


sauravkedia

Thats the fun part, when I try a single thread with 600 odd tickers, I get the error mentioned by you above. And when I try to run multiple threads, I get 429 error. As long as I am happy with 500 tickers in one thread, things are okay.

Vivek

1. Same KiteTicker instance shouldn't be used in different threads since you can run only one twisted reactor instance hence you get error `twisted.internet.error.ReactorAlreadyRunning`.
2. When more than 500 instruments are subscribed the message length exceed the maximum length set by server so it closes connection with 1009 error.
3. More than once connection can't be created since its rate limited.

For now I would suggest you guys to run KiteTicker in main thread or different thread as given here and subscribe to instruments in batch. Instead of subscribing more than 500 at once subscribe multiple times (though max 1000 instruments are allowed).

Since its announced that 1000 instruments are supported I will check backend team about that.

sauravkedia

Hi Vivek,

1. Looks alright.
2. Your team need to sort that max length setting. It is configurable. Further, can you advise how to subscribe in batches? Please share sample code. Why advertise a functionality which is not usable.
3. I thought we are allowed to run three connections and there was no rate limitation on websocket. There is a confusion here. What was declared is that three connections are supported and there is no rate limitation. Now, in Kite3, if there is only one connection allowed and/or there is rate limitation, please make a proper announcement, insert in changelog, explain to users how it affects them, then allow them to migrate and not let it slip like this. It breaks user codes and like you, we also struggle when some vendor abruptly changes things especially the breaking ones. Where is the mention of this in your changelog.

Further, I was able to establish multiple connections till about yesterday noon, after that I have started facing problems.

Regards
Saurav

sauravkedia

Between I have asked similar questions in my other thread (which has now got approved by your team). You can reply over there.

Vivek

1. You can send more than 500 instruments now, this has been fixed in backend
2. We have increased the limit for now but while announcing migration deadlines we will specify the rate limits for this as well.

sauravkedia

Thank you Vivek,

The problems have been fixed for now. In case there is any change in rate limits etc ,please announce well in advance and allow users to adjust/migrate.

Regards
Saurav