kiteTicker error

@SharadKothari
It seems to be happening because of the blocking of the main ticker thread. You can go through this thread to more about it.

And this is the error. Though it is not throwing error always, but the error is un predictable and worrysome if I am taking large volumes.

and here is the error

HINDCOPPER, -0.3000000000001819, 12.4689
Unhandled Error
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/twisted/python/log.py", line 103, in callWithLogger
return callWithContext({"system": lp}, func, *args, **kw)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/twisted/python/log.py", line 86, in callWithContext
return context.call({ILogContext: newCtx}, func, *args, **kw)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/twisted/python/context.py", line 122, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/twisted/python/context.py", line 85, in callWithContext
return func(*args,**kw)
--- ---
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/twisted/internet/selectreactor.py", line 149, in _doReadOrWrite
why = getattr(selectable, method)()
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/twisted/internet/tcp.py", line 243, in doRead
return self._dataReceived(data)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/twisted/internet/tcp.py", line 249, in _dataReceived
rval = self.protocol.dataReceived(data)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/twisted/protocols/tls.py", line 330, in dataReceived
self._flushReceiveBIO()
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/twisted/protocols/tls.py", line 295, in _flushReceiveBIO
ProtocolWrapper.dataReceived(self, bytes)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/twisted/protocols/policies.py", line 120, in dataReceived
self.wrappedProtocol.dataReceived(data)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/autobahn/twisted/websocket.py", line 291, in dataReceived
self._dataReceived(data)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/autobahn/websocket/protocol.py", line 1213, in _dataReceived
self.consumeData()
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/autobahn/websocket/protocol.py", line 1225, in consumeData
while self.processData() and self.state != WebSocketProtocol.STATE_CLOSED:
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/autobahn/websocket/protocol.py", line 1591, in processData
fr = self.onFrameEnd()
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/autobahn/websocket/protocol.py", line 1713, in onFrameEnd
self._onMessageEnd()
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/autobahn/twisted/websocket.py", line 319, in _onMessageEnd
self.onMessageEnd()
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/autobahn/websocket/protocol.py", line 634, in onMessageEnd
self._onMessage(payload, self.message_is_binary)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/autobahn/twisted/websocket.py", line 322, in _onMessage
self.onMessage(payload, isBinary)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/kiteconnect/ticker.py", line 71, in onMessage
self.factory.on_message(self, payload, is_binary)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/kiteconnect/ticker.py", line 670, in _on_message
self.on_ticks(self, self._parse_binary(payload))
File "/Users/sapan/Desktop/Sapan/coding/python/python_learning/ideas_with_kite_api-7.py", line 177, in on_ticks
helper_method_2(ticks= ticks)
File "/Users/sapan/Desktop/Sapan/coding/python/python_learning/ideas_with_kite_api-7.py", line 184, in helper_method_2
open_position = kite.positions()["net"]
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/kiteconnect/connect.py", line 441, in positions
return self._get("portfolio.positions")
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/kiteconnect/connect.py", line 826, in _get
return self._request(route, "GET", url_args=url_args, params=params, is_json=is_json)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/kiteconnect/connect.py", line 877, in _request
raise e
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/kiteconnect/connect.py", line 865, in _request
r = self.reqsession.request(method,
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/adapters.py", line 529, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.kite.trade', port=443): Read timed out. (read timeout=7)

builtins.AttributeError: 'NoneType' object has no attribute 'append'

The above error occurs when handling another initiating error in your codebase.

helper_method_2
open_position = kite.positions()["net"]
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/kiteconnect/connect.py", line 441, in positions
return self._get("portfolio.positions")

requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.kite.trade', port=443): Read timed out. (read timeout=7)

It's because of time-out happening while continously polling positions with every tick. Which is not recommended at all. Go through this article to know more.

@SAPANKHUNTIA,
Kite Connect is not suitable for latency-based or high-frequency trading. For HFT, you need to set up a colocation system at the exchange. It will cost above 18lakhs per annum.

Basically, you shouldn't block the thread that is receiving ticks. All your computations, DB dumps and data fetches should be moved to a secondary thread.

Hi @sujith, @rakeshr and team,

I tried something like this.

def helper_method_2(ticks):
    for company in ticks:
        Perform computation here
        place orders
        request open positions list
while True:
    def on_ticks(ws, ticks):
        for tick in ticks:
            print(ticks)
        new_thread = threading.Thread(target=helper_method_2(ticks = ticks))
        new_thread.start()
    #Assign callback
    kws.on_ticks=on_ticks

But i think my purpose won't be served here as the main thread will be blocked untill the new_thread is processed. Kindly share your words on this.

Again I am trying to do something like this

ticks_data = {}
def helper_method_2(ticks):
    for company in ticks:
        Perform computation here
        place orders
        request open positions list

def on_ticks(ws, ticks):
    global ticks_data 
    ticks_data= ticks

def on_connect(ws, response):
    ws.subscribe(instrument_tokens)
    ws.set_mode(ws.MODE_QUOTE, instrument_tokens)

kws.on_ticks = on_ticks
kws.on_connect = on_connect
# kws.on_close = on_close

kws.connect(threaded=True)

while True:
    t1 = threading.Thread(target=on_ticks(ws = ws, ticks = ticks))
    new_thread = threading.Thread(target=helper_method_2(ticks = ticks_data))
    
    t1.start()
    new_thread.start()
    
    t1.join()
    new_thread.join()

    kws.on_ticks=on_ticks

But again here I am stuck as I am confused what to assign to ws and ticks in the on_ticks(ws,ticks) function. and the same confusion arises here that the main thread will be blocked until the new_thread joins the flow.

Being a beginner I am finding it tough. Any suggestion will be very helpful.
Thank you.

hi @RED
Thank you.

Seems likee you are talking about the second method.
But it is throwing an error ticks and ws is not defined. so what to assign these two?


And again if i remove the thread.join() then will it be fine ?

Thank you @RED.
I have tried as you have mentioned and i was running the callback and calculations in two different threads. After 2:30 Hrs of smooth running it threw a similar error
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.kite.trade', port=443): Read timed out. (read timeout=7)
and the code stopped completely.

def helper_method_2(ticks):
      request open positions list
      for company in ticks:
            Perform computation here
            place orders
while True:
      def on_ticks(ws, ticks):
            global ticks_data
            ticks_data = ticks
      if not new_thread.is_alive():
            new_thread = threading.Thread(target=helper_method_2(ticks = ticks_data), daemon= True)
            new_thread.start()
      dt_now = datetime.datetime.now().time()
      kws.on_ticks=on_ticks

here is the complete error stack.
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1347, in getresponse
response.begin()
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 307, in begin
version, status, reason = self._read_status()
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 268, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 704, in readinto
return self._sock.recv_into(b)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1241, in recv_into
return self.read(nbytes, buffer)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1099, in read
return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/util/retry.py", line 531, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/packages/six.py", line 735, in reraise
raise value
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 447, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 336, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='api.kite.trade', port=443): Read timed out. (read timeout=7)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/sapan/Desktop/Sapan/coding/python/python_learning/ideas_with_kite_api-7_1.py", line 294, in
new_thread = threading.Thread(target=helper_method_2(ticks = ticks_data), daemon= True)
File "/Users/sapan/Desktop/Sapan/coding/python/python_learning/ideas_with_kite_api-7_1.py", line 121, in helper_method_2
open_position = kite.positions()["net"]
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/kiteconnect/connect.py", line 441, in positions
return self._get("portfolio.positions")
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/kiteconnect/connect.py", line 826, in _get
return self._request(route, "GET", url_args=url_args, params=params, is_json=is_json)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/kiteconnect/connect.py", line 877, in _request
raise e
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/kiteconnect/connect.py", line 865, in _request
r = self.reqsession.request(method,
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/adapters.py", line 529, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.kite.trade', port=443): Read timed out. (read timeout=7)

seems like again I am getting something wrong in polling positions. as i"m again stuck at the same issue.

File "/Users/sapan/Desktop/Sapan/coding/python/python_learning/ideas_with_kite_api-7_1.py", line 294, in
new_thread = threading.Thread(target=helper_method_2(ticks = ticks_data), daemon= True)
File "/Users/sapan/Desktop/Sapan/coding/python/python_learning/ideas_with_kite_api-7_1.py", line 121, in helper_method_2
open_position = kite.positions()["net"]

Is there any way to reconnect if this error arises?

Kindly help me with this.
Thank you.

Hi @SAPANKHUNTIA,

You are calling the thread wrong, it doesn't spawn a new worker thread instead it runs on main.

Code:
new_thread = threading.Thread(target=helper_method_2, args= (ticks_data ), daemon= True)

Pass args seperately.

Hi red , im tried to run this but im getting t1 not defined error ..... is there a way to define it outside the if condition .

ill try with to initialize
t1=None
before the start of the of the while loop