Connection error: 1006 - connection was closed uncleanly (None) DESPITE using threaded ticker

tapanjbhatt
Hello,

I use threaded ticker as advised on lot of discussion on this forum. The basic structure looks: in on_ticks(), we do not do any processing. newly arrived tick is just put into a python queue object using q.put() operation, which is non-blocking (refer https://docs.python.org/3/library/queue.html#queue.Queue.put).

def on_ticks(ws, ticks):
q1.put(ticks)

Entire processing takes place inside separate thread:

x = threading.Thread(target=process_ticks_in_thread)

def process_ticks_in_thread():
# read q1 and process ticks in orderly FIFO fashion here.

The code works 99% time of the live market. However, we consistently see at least few errors daily basis:

arrived 256265 : 2022-04-28 13:45:58 : 17272.85
arrived 260105 : 2022-04-28 13:45:58 : 36348.1
process 256265 : 2022-04-28 13:45:57 : 17273.45 : {'o': 17189.5, 'h': 17273.7, 'l': 17071.2, 'c': 17273.45}
process 260105 : 2022-04-28 13:45:57 : 36352.25 : {'o': 36189.95, 'h': 36362.35, 'l': 35952.9, 'c': 36352.25}
--------------------------------------------------------------------------------------------
TOACT: kite ticker was disconnected
TOACT: kite ticker was disconnected
ERROR:kiteconnect.ticker:Connection error: 1006 - connection was closed uncleanly (None)
ERROR:kiteconnect.ticker:Connection closed: 1006 - connection was closed uncleanly (None)
TOACT: kite ticker was disconnected
TOACT: kite ticker was disconnected
TOACT: kite ticker was disconnected
TOACT: kite ticker was disconnected
ERROR:kiteconnect.ticker:Connection error: 1006 - connection was closed uncleanly (WebSocket opening handshake timeout (peer did not finish the opening handshake in time))
ERROR:kiteconnect.ticker:Connection closed: 1006 - connection was closed uncleanly (WebSocket opening handshake timeout (peer did not finish the opening handshake in time))
connected
arrived 256265 : 2022-04-28 13:46:36 : 17271.65
arrived 260105 : 2022-04-28 13:46:36 : 36353.25
process 256265 : 2022-04-28 13:45:58 : 17272.85 : {'o': 17189.5, 'h': 17273.7, 'l': 17071.2, 'c': 17272.85}
process 256265 : 2022-04-28 13:45:59 : 17272.85 : {'o': 17189.5, 'h': 17273.7, 'l': 17071.2, 'c': 17272.85}

if you see here, we lost ticks for around 38 seconds. We feel quite uncomfortable about this as we calculate minute candles and missing end of minute tick means much damage to the math we are doing.

Can you help investigate and resolve this issue? We are having 24 hours full network connectivity, and despite that we see this error daily at least in one ticker (we run three at a time).

Any help to fully resolve this issue will be appreciated. In case we are not able to solve this issue fully, it is potential threat to the our trading model.
  • rakeshr
    peer did not finish the opening handshake in time
    Seems like issue of python GIL.
    You can check out this example as an alternative.
  • tapanjbhatt
    Hi @rakeshr ,

    Thanks for response. However one thing I would like to highight is we rarely see "peer did not finish the opening handshake in time". Checked last 7 days of log and we found only one isntance, that is posted here.

    However, below this error "Connection error: 1006 - connection was closed uncleanly (None)" occures consistently on daily basis, and whenever it occurs we have downtime of almost 1 minute. See today's log entries:

    --------------------------------------------------------------------------------------------
    arrived 256265 : 2022-04-29 11:52:58 : 17350.4
    arrived 260105 : 2022-04-29 11:52:58 : 36693.75
    process 256265 : 2022-04-29 11:52:57 : 17350.15 : {'o': 17329.25, 'h': 17357.35, 'l': 17241.9, 'c': 17350.15} in 0.0 sec
    process 260105 : 2022-04-29 11:52:57 : 36692.65 : {'o': 36474.05, 'h': 36716.65, 'l': 36345.7, 'c': 36692.65} in 0.0 sec
    --------------------------------------------------------------------------------------------
    TOACT: kite ticker was disconnected
    TOACT: kite ticker was disconnected
    TOACT: kite ticker was disconnected
    TOACT: kite ticker was disconnected
    TOACT: kite ticker was disconnected
    TOACT: kite ticker was disconnected
    TOACT: kite ticker was disconnected
    TOACT: kite ticker was disconnected
    TOACT: kite ticker was disconnected
    TOACT: kite ticker was disconnected
    TOACT: kite ticker was disconnected
    ERROR:kiteconnect.ticker:Connection error: 1006 - connection was closed uncleanly (None)
    TOACT: kite ticker was disconnectedERROR:kiteconnect.ticker:Connection closed: 1006 - connection was closed uncleanly (None)

    TOACT: kite ticker was disconnected
    TOACT: kite ticker was disconnected
    TOACT: kite ticker was disconnected
    TOACT: kite ticker was disconnected
    TOACT: kite ticker was disconnected
    TOACT: kite ticker was disconnected
    TOACT: kite ticker was disconnected
    TOACT: kite ticker was disconnected
    TOACT: kite ticker was disconnected
    TOACT: kite ticker was disconnected
    TOACT: kite ticker was disconnected
    TOACT: kite ticker was disconnected
    TOACT: kite ticker was disconnected
    TOACT: kite ticker was disconnected
    TOACT: kite ticker was disconnected
    TOACT: kite ticker was disconnected
    TOACT: kite ticker was disconnected
    TOACT: kite ticker was disconnected
    TOACT: kite ticker was disconnected
    TOACT: kite ticker was disconnected
    connected
    [256265, 260105]
    arrived 256265 : 2022-04-29 11:53:58 : 17348.75
    arrived 260105 : 2022-04-29 11:53:58 : 36686.35
    process 256265 : 2022-04-29 11:52:58 : 17350.4 : {'o': 17329.25, 'h': 17357.35, 'l': 17241.9, 'c': 17350.4} in 0.001 sec

    We lost one minute worth of ticks and the candle starting at 11:52 and 11:53 are calculated wrongly.

    Is there any way we can debug deeper and see lower level errors? This issues very critical for us. Meanwhile we are doing pilot on the celery example that you pointed.

    Thanks in advance.
  • tapanjbhatt
    tapanjbhatt edited May 2022
    hello @rakeshr @sujith @SRIJAN

    We had implemented fully multi-threaded ticker on 5th may and above issue vanished for around 20 days.

    We are handling only two callbacks: on_tick() and on_order_update(). Both does NOT do anything apart from adding the data to a python queue.

    The processing happpens on two separate threads, one handles the ticks from python Q and other handles order updates from another python Q.

    Since we are not doing any processing within calllbacks, the issue was gone.

    until TODAY.

    We saw below error despite of offloading entire workload from the main callback hooks:

    This error came at 14:30:20
    ERROR:kiteconnect.ticker:Connection closed: 1006 - connection was closed uncleanly (None)

    Another similar (but not exactly same) error came at 14:31:12

    ERROR:kiteconnect.ticker:Connection closed: 1006 - connection was closed uncleanly (peer dropped the TCP connection without previous WebSocket closing handshake)

    Why this keeps on occuring? We have clearly removed all processing from on_XXXX callbacks and have made them very lightweight. Though auto-reconnect feature worked, we are running risk of losing a full 15 min candle (as the issue occurred at 14:30) and other possible issues like impact cost/slippage if order is being placed at same time.
  • SRIJAN
    SRIJAN edited May 2022
    Please post the exact error stacktrace.
    This error
    "Connection closed: 1006 - connection was closed uncleanly (peer dropped the TCP connection without previous WebSocket closing handshake)"
    generally happens due to some internal error in your code which causes the websocket to disconnect.
  • tapanjbhatt
    tapanjbhatt edited May 2022
    hello @SRIJAN

    Our code is not generating any stacktrace, we checked complete log. I assume by stacktrace you mean the python's standard way to pring error which starrts with "Traceback (most recent call last):"

    To print stacktrace forcefully, we can wrap code within try...except but question is where to put try...except? in your sample we do not find any try...except.

    I have pasted the main part of our code here. Help us, where should we put what code so that it generates stacktrace?



    def on_order_update(ws, data):
    logging.info(f"Ticker on_order_update data: {data}")

    order_id = data['order_id']

    # if order_id in orders:
    filled_qty = data['filled_quantity']
    avg_price = data['average_price']

    if filled_qty > 0:
    print('order id: {0} filled qty: {1}'.format(order_id, filled_qty))

    order_update = {'order_id': order_id, 'filled_qty': filled_qty, 'avg_price': avg_price}
    order_updates_q.put(order_update)

    def on_ticks(ws, ticks):
    ticks_q.put(ticks)

    if __name__ == '__main__':

    logging.basicConfig(level=logging.DEBUG)

    # cleanup_db()

    kite = KiteConnect(api_key='<my api key>')
    access_token = get_today_token_from_db()
    kite.set_access_token(access_token)

    # if pivot_timeframe in overnight_timeframes:
    # init_ohlc()

    expity_to_trade = get_expiry_month(datetime.date.today(), expiry)
    fno_trading_symbols = get_trading_symbols(instruments, expity_to_trade)

    print(fno_trading_symbols)

    load_trade_state()
    init_pivots()
    init_ticks_data()
    init_position_status()
    load_open_positions()
    # print(ohlc)
    # print(pivot)


    kws = KiteTicker("<my api key>", access_token, reconnect_max_tries=300, reconnect_max_delay=5)

    kws.on_ticks = on_ticks
    kws.on_connect = on_connect
    kws.on_close = on_close
    kws.on_order_update = on_order_update

    kws.connect(threaded=True)

    count = 0
    x = threading.Thread(target=process_ticks_in_thread)
    x.start()

    y = threading.Thread(target=add_order_entry_and_process_order_update)
    y.start()

    while True:
    count += 1
    time.sleep(1)
    if not kws.is_connected():
    print('TOACT: kite ticker was disconnected')
  • tapanjbhatt
    @SRIJAN , just FYI, python indentation is getting lost when i use code formatting...but you shall be able to understand it.
  • SRIJAN
    SRIJAN edited May 2022
    Tracebacks are not automatically logged.
    Error can happen anywhere.
    You have to find those parts where errors are possible and wrap them in try-except block and log them.
    Did you check if there was no error output on the terminal?? Because you are not using any try-except , if there will be any error,it will be definitely displayed on the terminal.

    If there was no error,this connection error might be happening if you are doing heavy computation .

    You can try queue method as described here:
    https://kite.trade/forum/discussion/comment/25535/#Comment_25535

    Checkout this example:
    https://github.com/zerodha/kite-connect-python-example
Sign In or Register to comment.