Connection error: 1006 - connection was closed uncleanly (None)

zerotrader

I am getting the following error 10 minutes into my subscription. All my algo goes for a toss as the connection is not reliable.

I am subscribing to all live tickers in NSE about 2000. The processing involved inside on_ticks callback is fairly simple.
1) It converts the ticks to JSON for IPC bridge
2) writes the tick to a file as JSON

Please let me know how to overcome this error. Do I need to setup multiple subscribers in different processes ? or offload processing to a different thread? is there a network buffer setting I can increase ? disable nagles algorithm ??? or should I reconnect upon disconnection ???

#code below subscribes to about 2000 rics 
#instrlist is about 2000 

def on_ticks(ws, ticks):
    # Callback to receive ticks.
    for tick in ticks:
        ticker = instrticker_map.get(tick['instrument_token'])
        tick['symbol'] = ticker
        data = json.dumps(tick, cls=DateTimeEncoder)
        socket.send_multipart([ticker.encode(), data.encode()])
        file.write(data)
        file.write("\n")
    file.flush()
    print("Time {}".format(time_now()))


def on_connect(ws, response):
    # Callback on successful connect.
    ws.subscribe(instr_list)

    # Set tick in `full` mode.
    ws.set_mode(ws.MODE_FULL, instr_list)


def on_close(ws, code, reason):
    # On connection close stop the main loop
    # Reconnection will not happen after executing `ws.stop()`
    print("Error - {} - closing socket".format(time_now()))
    ws.stop()
    file.close()


def on_order_update(ws, data):
    pass

def on_message(ws, payload, is_binary):
    pass

def on_reconnect(ws, attempts_count):
    print("Error - %s - on reconnect, attempts %s " % (time_now(), attempts_count))
    pass

def on_noreconnect(ws):
    print("Error - {} - On no reconnect called".format(time_now()))
    pass

def on_error(ws, code, reason):
    print("Error - {} - code {} reason {}".format(time_now(), code, reason))
    pass



# Assign the callbacks.
kws.on_ticks = on_ticks
kws.on_connect = on_connect
kws.on_message = on_message
kws.on_order_update = on_order_update
kws.on_reconnect = on_reconnect
kws.on_noreconnect = on_noreconnect
kws.on_close = on_close
kws.on_error = on_error



# Infinite loop on the main thread. Nothing after this will run.
# You have to use the pre-defined callbacks to manage subscriptions.
kws.connect()

rakeshr

@zerotrader
You need to send socket.send_multipart([ticker.encode(), data.encode()]) to different thread, without blocking main on_tick thread, with kws.connect(threaded=True) .

zerotrader

thanks for the suggestions.. I moved the socket.send_multipart to a separate thread without blocking main on_tick (or the twisted reactor) thread. Here I used a sufficiently large queue to buffer for any processing lags or large burst of data.

The kws.connect(Threaded=True) is simply launching the twisted reactor in a thread ??? that some thing I didn't want.. as already all my project modules are separated into processes.

With the fixes above, it has improved from before .. but still connection drops in 2-3 hours.. or 3-4 times in a trading day. The re-connect is ok but that misses out on 3-10 seconds of data.

Please let me know how to reconnect faster than 3 seconds. Is the reconnect parameters like delay is configurable? I see there are on_reconnect and on_noreconnect handlers are exposed.

Vivek

@zerotrader Client side network disconnection is very common especially for WebSocket connections, If you don't want to miss out ticks while it was reconnecting then you have to run multiple ticker instances preferably in different machine and store it centrally somewhere like a DB or Redis.

Also don't do any operations in `on_tick` callback, create a new function and call it from `on_tick` asynchronously. For example in your case move file write also to different thread. Most common reason for WebSocket disconnection we have found was main thread getting blocked.

trade_then

@vivek and have you found a problem when the main thread was actually very fast by virtue of not getting blocked atall .
Please don`t reply it here. I dont want to highjack this thread. i am about to ask about it in my own post but found it tempting to start from here.

Thanks
Regards