python program snaps midway while storing live streaming into sqllite database

D11458
Hello @rakeshr @sujith ,

I am running python code to store websocket live streaming into a database, it works for a minute or so then stops midway. Noticed that this issue comes only while running a full program of live streaming data + storing into db. However if I simply run live streaming code, it works fine, no problems at all.
Please can someone assist.

Below error message.
Connection error: 1006 - connection was closed uncleanly (peer dropped the TCP connection without previous WebSocket closing handshake)
Connection closed: 1006 - connection was closed uncleanly (peer dropped the TCP connection without previous WebSocket closing handshake)
  • rakeshr
    It's because DB write operation is blocking the main ticker thread. You can go through Websocket streaming FAQs here.
  • D11458
    D11458 edited April 2021
    Thanks @rakeshr - I tried to build it based on your above link notes, however, its not working for some reason. Could you please assist with the execution here, below is the code:
    ======


    ===
  • rakeshr
    however, its not working for some reason
    Is it throwing the same disconnection error? We don't see any error log for this. Also, you will need to use queue manager(celery, etc) here because of the heavy DB write operations. Check example 1, on the same thread.
    I am running python code to store WebSocket live streaming into a database.
    If you are looking to store tick data to DB, you can go through this curated example of ours for python client.
  • D11458
    Hi @rakeshr - Thanks for reply. Sorry but 2nd part of screenshot was missing here. Yes the first part of my code above works at my end too as it is creating tables for each instrument. The next part which as per your advice, is created as a new thread and that's where I am stuck up. Could you please see if anything wrong? Below is the last part of the code:

    #Continued from above screenshot:
    while True:

    #Perform required data operation using tick data
    def on_ticks(ws, ticks):
    insert_ticks(ticks)
    print (ticks)
    def insert_ticks(ticks):
    cd=db.cursor()
    for tick in ticks:
    try:
    tok = "TOKEN"+str(tick['instrument_token'])
    vals = [tick['timestamp'],tick['last_price'], tick['volume']]
    query = "INSERT INTO {}(ts,price,volume) VALUES (?,?,?)".format(tok)
    cd.execute(query,vals)
    except:
    pass
    try:
    db.commit()
    except:
    db.rollback()


    #Assign callback
    kws.on_ticks=on_ticks
  • D11458
    FYI - I am calling around 500 instruments data with this program.
  • D11458
    Fixed it - at the moment looks like the fix with some changes working on new thread ....thanks and this query can be closed.
This discussion has been closed.