python program snaps midway while storing live streaming into sqllite database

D11458

Hello @rakeshr @sujith ,

I am running python code to store websocket live streaming into a database, it works for a minute or so then stops midway. Noticed that this issue comes only while running a full program of live streaming data + storing into db. However if I simply run live streaming code, it works fine, no problems at all.
Please can someone assist.

Below error message.
Connection error: 1006 - connection was closed uncleanly (peer dropped the TCP connection without previous WebSocket closing handshake)
Connection closed: 1006 - connection was closed uncleanly (peer dropped the TCP connection without previous WebSocket closing handshake)

rakeshr

It's because DB write operation is blocking the main ticker thread. You can go through Websocket streaming FAQs here.

D11458

Thanks @rakeshr - I tried to build it based on your above link notes, however, its not working for some reason. Could you please assist with the execution here, below is the code:
======


===

D11458

rakeshr

however, its not working for some reason

Is it throwing the same disconnection error? We don't see any error log for this. Also, you will need to use queue manager(celery, etc) here because of the heavy DB write operations. Check example 1, on the same thread.

I am running python code to store WebSocket live streaming into a database.

If you are looking to store tick data to DB, you can go through this curated example of ours for python client.

D11458

Hi @rakeshr - Thanks for reply. Sorry but 2nd part of screenshot was missing here. Yes the first part of my code above works at my end too as it is creating tables for each instrument. The next part which as per your advice, is created as a new thread and that's where I am stuck up. Could you please see if anything wrong? Below is the last part of the code:

#Continued from above screenshot:
while True:

#Perform required data operation using tick data
def on_ticks(ws, ticks):
insert_ticks(ticks)
print (ticks)
def insert_ticks(ticks):
cd=db.cursor()
for tick in ticks:
try:
tok = "TOKEN"+str(tick['instrument_token'])
vals = [tick['timestamp'],tick['last_price'], tick['volume']]
query = "INSERT INTO {}(ts,price,volume) VALUES (?,?,?)".format(tok)
cd.execute(query,vals)
except:
pass
try:
db.commit()
except:
db.rollback()


#Assign callback
kws.on_ticks=on_ticks

D11458

FYI - I am calling around 500 instruments data with this program.

D11458

Fixed it - at the moment looks like the fix with some changes working on new thread ....thanks and this query can be closed.