@rakeshr - The solution 2 you have proposed in this thread. That is using threaded=True and a while True loop in the main thread. Will that not generate hundreds of copies of on_ticks and helper_method functions every second as it continuously loops? How can it be advisable to implement such an approach?
Also, as per my understanding, all that threaded=True in kws.connect call does is to run the reactor in a separate thread. So, when later in while loop in the main thread, when you assign the kws.on_ticks to one of the hundreds of instances of on_ticks defined during the looping and when that particular on_ticks is triggered as tick is received and it takes more time to return before the next tick is received then you end up with the same scenario as you would have had with the reactor not running threaded. So, even if extremely inefficient, this is not at all a solution.
Am I missing something? One can try with this code after the kws.connect(threaded=True). On my system in 10 secs about 25000 (25k) twenty five thousand while_ticks functions were created.
# Infinite loop on the main thread. Nothing after this will run if not threaded. # You have to use the pre-defined callbacks to manage subscriptions. kws.connect(threaded=True)
from time import time, sleep
start_time = time()
all_ticks=[] functions=[]
while time() < start_time + 10:
def while_ticks(ws, ticks): print('ticked on while', while_ticks) helper_method(ticks)
def helper_method(ticks): all_ticks.extend(ticks) print('helped on while', helper_method) #sleep(2)
@SRIJAN - Thanks for your response pasted below. Though that will avoid the multiple definitions of on_tick in the while loop, but it is no different than if you had not run the reactor threaded. If your computations could have finished within one tick, then it would have not mattered if you were running the reactor threaded. And if your computations will not finish in one tick then the code below would have the same issue, if you were doing the computation within the on_tick without threaded. So, it is not a solution to the problem.
Maybe the wording needs improvement. Cos, it specifically mentions, "P.S: Don't forget to assign ticker callback for new thread." which is not at all needed. It should be that just issue minimal work like copying the tick elsewhere in the main on_tick which will certainly finish before the next tick come in and then worry about handling that data elsewhere in your code. That way there is no need for reassigning the callback within the scope of while loop as was explicitly suggested in the post earlier.
The idea here is not to nitpick or to point fingers, but to learn and share the learning. So, will appreciate if anyone will share further on this topic to improve shared learnings. Thanks for reading.
- AlgoXYZ
The solution proposed by Rakesh Sir is just a sample. You have to tweak the sample codes by applying some logic to make it work efficiently in real implementations.
I will explain.
1. The solution proposed by Rakesh Sir does not create multiple copies .
Actually,after kws.connect() is executed,it runs an infinite thread.
So,you can't write the tick processing code or any other code after connecting.
If processing of ticks is done inside on_ticks,the ticker can disconnect.
So,to process ticks without disrupting the main on_ticks thread, websocket is run in threaded mode by executing kws.connect(threaded=True). This only runs websocket in a separate thread. The code after that executes like a common program.
The infinite while loop is there just so the program doesn't stop. Not that it generates multiple threads. So, it's not that multiple copies of on_ticks or helper function are spawned,but yes,the functions are defined,and the callback is assigned again and again , infinite times,in the sample code.
This does put extra workload on the machine.
2. What you can do is shift all the code inside the while loop before the while loop. And,just write a pass statement inside the while loop.
@SRIJAN - Certainly threaded is better. As you can do more with two than one thread. But no separate callback to kws.on_ticks is needed. esp. under the while loop as the original thread emphasizes. That is just wasteful and not needed.
I was trying to figure out if there is a failsafe way to do things without getting into celery, maybe not. But then, have folks explored if this might be simpler, but does it suffer from the issues like solution 2 has.
Callback to on_ticks is needed after redefining on_ticks.
Because redefining on_ticks only redefines the on_ticks function in your program,but the on_ticks attribute of the KiteTicker object is still unchanged.
Callbacks are a must for using websocket.
However,this is right that you don't need to assign callback under while loop as I showed.
Why redifine on_ticks? Just define it the right way first time itself.
There is no difference between assigning kws.on_tick before or after kws.connect. As long as the code behaviour inside the before and after functions is same, there is no difference. So you can don't need to reassign it after connect, if your original on_tick is light weight enough. And If you take thar approach that, then there is a possibility that before the callback after the connect is assigned, the first tick would get passed to the on_tick function at address before kws.connect which may break things. So, just do it once, do it right.
# Callback for tick reception before connect. def on_ticks(ws,ticks): #logging.debug("Ticks: {}".format(ticks)) global all_ticks all_ticks.extend(ticks) print('ticked on def before connect')
# Assign the callbacks before connect. kws.on_ticks=on_ticks
#Connect # Infinite loop on the main thread. Nothing after this will run if not threaded. # You have to use the pre-defined callbacks to manage subscriptions. kws.connect(threaded=True)
# Callback for tick reception after connect. def on_ticks(ws,ticks): #logging.debug("Ticks: {}".format(ticks)) global all_ticks all_ticks.extend(ticks) print('ticked on def after connect')
# Assign the callbacks after connect. kws.on_ticks=on_ticks
@SRIJAN - Well you made it light weight by just dumping the tick into a global variable instead of doing calculus on it. If it is not threaded then it is game over. But with threaded you can still keep playing. It is not a production level solution, but you can explore. And as I mentioned, still with assigning the callback after connect, you run the risk that the first tick might get delivered to the earlier callback. If you want to do it after that is fine but then before the connect it should be set to default None, so that the ticks don't get passed to something which might break your code.
@AlgoTraderXYZ You seem to have replied on multiple threads with the same query. We have removed all your duplicate responses. Please refrain from doing so in the future. I have replied to your similar query here. *edit: typo
Am I missing something? One can try with this code after the kws.connect(threaded=True). On my system in 10 secs about 25000 (25k) twenty five thousand while_ticks functions were created.
Maybe the wording needs improvement. Cos, it specifically mentions, "P.S: Don't forget to assign ticker callback for new thread." which is not at all needed. It should be that just issue minimal work like copying the tick elsewhere in the main on_tick which will certainly finish before the next tick come in and then worry about handling that data elsewhere in your code. That way there is no need for reassigning the callback within the scope of while loop as was explicitly suggested in the post earlier.
The idea here is not to nitpick or to point fingers, but to learn and share the learning. So, will appreciate if anyone will share further on this topic to improve shared learnings. Thanks for reading.
- AlgoXYZ
The solution proposed by Rakesh Sir is just a sample.
You have to tweak the sample codes by applying some logic to make it work efficiently in real implementations.
I will explain.
1. The solution proposed by Rakesh Sir does not create multiple copies .
Actually,after kws.connect() is executed,it runs an infinite thread.
It is mentioned here:
https://github.com/zerodha/pykiteconnect#websocket-usage
So,you can't write the tick processing code or any other code after connecting.
If processing of ticks is done inside on_ticks,the ticker can disconnect.
So,to process ticks without disrupting the main on_ticks thread, websocket is run in threaded mode by executing kws.connect(threaded=True). This only runs websocket in a separate thread. The code after that executes like a common program.
The infinite while loop is there just so the program doesn't stop. Not that it generates multiple threads. So, it's not that multiple copies of on_ticks or helper function are spawned,but yes,the functions are defined,and the callback is assigned again and again , infinite times,in the sample code.
This does put extra workload on the machine.
2. What you can do is shift all the code inside the while loop before the while loop.
And,just write a pass statement inside the while loop.
Like-
kws.connect(threaded=True)
def computation(ws,ticks):
...
...
kws.on_ticks=computation
while True:
pass
The connection disrupts very easily without using threaded ticker.
The threaded ticker gives some time to do computations without closing the websocket connection.
But when doing really heavy computation,even the threaded ticker fails.
In that case,you have to use the queue method as described in solution 1 here:
https://kite.trade/forum/discussion/comment/25535/#Comment_25535
I was trying to figure out if there is a failsafe way to do things without getting into celery, maybe not. But then, have folks explored if this might be simpler, but does it suffer from the issues like solution 2 has.
https://chriskiehl.com/article/parallelism-in-one-line
Because redefining on_ticks only redefines the on_ticks function in your program,but the on_ticks attribute of the KiteTicker object is still unchanged.
Callbacks are a must for using websocket.
However,this is right that you don't need to assign callback under while loop as I showed.
There is no difference between assigning kws.on_tick before or after kws.connect. As long as the code behaviour inside the before and after functions is same, there is no difference. So you can don't need to reassign it after connect, if your original on_tick is light weight enough. And If you take thar approach that, then there is a possibility that before the callback after the connect is assigned, the first tick would get passed to the on_tick function at address before kws.connect which may break things. So, just do it once, do it right.
You seem to have replied on multiple threads with the same query. We have removed all your duplicate responses. Please refrain from doing so in the future.
I have replied to your similar query here.
*edit: typo