☰
›

Python On_ticks function doesnt run full; new tick data takes over

samirkelekar
samirkelekar in Market data (WebSockets)
I have written on_ticks to process tick data. But my processing takes time. Before it is over, new tick data comes in and a new on_ticks function with new tick data starts. How to handle this ? Any tips would be appreciated.
  • sujith
    sujith
    You should offload that task to another thread and add task to queue or ticks to queue, let it process ticks independently.
  • vaibhavsharma13
    vaibhavsharma13
    Hi @sujith will there be a new thread for each tick data processing?
  • tahseen
    tahseen
    @vaibhavsharma13 no, you won't have new thread for each tick. That would be terrible a code

    @samirkelekar @vaibhavsharma13

    For beginners

    1. Initialize internal queue Q in the beginning of the program
    2. Inside on_ticks you pass the data to the internal queue immediately
    3. No need to launch threads inside on_ticks
    4. You create thread in the main program for doing tasks
    5. Code inside this thread reads from Q the internal queue and processes the data

  • vaibhavsharma13
    vaibhavsharma13
    @tahseen thank you for your reply. What I understood is that I need to put tasks into a queue and run task of that queue by a single thread.
  • tahseen
    tahseen
    No, you need to pass the ticks JSON as is in a queue within on_ticks

    and there should be a separate thread that is running and consuming this queue of ticks to process

    this way you have decoupled on_ticks from computational load

    This is a very simple approach, you can use a proper message broker also instead of python queue
  • vaibhavsharma13
    vaibhavsharma13
    ty @tahseen can you tell which python library can be used for queuing tasks?
  • rakeshr
    rakeshr
    @vaibhavsharma13
    You can use celery, rq.
  • tahseen
    tahseen
    @vaibhavsharma13 am talking of basic queue of python


    from queue import Queue
    messageQueue = Queue()

    def read_queue():
        while True:
             bulk_ticks = messageQueue.get()
             # do something like NASA or SpaceX here with this data

    # this is the only thing you should write in on_ticks, anything more would be terrible coding
    def on_ticks(ws , ticks):
        messageQueue.put(ticks)

    # before starting KiteTicker, run read_queue in a thread so that it starts and gets hooked to the queue
    # below as an example, you can use new python 3 library also for threading
    import _thread
    _thread.start_new_thread(read_queue, ())

Sign In or Register to comment.