I have built a pair trading strategy using PyConnect.I have created a list of pairs in an excel file-they have the symbols and associated parameter which determine when a trade should be initiated and exited depending on if the stop loss or target is hit.When I run it for a single pair it works fine.
I then included around 10 pairs of stocks in my input file.I am collecting the ticks in a queue and passing it onto separate threads where the calculations are done and executions are performed.The code is calculating parameters for pairs for which no trades have been initiated and evaluating the conditions to check if the positions should be added.At the same it calculates the MTM of the existing positions.In this case,when the positions have been added for lets say two of the pairs and then have to calculate their MTM,the performance is considerably degraded.Since the majority of the pairs(out of a total of 10) have no positions,the code spends maximum time evaluating their parameters which results in delays in MTM calculations of the existing positions.For example,the SL or target price would already have been hit but since the MTM calculation was delayed the orders are sent out late.In some cases I observed that it was working with slightly delayed prices.
Is there any way to make this more efficient?Is anyone facing similar issues while trying to scale up their code?
Would like to know if anyone has tried multi processing and the challenges you faced,if any?
@kapilaggr - Threads in python share the same global variable of MainThread. You may check if there is a conflict which might cause your threads to use the values interchangeably. Instead you can think of subprocess module which will run as a separate process itself and there is no overlap as such. I am using multithreading in python however it is only for one symbol (one to download market data, one to check if there is a signal and the main thread). Let me know if you need further help. Performance depends on the PC, internet speed and various other factors. Very difficult to answer that part of the question. You can use some performance monitoring tools like Metricbeat, Elasticsearch, Grafana to monitor KiteConnect response time and CPU/Memory/Network of your PC to identify the bottlenecks.
Can you pls elaborate more on the sub process part.My code has these main modules-getting the ticks from kite connect,evaluating certain parameters to check if positions should be added,calculate the mtm once the positions have been added,sending out orders.Since I am using multiple pairs,module 2 and 3 run simultaneously.At any give point of time majority of the pairs do not have positions,only 20-25% pairs would have positions(till now the maximum number of pairs that i have used is 10).
As of now I am using only one function to perform all these steps.I am pushing the ticks into a queue.I am reading the input excel file into a pandas data frame and defining the number of thread as the number of rows in the excel. I am then looping over this number and crating a thread that calls my function.The function takes the queue and the index(the number being looped) as inputs and then performs the actions that I listed above.
Can you pls suggest a better and more efficient way of handling this.I feel the threading part here can be implemented in a better way.Also,would it improve the code if I define two separate functions for checking the signal to enter a position and to calculate the MTM.Pls share your thoughts on how this can be implemented.
Can you pls elaborate more on the sub process part.My code has these main modules-getting the ticks from kite connect,evaluating certain parameters to check if positions should be added,calculate the mtm once the positions have been added,sending out orders.Since I am using multiple pairs,module 2 and 3 run simultaneously.At any give point of time majority of the pairs do not have positions,only 20-25% pairs would have positions(till now the maximum number of pairs that i have used is 10).
As of now I am using only one function to perform all these steps.I am pushing the ticks into a queue.I am reading the input excel file into a pandas data frame and defining the number of thread as the number of rows in the excel. I am then looping over this number and crating a thread that calls my function.The function takes the queue and the index(the number being looped) as inputs and then performs the actions that I listed above.
Can you pls suggest a better and more efficient way of handling this.I feel the threading part here can be implemented in a better way.Also,would it improve the code if I define two separate functions for checking the signal to enter a position and to calculate the MTM.Pls share your thoughts on how this can be implemented.
Thanks and Regards,
Kapil