When i am using 5-6 scripts in websocket then data is coming perfectly fine and the same is getting updated properly in mysql database. Problem is that the moment i add more then 8-10 scripts in websocket...the data starts delayed and i sometimes get data 3-4 minutes delayed data in front end.
@nikhilbansal WebSocket doesn't send ticks update if price has not changed at that instance for the subscribed instrument.So, it can happen that, scrip you were checking might not have changed for few minutes.You can check this by adding few volatile instruments.
Thanks for your response. I have checked it for Volatile scripts. Actually i am using IIS to host my application and as i said it works absolutely fine for 4-5 scripts and this problem comes when i add 8-10 scripts or even more. it starts delaying (means prices does not match with terminal LTP vs my application LTP) for about 2-3 minutes.
i have put 1 second timer to update data on my mysql database.
Hi @nikhilbansal, You need to use multi-threading, the idea is to never block the main thread on which you are listening to ticks. I am guessing your DB writes are the reason for unavailability of the process when the tick arrives.
The right solution would be to do your DB writes or any other calculations on a separate thread and make sure you don't block the main thread.
@nikhilbansal create a RAM drive. When a tick arrives, dump the contents into a file in this RAM drive. Have another script "consume.py" to reach this file, parse it, store it in mysql and delete the file. If required, you can have multiple "consume1/2/3/....py" scripts running in parallel to match up to the speed of tick data. This method is highly scalable & reliable.
@ramatius thanks a ton. Not sure if i have understood this correctly. Can you please provide more details keeping in mind the .net or windows service as a solution?
(3) Create a .net C# app "stream.exe" which creates one file per tick, like "T:\tick_2018-10-10 13-12-44.data". If one tick is too fast, do it every 10 ticks (ignore other ticks).
(4) Create another .net C# app "consume.exe" that gets the directory listing and process each file, read it, generate query statements, execute the queries and then delete the file from T:\
(5) Your main app "trade.exe" can continue to use the mysql db for trading purposes
(6) In case you have many symbols, you can write "stream_set1.exe" that stores into T:\tick_set1_.data", "stream_set2.exe" that writes T:\tick_set2_.data and so on. The corresponding "consume_set1.exe", consume_set2.exe" can be created and puth the data into mysql.
Hope this gives you some idea on how to achieve it.
In a generic way what I would suggest is create a ticker connection and keep receiving ticks on the main thread. Inside onTick, create a new thread or use an existing second thread which will only do DB writes. In this way the DB writes are offloaded to slave thread and master will always listen to ticks.
WebSocket doesn't send ticks update if price has not changed at that instance for the subscribed instrument.So, it can happen that, scrip you were checking might not have changed for few minutes.You can check this by adding few volatile instruments.
Thanks for your response. I have checked it for Volatile scripts. Actually i am using IIS to host my application and as i said it works absolutely fine for 4-5 scripts and this problem comes when i add 8-10 scripts or even more. it starts delaying (means prices does not match with terminal LTP vs my application LTP) for about 2-3 minutes.
i have put 1 second timer to update data on my mysql database.
can you please help?
You need to use multi-threading, the idea is to never block the main thread on which you are listening to ticks. I am guessing your DB writes are the reason for unavailability of the process when the tick arrives.
The right solution would be to do your DB writes or any other calculations on a separate thread and make sure you don't block the main thread.
I am using .net application and mysql has option to manage pool of multi threads automatically.
I guess problem is in IIS server where my application is hosted and its adding delays
Can i use windows service to run ticker and update mysql db on every change in websocket data?
Not sure if switching to windows service will resolve this problem hence checking it with you guys
Thanks for your reply buddy.. i am not using python.. my application is in .net and using mysql database
(1) Install Dataram RAMDisk software
(2) Create a RAM drive with (say) 1GB in T:\
(3) Create a .net C# app "stream.exe" which creates one file per tick, like "T:\tick_2018-10-10 13-12-44.data". If one tick is too fast, do it every 10 ticks (ignore other ticks).
(4) Create another .net C# app "consume.exe" that gets the directory listing and process each file, read it, generate query statements, execute the queries and then delete the file from T:\
(5) Your main app "trade.exe" can continue to use the mysql db for trading purposes
(6) In case you have many symbols, you can write "stream_set1.exe" that stores into T:\tick_set1_.data", "stream_set2.exe" that writes T:\tick_set2_.data and so on. The corresponding "consume_set1.exe", consume_set2.exe" can be created and puth the data into mysql.
Hope this gives you some idea on how to achieve it.
@sujith just wanted to check if websocket supports multi threading and if yes, how can this be achieved by c# coding?
Inside onTick, create a new thread or use an existing second thread which will only do DB writes. In this way the DB writes are offloaded to slave thread and master will always listen to ticks.