It looks like you're new here. If you want to get involved, click one of these buttons!
.Net API client
.Net / VBA / Excel (3rd party)
Algorithms and Strategies
Mobile and Desktop apps
Market data (WebSockets)
In this Discussion
How to use KWS with messaging queue like kafka and spark streaming
If we are getting large number of ticks (Tokens) then the date received may not queued or dropped due to processing when inserting into db. Can anyone help how we can streamline the same using kafka messaging queue and spark streaming?
, did you tried to store in data frame ? if you are not into LFT(!Max 2000 orders per day which is like only 1000 buy orders) i think with decent broadband speed you should be able to store and then push them to db periodically ........
if you have subscribed to historical data you should be able to get TBT data....
You cannot do any FT trading here... too many restrictions.. dont bother breaking your head for Real-time processing.....
might be helpful for you.
The idea here is the Tick receiving module will be a producer and the post-processing module will be a consumer.
The producer keeps pushing messages to bus and consumer keeps reading one by one.