If we are getting large number of ticks (Tokens) then the date received may not queued or dropped due to processing when inserting into db. Can anyone help how we can streamline the same using kafka messaging queue and spark streaming?
@cheko, did you tried to store in data frame ? if you are not into LFT(!Max 2000 orders per day which is like only 1000 buy orders) i think with decent broadband speed you should be able to store and then push them to db periodically ........ if you have subscribed to historical data you should be able to get TBT data.... You cannot do any FT trading here... too many restrictions.. dont bother breaking your head for Real-time processing.....
The idea here is the Tick receiving module will be a producer and the post-processing module will be a consumer. The producer keeps pushing messages to bus and consumer keeps reading one by one.
if you have subscribed to historical data you should be able to get TBT data....
You cannot do any FT trading here... too many restrictions.. dont bother breaking your head for Real-time processing.....
This repository might be helpful for you.
The idea here is the Tick receiving module will be a producer and the post-processing module will be a consumer.
The producer keeps pushing messages to bus and consumer keeps reading one by one.