In general streaming data via websocket has its own problem with respect to speed. The abstractions being json, the data is parsed at your end and at client end making the processing of data very slow. If I have 200 instrument,even with distributed processing the time taken to parse/process one tick of 200 instruments is immense. My purpose of this request is to find an alternative to websocket which is atleast a bit faster an therefore gRPC comes to my mind. @sujith and @Kailash let me know your thoughts.
@gully Through sockets we send data in binary format and have optimisations to reduce the payload size. It's up to the client to scale it however it wants. Also, Clients parses it to native types such as Python client parses it to dictionary and Node client parses it to Object but you can write your own abstraction in client side. As you said by doing multiprocessing and dividing subscriptions to multiple connections can be useful to reduce the overhead in client side.
You get the data from Exchange->Process->.Convert To Json->Convert to binary->send->Client Receives- the data>From the binary form->converts to json-> Creates consumable objects from the json.->Starts consuming it
In the above process there is lot of transform on the same data. The json conversion is redundant for the api users(it makes sense for the Kite UI)
My proposal You receive the data-> Process->Convert to gRPC obj->send->Client receives the gRPC object model->Starts consuming it
@gully We get binary data from exchanges and slice the required bytes based on mode subscribed and send it to the client so it doesn't involve any JSON conversion and all. The only place where the binary is converted to JSON is in client side.
The process you have suggested involves converting the binary data to gRPC object which actually introduces an overhead and dependency. The only advantage I can see is we don't need to maintain different client library since gRPC comes with it but then we anyway had to have clients for HTTP API and might as well bundle this.
@vivek ahh!! yes I was not aware that the binary data received from exchange is sliced and then sent without any json conversion at the server side,in that case gRPC wouldnt fit the architecture..it would only slow things down..
You get the data from Exchange->Process->.Convert To Json->Convert to binary->send->Client Receives- the data>From the binary form->converts to json-> Creates consumable objects from the json.->Starts consuming it
In the above process there is lot of transform on the same data. The json conversion is redundant for the api users(it makes sense for the Kite UI)
My proposal
You receive the data-> Process->Convert to gRPC obj->send->Client receives the gRPC object model->Starts consuming it
The process you have suggested involves converting the binary data to gRPC object which actually introduces an overhead and dependency. The only advantage I can see is we don't need to maintain different client library since gRPC comes with it but then we anyway had to have clients for HTTP API and might as well bundle this.