gRPC for streaming data

In general streaming data via websocket has its own problem with respect to speed. The abstractions being json, the data is parsed at your end and at client end making the processing of data very slow. If I have 200 instrument,even with distributed processing the time taken to parse/process one tick of 200 instruments is immense.
My purpose of this request is to find an alternative to websocket which is atleast a bit faster an therefore gRPC comes to my mind.
@sujith and @Kailash let me know your thoughts.
  • Vivek
    @gully Through sockets we send data in binary format and have optimisations to reduce the payload size. It's up to the client to scale it however it wants. Also, Clients parses it to native types such as Python client parses it to dictionary and Node client parses it to Object but you can write your own abstraction in client side. As you said by doing multiprocessing and dividing subscriptions to multiple connections can be useful to reduce the overhead in client side.
  • gully
    gully edited May 2017
    @vivek I assume the current flow looks like this

    You get the data from Exchange->Process->.Convert To Json->Convert to binary->send->Client Receives- the data>From the binary form->converts to json-> Creates consumable objects from the json.->Starts consuming it

    In the above process there is lot of transform on the same data. The json conversion is redundant for the api users(it makes sense for the Kite UI)

    My proposal
    You receive the data-> Process->Convert to gRPC obj->send->Client receives the gRPC object model->Starts consuming it
  • Vivek
    @gully We get binary data from exchanges and slice the required bytes based on mode subscribed and send it to the client so it doesn't involve any JSON conversion and all. The only place where the binary is converted to JSON is in client side.

    The process you have suggested involves converting the binary data to gRPC object which actually introduces an overhead and dependency. The only advantage I can see is we don't need to maintain different client library since gRPC comes with it but then we anyway had to have clients for HTTP API and might as well bundle this.
  • gully
    @vivek ahh!! yes I was not aware that the binary data received from exchange is sliced and then sent without any json conversion at the server side,in that case gRPC wouldnt fit the would only slow things down..
This discussion has been closed.