Hi All, Did anyone notice any malformed Tick data (Full Quote with market depth, 184 bytes long) . I am yet to scan older data thoroughly, but looks like today I had more erroneous packets received than other days. Not sure about what the problem was today, but found one interesting fact. that is, first 20 bytes of the 184 bytes are okay. Is it server end problem or something else? Can anyone please be kind enough to shed some light on this ?
I am sorry for the inconvenience, I did mention 20 bytes, its actually first 12 bytes (Token, LastPrice and LastQuantity marked in BOLD). Here is the snapshot of HEX representation of 3 consecutive packets received for the particular token. I just found many of such malformed packets on almost all tokens, but for simplicity three packets being shown.
@sujith Its built using C (Not C++ or C#) codes. Data streaming is okay, but somehow data being received is corrupted at some points. I mentioned the hex dump so that you could understand first 12 bytes received are completely fine, but the rest of the frame is not. may be memcpy issue at the server side ?
@sujith seems like its not an issue with the client application. I am still examining it, but just to share an intermediate update on this, I found there are malformed packets received. As you are aware, websockets are very easy part of this data streaming, hence it doesn't matter which application it is literally. It would have been nice to have checksum at the end of each packet so that we would know the packet being received is properly delivered.
These are few consecutive packets again for example. Data will not be malformed in the midway of it. All the packets shown here is received sequentially as it is. As you can see, the time stamp of the bold packet is 1 sec less than the previous packet. Its quite strange that Data being pumped from server end is a past data and malformed. May be we need to investigate server side just in case if there is anything. Thread sync issue or anything ? I will try it in real time with wireshark, but would be quite tedious job to manage that huge data and find the exact malformed packet in it. As it is happening very randomly, so very time consuming.
@sujith Another point to note to validate its not client issue is, the timestamp starts from 64th byte which is getting recognised properly, but the BuyQuantity, SellQuantity started from 24th byte position. so if it is client side problem there will not be any case that only first 12 bytes received will be okay and last few bytes will be okay again. Volume starts at 20th byte position which you can see is malformed. There must be something wrong going on which we need to sort out. Thanks for understanding. Socket will not deliver few good bytes then few bad bytes, then again few good bytes. TCP will not deliver to the application layer.
There is still malformed data. One thing I forgot to mention in my earlier posts, that is the occurrence of this malformed data is noticed when subscribed to FULL (184 bytes) for many INSTRUMENTS (100-150) at once. lesser the number of subscribed Symbols, lesser the occurrence of this malformed data. I wish if we have Checksum on the data frame, that would really help to sort-out this problem.
I would request you to subscribe 100-150 Instruments and monitor any one Symbol for few hours. That will help you to understand the problem.
Here I just showed snapshot based on ATP=1308.49 and see this price was correct for most BNF PE, but suddenly one data arrived for NF CE with same ATP. Looks like some of the Data are not being written properly before delivery to WebSockets from Server.
Can you try some official client libraries? We use the same data on all our platforms. We haven't come across any issue like this. This could be happening at your end because of buffer reuse.
-- Valid Packet
Token: 17039874 | 2021-04-20 13:49:50 | LTP= 134.00 | ATP= 118.03 | LTQ= 75 | VOL= 2912100 | BID= 96825 | ASK= 118125 | OI= 1877400
HEX Debug | 0x01 0x04 0x02 0x02 0x00 0x00 0x34 0x58 0x00 0x00 0x00 0x4B 0x00 0x00 0x2E 0x1B 0x00 0x2C 0x6F 0x64 0x00 0x01 0x7A 0x39 0x00 0x01 0xCD 0x6D 0x00 0x00 0x30 0x16 0x00 0x00 0x3B 0x29 0x00 0x00 0x25 0xEE 0x00 0x00 0x3E 0xE4 0x60 0x7E 0x8E 0xA3 0x00 0x1C 0xA5 0x98 0x00 0x1E 0x70 0x17 0x00 0x1C 0x0F 0x02 0x60 0x7E 0x8E 0xA6 0x00 0x00 0x00 0x4B 0x00 0x00 0x34 0x21 0x00 0x01 0x00 0x00 0x00 0x00 0x00 0x4B 0x00 0x00 0x34 0x1C 0x00 0x01 0x00 0x00 0x00 0x00 0x04 0xFB 0x00 0x00 0x34 0x17 0x00 0x02 0x00 0x00 0x00 0x00 0x00 0x4B 0x00 0x00 0x34 0x12 0x00 0x01 0x00 0x00 0x00 0x00 0x01 0x77 0x00 0x00 0x34 0x0D 0x00 0x03 0x00 0x00 0x00 0x00 0x00 0x4B 0x00 0x00 0x34 0x58 0x00 0x01 0x00 0x00 0x00 0x00 0x03 0x39 0x00 0x00 0x34 0x5D 0x00 0x01 0x00 0x00 0x00 0x00 0x00 0x4B 0x00 0x00 0x34 0x62 0x00 0x01 0x00 0x00 0x00 0x00 0x05 0x46 0x00 0x00 0x34 0x67 0x00 0x03 0x00 0x00 0x00 0x00 0x02 0xA3 0x00 0x00 0x34 0x6C 0x00 0x05 0x00 0x00 | 184 Bytes
-- Malformed Packet
Token: 17039874 | 2021-04-20 13:49:50 | LTP= 134.00 | ATP= 889.88 | LTQ= 75 | VOL= 20000 | BID= 9875 | ASK= 10625 | OI= 8400
HEX Debug | 0x01 0x04 0x02 0x02 0x00 0x00 0x34 0x58 0x00 0x00 0x00 0x4B 0x00 0x01 0x5B 0x9C 0x00 0x00 0x4E 0x20 0x00 0x00 0x26 0x93 0x00 0x00 0x29 0x81 0x00 0x01 0x6B 0x48 0x00 0x01 0xA6 0x85 0x00 0x01 0x24 0x67 0x00 0x01 0x3E 0x61 0x60 0x7E 0x8E 0x56 0x00 0x00 0x20 0xD0 0x00 0x00 0x20 0xD0 0x00 0x00 0x19 0x96 0x60 0x7E 0x8E 0xA6 0x32 0x1D 0x00 0x00 0x4F 0x1C 0x00 0x00 0x47 0x1C 0x00 0x00 0x00 0x2E 0x1B 0x00 0x2C 0x6F 0x64 0x00 0x01 0x76 0x1F 0x00 0x01 0xCD 0x22 0x00 0x00 0x30 0x16 0x00 0x00 0x3B 0x29 0x00 0x00 0x25 0xEE 0x00 0x00 0x3E 0xE4 0x60 0x7E 0x8E 0xA3 0x00 0x1C 0xA5 0x98 0x00 0x1E 0x70 0x17 0x00 0x1C 0x0F 0x02 0x60 0x7E 0x8E 0xA7 0x00 0x00 0x00 0x4B 0x00 0x00 0x34 0x2B 0x00 0x01 0x00 0x00 0x00 0x00 0x00 0xE1 0x00 0x00 0x34 0x26 0x00 0x02 0x00 0x00 0x00 0x00 0x04 0xB0 0x00 0x00 0x34 0x21 0x00 0x02 0x00 0x00 0x00 0x00 0x00 0xE1 0x00 0x00 0x34 0x1C 0x00 0x02 0x00 0x00 0x00 0x00 0x00 0x4B 0x00 0x00 0x34 0x12 0x00 | 184 Bytes
-- Valid Packet
Token: 17039874 | 2021-04-20 13:49:51 | LTP= 134.00 | ATP= 118.03 | LTQ= 75 | VOL= 2912100 | BID= 96750 | ASK= 115425 | OI= 1877400
HEX Debug | 0x01 0x04 0x02 0x02 0x00 0x00 0x34 0x58 0x00 0x00 0x00 0x4B 0x00 0x00 0x2E 0x1B 0x00 0x2C 0x6F 0x64 0x00 0x01 0x79 0xEE 0x00 0x01 0xC2 0xE1 0x00 0x00 0x30 0x16 0x00 0x00 0x3B 0x29 0x00 0x00 0x25 0xEE 0x00 0x00 0x3E 0xE4 0x60 0x7E 0x8E 0xA3 0x00 0x1C 0xA5 0x98 0x00 0x1E 0x70 0x17 0x00 0x1C 0x0F 0x02 0x60 0x7E 0x8E 0xA7 0x00 0x00 0x00 0x4B 0x00 0x00 0x34 0x30 0x00 0x01 0x00 0x00 0x00 0x00 0x00 0x4B 0x00 0x00 0x34 0x2B 0x00 0x01 0x00 0x00 0x00 0x00 0x02 0xEE 0x00 0x00 0x34 0x26 0x00 0x02 0x00 0x00 0x00 0x00 0x04 0xFB 0x00 0x00 0x34 0x1C 0x00 0x03 0x00 0x00 0x00 0x00 0x00 0x4B 0x00 0x00 0x34 0x12 0x00 0x01 0x00 0x00 0x00 0x00 0x00 0x4B 0x00 0x00 0x34 0x58 0x00 0x01 0x00 0x00 0x00 0x00 0x03 0xCF 0x00 0x00 0x34 0x6C 0x00 0x03 0x00 0x00 0x00 0x00 0x01 0x77 0x00 0x00 0x34 0x76 0x00 0x03 0x00 0x00 0x00 0x00 0x00 0x4B 0x00 0x00 0x34 0x7B 0x00 0x01 0x00 0x00 0x00 0x00 0x02 0x0D 0x00 0x00 0x34 0x80 0x00 0x02 0x00 0x00 | 184 Bytes
Thanks
Which client library are you using? Can you elaborate on what is malformed data here? Which value is wrong or missing?
Its built using C (Not C++ or C#) codes. Data streaming is okay, but somehow data being received is corrupted at some points. I mentioned the hex dump so that you could understand first 12 bytes received are completely fine, but the rest of the frame is not. may be memcpy issue at the server side ?
These are few consecutive packets again for example. Data will not be malformed in the midway of it. All the packets shown here is received sequentially as it is. As you can see, the time stamp of the bold packet is 1 sec less than the previous packet. Its quite strange that Data being pumped from server end is a past data and malformed. May be we need to investigate server side just in case if there is anything. Thread sync issue or anything ? I will try it in real time with wireshark, but would be quite tedious job to manage that huge data and find the exact malformed packet in it. As it is happening very randomly, so very time consuming.
Token: 17056002 | 2021-04-28 15:26:49 | LTP= 6.45 | ATP= 9.65 | LTQ= 75 | VOL=28863750 | BID= 294600 | ASK= 437625 | BAR=0.673 | OI= 3541875
Token: 17056002 | 2021-04-28 15:26:50 | LTP= 6.50 | ATP= 9.64 | LTQ= 75 | VOL=28866750 | BID= 300750 | ASK= 435150 | BAR=0.691 | OI= 3541875
Token: 17056002 | 2021-04-28 15:26:50 | LTP= 6.45 | ATP= 9.64 | LTQ= 600 | VOL=28867350 | BID= 301125 | ASK= 432225 | BAR=0.697 | OI= 3541875
Token: 17056002 | 2021-04-28 15:26:51 | LTP= 6.45 | ATP= 9.64 | LTQ= 75 | VOL=28868850 | BID= 299250 | ASK= 434625 | BAR=0.689 | OI= 3541875
Token: 17056002 | 2021-04-28 15:26:52 | LTP= 6.45 | ATP= 9.64 | LTQ= 300 | VOL=28869225 | BID= 300075 | ASK= 430575 | BAR=0.697 | OI= 3541875
Token: 17056002 | 2021-04-28 15:26:51 | LTP= 6.45 | ATP= 6.00 | LTQ= 300 | VOL= 6315500 | BID= 252325 | ASK= 24075 | BAR=10.481 | OI= 939750
Token: 17056002 | 2021-04-28 15:26:53 | LTP= 6.45 | ATP= 9.64 | LTQ= 150 | VOL=28869225 | BID= 297675 | ASK= 432975 | BAR=0.688 | OI= 3541875
Token: 17056002 | 2021-04-28 15:26:54 | LTP= 6.45 | ATP= 9.64 | LTQ= 75 | VOL=28871775 | BID= 297675 | ASK= 432000 | BAR=0.689 | OI= 3541875
We took a RELIANCE tick dump for one whole day and we couldn't find any anomaly. I would suggest using one of the client libraries.
I would request you to subscribe 100-150 Instruments and monitor any one Symbol for few hours. That will help you to understand the problem.
Here I just showed snapshot based on ATP=1308.49 and see this price was correct for most BNF PE, but suddenly one data arrived for NF CE with same ATP. Looks like some of the Data are not being written properly before delivery to WebSockets from Server.
This could be happening at your end because of buffer reuse.