I was trying to optimize the websocket read-loop and one of the issues is that, Websocket data being received although sends number of packets in the first two bytes, but it does not send total length of the stream upfront. Which makes one to wait until one has received entire frame / message to know the data length. Because by now one has already filled it in some buffer.
This leads to duplication of Copy which has already been done or if we avoid that then renewal of buffer every time data comes in leads to another kind of penalty.
It would be nice if along with Total number of packets. Total length of stream is also sent along-with. So first two byte would contain number of packets and next 4 bytes would contain Length of entire message following it. It is very easy to do.
Total Length Of Message = ( 2 * number of messages ) + Total Of All Messages Lengths ( 2 * number of messages ) because every message contains its length in the beginning in form of 2 bytes.
In C programming if I have to optimally allocate memory to the data buffer, I must know the size of incoming data and for that MSG_PEEK flag is used with recv
Then with malloc exact size is allocated to the buffer. In next step now you can populate the buffer with data
Thanks for taking me to the C world! But I am holding to C# dearly, these days.
In order to know the size in advance that information has to be somewhere reachable to the application using it. I don`t see it anywhere, be it C# or C. All peeks can only look into the buffer that is already populated or sized to be populated. Unless PEEK is privy to the information from the socket implementation layer.
MSG_PEEK only looks into the input buffer without removing data from it. which is still client side information not server side.
I wrote a program to connect to the SMPP server, which was sending sending data to me. So I used MSG_PEEK as the flag in recv to read the size of the incoming data, to malloc the buffer variable size exactly as much the incoming data
Unless am understanding you incorrectly, this is something I have done.
@tahseen Thanks for replying. I have to admit I have not worked with sockets in C or with C itself for more than a decade, so memory about C stuff is blurry.
So it is not you who may not be understanding me, but may be me who isn`t understanding your solutions.
You say that you implemented a solution in C. But how is that helpful in C#. Because it would lead to DLLIMPORT ing a C lib.
According to my immature understanding this is what I am concluding. Even in C# we know about the message received length upfront, by reading the Length of Result. So one can initialize the data buffer according to it. This can work for small to medium messages being pushed by Websocket Server. But it is not guaranteed that message will appear on Client side in one rush. Problem starts then. Now in C# there is no way to know how much of message is still in the pipeline in various Network hubs, centers etc. from which that message is passing through. and they are free to disintegrate the data into as many chunks as they like according to their own logic of keeping their Network Hub/line performing and stable. and when any on those chunks arrives, we read it in a loop waiting for the EndOfMessage flag. Larger the message is more rounds in this loop.
This is exactly what is happening when count of subscribed ticks increases. One is waiting in the loop and Result.Length becomes the length of a fragment and not of entire messages which will arrive in other loops.
I maybe wrong here but you are saying MSG_PEEK was able to give you the length of data which had not even arrived in the socket buffer yet.
I think your system was working because the length of data being pushed by your server was Small or it was a LAN based network because in LAN packets are not often disintegrated even if the message was Large. Or default buffer for Incoming socket was enlarged to receive data in one go that is if it arrives in one attempt.
I think you are confusing reading the length of buffer in which Socket stores its incoming data and from which it reports the size of the buffered data ready to be read as the size of the entire message which has not even arrived. Or my own understanding is shallow and rusted to that I admit I don`t know much.
@sujith Yaa I know it sends number of packets information! but that is not the length of those packets upfront. then one would have to guess ( number of packets ) * ( kind of data subscribed per message length ). how does one know if all incoming are in full mode or ltp mode. that is known after reading the size of individual packets. One can know if the incoming message is binary then it would be tick data. If one has subscribed to only one mode like ltp or quote then it is fine but if some ticks are in ltp and others are in quote now one cannot calculate with number of packets.
Number of packets is the number of packets you have sent from your server it is not the number of packets I have received in my buffer in one go. I might be waiting in a loop until all packets arrive which are mentioned in Number of packets which are the first 2 bytes.
@trade_then We do send the total length of the bytes for any given packet(using which you can know mode). You can go through the message packet structure here and refer to B byte description.
@rakeshr Thanks for the input. Yes I know you send the length of a packet along with it. And it would have been sufficient if that was the only packet being received. What if there are fifty packets. Once the packet is received I can read the individual length of each packet. I know that. I am asking to know the length before receiving them all. Just like we know in 1st two bytes the number of packets we can know of length of all the packets in the next 2 or 4 bytes but we don`t. This would help in optimizing the speed of socket and also in other things. Because knowing the length before hand would allow us to save a full copy cycle which we now have to do.
Otherwise one would have to get into a lot of guess work.
Anyway this is a request feature intended. not a necessity.
I am able to do it even without Length in advance though it would have been nice if it was provided. On diagnosing it turns out that at least in my code This Loop is hardly invoked..
So with EndOfMessage checking in advance one is able to mitigate this issue.
In C programming if I have to optimally allocate memory to the data buffer, I must know the size of incoming data and for that MSG_PEEK flag is used with recv
Then with malloc exact size is allocated to the buffer. In next step now you can populate the buffer with data
This way you optimally do memory allocation
But I am holding to C# dearly, these days.
In order to know the size in advance that information has to be somewhere reachable to the application using it.
I don`t see it anywhere, be it C# or C.
All peeks can only look into the buffer that is already populated or sized to be populated.
Unless PEEK is privy to the information from the socket implementation layer.
MSG_PEEK only looks into the input buffer without removing data from it. which is still client side information not server side.
Thanks
Regards
Unless am understanding you incorrectly, this is something I have done.
Also check our https://github.com/statianzo/Fleck if that helps
Apologies, if I have not been able to understand you correctly
Thanks for replying. I have to admit I have not worked with sockets
in C or with C itself for more than a decade, so memory about C stuff is blurry.
So it is not you who may not be understanding me, but may be me who isn`t understanding your solutions.
You say that you implemented a solution in C.
But how is that helpful in C#. Because it would lead to DLLIMPORT ing a C lib.
According to my immature understanding this is what I am concluding.
Even in C# we know about the message received length upfront, by reading the Length of Result. So one can initialize the data buffer according to it.
This can work for small to medium messages being pushed by Websocket Server. But it is not guaranteed that message will appear on Client side in one rush. Problem starts then. Now in C# there is no way to know how much of message is still in the pipeline in various Network hubs, centers etc. from which that message is passing through. and they are free to disintegrate the data into as many chunks as they like according to their own logic of keeping their Network Hub/line performing and stable. and when any on those chunks arrives, we read it in a loop waiting for the EndOfMessage flag. Larger the message is more rounds in this loop.
This is exactly what is happening when count of subscribed ticks increases.
One is waiting in the loop and Result.Length becomes the length of a fragment and not of entire messages which will arrive in other loops.
I maybe wrong here but you are saying MSG_PEEK was able to give you the length of data which had not even arrived in the socket buffer yet.
I think your system was working because the length of data being pushed by your server was Small or it was a LAN based network because in LAN packets are not often disintegrated even if the message was Large. Or default buffer for Incoming socket was enlarged to receive data in one go that is if it arrives in one attempt.
I think you are confusing reading the length of buffer in which Socket stores its incoming data and from which it reports the size of the buffered data ready to be read as the size of the entire message which has not even arrived. Or my own understanding is shallow and rusted to that I admit I don`t know much.
Thanks
Regards
You can check out the documentation here.
Yaa I know it sends number of packets information! but that is not the length of those
packets upfront. then one would have to guess ( number of packets ) * ( kind of data subscribed per message length ). how does one know if all incoming are in full mode or ltp mode. that is known after reading the size of individual packets.
One can know if the incoming message is binary then it would be tick data. If one has subscribed to only one mode like ltp or quote then it is fine but if some ticks are in ltp and others are in quote now one cannot calculate with number of packets.
Number of packets is the number of packets you have sent from your server it is not
the number of packets I have received in my buffer in one go. I might be waiting in a loop until all packets arrive which are mentioned in Number of packets which are the first 2 bytes.
Thanks
Regards
We do send the total length of the bytes for any given packet(using which you can know mode). You can go through the message packet structure here and refer to B byte description.
Thanks for the input. Yes I know you send the length of a packet along with it.
And it would have been sufficient if that was the only packet being received.
What if there are fifty packets.
Once the packet is received I can read the individual length of each packet.
I know that. I am asking to know the length before receiving them all. Just like
we know in 1st two bytes the number of packets we can know of length of all the packets in the next 2 or 4 bytes but we don`t. This would help in optimizing the speed
of socket and also in other things. Because knowing the length before hand would allow us to save a full copy cycle which we now have to do.
Otherwise one would have to get into a lot of guess work.
Anyway this is a request feature intended. not a necessity.
Thanks
Regards
On diagnosing it turns out that at least in my code
This Loop is hardly invoked..
So with EndOfMessage checking in advance one is able to mitigate this issue.
And this assignment every time websocket callsback can also be avoided buy shifting it here or towards where buffer is.
Thanks
Regards