* * *

Author Topic: lnet, best algorithm to send files over a network using TLTCP and maximize speed  (Read 1893 times)

RickD

  • New member
  • *
  • Posts: 14
Hello all,

I build a basic framework with lnet to send files over a network using TLTCP.

The whole thing works great, however i only get speeds up to 8% of the maximum network capacity.

Right now i use a pingpong system like this:

client   : request packet
server : send packet
client   : process packet and request new one
server : send new packet

I fixed part of the problem by using bigger packets (started with 200 bytes, now use 1250 bytes) which lowers the number of requests. And increases the transfer speed.

For some reason around 1250 bytes seems the maximum size of a packet (i guess lnet has an upper limit for packet size), so i can't improve much more on that.

My next idea is to use OnCanSend and just rapid fire the packets without requests to remove most of the communication latency (does not seem very elegant though).

However before i try that i want to check here what the best way to tackle this problem is. I'm sure a lot of other people have worked on this and there is no need to reinvent the wheel.

So any ideas, tips and tricks on this are much appreciated!

Regards, Rick







RickD

  • New member
  • *
  • Posts: 14
An update:

I rebuild the project with OnCanSend to just fire packets when possible, but most packets seem to be lost in the void of my home network or are received damaged. Also no error event seems to be triggered.

Not really sure what goes wrong. As far as i know tcp/ip should take care of the syncing and make sure the packets are received in order and not damaged (in theory).


GetMem

  • Hero Member
  • *****
  • Posts: 2247
Quote
For some reason around 1250 bytes seems the maximum size of a packet (i guess lnet has an upper limit for packet size), so i can't improve much more on that.
This is the limitation of the network.

You can try something like this:
1. client   : request file
2. server : send filesize
3. server : send file in chunks
Code: Pascal  [Select]
  1. const
  2.   BufferSize = 1024;
  3. var
  4.   Fs: TFileStream;
  5.   TotalCount: Integer;
  6.   Buffer: Pointer;
  7.   BytesToRead: Integer;  
  8. begin
  9.   Fs := TFileStream.Create('pathtoyourfile', fmOpenRead or fmshareDenyWrite);
  10.   try
  11.     TotalCount := 0;
  12.     GetMem(Buffer, BufferSize);
  13.     repeat
  14.       if TotalCount + BufferSize > Fs.Size then
  15.         BytesToRead := FS.Size - TotalCount
  16.       else
  17.         BytesToRead := BufferSize;
  18.       Fs.Read(Buffer^, BytesToRead);
  19.       FCon.Send(Buffer, BufferSize);
  20.       TotalCount := TotalCount + BytesToRead;
  21.     until TotalCount >= FS.Size;
  22.     FreeMem(Buffer);
  23.   finally
  24.     Fs.Free;
  25.   end;                  
  26. end;
  27.  
4. client : receive file size
5. client : receive file in loop(almost same code as above), you should read until the streamsize becomes equal to the filesize received previously(step 4)
« Last Edit: January 08, 2017, 07:29:13 am by GetMem »

RickD

  • New member
  • *
  • Posts: 14
Quote
For some reason around 1250 bytes seems the maximum size of a packet (i guess lnet has an upper limit for packet size), so i can't improve much more on that.
This is the limitation of the network.

You can try something like this:
1. client   : request file
2. server : send filesize
3. server : send file in chunks
Code: Pascal  [Select]
  1. const
  2.   BufferSize = 1024;
  3.  
  4. [snip]
  5.  
  6.     Fs.Free;
  7.   end;                  
  8. end;
  9.  
4. client : receive file size
5. client : receive file in loop(almost same code as above), you should read until the streamsize becomes equal to the filesize received previously(step 4)

I used a similar, but event based method (like the example on the lnet site, which is supposed to work correctly):

Note that the block_size:=200

Code: Pascal  [Select]
  1. procedure TLTCPTest.OnCanSendL(aSocket: TLSocket);
  2. var
  3. Sent,i: Integer;
  4. begin
  5.  
  6.  if (global_packet_index>0) then
  7.  begin
  8.         repeat
  9.                 begin
  10.                 try
  11.                 BlockRead(F, packet_buffer[5] , block_size, i);
  12.                 except end;
  13.                 inc(global_packet_index);
  14.                 if (i<block_size) then begin
  15.                 packet_buffer[4]:='3';                                                  // last packet
  16.                 try
  17.                 close(f);
  18.                 except end;
  19.                 writeln(': upload done ');
  20.                 global_packet_index:=0;                                                                                        
  21.                 end;
  22.                 Sent := FCon.SendMessage(packet_buffer, aSocket); // remember, don't use the aSocket directly!                                         
  23.                 //writeln('---> ',sent);
  24.                 end;
  25.         until ( (sent=0) or (global_packet_index=0) );
  26.  end;
  27.  
  28. end;
  29.  

The result : no error events are fired and , sent is always > 0. However the packets are not received correctly even though OnCanSend should handle the network syncing.

Seems to me that everything is correct, but the system just can't handle the speed without any extra syncing. Also note that if i activate the //writeln('---> ',sent); line, more packets are received correctly , which indicates the extra writeln pause makes things better.

But if anybody sees any mistakes in my method, please let me know ..

GetMem

  • Hero Member
  • *****
  • Posts: 2247
Quote
The result : no error events are fired and , sent is always > 0. However the packets are not received correctly even though OnCanSend should handle the network syncing.
Seems to me that everything is correct, but the system just can't handle the speed without any extra syncing. Also note that if i activate the //writeln('---> ',sent); line, more packets are received correctly , which indicates the extra writeln pause makes things better.
But if anybody sees any mistakes in my method, please let me know ..
There is not enough code to see the mistake(if any). I will make a small demo application today or tomorrow.
« Last Edit: January 09, 2017, 04:06:10 pm by GetMem »

GetMem

  • Hero Member
  • *****
  • Posts: 2247
Ok. I made a very basic demo. Just change the parameters for Client/FTCPClient.GetFile. You need to provide remote/local files.
https://drive.google.com/open?id=0B9Me_c5onmWoeHg2M2pldW5wZ2s

PS: I like Indy/Synapse much better then lnet.

RickD

  • New member
  • *
  • Posts: 14
Ok. I made a very basic demo. Just change the parameters for Client/FTCPClient.GetFile. You need to provide remote/local files.
https://drive.google.com/open?id=0B9Me_c5onmWoeHg2M2pldW5wZ2s

PS: I like Indy/Synapse much better then lnet.

Thanks for posting this. I will look into it later this week.

Awesome Programmer

  • Sr. Member
  • ****
  • Posts: 288
  • Programming is FUN only when it works :)
    • Cool Technology
Ok. I made a very basic demo. Just change the parameters for Client/FTCPClient.GetFile. You need to provide remote/local files.
https://drive.google.com/open?id=0B9Me_c5onmWoeHg2M2pldW5wZ2s

PS: I like Indy/Synapse much better then lnet.

I have looked at your demo for LNET, but I am not understanding your uclient receive procedure or event. Your code doesn't take into account that the file might be received in chunks not in one BIG CHUNK. You just read once for a file and then execute rest of the code. Maybe that's why you made your Buffersize big 1048578 in size. Maybe that is big enough of a size for most of the files in the world. However, you may not receive or read file in one big chunk... If you could clarify your code, that would so helpful.

 

Recent

Get Lazarus at SourceForge.net. Fast, secure and Free Open Source software downloads Open Hub project report for Lazarus