Lazarus

Programming => Networking and Web Programming => Topic started by: r.lukasiak on June 16, 2022, 10:45:02 pm

Title: Sending and receiving files with lNet
Post by: r.lukasiak on June 16, 2022, 10:45:02 pm
Hi everyone,

I'm trying to use lNet to send (on client side) and recive (on server side) files. Following https://lnet.wordpress.com/usage/sockets-protocols-and-sending/ and https://forum.lazarus.freepascal.org/index.php?topic=5146.0 I've managed to achieve it, somehow...
For smal files (<1kb) it works perfectly, it's sent in 1 piece but files I need to send are ~100kb and here I'm facing a problem.

first, on the client side, I did it exactly as indicated in mentioned posts:
Code: Pascal  [Select][+][-]
  1. procedure TApp.LTCPCanSend(aSocket: TLSocket);
  2. var
  3.   n: Integer;
  4.    
  5. begin
  6.   repeat
  7.      n:=LTCP.SendMessage(mtFile+h+Lbuffer, aSocket);
  8.      Delete(Lbuffer,1,n);
  9.   until (n = 0) or (length(Lbuffer)=0);
  10. end;
  11.  
but it's send in chunks of 1408 bytes, some chunks differ. The problem is that the message consists of a code (for the server to know what type of message it is), hash and a JPG file. IF it's send in irregular chunks, the server recognizes only the first one, next chunks are sent without the code and hash.

Then I got an idea to chop the message into 1024-byte chunks and send each piece with the code and hash so the server knows what file it belongs to but even though I send it in 1024 bytes, the server still receives random pieces.
Code: Pascal  [Select][+][-]
  1.   repeat
  2.       l:=length(Lbuffer);
  3.       if l>1024+length(mtFile+h) then l:=1024+length(mtFile+h);
  4.       n:=LTCP.SendMessage(mtFile+h+copy(Lbuffer,1,l), aSocket);
  5.       Delete(Lbuffer,1,n);
  6.       if length(Lbuffer)>0 then Lbuffer:=mtFile+h+Lbuffer;
  7.   until (n = 0) or (length(Lbuffer)=0);  
  8.  

on the server side I have:
Code: Pascal  [Select][+][-]
  1. procedure TKTCPServer.OnReceive(aSocket: TLSocket);
  2. var
  3.   s: string;
  4.   f: file;
  5. begin
  6.   if aSocket.GetMessage(s) > 0 then
  7.     begin
  8.       Writeln('received '+IntToStr(length(s))+' bytes: '+copy(s,2,34)+'...');
  9.     end;
  10. end;

and I get:
Code: Pascal  [Select][+][-]
  1. ..
  2.  
  3. 15:20:49.598 received 6343 bytes: @C15B4629CF2991BD8EC81DA24BDB39...
  4. 15:20:49.599 received 1024 bytes: i¹£4Ú9¦!sE%-QEQEQI@...
  5. 15:20:49.603 received 2048 bytes: uP[C#*°#­Jná+Ì9¦"ÕEù...
  6. 15:20:49.612 received 1024 bytes: Èþ̽٨û2ÿyªj*Cöaý...
  7. 15:20:49.725 received 1267 bytes: �Q(¢PÒfR¢(¢...
  8. 15:20:49.728 received 2372 bytes: u¢(¢Ñ1E1KIÅ...
  9. 15:20:49.739 received 2372 bytes: �\RcÓ3ÒQøPþ44£�...
  10. 15:20:49.749 received 2372 bytes: ´ûT¡▒FA$SrIQÁ�...
  11. 15:20:49.757 received 2372 bytes: ¤ÛÎihh£Úo©Å16Ef�...
  12.  
  13. ...
  14.  
only the first chunk contains the code and hash but it's 6343 bytes so I guess it also contains some part of other chunks that were sent .

What am I doing wrong?

Any ideas how I can send a file and make sure the server identifies all the chunks as one file?

Thanks in advance,
Robert
Title: Re: Sending and receiving files with lNet
Post by: rvk on June 17, 2022, 05:13:19 pm
Quote from: r.lukasiak on June 16, 2022, 10:45:02 pm
Any ideas how I can send a file and make sure the server identifies all the chunks as one file?
So why not parse the result from aSocket.GetMessage(s) correctly and split the received chunks back to individual packets?

But if you do that you can just as wel send everything in one go.
It would be better to investigate why a file of 100KB would be a problem.

Quote from: r.lukasiak on June 16, 2022, 10:45:02 pm
IF it's send in irregular chunks, the server recognizes only the first one, next chunks are sent without the code and hash.
So yeah. Try to find out why this goes wrong at the server end.
You should be able to just read the entire file and check the hash at the end.
Irregular chunks shouldn't be a problem. But you need to create a mechanism where you know the file is received completely.
You could send a filesize in the beginning and read until you have that number.
Another option would be to use the timeout but the downside of that is that you can get delays on the internet and it would resume after say a second.

Isn't there a ready made SendFile and ReceiveFile for this in lNet ??
Title: Re: Sending and receiving files with lNet
Post by: dje on June 19, 2022, 02:57:21 am
Any reason you are using yet another internet library? My advice is to look at the ssockets unit (fcl-net package) which comes with FreePascal/Lazarus.

There is an example of client/server programming in:
isockcli.pp and isocksvr.pp

Whenever I need to transfer data, I use fphttpserver & fphttpclient.

I prefer Indy, but I also like to keep project dependencies to libraries which come with FreePascal/Lazarus. It makes porting a whole lot easier.
Title: Re: Sending and receiving files with lNet
Post by: MarkMLl on June 19, 2022, 08:43:01 am
Quote from: r.lukasiak on June 16, 2022, 10:45:02 pm
but it's send in chunks of 1408 bytes, some chunks differ. The problem is that the message consists of a code (for the server to know what type of message it is), hash and a JPG file. IF it's send in irregular chunks, the server recognizes only the first one, next chunks are sent without the code and hash.

Then I got an idea to chop the message into 1024-byte chunks and send each piece with the code and hash so the server knows what file it belongs to but even though I send it in 1024 bytes, the server still receives random pieces.

TCP is a stream-oriented protocol, what you're doing quite simply won't work reliably. You /can/ rely on stuff arriving in the correct order, you /can't/ rely on block boundaries being preserved.

You need to put your own transfer protocol on top of TCP, or find a preexisting one. Then received TCP messages go into a buffer, and you parse this byte-by-byte to extract the higher-level transfer blocks.

Alternatively, use UDP (where block boundaries are honoured but you have to handle retries and the possibility of out-of-order reception) or SCTP.

Having said which, I agree with

Quote from: derek.john.evans on June 19, 2022, 02:57:21 am
Any reason you are using yet another internet library? My advice is to look at the ssockets unit (fcl-net package) which comes with FreePascal/Lazarus.

There is an example of client/server programming in:
isockcli.pp and isocksvr.pp

Whenever I need to transfer data, I use fphttpserver & fphttpclient.

I prefer Indy, but I also like to keep project dependencies to libraries which come with FreePascal/Lazarus. It makes porting a whole lot easier.

I use LNet for Telnet support on occasion (low-overhead debugging access to a strictly-local program) and TBH it's nothing but trouble. It predates routine use of threads, has some nasty failure modes, and the sourcecode isn't exactly easy reading.

MarkMLl
Title: Re: Sending and receiving files with lNet
Post by: r.lukasiak on June 21, 2022, 01:28:29 am
@rvk, thank you very much for the answer. I could modify the server so as soon as it recognizes the type of message to be a file, it'd assume that everything that comes next is a part of the file instead of parsing each chunk. However, this solution may fail if something happens on the way, if one of chunks is lost, it'd be still stuck to the expected file size. For now this solution might work, I'll implement it, I'll add the file size to the header and see what other issues I may face :-)

@derek.john.evans and MarkMLl, no, there is no particular reason for using lNet, it's just that I'm new to Lazarus+Networking so I just googled "Lazarus WebSockets", first I found was LazWebSockets by @Warfley, it seems pretty cool, it's threaded etc (I'm learning threads and WebSockets, I need them both) but there is not much documentation and it was hard to find any help so I switched to the second solution google gave me - lNet, it seems pretty complete, so far it works as needed, however not exactly as expected (but I guess it's just due to lack of knowledge on WebSockets and networking in general).

Quote
My advice is to look at the ssockets unit (fcl-net package) which comes with FreePascal/Lazarus.
I didn't even know it existed  :o, I will take a look.  I reviewed the demos you mentioned and I reviewed its source code. It seems to work a little bit different to lNet, I will give it a try. I found this https://wiki.freepascal.org/Sockets (https://wiki.freepascal.org/Sockets) but it's rather outdated, it mentions visual component, is it still the case? I can't find any and visual components might help me, especially for the client side.

Quote
but I also like to keep project dependencies to libraries which come with FreePascal/Lazarus. It makes porting a whole lot easier.
so do I, I used LazWebSockets and lNet only becasue I didn't know ssockets unit existed.
TinyPortal © 2005-2018