Recent

Author Topic: Rejecting file with indy  (Read 1824 times)

torbente

  • Sr. Member
  • ****
  • Posts: 325
    • Noso Main Page
Rejecting file with indy
« on: January 04, 2021, 12:43:53 am »
Hi everyone;
I created a p2p networkd using indy (IdTCPServer and multiple TIdTCPClients) for mostly exchange text lines. But sometimes, a file needs to be sent. On this cases, the sender first sends a textline, something like:
FILE {filename} {hashofthefile}
And then the file itself is sent.
The receiver read the line first and then receives the file.
Server Example of code:

Code: Pascal  [Select][+][-]
  1. LLine := AContext.Connection.IOHandler.ReadLn(IndyTextEncoding_UTF8);  
  2. AFileStream := TFileStream.Create(UpdateZipName, fmCreate);
  3. AContext.Connection.IOHandler.ReadStream(AFileStream);

IT is working perfect right now. But i want be able to NOT DOWNLOAD the file and clear it from the buffer.

If i use
Code: Pascal  [Select][+][-]
  1. Conexiones[Slot].context.Connection.IOHandler.InputBuffer.Clear;  

it deletes the file, but also all that is inmediately after the file, including text lines and/or other files.

Of course, i could just download the file and delete it inmediately, but it is only an imperfect solution.
Noso Cryptocurrency Main Developer
https://github.com/DevTeamNoso/NosoWallet

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 1312
    • Lebeau Software
Re: Rejecting file with indy
« Reply #1 on: January 05, 2021, 02:30:35 am »
On this cases, the sender first sends a textline, something like:
FILE {filename} {hashofthefile}

You should include the file's size, too.  That way, the receiver can pre-size the target file if it wants to (that can speed up disk I/O), or reject the transfer if the file is too large, etc.  If nothing else, this will also allow you to reuse the same TCP connection for multiple transfers, or at least continued communication, without creating a new TCP connection.

I would not send the hash up front, though.  That requires the sender to actually hash the entire file before even thinking of starting a transfer, and doing that will prevent the transfer from streaming bytes dynamically from other sources than just a file on a hard drive.  If you are going to send a hash, it should be sent after the transfer is finished.  The receiver has to hash the bytes as they are being received, so the sender can do the same while sending the bytes.  You can't compare hashes until the transfer is finished anyway, so may as well not waste the processing time at the beginning of the transfer.

I would also suggest making the receiver send a preliminary response back to the sender BEFORE allowing the sender to begin sending the actual file bytes.  And to make the receiver send a final response AFTER the transfer has finished.  This way, the receiver can reject a transfer right away (file too big, can't create the target file, etc), and it can report whether all bytes were received and the hashes compared equal.

The receiver read the line first and then receives the file.
Server Example of code:

Code: Pascal  [Select][+][-]
  1. LLine := AContext.Connection.IOHandler.ReadLn(IndyTextEncoding_UTF8);  
  2. AFileStream := TFileStream.Create(UpdateZipName, fmCreate);
  3. AContext.Connection.IOHandler.ReadStream(AFileStream);

This use of ReadStream() is assuming the sender is sending the file size before the file data.  Is the sender actually doing that?

But i want be able to NOT DOWNLOAD the file and clear it from the buffer.

The only way to abort TIdIOHandler.ReadStream() once it has begun its work is to close the TCP connection, or at least raise an exception in the TIdIOHandler.OnWork... events.  But that won't remove any unread data from the TIdIOHandler.InputBuffer.  Your best option is to allow your protocol to reject the transfer before ReadStream() is called at all.  Once ReadStream() is called, you have to assume the transfer will progress to completion, unless an error occurs along the way, in which case the only sane thing to do is close the connection anyway.

If i use
Code: Pascal  [Select][+][-]
  1. Conexiones[Slot].context.Connection.IOHandler.InputBuffer.Clear;  

it deletes the file, but also all that is inmediately after the file, including text lines and/or other files.

All the more reason to send the file size up front, when possible.  That way, you can calculate how many bytes remain in the transfer, and then Discard() them without discarding what follows the transfer.
« Last Edit: January 05, 2021, 02:34:37 am by Remy Lebeau »
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

torbente

  • Sr. Member
  • ****
  • Posts: 325
    • Noso Main Page
Re: Rejecting file with indy
« Reply #2 on: January 05, 2021, 06:52:58 pm »
EDIT: I got the filesize in bytes using filesize. Now i can send that info too, but how i could clear that specified number of bytes from inputbuffer?

Thanks you a lot for your long reply.

Quote
You should include the file's size, too.

I have no idea how to get the filesize. Anyway, the TCP connection is always open until one of the parts (server/client) close it. (using regular pings client/server)

Quote
I would not send the hash up front, though. 

99% of times, the file hash is already stored on memory (files are hashed inmediately after been created). Also, the hash is the main information to let know the receiver if it is the required file.

Quote
I would also suggest making the receiver send a preliminary response back to the sender BEFORE allowing the sender to begin sending the actual file bytes.

It is been done before (almost always):
Receiver: REQUEST {filename} {filehash}
sender: FILE {filename} {filehash}

But sometimes the sender generates a new file that must be shared with all the peers in the network. In those cases, the sender sends this:

sender: FILE {filename} {filehash} {UserWhoCreatedThefile} {SignToVerify}

The receiver, in this case, should verify if {UserWhoCreatedThefile} and {SignToVerify} are valid. If so, then download the file and re-send to the other peers; if not, just ignore the file.

So the receiver is doing this:

Code: Pascal  [Select][+][-]
  1. LLine := AContext.Connection.IOHandler.ReadLn(IndyTextEncoding_UTF8);
  2. // verify is {UserWhoCreatedThefile} {SignToVerify} is valid
  3. if Validated then
  4.    begin
  5.    AFileStream := TFileStream.Create(UpdateZipName, fmCreate);
  6.    AContext.Connection.IOHandler.ReadStream(AFileStream);
  7.    AFileStream.Free;
  8.    end
  9. else
  10.    begin
  11.    // discard it; how??
  12.    end;
  13.  

For your reply, i guess i could send the filesize in advance (even when i do not know how to get that info) so, if the receiver do not want to receive the file, it could clear that size from the inputbuffer (what i neither know how could be done) without affect the other incoming information.

Againn, thank you for your help

« Last Edit: January 05, 2021, 07:25:34 pm by torbente »
Noso Cryptocurrency Main Developer
https://github.com/DevTeamNoso/NosoWallet

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 1312
    • Lebeau Software
Re: Rejecting file with indy
« Reply #3 on: January 05, 2021, 09:34:26 pm »
EDIT: I got the filesize in bytes using filesize. Now i can send that info too, but how i could clear that specified number of bytes from inputbuffer?

I already told you how.  Take the number of bytes you already received, and subtract that from the number of bytes you are expecting, and that is the number of bytes you can pass to TIdIOHandler.Discard().  But, I really strongly urge you not to do this.  If an unexpected error happens during the transfer, you really don't know the state of the socket anymore, so the only sane thing to do is close the connection.  If you are going to reject a file, it needs to be done before the transfer begins, not during the transfer.

Quote
I would not send the hash up front, though. 

99% of times, the file hash is already stored on memory (files are hashed inmediately after been created). Also, the hash is the main information to let know the receiver if it is the required file.

IF you already know the hash up front, then fine.

Quote
I would also suggest making the receiver send a preliminary response back to the sender BEFORE allowing the sender to begin sending the actual file bytes.

It is been done before (almost always):
Receiver: REQUEST {filename} {filehash}
sender: FILE {filename} {filehash}

As long as the FILE acts as a response saying "yes, I will start sending the file bytes now", then fine.  Sending the file name and hash in that reply is a bit redundant, though.

But sometimes the sender generates a new file that must be shared with all the peers in the network. In those cases, the sender sends this:

sender: FILE {filename} {filehash} {UserWhoCreatedThefile} {SignToVerify}

The receiver, in this case, should verify if {UserWhoCreatedThefile} and {SignToVerify} are valid. If so, then download the file and re-send to the other peers; if not, just ignore the file.

The receiver should not just ignore the file in that situation.  If the FILE command is unsolicited, then the receiver should send back a reply accepting or rejecting the transfer before the file bytes are sent.  It is bad protocol design to have unacknowledged commands.  If adding that response is not possible, then the sender should at least include the file size in the FILE command so that the receiver knows how many bytes it needs to ignore.

For your reply, i guess i could send the filesize in advance ... so, if the receiver do not want to receive the file, it could clear that size from the inputbuffer

Yes, exactly.

what i neither know how could be done

Have a look at the TIdIOHandler.Discard() method.  Or, you could just keep using TIdIOHandler.ReadStream() and simply set its AStream parameter to nil to discard all of the received data.
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

torbente

  • Sr. Member
  • ****
  • Posts: 325
    • Noso Main Page
Re: Rejecting file with indy
« Reply #4 on: April 11, 2021, 01:17:33 am »
I retake this thread since now i have a related issue and im stuck (again  %)):
I checked the internet couple days looking for a solution without result.

How i can get an error during the download of the file, so i can close the connection?
(Note: This function is running in a thread)

Code: Pascal  [Select][+][-]
  1. LLine := AContext.Connection.IOHandler.ReadLn(IndyTextEncoding_UTF8);
  2. // Line contains filename, hash and size
  3. if Validated then // the file is valid, so download it
  4.    begin
  5.    try
  6.       AFileStream := TFileStream.Create(UpdateZipName, fmCreate);
  7.       AContext.Connection.IOHandler.ReadStream(AFileStream);
  8.       // If an error happens during the download, how i could catch it?
  9.       // Or how i could know the size of the data already downloaded?
  10.       // Try...except block somehow is not working...
  11.       AFileStream.Free;
  12.    except on E:Exception do // Disconnect
  13.    end
  14. else  // Not a valid file, so discard it, but keeps the connection
  15.    begin
  16.    AContext.Connection.IOHandler.DiscardAll();
  17.    end;
Noso Cryptocurrency Main Developer
https://github.com/DevTeamNoso/NosoWallet

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 1312
    • Lebeau Software
Re: Rejecting file with indy
« Reply #5 on: April 12, 2021, 08:40:19 pm »
How i can get an error during the download of the file, so i can close the connection?
(Note: This function is running in a thread)

Indy raises an exception on an error.  Simply catch the exception to Disconnect() the connection.  Or, in the case of an Indy TCP server, just let the server catch the exception, it will close the connection for you, and fire the OnException event.

Code: Pascal  [Select][+][-]
  1. AFileStream := TFileStream.Create(UpdateZipName, fmCreate);
  2. AContext.Connection.IOHandler.ReadStream(AFileStream);
  3. // If an error happens during the download, how i could catch it?
  4. // Or how i could know the size of the data already downloaded?
  5. // Try...except block somehow is not working...

A try..except will work just fine.  You should also use a try..finally to ensure the TFileStream is freed whether an exception is raised or not.

Code: Pascal  [Select][+][-]
  1. AContext.Connection.IOHandler.DiscardAll();

Note that DiscardAll() will not return until the connection has been closed.
« Last Edit: April 12, 2021, 08:42:09 pm by Remy Lebeau »
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

 

TinyPortal © 2005-2018