Lazarus

Programming => General => Topic started by: Hartmut on September 11, 2019, 07:24:38 pm

Title: [SOLVED] unzipping a file gives no error when harddisk is full
Post by: Hartmut on September 11, 2019, 07:24:38 pm
For unzipping a certain file I use the following program:

Code: Pascal  [Select][+][-]
  1. {$mode objfpc}{$H+}
  2.  
  3. uses classes,sysutils,zipper;
  4.  
  5. function unzip_file(fspecZ,fspecP,destDir: ansistring): ansistring;
  6.    {unzips packed file 'fspecP' from zipfile 'fspecZ' to destination 'destDir'}
  7.    var Z: TUnZipper;
  8.        SL: TStringList;
  9.    begin
  10.    Z:=TUnZipper.Create;
  11.    SL:=TStringList.Create;
  12.  
  13.    try
  14.       SL.Add(fspecP);
  15.       Z.FileName:=fspecZ;
  16.       Z.OutputPath:=destDir;
  17.       Z.UnZipFiles(SL);
  18.  
  19.       Z.Free;
  20.       SL.Free;
  21.       exit(''); {if no Error}
  22.  
  23.    except
  24.       on E: EZipError do
  25.          begin
  26.          Z.Free; SL.Free; exit('Error 1 ' + E.Message);
  27.          end;
  28.       on E: Exception do  {all other Exceptions: }
  29.          begin
  30.          Z.Free; SL.Free; exit('Error 2 ' + E.Message);
  31.          end;
  32.    end;
  33.    end;
  34.  
  35. var fspecZ,fspecP,destDir,se: ansistring;
  36.  
  37. begin
  38. fspecZ:='h:\tmp\Dylan.zip';
  39. fspecP:='Dylan.wav';       // this file is 147 MB big
  40. destDir:='d:\Tst\unzip\';  // this drive has only 137 MB free
  41. se:=unzip_file(fspecZ,fspecP,destDir);
  42. writeln('result="', se, '"');
  43. end.

But when I unzip a file, which is bigger than the free space on the destination harddisk, I get no error. The file is just truncated. I would expect to get an error, because the file is invalid.
Do I something wrong?
Is this a bug?

I use FPC 3.0.4 on Windows 7. My little project is attached. Thanks in advance.
Title: Re: unzipping a file gives no error when harddisk is full
Post by: jamie on September 11, 2019, 08:50:59 pm
I have not looked at that package yet but there should be an expanded disk space requirement method before you decompress it.

 Also, knowing the file size is already larger than the target before unzipping should be a good indicator or trouble ahead.
 
 
Title: Re: unzipping a file gives no error when harddisk is full
Post by: Hartmut on September 11, 2019, 09:29:25 pm
Quote from: jamie on September 11, 2019, 08:50:59 pm
...there should be an expanded disk space requirement method before you decompress...
What do you mean by that? Sorry, I don't understand this...
Title: Re: unzipping a file gives no error when harddisk is full
Post by: jamie on September 11, 2019, 10:40:06 pm
I meant there should be a check point method call done in the Tzipper, a method you can call to determine the total unexpanded file size before you actually unzip the file.

 I looked at it, there does not seem to be a way provided to determine required disk space before performing a decompress operation.

 Maybe you could suggest an improvement update ?  ;)
Title: Re: unzipping a file gives no error when harddisk is full
Post by: winni on September 11, 2019, 11:33:16 pm
Hi!

Unit zipper.pp:

In the  record Local_File_Header_Type - which is the internal header for each internal file - you will find the field Uncompressed_Size.

Add this field for all files contained in the zip file and you'll get the needed space on your harddisk.

Winni
Title: Re: unzipping a file gives no error when harddisk is full
Post by: sstvmaster on September 12, 2019, 12:59:20 am
This is my solution:
Code: Pascal  [Select][+][-]
  1. procedure TForm1.Button1Click(Sender: TObject);
  2. var
  3.   Z: TUnZipper;
  4.   i: Integer;
  5.   TotalSize: Int64;
  6.   EntrySize: Int64;
  7.   FreeSpace: Int64;
  8. begin
  9.   z := TUnZipper.Create;
  10.   z.FileName := 'h:\tmp\Dylan.zip';
  11.   z.OutputPath := 'd:\Tst\unzip\';
  12.  
  13.   z.Examine; // needed!!!
  14.  
  15.  
  16.   { ** Total size of all files in the archive ** }
  17.   TotalSize := 0;
  18.   for i := 0 to z.Entries.Count - 1 do
  19.   begin
  20.     Inc(TotalSize, z.Entries.Entries[i].Size);
  21.   end;
  22.   ShowMessage('The size of archive is: ' + IntToStr(TotalSize) + ' bytes.');
  23.   { ** }
  24.  
  25.  
  26.   { ** Size of an entry in the archive ** }
  27.   EntrySize := 0;
  28.   for i := 0 to z.Entries.Count - 1 do
  29.   begin
  30.     if z.Entries[i].ArchiveFileName = 'Dylan.wav' then
  31.     begin
  32.       EntrySize := z.Entries[i].Size;
  33.     end;
  34.   end;
  35.   ShowMessage('The size of entry is: ' + IntToStr(EntrySize) + ' bytes.');
  36.   { ** }
  37.  
  38.  
  39.   { ** how much free space is there? ** }
  40.   FreeSpace := DiskFree(Ord('d') - 64);  // "d" is the drive letter
  41.   ShowMessage('There are : ' + IntToStr(FreeSpace) + ' bytes free.');
  42.   { ** }
  43.  
  44.   if FreeSpace > EntrySize then // or TotalSize
  45.   begin
  46.     // do unzip here
  47.   end
  48.   else
  49.   begin
  50.     // error message
  51.   end;
  52.  
  53.  
  54.   z.Free;
  55. end;
  56.  
Title: Re: unzipping a file gives no error when harddisk is full
Post by: valdir.marcos on September 12, 2019, 05:04:56 am
Quote from: Hartmut on September 11, 2019, 07:24:38 pm
For unzipping a certain file I use the following program:
Code: Pascal  [Select][+][-]
  1. {$mode objfpc}{$H+}
  2.  
  3. uses classes,sysutils,zipper;
  4.  
  5. function unzip_file(fspecZ,fspecP,destDir: ansistring): ansistring;
  6.    {unzips packed file 'fspecP' from zipfile 'fspecZ' to destination 'destDir'}
  7.    var Z: TUnZipper;
  8.        SL: TStringList;
  9.    begin
  10.    Z:=TUnZipper.Create;
  11.    SL:=TStringList.Create;
  12.  
  13.    try
  14.       SL.Add(fspecP);
  15.       Z.FileName:=fspecZ;
  16.       Z.OutputPath:=destDir;
  17.       Z.UnZipFiles(SL);
  18.  
  19.       Z.Free;
  20.       SL.Free;
  21.       exit(''); {if no Error}
  22.  
  23.    except
  24.       on E: EZipError do
  25.          begin
  26.          Z.Free; SL.Free; exit('Error 1 ' + E.Message);
  27.          end;
  28.       on E: Exception do  {all other Exceptions: }
  29.          begin
  30.          Z.Free; SL.Free; exit('Error 2 ' + E.Message);
  31.          end;
  32.    end;
  33.    end;
  34.  
  35. var fspecZ,fspecP,destDir,se: ansistring;
  36.  
  37. begin
  38. fspecZ:='h:\tmp\Dylan.zip';
  39. fspecP:='Dylan.wav';       // this file is 147 MB big
  40. destDir:='d:\Tst\unzip\';  // this drive has only 137 MB free
  41. se:=unzip_file(fspecZ,fspecP,destDir);
  42. writeln('result="', se, '"');
  43. end.

But when I unzip a file, which is bigger than the free space on the destination harddisk, I get no error. The file is just truncated. I would expect to get an error, because the file is invalid.
Do I something wrong?
Is this a bug?

I use FPC 3.0.4 on Windows 7. My little project is attached. Thanks in advance.

Quote from: jamie on September 11, 2019, 10:40:06 pm
I meant there should be a check point method call done in the Tzipper, a method you can call to determine the total unexpanded file size before you actually unzip the file.
I looked at it, there does not seem to be a way provided to determine required disk space before performing a decompress operation.
Maybe you could suggest an improvement update ?  ;)
I have just opened a feature request:
unzipping a file gives no error when harddisk is full
https://bugs.freepascal.org/view.php?id=36068
Title: Re: unzipping a file gives no error when harddisk is full
Post by: jamie on September 12, 2019, 02:18:35 pm
Yes, I see that it has been fixed but you need to use the latest fpc to get the fix..
Title: Re: unzipping a file gives no error when harddisk is full
Post by: sstvmaster on September 12, 2019, 02:30:39 pm
And what is the difference between:

OLD:
Code: Pascal  [Select][+][-]
  1. FOutFile.Write(Buf^,Count);

NEW:
Code: Pascal  [Select][+][-]
  1. FOutFile.WriteBuffer(Buf^,Count);

Raise it an exception now?
Title: Re: unzipping a file gives no error when harddisk is full
Post by: wp on September 12, 2019, 02:36:02 pm
Quote from: sstvmaster on September 12, 2019, 02:30:39 pm
And what is the difference between:

OLD:
Code: Pascal  [Select][+][-]
  1. FOutFile.Write(Buf^,Count);

NEW:
Code: Pascal  [Select][+][-]
  1. FOutFile.WriteBuffer(Buf^,Count);

Raise it an exception now?

Yes.

Code: Pascal  [Select][+][-]
  1.   procedure TStream.WriteBuffer(const Buffer; Count: Longint);
  2.  
  3.   var
  4.     r,t : Longint;
  5.  
  6.     begin
  7.       T:=0;
  8.       Repeat
  9.          r:=Write(PByte(@Buffer)[t],Count-t);
  10.          inc(t,r);
  11.       Until (t=count) or (r<=0);
  12.       if (t<Count) then
  13.          Raise EWriteError.Create(SWriteError);
  14.     end;
Title: Re: unzipping a file gives no error when harddisk is full
Post by: sstvmaster on September 12, 2019, 03:01:32 pm
That means it raises an exception on a 4gb large zip at the last, lets say, 10 bytes of unzipping?
Title: Re: unzipping a file gives no error when harddisk is full
Post by: jamie on September 12, 2019, 04:39:53 pm
I noticed the use of LongWord for uncompressed sizes etc in the Local_File_Header_Type?

Does that mean the complete zip file or per internal file being uncompressed?

 Because I think LongWord is 32 bit so that is what say 4G limit?

PS.
 WP code

Looks like the Write method is implemented with returning < 0 for count output if it fails?

 I guess it is an abstraction so you are left up to doing as you please. I wonder if all the other stream implementations follow this path in other controls?

Title: Re: unzipping a file gives no error when harddisk is full
Post by: winni on September 12, 2019, 06:28:28 pm
Hi!

The old versions of Zip were restricted to 4 GB. Now they expanded it to 16 Exabytes - not really a restriction.

But dont forget:

Fat32 is restricted to 4 GB
Microsoft said that you should not use encrypted directories greater than 4 GB - they might crash.

Winni
Title: Re: unzipping a file gives no error when harddisk is full
Post by: Hartmut on September 12, 2019, 07:17:41 pm
During my absence a bug report was created and the bug was fixed. Great!

I could download the new version from SVN but I struggled how to compile the FPC including fpc.cfg. I created a new topic for that in https://forum.lazarus.freepascal.org/index.php/topic,46724.0.html

Thanks a lot to all who helped me so quickly!
Title: Re: unzipping a file gives no error when harddisk is full
Post by: Hartmut on September 13, 2019, 07:33:43 pm
@valdir.marcos:
Thank you very much for creating a bug report in https://bugs.freepascal.org/view.php?id=36068
The bug has been fixed after a few hours.
I have verified the fix and it works.
But I can't add a Note to the bug report nor close it. Maybe I don't have the rights to do this (I am only a "reporter").
Can you add a note and close the report?
Text could be: The fix has been verified by Hartmut with Revision 42985 on Windows 7. Now if the disk is full during unzipping an Exception "Stream write error" is raised. The bug has been fixed. Thanks for fixing.
Title: Re: [SOLVED] unzipping a file gives no error when harddisk is full
Post by: winni on September 14, 2019, 09:12:41 pm
Hi
!


But let us remember that this Zip version works only with the old Zip format  < 4 GB.
It will fail with the new version with the maximum of 16 Exabyte.

There should be a notice in the doc file.

Winni
Title: Re: unzipping a file gives no error when harddisk is full
Post by: valdir.marcos on September 20, 2019, 02:42:56 pm
Quote from: Hartmut on September 13, 2019, 07:33:43 pm
@valdir.marcos:
Thank you very much for creating a bug report in https://bugs.freepascal.org/view.php?id=36068
The bug has been fixed after a few hours.
I have verified the fix and it works.
But I can't add a Note to the bug report nor close it. Maybe I don't have the rights to do this (I am only a "reporter").
Can you add a note and close the report?
Text could be: The fix has been verified by Hartmut with Revision 42985 on Windows 7. Now if the disk is full during unzipping an Exception "Stream write error" is raised. The bug has been fixed. Thanks for fixing.
I don't think it's necessary since the ticket is already closed.
Title: Re: [SOLVED] unzipping a file gives no error when harddisk is full
Post by: valdir.marcos on September 20, 2019, 02:44:31 pm
Quote from: winni on September 14, 2019, 09:12:41 pm
Hi!
But let us remember that this Zip version works only with the old Zip format  < 4 GB.
It will fail with the new version with the maximum of 16 Exabyte.
There should be a notice in the doc file.
Winni
Have you already filed a bug report about documentation?
Title: Re: unzipping a file gives no error when harddisk is full
Post by: wp on September 20, 2019, 04:05:09 pm
Quote from: valdir.marcos on September 20, 2019, 02:42:56 pm
the ticket is already closed.
No it is not, it is only "resolved", i.e. the developer solved the issue either by fixing it himself, by apply a patch, or by discarding it. Now it's your turn as reporter to "close" the issue, i.e. you do the final test and verify that the issue really is solved. A "closed" issue is removed from most of the bugtracker lists which helps the developers by thinning out the bug lists. If you don't agree that the issue is solved you can re-open it by clicking on the "Reopen" button (this can happen even after you "closed" the ticket and discover later that the issue still exists under some circumstances.)

BTW, I would not close this particular issue because it does not consider the argument brought in by sstvmaster (reply #10): The unzipper should check the available disk space *before* beginning to write to disk. The current solution allows to write gigabytes to the HD and notice only at the end that the disk is overrunning. I think Michael was not aware of this aspect when resolving the issue. At least it should be brought to his attention to make him reconsider his solution.
Title: Re: unzipping a file gives no error when harddisk is full
Post by: valdir.marcos on September 20, 2019, 05:01:57 pm
Quote from: wp on September 20, 2019, 04:05:09 pm
Quote from: valdir.marcos on September 20, 2019, 02:42:56 pm
the ticket is already closed.
No it is not, it is only "resolved", i.e. the developer solved the issue either by fixing it himself, by apply a patch, or by discarding it. Now it's your turn as reporter to "close" the issue, i.e. you do the final test and verify that the issue really is solved. A "closed" issue is removed from most of the bugtracker lists which helps the developers by thinning out the bug lists. If you don't agree that the issue is solved you can re-open it by clicking on the "Reopen" button (this can happen even after you "closed" the ticket and discover later that the issue still exists under some circumstances.)
Thanks for your information.
I thought it was closed because of a new menu option to "Reopen" (Reabrir) (image attached) appeared and no way to add more information on Activities panel.

Quote
BTW, I would not close this particular issue because it does not consider the argument brought in by sstvmaster (reply #10): The unzipper should check the available disk space *before* beginning to write to disk. The current solution allows to write gigabytes to the HD and notice only at the end that the disk is overrunning. I think Michael was not aware of this aspect when resolving the issue. At least it should be brought to his attention to make him reconsider his solution.
I will reopen the ticket and write down all this extra information.
Thanks again.
Title: Re: unzipping a file gives no error when harddisk is full
Post by: PascalDragon on September 21, 2019, 12:03:12 pm
Quote from: wp on September 20, 2019, 04:05:09 pm
BTW, I would not close this particular issue because it does not consider the argument brought in by sstvmaster (reply #10): The unzipper should check the available disk space *before* beginning to write to disk. The current solution allows to write gigabytes to the HD and notice only at the end that the disk is overrunning. I think Michael was not aware of this aspect when resolving the issue. At least it should be brought to his attention to make him reconsider his solution.
I don't agree here. It should be up to the application to decide whether it wants to prohibit extracting to a volume without enough free space or not (e.g. there could be a valid case that one extracts something while deleting something else at the same time).
Not to mention that the SysUtils.DiskFree function might not return something useful on all systems.
Title: Re: unzipping a file gives no error when harddisk is full
Post by: sstvmaster on September 21, 2019, 07:47:31 pm
Quote from: PascalDragon on September 21, 2019, 12:03:12 pm
I don't agree here. It should be up to the application to decide whether it wants to prohibit extracting to a volume without enough free space or not
Shure, the App could do that.

Quote from: PascalDragon on September 21, 2019, 12:03:12 pm
(e.g. there could be a valid case that one extracts something while deleting something else at the same time).
OK, but i never saw something like this.

Quote from: PascalDragon on September 21, 2019, 12:03:12 pm
Not to mention that the SysUtils.DiskFree function might not return something useful on all systems.
Then the developer should solve this problem.
Title: Re: [SOLVED] unzipping a file gives no error when harddisk is full
Post by: jamie on September 21, 2019, 10:43:19 pm
Maybe at least  property to report the required space could be in order?
Title: Re: [SOLVED] unzipping a file gives no error when harddisk is full
Post by: sstvmaster on September 22, 2019, 12:33:40 am
This could be a solution:
Code: Pascal  [Select][+][-]
  1. --- C:/lazarus/fpc/3.0.4/source/packages/paszlib/src/zipper.pp  Sat Sep 21 23:55:13 2019
  2. +++ C:/Users/Ich/Desktop/ZipperSizeUncompressed/zipper.pp       Sun Sep 22 00:18:14 2019
  3. @@ -493,0 +494 @@
  4. +    FUnzippedSize: Int64;
  5. @@ -549,0 +551 @@
  6. +    Property UnZippedSize: Int64 Read FUnZippedSize;
  7. @@ -2334,0 +2337,8 @@
  8. +
  9. +  // Unzipped size of file
  10. +  FUnzippedSize := 0;
  11. +  for i:=0 to FEntries.Count - 1 do
  12. +  begin
  13. +    Inc(FUnZippedSize, FEntries.Entries[i].Size);
  14. +  end;
  15. +
  16.  

This is a diff against fpc 3.0.4.
Title: Re: [SOLVED] unzipping a file gives no error when harddisk is full
Post by: jamie on September 22, 2019, 01:00:10 am
yes, looks good..
TinyPortal © 2005-2018