Recent

Author Topic: [SOLVED] unzipping a file gives no error when harddisk is full  (Read 590 times)

Hartmut

  • Sr. Member
  • ****
  • Posts: 265
[SOLVED] unzipping a file gives no error when harddisk is full
« on: September 11, 2019, 07:24:38 pm »
For unzipping a certain file I use the following program:

Code: Pascal  [Select]
  1. {$mode objfpc}{$H+}
  2.  
  3. uses classes,sysutils,zipper;
  4.  
  5. function unzip_file(fspecZ,fspecP,destDir: ansistring): ansistring;
  6.    {unzips packed file 'fspecP' from zipfile 'fspecZ' to destination 'destDir'}
  7.    var Z: TUnZipper;
  8.        SL: TStringList;
  9.    begin
  10.    Z:=TUnZipper.Create;
  11.    SL:=TStringList.Create;
  12.  
  13.    try
  14.       SL.Add(fspecP);
  15.       Z.FileName:=fspecZ;
  16.       Z.OutputPath:=destDir;
  17.       Z.UnZipFiles(SL);
  18.  
  19.       Z.Free;
  20.       SL.Free;
  21.       exit(''); {if no Error}
  22.  
  23.    except
  24.       on E: EZipError do
  25.          begin
  26.          Z.Free; SL.Free; exit('Error 1 ' + E.Message);
  27.          end;
  28.       on E: Exception do  {all other Exceptions: }
  29.          begin
  30.          Z.Free; SL.Free; exit('Error 2 ' + E.Message);
  31.          end;
  32.    end;
  33.    end;
  34.  
  35. var fspecZ,fspecP,destDir,se: ansistring;
  36.  
  37. begin
  38. fspecZ:='h:\tmp\Dylan.zip';
  39. fspecP:='Dylan.wav';       // this file is 147 MB big
  40. destDir:='d:\Tst\unzip\';  // this drive has only 137 MB free
  41. se:=unzip_file(fspecZ,fspecP,destDir);
  42. writeln('result="', se, '"');
  43. end.

But when I unzip a file, which is bigger than the free space on the destination harddisk, I get no error. The file is just truncated. I would expect to get an error, because the file is invalid.
Do I something wrong?
Is this a bug?

I use FPC 3.0.4 on Windows 7. My little project is attached. Thanks in advance.
« Last Edit: September 13, 2019, 07:34:11 pm by Hartmut »

jamie

  • Hero Member
  • *****
  • Posts: 1899
Re: unzipping a file gives no error when harddisk is full
« Reply #1 on: September 11, 2019, 08:50:59 pm »
I have not looked at that package yet but there should be an expanded disk space requirement method before you decompress it.

 Also, knowing the file size is already larger than the target before unzipping should be a good indicator or trouble ahead.
 
 

Hartmut

  • Sr. Member
  • ****
  • Posts: 265
Re: unzipping a file gives no error when harddisk is full
« Reply #2 on: September 11, 2019, 09:29:25 pm »
...there should be an expanded disk space requirement method before you decompress...
What do you mean by that? Sorry, I don't understand this...

jamie

  • Hero Member
  • *****
  • Posts: 1899
Re: unzipping a file gives no error when harddisk is full
« Reply #3 on: September 11, 2019, 10:40:06 pm »
I meant there should be a check point method call done in the Tzipper, a method you can call to determine the total unexpanded file size before you actually unzip the file.

 I looked at it, there does not seem to be a way provided to determine required disk space before performing a decompress operation.

 Maybe you could suggest an improvement update ?  ;)

winni

  • Full Member
  • ***
  • Posts: 182
Re: unzipping a file gives no error when harddisk is full
« Reply #4 on: September 11, 2019, 11:33:16 pm »
Hi!

Unit zipper.pp:

In the  record Local_File_Header_Type - which is the internal header for each internal file - you will find the field Uncompressed_Size.

Add this field for all files contained in the zip file and you'll get the needed space on your harddisk.

Winni
« Last Edit: September 11, 2019, 11:35:04 pm by winni »

sstvmaster

  • Full Member
  • ***
  • Posts: 116
Re: unzipping a file gives no error when harddisk is full
« Reply #5 on: September 12, 2019, 12:59:20 am »
This is my solution:
Code: Pascal  [Select]
  1. procedure TForm1.Button1Click(Sender: TObject);
  2. var
  3.   Z: TUnZipper;
  4.   i: Integer;
  5.   TotalSize: Int64;
  6.   EntrySize: Int64;
  7.   FreeSpace: Int64;
  8. begin
  9.   z := TUnZipper.Create;
  10.   z.FileName := 'h:\tmp\Dylan.zip';
  11.   z.OutputPath := 'd:\Tst\unzip\';
  12.  
  13.   z.Examine; // needed!!!
  14.  
  15.  
  16.   { ** Total size of all files in the archive ** }
  17.   TotalSize := 0;
  18.   for i := 0 to z.Entries.Count - 1 do
  19.   begin
  20.     Inc(TotalSize, z.Entries.Entries[i].Size);
  21.   end;
  22.   ShowMessage('The size of archive is: ' + IntToStr(TotalSize) + ' bytes.');
  23.   { ** }
  24.  
  25.  
  26.   { ** Size of an entry in the archive ** }
  27.   EntrySize := 0;
  28.   for i := 0 to z.Entries.Count - 1 do
  29.   begin
  30.     if z.Entries[i].ArchiveFileName = 'Dylan.wav' then
  31.     begin
  32.       EntrySize := z.Entries[i].Size;
  33.     end;
  34.   end;
  35.   ShowMessage('The size of entry is: ' + IntToStr(EntrySize) + ' bytes.');
  36.   { ** }
  37.  
  38.  
  39.   { ** how much free space is there? ** }
  40.   FreeSpace := DiskFree(Ord('d') - 64);  // "d" is the drive letter
  41.   ShowMessage('There are : ' + IntToStr(FreeSpace) + ' bytes free.');
  42.   { ** }
  43.  
  44.   if FreeSpace > EntrySize then // or TotalSize
  45.   begin
  46.     // do unzip here
  47.   end
  48.   else
  49.   begin
  50.     // error message
  51.   end;
  52.  
  53.  
  54.   z.Free;
  55. end;
  56.  
Lazarus 2.0.4 x32
Lazarus 2.1.0 Trunk x32
OS Win 7 32bit

valdir.marcos

  • Hero Member
  • *****
  • Posts: 795
Re: unzipping a file gives no error when harddisk is full
« Reply #6 on: September 12, 2019, 05:04:56 am »
For unzipping a certain file I use the following program:
Code: Pascal  [Select]
  1. {$mode objfpc}{$H+}
  2.  
  3. uses classes,sysutils,zipper;
  4.  
  5. function unzip_file(fspecZ,fspecP,destDir: ansistring): ansistring;
  6.    {unzips packed file 'fspecP' from zipfile 'fspecZ' to destination 'destDir'}
  7.    var Z: TUnZipper;
  8.        SL: TStringList;
  9.    begin
  10.    Z:=TUnZipper.Create;
  11.    SL:=TStringList.Create;
  12.  
  13.    try
  14.       SL.Add(fspecP);
  15.       Z.FileName:=fspecZ;
  16.       Z.OutputPath:=destDir;
  17.       Z.UnZipFiles(SL);
  18.  
  19.       Z.Free;
  20.       SL.Free;
  21.       exit(''); {if no Error}
  22.  
  23.    except
  24.       on E: EZipError do
  25.          begin
  26.          Z.Free; SL.Free; exit('Error 1 ' + E.Message);
  27.          end;
  28.       on E: Exception do  {all other Exceptions: }
  29.          begin
  30.          Z.Free; SL.Free; exit('Error 2 ' + E.Message);
  31.          end;
  32.    end;
  33.    end;
  34.  
  35. var fspecZ,fspecP,destDir,se: ansistring;
  36.  
  37. begin
  38. fspecZ:='h:\tmp\Dylan.zip';
  39. fspecP:='Dylan.wav';       // this file is 147 MB big
  40. destDir:='d:\Tst\unzip\';  // this drive has only 137 MB free
  41. se:=unzip_file(fspecZ,fspecP,destDir);
  42. writeln('result="', se, '"');
  43. end.

But when I unzip a file, which is bigger than the free space on the destination harddisk, I get no error. The file is just truncated. I would expect to get an error, because the file is invalid.
Do I something wrong?
Is this a bug?

I use FPC 3.0.4 on Windows 7. My little project is attached. Thanks in advance.

I meant there should be a check point method call done in the Tzipper, a method you can call to determine the total unexpanded file size before you actually unzip the file.
I looked at it, there does not seem to be a way provided to determine required disk space before performing a decompress operation.
Maybe you could suggest an improvement update ?  ;)
I have just opened a feature request:
unzipping a file gives no error when harddisk is full
https://bugs.freepascal.org/view.php?id=36068

jamie

  • Hero Member
  • *****
  • Posts: 1899
Re: unzipping a file gives no error when harddisk is full
« Reply #7 on: September 12, 2019, 02:18:35 pm »
Yes, I see that it has been fixed but you need to use the latest fpc to get the fix..

sstvmaster

  • Full Member
  • ***
  • Posts: 116
Re: unzipping a file gives no error when harddisk is full
« Reply #8 on: September 12, 2019, 02:30:39 pm »
And what is the difference between:

OLD:
Code: Pascal  [Select]
  1. FOutFile.Write(Buf^,Count);

NEW:
Code: Pascal  [Select]
  1. FOutFile.WriteBuffer(Buf^,Count);

Raise it an exception now?
Lazarus 2.0.4 x32
Lazarus 2.1.0 Trunk x32
OS Win 7 32bit

wp

  • Hero Member
  • *****
  • Posts: 6140
Re: unzipping a file gives no error when harddisk is full
« Reply #9 on: September 12, 2019, 02:36:02 pm »
And what is the difference between:

OLD:
Code: Pascal  [Select]
  1. FOutFile.Write(Buf^,Count);

NEW:
Code: Pascal  [Select]
  1. FOutFile.WriteBuffer(Buf^,Count);

Raise it an exception now?

Yes.

Code: Pascal  [Select]
  1.   procedure TStream.WriteBuffer(const Buffer; Count: Longint);
  2.  
  3.   var
  4.     r,t : Longint;
  5.  
  6.     begin
  7.       T:=0;
  8.       Repeat
  9.          r:=Write(PByte(@Buffer)[t],Count-t);
  10.          inc(t,r);
  11.       Until (t=count) or (r<=0);
  12.       if (t<Count) then
  13.          Raise EWriteError.Create(SWriteError);
  14.     end;
Lazarus trunk / fpc 3.0.4 / all 32-bit on Win-10

sstvmaster

  • Full Member
  • ***
  • Posts: 116
Re: unzipping a file gives no error when harddisk is full
« Reply #10 on: September 12, 2019, 03:01:32 pm »
That means it raises an exception on a 4gb large zip at the last, lets say, 10 bytes of unzipping?
Lazarus 2.0.4 x32
Lazarus 2.1.0 Trunk x32
OS Win 7 32bit

jamie

  • Hero Member
  • *****
  • Posts: 1899
Re: unzipping a file gives no error when harddisk is full
« Reply #11 on: September 12, 2019, 04:39:53 pm »
I noticed the use of LongWord for uncompressed sizes etc in the Local_File_Header_Type?

Does that mean the complete zip file or per internal file being uncompressed?

 Because I think LongWord is 32 bit so that is what say 4G limit?

PS.
 WP code

Looks like the Write method is implemented with returning < 0 for count output if it fails?

 I guess it is an abstraction so you are left up to doing as you please. I wonder if all the other stream implementations follow this path in other controls?

« Last Edit: September 12, 2019, 05:00:14 pm by jamie »

winni

  • Full Member
  • ***
  • Posts: 182
Re: unzipping a file gives no error when harddisk is full
« Reply #12 on: September 12, 2019, 06:28:28 pm »
Hi!

The old versions of Zip were restricted to 4 GB. Now they expanded it to 16 Exabytes - not really a restriction.

But dont forget:

Fat32 is restricted to 4 GB
Microsoft said that you should not use encrypted directories greater than 4 GB - they might crash.

Winni

Hartmut

  • Sr. Member
  • ****
  • Posts: 265
Re: unzipping a file gives no error when harddisk is full
« Reply #13 on: September 12, 2019, 07:17:41 pm »
During my absence a bug report was created and the bug was fixed. Great!

I could download the new version from SVN but I struggled how to compile the FPC including fpc.cfg. I created a new topic for that in https://forum.lazarus.freepascal.org/index.php/topic,46724.0.html

Thanks a lot to all who helped me so quickly!

Hartmut

  • Sr. Member
  • ****
  • Posts: 265
Re: unzipping a file gives no error when harddisk is full
« Reply #14 on: September 13, 2019, 07:33:43 pm »
@valdir.marcos:
Thank you very much for creating a bug report in https://bugs.freepascal.org/view.php?id=36068
The bug has been fixed after a few hours.
I have verified the fix and it works.
But I can't add a Note to the bug report nor close it. Maybe I don't have the rights to do this (I am only a "reporter").
Can you add a note and close the report?
Text could be: The fix has been verified by Hartmut with Revision 42985 on Windows 7. Now if the disk is full during unzipping an Exception "Stream write error" is raised. The bug has been fixed. Thanks for fixing.