[SOLVED] Request: test zip file for zip64 support

[BigChimp]

Hi all,

I'm working on support for the zip64 format (e.g. large files and large amount of files) for FreePascal tzipper.

I think I've about figured it out and my tests create a zip file with 65537 empty files.
Info-Zip's unzip.exe (version 6) and DotNetZip can extract it.
7zip's 7za archiver (9.20 and 9.22 beta) has trouble and say it's invalid. Unfortunately, there doesn't seem to be more diagnostic output available.

Request: could you download my generated test zip at https://bitbucket.org/reiniero/fpc_laz_patch_playground/downloads/z64_65537emptyfiles.zip and test it with your favourite utility, and report success/failure?
Thanks a lot.

Background:
Current work in progress at
https://bitbucket.org/reiniero/fpc_laz_patch_playground/src
directory zip64
I think I've got compression and extraction working.

I was a bit too optimistic by assuming the code could support qword-sized files, but it uses streams which can only go to int64.
So I'll need to build in some exceptions for large file sizes when compressing/uncompressing zip files, and rework the various position/size variables to use int64 again instead of qword.

[taazz]

winRar 4.20 beta 2 32bit opens the file correctly but using "test archive files" command reports a corrupt archive.

Abbrevia already supports 64bit zip files and has some test cases on it which you could use as a starting point probably https://sourceforge.net/p/tpabbrevia/code/HEAD/tree/trunk/tests/.
Make sure to read the "readme.txt" file for info on where you can find the test files.

[bonmario]

I've tried with the command unzip on my Ubuntu Linux 12.10 64 bit, and i have this error:

error:  expected central file header signature not found (file #65538).
  (please check that you have transferred or created the zipfile in the
  appropriate BINARY mode and that you have compiled UnZip properly)
 extracting: 65537


Hi, Mario

[BigChimp]

@taazz: thanks! I suppose there's no extra diagnostic output available from winrar to indicate what it thinks is corrupted?

I've tried with the command unzip on my Ubuntu Linux 12.10 64 bit, and i have this error:

error:  expected central file header signature not found (file #65538).
  (please check that you have transferred or created the zipfile in the
  appropriate BINARY mode and that you have compiled UnZip properly)
 extracting: 65537

Grazie, Mario

AFAIU, that's normal for Info-Zip's unzip when reading zip64 files.
Running

Code: [Select]

unzip -t z64_65537emptyfiles.zip
to test the archive, does indicate all files should be ok.

File 65538 doesn't actually exist - I'm wondering where that's coming from... but I've seen it in other posts where people work with info-zip.

from my notes in the test program:

Code: [Select]

    {Note:
    info-zip unzip 6.0 works with this file, though it prints some warnings which
    are apparently common if the zip64 format is used correctly:
    warning:  76 extra bytes at beginning or within zipfile
      (attempting to process anyway)
    error:  reported length of central directory is
      -76 bytes too long (Atari STZip zipfile?  J.H.Holm ZIPSPLIT 1.1
      zipfile?).  Compensating...
    error:  expected central file header signature not found (file #65538).
      (please check that you have transferred or created the zipfile in the
      appropriate BINARY mode and that you have compiled UnZip properly)

    Ionic's DotNetZip library unzip.exe utility verison 1.9.1.8 works

    7za 9.20 and 9.22 beta give invalid zip error
    probably an error in their implementation...
    }


[BigChimp]

Oh, I've uploaded
https://bitbucket.org/reiniero/fpc_laz_patch_playground/downloads/tczipper.exe
(Windows x86) that lets you run the tests yourself, and more importantly:

Code: [Select]

tczipper bla.zip
will read bla.zip and try to decompress all files (resulting files are not extracted but discarded so they don't fill your disk).

This allows you to use the code to test any zip file you may have on your system.

[Bart]

Windows Explorer (Win7-64) cannot open the file.

[Window Title]
Fout in Gecomprimeerde (gezipte) mappen

[Content]
Kan de map niet openen.

De gecomprimeerde (gezipte) map C:\Users\Bart\Downloads\z64_65537emptyfiles.zip is ongeldig.

[OK]

AFAIK you are Dutch so I won't translate 

Bart

[taazz]

from a quick look around I did not see any settings for detail reporting or loging. For the time being I'm running a "repair archive" command to see what it will report.

EDIT:
Repair completed and the repaired archive could be opened by windows explorer on windows 7 32 bit and winrar 4.20 with out problems. I have a diff from the two files the black spots are where the differences are. Hope it helps a bit.

[BigChimp]

Windows Explorer (Win7-64) cannot open the file.

Takkeding

AFAIU Win7's built in zip support should support the zip64 format... so either it's having problems with something else in the zip or I'm still missing something...

[jwdietrich]

In addition I tried some Mac-based unzip utilities with your large zip file. None of them could successfully retrieve 65537 empty files. These are the results:

Apple's built-in Archive Utility: Works several minutes in order to create an empty folder.
Keka: Exits with error code 2.
Stuffit Expanders: Creates an empty folder.

[BlueIcaro]

Hi, my winrar 64 bits on W7/64 can open it.
But 7z / 64 bit, say :"Can't open as compress file"

/BLueIcaro

[JuhaManninen]

Ark (part of KDE) on a 64-bit Linux opens it without problems.
It takes some time though, around 10 minutes.

[Rails]

Squeeze (XFCE Desktop) fails with Exit code 768, whatever that means. 


It did find a whole bunch of empty files though, but the highest file name reported is 65499. It also reports that all the files in the archive have a CRC-32 of 00000000.

Edit: I should mention that squeeze's  release number is only at 0.23, so it isn't the best (un)archiver out there yet

[sam707]

I thought that zip was paleolithic sumthing and that human beings found out fire , invented wheel , etc ...

http://www.7-zip.org/sdk.html
http://en.wikipedia.org/wiki/LZMA

anyone requesting for ARJ ? PKARC ? MSDOS 0.9 ? LOL

LZMA SDK for free pascal download >> http://www.birtles.org.uk/programming/LZMA.442b.7z Compatilbe with Borland Delphi, Borland Kylix, Free Pascal and the Lazarus IDE

I enjoy my quad-core on it 

[BigChimp]

Thanks guys, especially taazz: exactly what I needed! I'll have a look to see what the changes entail.

@Sam: that's nice. Looking forward to a patch with 7z support from you for FPC. Until then, there's already support for zip in FPC and improving it to support zip64 makes sense to me...

[BigChimp]

It did find a whole bunch of empty files though, but the highest file name reported is 65499. It also reports that all the files in the archive have a CRC-32 of 00000000.

Yes, unzip shows CRC as 0 as well which is probably correct as the original data that is compressed is also 0 bytes.
The 65499 number of files is rather more worrying... as the other tools do not have problems with it, I think it's indeed a problem with squeeze as you surmised.