Can I append data to the end of a compilied program?

[wpflum]

I'm trying to 'idiot proof' an down system backup data view program and I'm trying to make sure that the program always has the datafile it needs to run no matter what.  What the program and data is is a nightly copy of items and prices from our main database that the program searches to display the price of an item in the stores in case there is a failure of the main system or communication break between the stores and the main server.  Right now I have it working so that a group of tables can be read directly from a zipped file so only one file has to be updated and no one has to unzip it to use it but I'd like to go one better.  Since the program and data are only around 2 Meg I'd like to be able to add the data directly to the program so that only the 'program' has to be updated to be used.  This way if the end user needs to grab the file from another PC to use on theirs then all they have to do is copy the exe file and everything is there.  Now I don't want to recompile the program each night so what I was thinking if was creating the program and then appending on the zipped data file to the end of it.  I've seen some info on the web that seems to indicate that I might be able to do it but I'm short on details on how to do it from within Lazarus/Pascal.

Any ideas?

[karaba]

yes it is you write a simple utility that will open you exe as a tfilestream seek to the end of the file and start appending data from the compressed file.There now you have an exe with the required data at the end. Now you have 2 problems 1) you do not know where the exe ned and the data file begins and 2 there is no compression component based on the tstream that will be able to read and decode from any arbitrary position in a file downwards all assume that the compression info are at the top of the stream never allowing any other variation.

to solve 1) its easy just append a int64 at the end of the file with the size of the data file now you know that the data filestarts at filesize-8-Int64Value. To solve the second whell 2 ways
  1) copy the data in to a temp file and open that easy but a bit long timed depending on the size of the data file size.
2) same as before but instead of a temp file use a memorystream that is a bit faster but too much memory will be used.
3) write your own decompression that can avoid all the above problems and use it directly on the file.

There, that's the main theory behind it. some variations exist mainly on whether you will save the data files size or the exe file size and where you will save it eg at the of the file as a resource in the exe it self or as a const in the program the 3rd case requires that a patching mechanism should be written to find the const in the exe and change its value after the compilation. This is done once if the value kept is the size of the exe or every time the data file is appended if its the size of the data file.

If you need help with a specific issue of the above process fill free to ask again.

[BigChimp]

Appending data in a zip file is exactly what I did in my CheckRide remote control tool
https://bitbucket.org/reiniero/checkride

Have a look here
https://bitbucket.org/reiniero/checkride/src/1eaaf0fad9f7b87c5fb182b25f7de06735da886a/Source/clientcustomizer.pas?at=default
you could use the resourcezipper units etc as well...

[wpflum]

yes it is you write a simple utility that will open you exe as a tfilestream seek to the end of the file and start appending data from the compressed file.There now you have an exe with the required data at the end. Now you have 2 problems 1) you do not know where the exe ned and the data file begins and 2 there is no compression component based on the tstream that will be able to read and decode from any arbitrary position in a file downwards all assume that the compression info are at the top of the stream never allowing any other variation.

to solve 1) its easy just append a int64 at the end of the file with the size of the data file now you know that the data filestarts at filesize-8-Int64Value. To solve the second whell 2 ways
  1) copy the data in to a temp file and open that easy but a bit long timed depending on the size of the data file size.
2) same as before but instead of a temp file use a memorystream that is a bit faster but too much memory will be used.
3) write your own decompression that can avoid all the above problems and use it directly on the file.

There, that's the main theory behind it. some variations exist mainly on whether you will save the data files size or the exe file size and where you will save it eg at the of the file as a resource in the exe it self or as a const in the program the 3rd case requires that a patching mechanism should be written to find the const in the exe and change its value after the compilation. This is done once if the value kept is the size of the exe or every time the data file is appended if its the size of the data file.

If you need help with a specific issue of the above process fill free to ask again.

How does this sound to determine the start of the appended file.  Since I'm appending a zip file the header for each file in the zip is '50 4b 03 04'  so if I know how many files are in the archive then all I have to do is read backward from the end of the exe and count each set of headers until I read the last one and at that point I have the entire zip that I can then uncompress in a stream and use. 

In my head this sounds like it will work but the voices lie a lot so how does it sound to you?

[karaba]

can you guaranty that the sequence of bytes  '50 4b 03 04' will not be found inside the compressed data or inside the application exe at all? in other words are you sure that the count is the number of files and not something else? Why would you want to spend time searching for something that you can tell exactly where it is with 8 bytes more space? If you do not care about speed or do not have any speed problems (loading the application will be a lot slower if you search, count calculate and then load instead of jump and load).

[wpflum]

I was trying to minimize the changes to the legacy code that produces the zipped databases.  I guess the bit to add the length on to the end shouldn't be too hard to add into the script that would append the zip file to the program file.  I'll give that a shot.