Examples of parsing binary files from start to end. Do stuff with found values

[Gizmo]

Hi

I have Googled for a reply to this, but most of the answers seem to relate to either a) not using file streams but instead using the older methods of AssignFile, EOF, BlockRead, BlockWrite etc etc whereas I need to use streams (as per the guidance in the Lazarus notes when reading binary data - http://wiki.freepascal.org/File_Handling_In_Pascal..."For opening files for direct access TFileStream should be used.") or b) they relate to finding one value and then stopping.

In my case, I need to code an app that wil

  a) Read a binary data file from start to end
  b) On the way, if a magic file marker is found, I need it to then read in 1019 bytes (1024 less the 5 byte magic marker) and 'do stuff' with the values in between.
  c) Once the end of the file is reached, I need the found values to be output to a memo list.

I don't expect anyone to give me pseudo code for this but what I was hoping for was an example somewhere on the net where the same kind of thing is done using ideally Free Pascal and Lazarus, though a Delphi example would also suffice.

I posted the following question on StackOverflow (http://stackoverflow.com/questions/9944550/using-free-pascal-lazarus-to-parse-a-large-binary-file-for-specific-values) which shows the code I have so far, if that helps or if anyone can give me some guidance as to whether I'm on the right tracks.

Ted

[BigChimp]

Quick glance through the article; have a different take on it. It says you'll need TFileStream for direct access.... in the Streams section. It doesn't say you have to use TFIlestream for any binary file - i.e. the old way of accessing binary files will still work perfectly well.

BTW, I don't think the code on StackOverflow is complete; it would help if you post e.g. a compilable console program and a test file...

Regards,
BigChimp

[User137]

Filestream function and property names are very self explanatory. You have Size property telling how much data there is, and you can use read and write functions to manipulate the data. Different opening modes can be found with ctrl+clicks in Lazarus, from help or google. You mainly want to use fmCreate when saving files, and fmOpenRead when reading it.

As for data buffers, for example if you use dynamic array of byte named varData. Then filestream refers to actual data in varData[0], not just varData because that's the pointer to where actual arrays memory is.

[ludob]

It'll depend on the size of your file. If it is a few 100M then FileStream is not a good idea and you need to read the file in chunks.

The approach take in the sample on stackoverflow assumes the the file contains blocks of MFTRecordsStore and you test the first 5 bytes of each block for 'FILE0'. If this is a memory dump I assume 'FILE0' can be anywhere and not just at 1024 bytes boundaries.

A technique I have used in the past for single pass scanning of large files is reading with blockread a chunk of data (64k) in buffer and access this buffer as circular buffer: when you reach the end of the buffer, read in the next chunk and reset your buffer pointer. Put this in a GetNextByte routine. Then the only thing you need to do is a loop that calls GetNextByte while searching for the 'FILE0' sequence. If found, get the next 1019 bytes in the same way.

[User137]

It'll depend on the size of your file. If it is a few 100M then FileStream is not a good idea and you need to read the file in chunks.

Why not? You can seek forward with TFileStream too (or backwards, although it may be slow), reading a header and skipping the actual data between. This can be simply done with .Position property. With Filestream the data can be truly dynamic, and contain data that  may vary alot.

[eny]

Why not?

although it may be slow

ludob already described the way to handle big files efficiently: reading data in 64K blocks and process them in memory is much faster than reading 100M individual bytes from a file. This is exactly what TS requested.

[Gizmo]

Hey ludob....I don't suppose you'd be prepared to share (publically here or privately via PM?) a sample of your "technique that you have used in the past for single pass scanning of large files was  reading with blockread a chunk of data (64k) in buffer and access this buffer as circular buffer: when you reach the end of the buffer, read in the next chunk and reset your buffer pointer. Put this in a GetNextByte routine" code? If not, I will just happily take your advice and get working on it. It sounds very logical and sensible. I have also read page 526 of the Lazarus book that has another good example of using BlockRead etc.

[ludob]

No problem. I've cut out the relevant routines and attached. It was written for delphi 6. It was a tool that would take all print jobs in a print queue, translate printer escape sequences and spool it to another queue. The complete program is closed source.
function ReadChar(var C: Char): Boolean; is the routine that gets a char form the cyclic buffer and fills it when empty.
procedure SubstituteInJob(pJob: PJobData); is the main body that does the processing. For what you need this is probably overkill. It scans for a large number of signatures at the same time and replaces them with another block of data when found. The complexity of this routine is mainly linked to the fact that nothing can be written to the output as long as one or more signatures are partially matching. pJob: PJobData is the linked list of the sequences and their replacements.
procedure WriteChar(C: Char); is also using the cyclic buffer technique but to write the data.

[Gizmo]

Wowser ludob! That's some fancy work there! Thanks for sharing. I will try to digest it and and understand it though I think, as you say, it may be too complex overall for my needs. It certainly helps with my understanding though.

'All' I need, essentially, is some code to say "Read in some bytes [64Kb as stated, perhaps], see if the string FILE0 appears in it anywhere. If and when it does, read 1K bytes from wherever the hit appears and then stagger forward in certain [so far undefined increments] jumps and read byte ranges until you get to the end of that 1K block. Then carry on reading repeating the action until you get to the end of the file"

[User137]

although it may be slow

ludob already described the way to handle big files efficiently: reading data in 64K blocks and process them in memory is much faster than reading 100M individual bytes from a file. This is exactly what TS requested.

By slow i meant going the file backwards from end to beginning -.-  The speed when reading forward is very fast.

Well, if you can use optimization trick like that then ok. Assuming the data is aligned in 64kb chunks. You would lose some of the efficiency gain if the file information must be parsed from between the chunks.

And.. theoretically a class as general as TFileStream might be internally using optimizations which take most speed out of the harddisk.

[eny]

By slow i meant going the file backwards from end to beginning -.-  The speed when reading forward is very fast.

It depends on how the data is read.
When reading few bytes at a time, it becomes relatively slow.

Well, if you can use optimization trick like that then ok. Assuming the data is aligned in 64kb chunks.

The most benefit is when the chunks are a power of the cluster size.
Although I'm not sure about the implementation of the hardware killer code in the latest Window$ versions.

And.. theoretically a class as general as TFileStream might be internally using optimizations which take most speed out of the harddisk.

There is no optimization done in TFileStream. For Window$ it mappes directly to kernel functions that do file handling. I guess it's the same for other implementations.

Attached an example of an OO solution for the buffering: a class that buffers reading from streams (disks or otherwise). All complexity of the buffering is handled by a 'SequentialStreamReadBuffer' class. The main program can remain surprinsingly simple.
All required extensions (like reading words, ints, records etc.) can be implemented in a generic way in that class.
A demo is provided.

[ludob]

Using TFileStream for small files is very easy since you can load everything in memory, copy to a string and scan the string from begin to end. For huge files the amount of memory used can be simply too much for the system. The other downside is that you only start scanning when the file is read while the chunk approach scans while reading.
You can also use TFileStream to read small chunks but then you have exactly the same problem to solve, what if the sequence searched is on a boundary? You'll end up with a similar approach of what I have done. If you want you can change in my code

Code: [Select]

ReadFile(fHandleR, bufi, sizeof(bufi), bytesread, nil); with

Code: [Select]

bytesread:=FileStream.Read(bufi,sizeof(bufi));. It'll work also and very probably at the same speed. Random access is also possible with direct file access (seek).

Assuming the data is aligned in 64kb chunks. You would lose some of the efficiency gain if the file information must be parsed from between the chunks.

Look at the code. No alignment is assumed. For the code that searches the sequence, it reads character per character and isn't aware of the read boundaries.

And.. theoretically a class as general as TFileStream might be internally using optimizations which take most speed out of the harddisk.

TFileStream uses internally exactly the same functions (CreateFile, ReadFile). The equivalent FileStream.Read code above ends up with exactly the same Readfile command. When you start to do smart things like TStream.ReadByte you'll end up doing a ReadBuffer(b,1) which will be very slow

[User137]

When you start to do smart things like TStream.ReadByte you'll end up doing a ReadBuffer(b,1) which will be very slow

I guess that's true 

But lets be clear of something. This application opened my 400Mb AVI file. When checking task manager while application is running, it has only reserved the usual 1.8Mb or so. So 1 fact is that TFileStream does not load the whole file in memory for its operations. Not saying anyone claimed so.

Code: [Select]

  //TForm
  private
    F: TFileStream;

procedure TForm1.FormCreate(Sender: TObject);
begin
  F:=TFileStream.Create('E:\temp\video.avi', fmOpenRead);
end;

procedure TForm1.FormDestroy(Sender: TObject);
begin
  F.Free;
end;
Then if you imagine a huge 2Gb compressed file where you need to find a certain file from its records. Now assuming there is no index list or other optimizations at beginning of file:

Code: [Select]

type
  TFileHeader = packed record
    filename: string[30];
    size: cardinal;
    compression: TCompressionData; // Whatever there is...
  end;

  TFileData = PByte;This is a simple file structure with Header, Data, Header, Data, and so on...

Code: [Select]

var F: TFileStream; h: TFileHeader; data: TFileData;
...
  // search would do something like:
  repeat
    F.ReadBuffer(h, sizeof(h));
    if h.filename = searchString then begin
      data:=getmem(h.size);
      F.ReadBuffer(data, h.size); // Read data (this would read an entire file full of data be it 10 bytes or 100Mb)
      // Do something with data, decompress it etc.
     
      FreeMem(data); // Release data when its not needed
      break; // Break out from loop
    end; 
    F.Position:=F.Position+h.size; // Seek to next header
  until F.Position>=F.Size-sizeof(TFileHeader);

[ludob]

But lets be clear of something. This application opened my 400Mb AVI file. When checking task manager while application is running, it has only reserved the usual 1.8Mb or so.

I haven't said anything different. It gets nasty when you read the complete data into a string or a buffer which is what you typically would do with small files because it gives you a nice and easy linear buffer and you can use pos(subst,str) to quickly find the data. The very nice SequentialStreamReadBuffer posted by eny is using the exactly same chuncked reading technique with TFileStream.

This is a simple file structure with Header, Data, Header, Data, and so on...

I understood the header could start at any byte location. Here you assume it starts at a multiple of sizeof(TFileHeader). The file is a memory dump and hasn't a particular layout.