* * *

Author Topic: Working with binary files and searching for hex values  (Read 2565 times)

DumDum

  • New member
  • *
  • Posts: 31
Working with binary files and searching for hex values
« on: January 02, 2018, 09:52:07 am »
Hi,

New here so please be gentle.  I am looking for a way to make simple program which can search through a binary file and find specific hex values. Also, would like the program to spit out at what address the search input was found.

If someone could point me in the direction of some research or things to try that would be awesome.

Thanks!

howardpc

  • Hero Member
  • *****
  • Posts: 2644
Re: Working with binary files and searching for hex values
« Reply #1 on: January 02, 2018, 10:01:13 am »
What size hex values are you seeking (byte, word, longword, something else)?

DumDum

  • New member
  • *
  • Posts: 31
Re: Working with binary files and searching for hex values
« Reply #2 on: January 02, 2018, 10:19:50 am »
For an example I am looking at hex in this format: FF FA 2F FD 88 6D. What category would this fall under? I'm sorry for my lack of terminology.

Thaddy

  • Hero Member
  • *****
  • Posts: 5798
Re: Working with binary files and searching for hex values
« Reply #3 on: January 02, 2018, 10:47:14 am »
Something like this?:
Code: Pascal  [Select]
  1. program untitled;
  2. {$ifdef fpc}{$mode delphi}{$H+}{$I-}{$endif}
  3. uses sysutils;
  4. var
  5.   a:AnsiString = 'whatever you want';
  6.   b:TCharArray;
  7.   i:integer;
  8. begin
  9.   b:=a.ToCharArray;
  10.   for i := Low(b) to High(b) do write(byte(b[i]).ToHexString:3);
  11. end.

In a GUI app, use writestr.
« Last Edit: January 02, 2018, 10:48:57 am by Thaddy »
recommends {$macro on}{$define Silly:=ObjFpc}

Eugene Loza

  • Hero Member
  • *****
  • Posts: 561
    • My "almost daily" development blog
Re: Working with binary files and searching for hex values
« Reply #4 on: January 02, 2018, 10:53:02 am »
I'd use a simple File of Byte or TFileStream to read a buffer and then search for a specific record there. Thou, I strongly believe there should be much more optimal solution. Anyway, you can easily get started with something like this (pseudocode, it might (and should) contain errors):
Code: Pascal  [Select]
  1. var
  2.   F: File of Byte;
  3.   b: byte;
  4.   Index: integer;
  5.   A: array [0..4] of byte;
  6.   i: integer;
  7.   flg: boolean;
  8. begin
  9.   A[0] := $FF;
  10.   A[1] := $FA;
  11.   A[2] := $2F;
  12.   A[3] := $88;
  13.   A[4] := $6D;
  14.   AssignFile(F,'myfile.bin');
  15.   Reset(F);
  16.   index := 0;
  17.   repeat
  18.     Read(F, b);
  19.     if b = a[0] then
  20.     begin
  21.        flg := true;
  22.        i := 1;
  23.        while (b = a[i]) and (i <=4) do
  24.        begin
  25.          Read(F, b);
  26.          inc(i);
  27.          if b<>a[i] then begin
  28.            Break;           //<--------------- there's a bug in here, as if b = a[0] or in case the a[i] sequence is repititive (i.e. $FF $FA $FF $FB) you'll miss this start of the sequence, but it shouldn't be too complex to fix (just save values of b in a buffer, and revert to "last checked for sequence start")
  29.            flg := false;
  30.        end;
  31.        if flg then WriteLn('HORRAY!!! I've found it at ', Index);
  32.    end;
  33.    inc(Index);
  34.  until EOF(F);
  35.  CloseFile(F);
  36. end;
  37.  
Lazarus 1.9 + FPC 3.1.1 Debian Jessie 64 bit.

My Free and Open Source games in Lazarus/FreePascal/CastleGameEngine:
https://decoherence.itch.io/
(and some ancient games in Turbo Pascal too)
Sources are here: https://github.com/eugeneloza?tab=repositories

Thaddy

  • Hero Member
  • *****
  • Posts: 5798
Re: Working with binary files and searching for hex values
« Reply #5 on: January 02, 2018, 11:06:07 am »
Way too complex..
Code: Pascal  [Select]
  1. program untitled;
  2. {$mode objfpc}
  3. uses classes, sysutils;
  4. // all you need:
  5. procedure DumpAsHex(const s:TStream;ItemsPerLine:integer =10);
  6. var i:integer;
  7. begin
  8.   for i:=0 to pred(s.Size) do
  9.   begin
  10.     if (i<>0) and (i mod ItemsPerline  = 0)   then writeln;
  11.     write(s.readbyte.ToHexString:3);
  12.   end;
  13. end;
  14. var
  15.   stream:TStream; //Whatever stream...
  16. begin
  17.   Stream:= TSTringStream.Create('Whatever you want');
  18.   DumpAsHex(Stream);
  19.   Stream.Free;
  20. end.
« Last Edit: January 02, 2018, 11:14:19 am by Thaddy »
recommends {$macro on}{$define Silly:=ObjFpc}

howardpc

  • Hero Member
  • *****
  • Posts: 2644
Re: Working with binary files and searching for hex values
« Reply #6 on: January 02, 2018, 11:17:51 am »
A slow, brute-force search could be something like this:

Code: Pascal  [Select]
  1. program SearchBinary;
  2.  
  3. {$Mode objfpc}{$H+}
  4. {$IfDef windows}{$AppType console}{$EndIf}
  5.  
  6. uses sysutils, Classes;
  7.  
  8. type
  9.   T6ByteArray = array[0..5] of Byte;
  10.  
  11. const
  12.   ArrayToFind: T6ByteArray = (($FF), ($FA), ($2F), ($FD), ($88), ($6D));
  13.   FileNameToSearch: String = 'test.bin';
  14.  
  15. var
  16.   fs: TFileStream;
  17.   fLength, p: Int64;
  18.   b: Byte;
  19.   found: Boolean = False;
  20.  
  21.   function Located: Boolean;
  22.   var
  23.     x: Integer;
  24.  
  25.       function ByteFound: Boolean;
  26.       begin
  27.         b:=fs.ReadByte;
  28.         Exit(b = ArrayToFind[x]);
  29.       end;
  30.  
  31.   begin
  32.     for x:=0 to 5 do
  33.       if not ByteFound then
  34.         Exit(False);
  35.     Result:=True;
  36.   end;
  37.  
  38. begin
  39.   if not FileExists(FileNameToSearch) then begin
  40.     WriteLn('"',FileNameToSearch,'" could not be found');
  41.     WriteLn('Press [Enter] to quit');
  42.     ReadLn;
  43.     halt;
  44.   end
  45.   else begin
  46.     fs:=TFileStream.Create(FileNameToSearch, fmOpenRead);
  47.     fLength:=fs.Size;
  48.     try
  49.       p:=0;
  50.       while (p <= fLength - SizeOf(T6ByteArray)) do begin
  51.         fs.Position:=p;
  52.         if Located then begin
  53.           found:=True;
  54.           Break;
  55.         end
  56.         else begin
  57.           Inc(p);
  58.           if (p mod 1000) = 0 then
  59.             Write('.');
  60.         end;
  61.       end;
  62.     finally
  63.       fs.Free;
  64.     end;
  65.     if found then
  66.       WriteLn('Byte sequence found at position ',p)
  67.     else WriteLn('Byte sequence not found');
  68.   end;
  69. end.

Thaddy

  • Hero Member
  • *****
  • Posts: 5798
Re: Working with binary files and searching for hex values
« Reply #7 on: January 02, 2018, 01:03:40 pm »
I have no time to test this, but:
Code: Pascal  [Select]
  1. program untitled;
  2. {$if FPC_FULLVERSION < 30101}{$ERROR this demo needs version 3.1.1}{$endif}
  3. {$mode delphi}{$macro on}
  4. uses classes, sysutils, strutils;
  5. var
  6.   Stream:TmemoryStream;
  7.   af:Array of byte;
  8.   S,f:AnsiString;
  9.   R:SizeIntArray;
  10.   i:integer;
  11. begin
  12.   af:=[$FF, $FA, $2F, $FD, $88, $6D];
  13.   f:=AnsiString(af);
  14.   If FileExists(Paramstr(1)) then
  15.   begin
  16.     Stream:= TMemorystream.Create;
  17.     try
  18.       Stream.LoadFromFile(Paramstr(1));
  19.       SetString(s, Stream.Memory, Stream.Size);
  20.       FindMatchesBoyerMooreCaseSensitive(s,f,r,true);
  21.       if length(r) > 0 then for i := Low(r) to High(r) do writeln(r[i]) else writeln('no matches found');
  22.     finally
  23.       Stream.free;
  24.     end
  25.   end else writeln('No such File');
  26. end.
May work too....?
« Last Edit: January 02, 2018, 01:11:09 pm by Thaddy »
recommends {$macro on}{$define Silly:=ObjFpc}

DumDum

  • New member
  • *
  • Posts: 31
Re: Working with binary files and searching for hex values
« Reply #8 on: January 02, 2018, 05:23:13 pm »
Thank you all for your input!

Thaddy, I tried your code but to no avail.

Howard, yours worked without a hitch!
Is there a way to convert the outputted binary address to a hex address instead when pointing to the location of the succeeded search?

Xor-el

  • Jr. Member
  • **
  • Posts: 97
Re: Working with binary files and searching for hex values
« Reply #9 on: January 02, 2018, 05:28:32 pm »
Guess am late to this discussion but here is a Unit I wrote sometime back to do exactly this

https://github.com/Xor-el/PatternFinderPascal

Just follow the samples in the unit tests.

Regards..

ASerge

  • Hero Member
  • *****
  • Posts: 704
Re: Working with binary files and searching for hex values
« Reply #10 on: January 02, 2018, 06:13:36 pm »
I suggest using Shift Or Algorithm.
Advantage:
1. When reading does not come back (in contrast @howardpc algorithm).
2. Uses little memory and can work with files that do not fit in memory (in contrast @Thaddy algorithm).
3. Search can be case insensitive.
3. Fast.

Disadvantage:
1. Use some memory for prepare.
2. Limit search mask length for SizeOf(PtrInt)*8 bytes (32 or 64).

Code: Pascal  [Select]
  1. unit uBitShiftSearch;
  2.  
  3. {$mode objfpc}{$H+}
  4.  
  5. interface
  6.  
  7. uses Classes;
  8.  
  9. // BitShiftPrepare can be called up several times to customize the search mask.
  10. // For additional search cases, set ClearLastSearch to False
  11. procedure BitShiftPrepare(const What; LengthWhat: Integer; ClearLastSearch: Boolean = True);
  12.  
  13. // If found, the current position of the stream indicates the beginning
  14. // of the fragment. If not, then at the end of the stream
  15. // To find the next fragment, increase the stream position by the fragment
  16. // length and call the function again
  17. function BitShiftFound(Where: TStream): Boolean;
  18.  
  19. implementation
  20.  
  21. var
  22.   GWhatBits: array [Byte] of IntPtr;
  23.   GWhatHighIndex: Integer;
  24.  
  25. procedure BitShiftPrepare(const What; LengthWhat: Integer; ClearLastSearch: Boolean = True);
  26. var
  27.   iBit: Integer;
  28.   BitData: IntPtr;
  29. begin
  30.   if LengthWhat > SizeOf(IntPtr) * 8 then
  31.     LengthWhat := SizeOf(IntPtr) * 8; // or raise error
  32.   GWhatHighIndex := LengthWhat - 1;
  33.   if ClearLastSearch then
  34.     FillChar(GWhatBits, SizeOf(GWhatBits), 0);
  35.   BitData := 1;
  36.   iBit := 0;
  37.   while iBit <= GWhatHighIndex do
  38.   begin
  39.     GWhatBits[PByte(@What)[iBit]] := GWhatBits[PByte(@What)[iBit]] or BitData;
  40.     BitData := BitData shl 1;
  41.     Inc(iBit);
  42.   end;
  43. end;
  44.  
  45. function BitShiftFound(Where: TStream): Boolean;
  46. var
  47.   iBufPos, BufLen: Integer;
  48.   iWherePos, Size: Int64;
  49.   Mask, Vect: IntPtr;
  50.   Buffer: array [Byte] of Byte;
  51. begin
  52.   Mask := 1 shl GWhatHighIndex;
  53.   Vect := 0;
  54.   Size := Where.Size;
  55.   iWherePos := Where.Position;
  56.   while iWherePos < Size do
  57.   begin
  58.     BufLen := Where.Read(Buffer, SizeOf(Buffer));
  59.     iBufPos := Low(Buffer);
  60.     while (iBufPos < BufLen) and ((Vect and Mask) = 0) do
  61.     begin
  62.       Vect := (Vect shl 1) or 1;
  63.       Vect := Vect and GWhatBits[Buffer[iBufPos]];
  64.       Inc(iBufPos);
  65.     end;
  66.     if (Vect and Mask) <> 0 then
  67.     begin
  68.       Where.Position := iWherePos + iBufPos - GWhatHighIndex - 1;
  69.       Exit(True);
  70.     end;
  71.     Inc(iWherePos, BufLen);
  72.   end;
  73.   Result := False;
  74. end;

and test

Code: Pascal  [Select]
  1. procedure TForm1.Button1Click(Sender: TObject);
  2. const
  3.   ByteToSearch: AnsiString = 'loadlibraryw'; // Made deliberately in lowercase
  4. var
  5.   Stream: TFileStream;
  6. begin
  7.   BitShiftPrepare(ByteToSearch[1], Length(ByteToSearch));
  8.   BitShiftPrepare(UpperCase(ByteToSearch)[1], Length(ByteToSearch), False);
  9.   Stream := TFileStream.Create('c:\windows\system32\kernel32.dll', fmOpenRead or fmShareDenyNone);
  10.   try
  11.     Stream.Position := 100; // For test only
  12.     if BitShiftFound(Stream) then
  13.       Label1.Caption := Format('Found as position %d($%0:x)', [Stream.Position])
  14.     else
  15.       Label1.Caption := 'Not found';
  16.   finally
  17.     Stream.Free;
  18.   end;
  19. end;

howardpc

  • Hero Member
  • *****
  • Posts: 2644
Re: Working with binary files and searching for hex values
« Reply #11 on: January 02, 2018, 06:22:01 pm »
Is there a way to convert the outputted binary address to a hex address instead when pointing to the location of the succeeded search?

You can use the HexStr function to convert a decimal int64 position to a hex value (it is a byte index from the start of the file, not an "address" in the usual meaning of that term).
In the code I gave simply replace the penultimate WriteLn with this:
Code: Pascal  [Select]
  1. WriteLn('Byte sequence found at position $',hexStr(p, 8))

Or you could use the Format function for display, as ASerge shows you.

DumDum

  • New member
  • *
  • Posts: 31
Re: Working with binary files and searching for hex values
« Reply #12 on: January 02, 2018, 06:27:17 pm »
Thank you all so much for your replies! I am learning a lot here.

If I were to search for this byte array however would like to omit one of the variables as a null.  So basically as an example.

ex.  Searching for 9A 4E 0B D7 40
      I want to find 9A 43 ** D7 40
     
Say if in this particular file 0B didn't exist but the rest did.
How would I go about searching for the values, can a null be used here?

howardpc

  • Hero Member
  • *****
  • Posts: 2644
Re: Working with binary files and searching for hex values
« Reply #13 on: January 02, 2018, 06:34:54 pm »
Thank you all so much for your replies! I am learning a lot here.

If I were to search for this byte array however would like to omit one of the variables as a null.  So basically as an example.

ex.  Searching for 9A 4E 0B D7 40
      I want to find 9A 43 ** D7 40
     
Say if in this particular file 0B didn't exist but the rest did.
How would I go about searching for the values, can a null be used here?

Your question is not clear.
Do you want to look for the 4-byte sequence 9A 4E D7 40
or the 5-byte sequence                                 9A 4E 00 D7 40
or the entire range of 5-byte sequences         9A 4E ZZ D7 40  where ZZ might have any value from $00 to $FF?

DumDum

  • New member
  • *
  • Posts: 31
Re: Working with binary files and searching for hex values
« Reply #14 on: January 02, 2018, 06:49:56 pm »
Sorry about that Howard.

The second option.

 

Recent

Get Lazarus at SourceForge.net. Fast, secure and Free Open Source software downloads Open Hub project report for Lazarus