[Solved] ReadBuffer(AnsiString[1], 1500); - AnsiString is empty

[FiftyTifty]

Aaaaaand it's me again!

It seems that using ReadBuffer doesn't write to the passed AnsiString, that I'm using to hold the raw bytes (i.e, the actual hex bytes as they appear in the file through a hex viewer). But, ehm, shouldn't that just work?

Here's the function in question:

Code: [Select]

function GetWholeNPCData(msMAPFile: TMemoryStream; dwordPointer: DWord): AnsiString;
var
  bufferNPCData: AnsiString;
begin

  {$IFDEF DEBUG} ShowMessage('Putting entire block of NPC data into buffer!'); {$ENDIF}
  SetLength(bufferNPCData, 1500);

  msMAPFile.Seek(dwordPointer, 0);
  {$IFDEF DEBUG} ShowMessage('Used pointer to get to NPC data!'); {$ENDIF}
  msMAPFile.ReadBuffer(bufferNPCData[1], 1500); // <-- No bytes are written to the AnsiString?
  {$IFDEF DEBUG} ShowMessage('Put NPC data into buffer! First byte: '+bufferNPCData[1]); {$ENDIF}

  Result := bufferNPCData;
  {$IFDEF DEBUG} ShowMessage('Sent the buffer!'); {$ENDIF}

end;

It's not that the byte is even different. It's that there are no bytes at all in the AnsiString. Calling ShowMessage(bufferNPCData[1]) just returns a blank.

Repo's here.

[Handoko]

Try this:

Code: Pascal  [Select][+][-]

1. const
2.   DataSize = 30;
3.  
4. var
5.   Buffer: AnsiString;
6.  
7. begin
8. // ...
9.   SetLength(Buffer, DataSize);
10.   MyStream.ReadBuffer(Pointer(Buffer)^, DataSize);
11.   ShowMessage(Buffer);
12. // ...
13. end;

Alternatively, you may use WriteAnsiString and ReadAnsiString.

[wp]

It's not that the byte is even different. It's that there are no bytes at all in the AnsiString. Calling ShowMessage(bufferNPCData[1]) just returns a blank.

ShowMessage "thinks" that a string has terminated if a zero byte has been found. So, if bufferNPCData[1] is #0 then ShowMessage will display an empty string.

Code: Pascal  [Select][+][-]

1. procedure TForm1.Button1.Click(Sender: TObject);
2. begin
3.   ShowMessage(#0'Hello world');
4. end;

I looked at the MAP files in your repository. They are all binary files. There's a high chance that they contain zero byte and other non-printable characters. Therefore, accessing their contents on the basis of strings is conceptually doubtful, I would define the buffer as an array of bytes instead because this would inhibit me from checking the contents of the buffer by means of string procedures.

[FiftyTifty]

Try this:
-snip-
Alternatively, you may use WriteAnsiString and ReadAnsiString.

The pointer didn't work either. ShowMessage() still returns blanks, and the code to find the series of FF bytes only returns 0's.

ShowMessage "thinks" that a string has terminated if a zero byte has been found. So, if bufferNPCData[1] is #0 then ShowMessage will display an empty string.

I looked at the MAP files in your repository. They are all binary files. There's a high chance that they contain zero byte and other non-printable characters. Therefore, accessing their contents on the basis of strings is conceptually doubtful, I would define the buffer as an array of bytes instead because this would inhibit me from checking the contents of the buffer by means of string procedures.

But in order to find the structures and get to the data inside them, I need to:

Find occurrences of each set of FF bytes, starting with the longest (16 FF's) to the shortest (6 FF's)
Store the index of the found sets of FF bytes
Store the length of the found FF bytes
Depending on the number of the FF bytes, skip a certain number of bytes in the NPC structure
Push the next bunch of bytes to the GUI
When user saves the file, write the GUI entries back to their respective bytes


Far as I know, we are only able to search for a series of values with strings. Far as I know, there's no pos() function for arrays, strings are designed for this sorta thing.

Geh.

[Handoko]

The pointer didn't work either. ShowMessage() still returns blanks, and the code to find the series of FF bytes only returns 0's.

That's weird. It works on my computer. I just wrote my File > TStream > AnsiString module some days ago, tested with many random texts and all are working as expected. I guess your issue is what WP said, the data itself: #0 is not allowed.

Try to make sure your data do not have #0 nor any multibyte characters. Or perhaps you can sanitize the data by doing a for loop to remove the 'troublesome' characters.

[Thaddy]

The Windows version of ShowMessage probably uses the OS somewhere and therefor a PChar conversion. PChars can only contain #0 in one place: as string terminator.
Use buffers but not PChar buffers on binary data, but just pointer to or var buffer + length parameters.

[Handoko]

O yes, I forgot to mention I use Linux. My suggestion works on Linux but never tried on Windows.

[FiftyTifty]

That's weird. It works on my computer. I just wrote my File > TStream > AnsiString module some days ago, tested with many random texts and all are working as expected. I guess your issue is what WP said, the data itself: #0 is not allowed.

Try to make sure your data do not have #0 nor any multibyte characters. Or perhaps you can sanitize the data by doing a for loop to remove the 'troublesome' characters.

The data itself is just hex. I'm loading a .MAP file into memory, skipping a bunch of bytes, and trying to get to bundles of Byte and Word values. Removing any bytes will just make it a complete nightmare, as I'd have to do some insane amount of micromanaging indexes, removed byte positions, etc.

The Windows version of ShowMessage probably uses the OS somewhere and therefor a PChar conversion. PChars can only contain #0 in one place: as string terminator.
Use buffers but not PChar buffers on binary data, but just pointer to or var buffer + length parameters.

Isn't a buffer just another term for a variable that holds data read from a file, or from memory? And I'll be needing to chuck the raw bytes into text, in order to navigate through the data, no? There aren't any pointers (afaik) aside from the one in the header that takes me near to the beginning of the NPC data, so I have to parse it all manually.

[Handoko]

The data itself is just hex. I'm loading a .MAP file into memory, skipping a bunch of bytes, and trying to get to bundles of Byte and Word values. Removing any bytes will just make it a complete nightmare, as I'd have to do some insane amount of micromanaging indexes, removed byte positions, etc.

No, I don't suggest you to remove the actual data. I mean, you can make some sample data for testing purposes to find out where the problem is. Once it works, then you can improve your code further supports the 'troublesome' data. I tested my module with some random texts (alphabets, numbers, common punctuation marks), it works.

[Handoko]

Isn't a buffer just another term for a variable that holds data read from a file, or from memory? And I'll be needing to chuck the raw bytes into text, in order to navigate through the data, no? There aren't any pointers (afaik) aside from the one in the header that takes me near to the beginning of the NPC data, so I have to parse it all manually.

So you use AnsiString (as buffer) to store the data? I don't think it is okay. I use TMemoryStream to store data. And I write my own function to display it to the screen, I do it because I've already known what WP said: ShowMessage won't work correctly if there is a #0 in the middle of AnsiString.

[FiftyTifty]

So you use AnsiString (as buffer) to store the data? I don't think it is okay. I use TMemoryStream to store data. And I write my own function to display it to the screen, I do it because I've already known what WP said: ShowMessage won't work correctly if there is a #0 in the middle of AnsiString.

This is how the code's set up at the moment:

Load file into TMemoryStream
Get the pointer to NPC data from the header
Load next 1500 bytes into AnsiString
Search AnsiString for substrings that consist of certain combinations of FF characters
Store not-0 Pos() results in a TStringList

The reason I am using the AnsiString, is so I can actually search the small portion of data via Pos().

[wp]

there's no pos() function for arrays, strings are designed for this sorta thing.

No. Do you know CompareByte? http://www.freepascal.org/docs-html/rtl/system/comparebyte.html

[rvk]

So the problem is not that #0 causes problems, but maybe #ff causes the problem?

The files looks like this Thaddy, so I guess you know that it's no use to go with "string" for that

Yeah, array with comparebyte is definitely the way to go here.

[FiftyTifty]

No. Do you know CompareByte? http://www.freepascal.org/docs-html/rtl/system/comparebyte.html

Nope, didn't know about it. This is my first program that deals with bytes in unknown positions, of varying size. This function seems awfully...Low level. With Pos(), the functions returns the index of the substring's first occurrence. Nice and easy.

CompareByte(), on the other hand, would involve a loop that increments Seek(), a check to make sure there's at least 16 after the current position, adding an index to the TStringList if there's a match...Gruh. Is there no equivalent to Cheat Engine's AOBScan (Array Of Byte Scan)?

Just did a bit of Googling after typing that out, and found a user-made function that supposedly does that. Cleaned up the variable names and formatted it, as I want to figure out how it works.

Code: [Select]

function AOBScan(pbyteAOB: PByte; cardinalAOBSize: Cardinal;
                 pbyteBufferToScan: PByte; cardinalSizeOfBuffer: Cardinal): PByte;
var
  iCounter, iSecondCounter: Cardinal;
  bAOBExists: Boolean;

begin

  Result := nil;

  if cardinalAOBSize > cardinalSizeOfBuffer then Exit;
 
  if cardinalAOBSize = 0 then
  begin

    Result := pbyteBufferToScan;
    Exit;

  end;

  for iCounter := 0 to cardinalSizeOfBuffer - cardinalAOBSize do //Ensures that we don't scan beyond the allocated buffer
  begin

    if PByte(Cardinal(pbyteBufferToScan) + iCounter)^ = pbyteAOB^ then
    begin

      bAOBExists := True; //assume we ok
      for iSecondCounter := 1 to cardinalAOBSize - 1 do
      begin

        if PByte(Cardinal(pbyteBufferToScan) + iCounter + iSecondCounter)^ <> PByte(Cardinal(pbyteAOB) + iSecondCounter)^ then
        begin

          bAOBExists := False;
          Break;
        end;

      end;

      if bAOBExists then //after successful loop
      begin

        Result := PByte(Cardinal(pbyteBufferToScan) + iCounter);
        Exit;

      end;

    end;

  end;

end;

Looking at if PByte(Cardinal(pbyteBufferToScan) + iCounter)^ = pbyteAOB^,  Cardinal(pbyteBufferToScan) according to the wiki, a Cardinal gets us the memory address of the passed variable.

That line can be summed in ASM as one command: cmp [pbyteBufferToScan + iCounter] , [pbyteAOB]
So it compares the bytes of the pbyteAOB, to the current "position" we're checking in the buffer.

No idea what the second if statement does. Might be some sort of "Are we at the end of the buffer?" check?

[FiftyTifty]

The files looks like this Thaddy, so I guess you know that it's no use to go with "string" for that

Yeah, array with comparebyte is definitely the way to go here.

Gah. What thought reading into the AnsiString would do, is give me the raw bytes but as a string. Y'know, like this (sans the spaces):

Code: [Select]

ansistringBytes := 'D7 FC 05 F8 B7 07 4F 09 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 B3 09 DD FB D8 F9 C0 05 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0C 00 04 00 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 06 00 03 00 03 00 FF FF FF FF FF FF FF FF FF FF FF FF 02 00 00 00 02 00 01 00 FF FF FF FF FF FF FF FF FF FF FF FF';