How to avoid storing conflicting results?

[FiftyTifty]

Now that I've moved on from asking basic "Why this don't work!?" questions, I've gotten to the concept phase. Title doesn't describe it too well, but, ehm, character limits.

Situation: I have a TMemoryStream[, that I search through for seven FF hex byte combinations, from 6-18 bytes in length. These are separators that precede a placed NPC's data, and since they vary in size, I need to get their index in the stream, without having invalid indexes.

...An example might explain it better. Say I have the following separator in the Stream:

Code: [Select]

FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF

The above separator is 18 bytes in length. If I search the whole TMemoryStream for 18 FF bytes, I'm A-OK. But I then have to search the stream for smaller sequences of FF bytes, the smallest being 6 FF bytes.

If I were to search for 6 FF bytes, the above stream would give me thirteen indexes, which are "invalid". The "sub-indexes" are invalid, as I need to get entire sequence of FF bytes, to know how many bytes I need to move forward to get to the wanted NPC data.

I thought two TStringLists would do the trick, one for the indices, the other for the length of the valid separators. But they're too rigid; if I use .Sorted := true;, I lose the 1:1 nature of the two TStringLists (can't do FirstList[iCounter]; SecondList[iCounter];).

So yeah, kinda stuck 'ere.


Github repo's here.

And here's the function that adds the indexes and separator lengths:

Code: [Select]

Function GetNPCData(msMAPFile: TMemoryStream; dwordPointer: DWord;
                    Out tstrlistSeparatorPos,
                    tstrlistSeparatorLength: TStringList): recordNPCArray;
var
  msSubStream: TMemoryStream;
  iCounter, byteArrayLength: Byte;
  iSubCounter: Word;
  iPosResult: integer;
  tpaCurrentArray: TPatternArray;
begin

  if Length(arrayNPCDataSeparators) = 0 then
     FillMetaArrays();

  msMAPFile.Seek(dwordPointer, 0);

  msSubStream := TMemoryStream.Create;
  msSubStream.CopyFrom(msMAPFile, 2000);


  for iCounter := 1 to 7 do begin //For each separator

    tpaCurrentArray := arrayNPCDataSeparators[iCounter];
    byteArrayLength := Length(tpaCurrentArray);

    for iSubCounter := 1 to (msSubStream.Size - (byteArrayLength - 1) ) do begin

      iPosResult := SearchTMemStream(msSubStream, tpaCurrentArray);

      if iPosResult > -1 then begin

        tstrlistSeparatorPos.Add(IntToStr(iPosResult));
        tstrlistSeparatorLength.Add(IntToStr(byteArrayLength));

      end; //End "If separator found in stream"

    end; //End "For bytes in stream"

  end; //End "For each separator"


  msSubStream.Free;

end;


[Handoko]

I didn't follow your posts from the beginning, I may not fully understand what you want. But try my code:

Code: Pascal  [Select][+][-]

1. procedure TForm1.Button1Click(Sender: TObject);
2. const
3.   SeparatorChar = Char($FF);
4.   SeparatorMinCount = 6;
5. var
6.   TestStream: TMemoryStream;
7.   TestString: String;
8.   i: Integer;
9.  
10.   FoundSeparatorCount: Integer = 0;
11.   FoundPosition: Integer = 0;
12.  
13. begin
14.  
15.   // Generate data for TestString
16.   TestString := 'testing';
17.   for i := 1 to 8 do
18.     TestString := TestString + SeparatorChar;
19.   TestString := TestString + 'and test again';
20.  
21.   // Copy TestString to TestStream
22.   TestStream := TMemoryStream.Create;
23.   for i := 1 to Length(TestString) do
24.     TestStream.WriteByte(Ord(TestString[i]));
25.  
26.   // Finding the Separator
27.   TestStream.Position := 0;
28.   for i := 1 to TestStream.Size do
29.     if TestStream.ReadByte = Ord(SeparatorChar) then
30.       Inc(FoundSeparatorCount)
31.     else
32.       if (FoundSeparatorCount > 0) then Break;
33.   if (FoundSeparatorCount >= SeparatorMinCount) then
34.     FoundPosition := TestStream.Position-1;
35.   TestStream.Position := 0;
36.  
37.   // Show the result
38.   if (FoundPosition <= 0) then
39.     ShowMessage('Cannot find separator in the TestStream.')
40.   else
41.     ShowMessage('Position after separator in TestStream = ' + IntToStr(FoundPosition));
42.  
43.   TestStream.Free;
44.  
45. end;

Note:
The index of first data in TMemoryStream is 0 (not 1).

[FiftyTifty]

I didn't follow your posts from the beginning, I may not fully understand what you want. But try my code:

-snip-

Note:
The index of first data in TMemoryStream is 0 (not 1).

To sum it up:

Got a bunch of binary files, with between 0-8 NPC data structures of varying positions, count and size. I need to parse a block of bytes, in order to pull the NPC structures from it. Each structure is separated by a series of FF FF Words, that varies between 3-9 Words (6-18 bytes) in length. There can be one 18x FF structure, and several 6x FF structures. There may also be FF bytes within the structure itself, that are used by the game engine.

Here's a .txt file with a bit of info: https://github.com/MajinCry/Xebra---Digimon-World/blob/master/Digimon%20Struc%20Dumps.txt

The numbers between the brackets are the number of FF bytes. The fewer FF bytes, the more data I've to skip at the end of the FF byte sequence. If I were to check a 6 byte FF separator against a structure that has a 14 byte structure, I'd get several different results. E.g:

FF FF FF FF FF FF FF FF FF FF FF FF FF FF

Would return the indexes 0, 1, 2, 3, 4, 5, 6, 7 and (?) 8. Then I check for 8 byte FF separators, 10 byte FF separators...Etc. A whole bunch of useless indexes are returned. I designed the arrays to check against to be used from longest -> shortest, but I'll still be given a bunch of redundant indexes and separator lengths.

Edit: You can look at one of the MAP files here: https://github.com/MajinCry/Digimon-World-MAP-File-Editor/blob/master/MAP1%20Files/MAYO03.MAP and it's pointer to the NPC data block is 00020b54

[Handoko]

Edit: You can look at one of the MAP files here: https://github.com/MajinCry/Digimon-World-MAP-File-Editor/blob/master/MAP1%20Files/MAYO03.MAP and it's pointer to the NPC data block is 00020b54

I didn't see any FF in the map files.

Have you tried my code? It runs correctly to find FF in TMemoryStream and able to provide the position of the next non FF character in TMemoryStream. If the FoundPosition = 0, it means no FF in the stream. You can configure minimal FF series to be consider to be a valid separator by setting the const: SeparatorMinCount.

[Handoko]

Lets talk it in different way.

Here, for example is the data in the TMemory Stream:

   10 10 FF FF FF 20 20 20 FF FF FF FF FF FF FF 30 FF FF 40 40
    ^     ^        ^        ^                    ^  ^     ^
pos 0     2        5        8                   15 16    18

By running the function, what kind of the result do you want to get?

[FiftyTifty]

I didn't see any FF in the map files.

Have you tried my code? It runs correctly to find FF in TMemoryStream and able to provide the position of the next non FF character in TMemoryStream. If the FoundPosition = 0, it means no FF in the stream. You can configure minimal FF series to be consider to be a valid separator by setting the const: SeparatorMinCount.

If you look at the header, and get the pointer (first byte determines where the pointer is), you'll promptly find a bunch of FF bytes. For example, in MAY003.MAP: https://snag.gy/o9Ewae.jpg

The first separator is always a dud, it has no NPC data. The first actual NPC structure's byte also has the name for the total number of NPC structures in the .MAP file. The maximum is 8, minimum is 0. In this .map file, there are 5 NPC structures.

Lets talk it in different way.

Here, for example is the data in the TMemory Stream:

   10 10 FF FF FF 20 20 20 FF FF FF FF FF FF FF 30 FF FF 40 40
    ^     ^        ^        ^                    ^  ^     ^
pos 0     2        5        8                   15 16    18

By running the function, what kind of the result do you want to get?

   10 10 FF FF FF 20 20 20 FF FF FF FF FF FF FF 30 FF FF 40 40
    ^     ^        ^        ^                    ^  ^     ^
pos 0     2        5        8                   15 16    18

The problem arises with larger FF byte separators being compared against smaller byte arrays. Taking a look at your script again, it seems fairly robust. First looked at it when it was 3am, so...Yeah.

I want to keep the code comparing byte arrays and such, rather than concocting strings, as a learning experience. Was advised in the previous thread, so why not.