[SOLVED] Bug in DynArrays

[Key-Real]

Hi,

I'm working on the PlayStation 1 Port.
I run into a issue with setlength()

It is a bigger setup, so I can't provide an easy example.

I'll try to simplify the situation:
got:

Code: Pascal  [Select][+][-]

1. Type
2. Vector2 = record
3.                        x, y : longint;
4.                 end;
5. v3dObject = record
6.                        ...
7.                        positions : array of Vector3;
8.                        uvs : array of Vector2;
9.                        ...
10.                      end;
11.  
12. function importObject: v3dObject;
13. begin
14. ....
15.    setLength(result.positions, 12345);
16. ....
17.    setLength(result.uvs, 12345);                <--   Here it crash
18. ....
19. end;
20.  

observations:
if i do in the main program

Code: Pascal  [Select][+][-]

1. ...
2. var
3.      a1 : array of dword;
4.      a2 : array of dword;
5. begin
6.      setLength(a1, 12345);
7.      setLength(a2, 12345);
8.      ....
9. end.
10.  

everything works, i do not do those 2 setLength, it crashes as described above.

further observations:

he goes into:

Code: Pascal  [Select][+][-]

1. procedure fpc_dynarray_setlength(var p : pointer;pti : pointer; dimcount : sizeint;dims : pdynarrayindex);[Public,Alias:'FPC_DYNARR_SETLENGTH']; compilerproc;
2. ....
3.      if assigned(p) then
4.         ....
5.      else
6.        begin
7.           oldlen:=0;
8.           {$IFDEF PS1}writeln('!!!!!!! AllocMem');{$ENDIF}
9.               newp:=AllocMem(size);                                                      <---  INTO THIS
10.           {$IFDEF PS1}writeln('!!!!!!! AllocMem done');{$ENDIF}
11.        end;
12.  

there now we are in heap.inc:

Code: Pascal  [Select][+][-]

1. function SysAllocMem(size: ptruint): pointer;
2. begin
3.   {$IFDEF PS1}writeln('!!!!!!! SysAllocMem()');{$ENDIF}
4.   result := SysGetMem(size);                                                     <--- INTO THIS
5.   if Assigned(result) then
6.     FillChar(result^, SysMemSize(result), 0);
7. end;
8.  

than:

Code: Pascal  [Select][+][-]

1. function SysGetMem(size : ptruint):pointer;
2. ...
3.   if (size < GrowHeapSize2 div 2) and (size <= HeapInc.MaxVarHeaderAndPayload - HeapInc.VarHeaderSize) then begin
4.     {$IFDEF PS1}writeln('!!!!!!! AllocVar()');{$ENDIF}
5.     result := ts^.AllocVar(size, false);                                          <--- INTO THIS
6.   end else begin
7.   {$IFDEF PS1}writeln('!!!!!!! AllocHuge()');{$ENDIF}
8.     result := ts^.AllocHuge(size);
9.   end;
10. ...
11.  

than:

Code: Pascal  [Select][+][-]

1. function HeapInc.ThreadState.AllocVar(size: SizeUint; isArena: boolean): pointer;
2. ...
3.   {$IFDEF PS1}writeln('!!!!!!! in AllocVar');{$ENDIF}
4.     binIndex := VarSizeToBinIndexUp(size + VarHeaderSize);
5.     size := BinIndexToVarSize(binIndex);
6.     fv := varFree.Find(binIndex);
7.    
8.  
9. <---    Here is an imporatant place A, see below    ---->
10.  
11. OK NOW HE FINDS THE fv, so later he goes into
12.    ...
13.     end else begin
14.       varFree.Remove(fv);                          <--- he goes into tihs
15.     end;
16.    ...
17.  

so:

Code: Pascal  [Select][+][-]

1. procedure HeapInc.VarFreeMap.Remove(c: pFreeVarChunk);
2. ...
3. begin
4.     prev := c^.prev;
5.     next := c^.next;
6.    
7.     if Assigned(next) then
8.       next^.prev := prev;
9.  
10.     if Assigned(prev) then begin
11.       prev^.next := next;                          <-- HE CRUSH HERE !!!!!!!!!
12.     end else
13.     ...
14.  

conclusion:
he find a not existing item in the linked list because he finds fv := varFree.Find(binIndex);

observation:
if i do in the Important Place A:

Code: Pascal  [Select][+][-]

1. {$IFDEF PS1}fv:= nil;{$ENDIF}
2.  

everything works.

so now:
IDEAS??????

[runewalsh]

I’d bet on memory corruption (double / incorrect FreeMem) on your part...

Or maybe varFree somehow ended up uninitialized? It it assumed to be zero-initialized, being a global (or thread-local) variable. Try FillChar(HeapInc.thisTs, sizeof(HeapInc.thisTs), 0) somewhere in the initialization routines (InitHeap / InitHeapThread / RelocateHeap, IDK how PS1 works).

[Key-Real]

the fillchar() unfortunly changes nothing.

i implemented, in sysheap.inc on my target:

function SysOSAlloc (size: ptruint): pointer;
procedure SysOSFree(p: pointer; size: ptruint);


what else i have to implement?

[runewalsh]

Most likely the memory corruption, the freelist is easily corrupted with incorrect overwrites.

Example:

Code: Pascal  [Select][+][-]

1. var
2.         a, b: array of int32;
3. begin
4.         SetLength(a, 200);
5.         FillDWord(pointer(a)^, length(a) * 2 { writes 200 items past the end }, $ABABABAB);
6.         SetLength(b, 200);
7. end.
8.  

It crashes the EXACTLY SAME way as your code, and due to the nature of the corruption, heaptrc won't help you (without scanning the entire heap for corruptions every time). :\

[Key-Real]

so the positions array overwrites the uvs array?

because first setlength(positions,123);
and then uvs follow.
?

can't be, i comment out all the writes to positions array

[runewalsh]

Something is overwriting something, perhaps in an unrelated place.
Another example that crashes (almost) the same way (but only without heaptrc, for... reasons):

Code: Pascal  [Select][+][-]

1. var
2.         a, b: pByte;
3. begin
4.         a := GetMem(1000);
5.         b := GetMem(1000);
6.         FreeMem(b);
7.         b[0] := 1; // corruption
8.         b := GetMem(1000); // crash
9. end.

[Thaddy]

Simple observation: you do not know jack dung about alignment. You just assume memory is contiguous.
Sequential <> contiguous.
Back to school.
The use of getmem() in the hands of people like you makes me shiver. 
Horror!! 
You don't have a license to use getmem(). 
Period.

[PascalDragon]

The use of getmem() in the hands of people like you makes me shiver. 
Horror!! 
You don't have a license to use getmem(). 
Period.

Key-Real is trying to debug an issue with dynamic arrays where they stepped down into where SysGetMem is used. So they aren't using GetMem directly, so please keep your opinion to yourself. And runewalsh is only demonstrating a possible cause. And he definitely is allowed to demonstrate using GetMem considering that he implemented the current heap implementation.

[Key-Real]

Something is overwriting something, perhaps in an unrelated place.
Another example that crashes (almost) the same way (but only without heaptrc, for... reasons):

Code: Pascal  [Select][+][-]

1. var
2.         a, b: pByte;
3. begin
4.         a := GetMem(1000);
5.         b := GetMem(1000);
6.         FreeMem(b);
7.         b[0] := 1; // corruption
8.         b := GetMem(1000); // crash
9. end.

It was indeed memory corruption as You say. Thx a lot.