Recent

Author Topic: assignments tracking  (Read 4066 times)

mm7

  • Full Member
  • ***
  • Posts: 193
  • PDP-11 RSX Pascal, Turbo Pascal, Delphi, Lazarus
assignments tracking
« on: February 26, 2020, 03:27:14 pm »
Hi
I am maintaining quite big and quite convoluted program.
I want to track assignments of a specific class
because it is very hard to find a root cause of exception if an object  was created in one place, used in another (assigned to multiple vars, fields of other objects, arrays, lists...), deleted in third, and then used somewhere else.
So I need something like ref tracking, but not just ref counter, rather an assignment function with possibility of logging callstack to file.

My first idea was to overload := operator. But it is not allowed for equal types.

Second idea if I can use interface ref count TInterfacedObject._AddRef function.

I tried example from here https://wiki.freepascal.org/How_To_Use_Interfaces
But TInterfacedObject._AddRef was never called (in Linux).
Does it work only for COM/Corba? What should call it?

Any other ideas how to track all assignments of a type (class)?

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9792
  • Debugger - SynEdit - and more
    • wiki
Re: assignments tracking
« Reply #1 on: February 26, 2020, 04:25:20 pm »
Have a look at TRefCountedObject.

You do have to manually call Addref/ReleaseRef.

But if you define WITH_REFCOUNT_DEBUG, then you can pass in a name for each ref.

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9792
  • Debugger - SynEdit - and more
    • wiki
Re: assignments tracking
« Reply #2 on: February 26, 2020, 04:43:24 pm »
I still need to commit this, but here is the code to dump leaks;

Code: Pascal  [Select][+][-]
  1. {$IFDEF WITH_REFCOUNT_DEBUG}
  2. uses LazLoggerBase;
  3. {$IFDEF WITH_REFCOUNT_LEAK_DEBUG}
  4. var FUnfreedRefObjList: TRefCountedObject = nil;
  5.  
  6. procedure DumpRefCountedObjects;
  7. var
  8.   o: TRefCountedObject;
  9. begin
  10.   o := FUnfreedRefObjList;
  11.   while o <> nil do begin
  12.     DebugLn([DbgSName(o), ' Cnt:', o.FRefCount]);
  13.     if o.FDebugList <> nil then
  14.       DebugLn('  ', o.FDebugList.CommaText);
  15.     o := o.FDebugNext;
  16.   end;
  17. end;
  18. {$ENDIF}
  19. {$ENDIF}
  20.  

mm7

  • Full Member
  • ***
  • Posts: 193
  • PDP-11 RSX Pascal, Turbo Pascal, Delphi, Lazarus
Re: assignments tracking
« Reply #3 on: February 26, 2020, 06:59:29 pm »
Thank you Martin,
I'll try TRefCountedObject

mm7

  • Full Member
  • ***
  • Posts: 193
  • PDP-11 RSX Pascal, Turbo Pascal, Delphi, Lazarus
Re: assignments tracking
« Reply #4 on: February 26, 2020, 09:15:21 pm »
Hi Martin, I tried.
Code: Pascal  [Select][+][-]
  1. {$mode objfpc} {$H+}
  2.  
  3. program assignments_tracking;
  4.  
  5. uses LazClasses;
  6.  
  7. type MyTRefCountedObject = class(TRefCountedObject)
  8. end;
  9.  
  10. var O1,O2,O3: MyTRefCountedObject;
  11. begin
  12.   O1 := MyTRefCountedObject.Create;
  13.   O2:=O1;
  14.   O3:=O1;
  15.   Writeln('O1.RefCount:',O1.RefCount);
  16.   O1.Free;
  17. end.
  18.  
Return
Code: Pascal  [Select][+][-]
  1. RefCount:0
  2.  

I expected O1.RefCount should be 3. Because the object is assigned to 3 vars.
Am I missing something?

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9792
  • Debugger - SynEdit - and more
    • wiki
Re: assignments tracking
« Reply #5 on: February 26, 2020, 09:58:40 pm »
You do have to manually call Addref/ReleaseRef.

The compiler does not do this for you.
Code: Pascal  [Select][+][-]
  1.   O1 := MyTRefCountedObject.Create;
  2.   O1.AddReference({$IFDEF WITH_REFCOUNT_DEBUG}'Just created', self  {$ENDIF});
  3.   O2:=O1;
  4.   O2.AddReference();

It is just a framework.
It has some very limited support for weak refs. But I would not go to far into using them. (overloading := would not break circle refs either)

ReleaseRef must have matching params !!! That ensures your AddRef/ReleaseRef are correctly paired.

Search components/fpdebug for WITH_REFCOUNT_DEBUG, you get plenty of examples.

As you already have a codebase, it will unfortunately mean a lot of work to put it all together.



If you are just looking to find a single (or a few) accesses to no longer existing objects:

On linux: Use valgring (note, this needs "uses cmem").
This gives you where unallocated mem was read or written. Where it was previously allocated, and where it was freed.
It is worth getting a linux VM, just for having valgrind.


Otherwise:

Make sure you use heaptrc. And set the environment HEAPTRC="keepreleased"
That means memory will not get re-used (or at least not soon)

Heaptrc will (with delay) tell you if your app wrote (only wrote) to unused memory.
IIRC it will tell you where the mem got allocated.
From there it is still a lot of work, and a lot of try and error.

There are two features of the debugger I often use in cases like this.

Debugger History: Setting a breakpoint in the destructor (assuming I know the class that is involved). Record the stack (and sometimes some watches, like the address of self), and autocontinue the app.
That way I do not have to keep pressing F9. But I can later find a list of all calls in the History window.
Then when the memory gets accessed (assuming it causes a crash, right then), I can get the address of the object that was incorrectly accessed, and go through the list of destroy calls.

The other is, if I know the exact instance that gets accessed after destroy, and which field is accessed, but not when. Break in the destroy, and set a watchpoint on the field. The debugger will stop if the memory is accessed (read or write). But you must already know which instance causes the issue.
« Last Edit: February 26, 2020, 10:02:20 pm by Martin_fr »

mm7

  • Full Member
  • ***
  • Posts: 193
  • PDP-11 RSX Pascal, Turbo Pascal, Delphi, Lazarus
Re: assignments tracking
« Reply #6 on: February 26, 2020, 10:51:10 pm »
Martin, MANY THANKS for so detailed explanation!
I see that unfortunately no magic wand exist. :(

It looks like Valgrind is the way I should start with. I am on Linux for many years.



mm7

  • Full Member
  • ***
  • Posts: 193
  • PDP-11 RSX Pascal, Turbo Pascal, Delphi, Lazarus
Re: assignments tracking
« Reply #7 on: February 27, 2020, 03:05:15 pm »
Martin,
many thanks again!

Valgrind is is the "magic wand"! Previously I used for profiling only.

I have not found integration with it in Lazarus. May be some plugin can be installed. Currently I run it in separate shell, read file names and rows and manually navigate in Lazarus.
It works, but is not very convenient.

Secondly, sometimes it finds things that look like false positives to me. I.e.
Code: Pascal  [Select][+][-]
  1. ==12776== Conditional jump or move depends on uninitialised value(s)
  2. ==12776==    at 0x45D404: SYSTEM_$$_INDEXBYTE$formal$INT64$BYTE$$INT64 (in /home/mark/MyProjects/freeship-plus-in-lazarus/FreeShip)
  3. ==12776==    by 0x6E494B: IMGLIST$_$TCUSTOMIMAGELISTRESOLUTION_$__$$_READDATA$TSTREAM (imglist.inc:907)
  4. ==12776==    by 0x6E9634: IMGLIST$_$TCUSTOMIMAGELIST_$__$$_READDATA$TSTREAM (imglist.inc:2178)
  5. ==12776==    by 0x695CC5: CLASSES$_$TREADER_$__$$_DEFINEBINARYPROPERTY$ANSISTRING$TSTREAMPROC$TSTREAMPROC$BOOLEAN (in /home/mark/MyProjects/freeship-plus-in-lazarus/FreeShip)
  6. ==12776==    by 0x6E7886: IMGLIST$_$TCUSTOMIMAGELIST_$__$$_DEFINEPROPERTIES$TFILER (imglist.inc:1622)
  7. ==12776==    by 0x697A4E: CLASSES$_$TREADER_$__$$_READPROPERTY$TPERSISTENT (in /home/mark/MyProjects/freeship-plus-in-lazarus/FreeShip)
  8. ==12776==    by 0x696DB8: CLASSES$_$TREADER_$__$$_READDATA$TCOMPONENT (in /home/mark/MyProjects/freeship-plus-in-lazarus/FreeShip)
  9. ==12776==    by 0x696E7B: CLASSES$_$TREADER_$__$$_READDATA$TCOMPONENT (in /home/mark/MyProjects/freeship-plus-in-lazarus/FreeShip)
  10. ==12776==    by 0x698874: CLASSES$_$TREADER_$__$$_READROOTCOMPONENT$TCOMPONENT$$TCOMPONENT (in /home/mark/MyProjects/freeship-plus-in-lazarus/FreeShip)
  11. ==12776==    by 0x70065A: LRESOURCES_$$_INITLAZRESOURCECOMPONENT$TCOMPONENT$TCLASS$$BOOLEAN (lresources.pp:3183)
  12. ==12776==    by 0x6F8710: LRESOURCES_$$_INITRESOURCECOMPONENT$TCOMPONENT$TCLASS$$BOOLEAN (lresources.pp:804)
  13. ==12776==    by 0x4BD695: FORMS$_$TCUSTOMFORM_$__$$_PROCESSRESOURCE (customform.inc:2069)
  14.  

It is in Lazarus standard code. It loads an image from a resource when it loads main form at start. I've checked var Signature in procedure TCustomImageListResolution.ReadData(AStream: TStream);  approximately in imglist.inc: 907, var Signature is always populated.
I've even checked output of AStream.Read(Signature, SizeOf(Signature));
It is always 2. Why Valgrind is not happy with "if Signature = SIG_LAZ4" - "Conditional jump or move depends on uninitialised value(s)"

mm7

  • Full Member
  • ***
  • Posts: 193
  • PDP-11 RSX Pascal, Turbo Pascal, Delphi, Lazarus
Re: assignments tracking
« Reply #8 on: February 28, 2020, 03:28:41 pm »
I've found the issue. Compiled RTL with debug and Valgrind. And figured out that there is buffer overrun on zerobased ansistring comparison.

In rtl/inc/astrings.inc:587
Code: Pascal  [Select][+][-]
  1. Function fpc_CharArray_To_AnsiStr(const arr: array of ansichar; {$ifdef FPC_HAS_CPSTRING}cp : TSystemCodePage;{$endif FPC_HAS_CPSTRING}zerobased: boolean = true): RawByteString; compilerproc;
  2. var
  3.   i  : SizeInt;
  4. {$ifndef FPC_HAS_CPSTRING}
  5.   cp : TSystemCodePage;
  6. {$endif FPC_HAS_CPSTRING}
  7. begin
  8.   if (zerobased) then
  9.     begin
  10.       if (arr[0]=#0) Then
  11.         i := 0
  12.       else
  13.       begin
  14.         i:=IndexChar(arr,high(arr)+1,#0);
  15.         if i = -1 then
  16.           i := high(arr)+1;
  17.       end;
  18.     end
  19.   else
  20. ......
  21.  
587:        i:=IndexChar(arr,high(arr)+1,#0);

That is what Valgrind detected!
I think there should not be +1 here.
Though logically it works correctly. It is wrong just from purity perspective. Why it reads outside of array?
Also it is weird that it is not caught by range check given I've compiled RTL with -Cr option.

Should I report this to FPC people? Where?
Submitted bug 0036743
« Last Edit: February 28, 2020, 06:25:26 pm by mm7 »

Thaddy

  • Hero Member
  • *****
  • Posts: 14201
  • Probably until I exterminate Putin.
Re: assignments tracking
« Reply #9 on: February 28, 2020, 06:48:23 pm »
Although not part of the actual chararray payload the terminator #0 at length+1 is there to support C style PAnsiChar. In that sense this is not a bug. Delphi does the same.
Specialize a type, not a var.

garlar27

  • Hero Member
  • *****
  • Posts: 652
Re: assignments tracking
« Reply #10 on: February 28, 2020, 11:21:29 pm »
Hi
I am maintaining quite big and quite convoluted program.
I want to track assignments of a specific class
because it is very hard to find a root cause of exception if an object  was created in one place, used in another (assigned to multiple vars, fields of other objects, arrays, lists...), deleted in third, and then used somewhere else.
So I need something like ref tracking, but not just ref counter, rather an assignment function with possibility of logging callstack to file.

My first idea was to overload := operator. But it is not allowed for equal types.

Second idea if I can use interface ref count TInterfacedObject._AddRef function.

I tried example from here https://wiki.freepascal.org/How_To_Use_Interfaces
But TInterfacedObject._AddRef was never called (in Linux).
Does it work only for COM/Corba? What should call it?

Any other ideas how to track all assignments of a type (class)?
I'm curious,
The problem you have is that changes in the objects you create, destroy and create again is not reflected in the places you assigned it?

something like that:
Code: Pascal  [Select][+][-]
  1. type
  2.    TMyClass1 = class
  3.       Value: Integer;
  4.    end;
  5.  
  6.    { TMyClass2 }
  7.  
  8.    TMyClass2 = class
  9.       SomeObject: TMyClass1;
  10.       procedure DoSomething();
  11.    end;
  12.  
  13. { TMyClass2 }
  14.  
  15. procedure TMyClass2.DoSomething();
  16. begin
  17.    if Assigned(SomeObject) then begin
  18.       //some code
  19.       SomeObject.Value := SomeObject.Value *2;
  20.    end;
  21. end;
  22.  
  23. var
  24.    TheObj1: TMyClass1;
  25.    TheObj2: TMyClass2;
  26. begin
  27.    try
  28.       TheObj1 := TMyClass1.Create;
  29.       TheObj1.Value := 1;
  30.  
  31.       TheObj2 := TMyClass2.Create;
  32.       TheObj2.SomeObject := TheObj1;
  33.  
  34.       TheObj2.DoSomething();
  35.  
  36.       FreeAndNil(TheObj1);
  37.  
  38.       TheObj1 := TMyClass1.Create;
  39.       TheObj1.Value := 5;
  40.  
  41.       TheObj2.DoSomething(); // <----  ACCES VIOLATION HERE
  42.  
  43.    finally
  44.        FreeAndNil(TheObj1);
  45.        FreeAndNil(TheObj2);
  46.    end;
  47. end;
  48.  

or it is something different?

mm7

  • Full Member
  • ***
  • Posts: 193
  • PDP-11 RSX Pascal, Turbo Pascal, Delphi, Lazarus
Re: assignments tracking
« Reply #11 on: February 29, 2020, 12:49:35 am »
Although not part of the actual chararray payload the terminator #0 at length+1 is there to support C style PAnsiChar. In that sense this is not a bug. Delphi does the same.
Exactly. It is not part of the actual source char array. And it has nothing to do with C-strings yet.
In this scope it is just an array of ansichar of some length, and program makes IndexChar to search in it within Length+1. Means outside. So doing buffer overrun.


@garlar27
Yes it is like this. Sometimes even worse, when memory where TheObj2.SomeObject is still pointing to, some new data of another object is already allocated. It will not throw access violation, but rather will do random things.

@Martin_fr
I tried to attach GDB from Lazarus to Valgrind like they do in pure GDB with "target remote | vgdb" command.
I assigned `--eval-command="target remote | vgdb' into Debugger_Startup_Options, but it does not work, shows error panel:
Initialization output:
~"Remote debugging using | vgdb\n"
&"relaying data between gdb and process 21926\n"
=thread-group-started,id="i1",pid="42000"
&"\nwarning: "
&"remote target does not support file transfer, attempting to access files from local filesystem.\n"
=thread-group-exited,id="i1"
&"/home/mark/MyProjects/freeship-plus-in-lazarus/FreeShip (deleted): No such file or directory.\n"
 
« Last Edit: February 29, 2020, 01:22:29 am by mm7 »

Thaddy

  • Hero Member
  • *****
  • Posts: 14201
  • Probably until I exterminate Putin.
Re: assignments tracking
« Reply #12 on: February 29, 2020, 09:44:10 am »
The bugreport is closed by Jonas as no change required, as I already suspected. It is simply not a bug.
Specialize a type, not a var.

mm7

  • Full Member
  • ***
  • Posts: 193
  • PDP-11 RSX Pascal, Turbo Pascal, Delphi, Lazarus
Re: assignments tracking
« Reply #13 on: February 29, 2020, 03:25:20 pm »
The bugreport is closed by Jonas as no change required, as I already suspected. It is simply not a bug.
Too bad the bugreport is closed because it is the bug.
Valgrind sees it as a bug. That is how it was found.

Ok, Thaddy, tell me if this is not a bug?
Code: Pascal  [Select][+][-]
  1. program Project1;
  2. var arr: array[0..9] of char;
  3.   i,j:integer;
  4. begin
  5.  
  6.   // manual CharIndex(arr, high(arr)+1,#0);
  7.   i:=-1;
  8.   for j:=0 to high(arr) + 1 do
  9.     if arr[j] = #0
  10.     then i:=j;
  11.  
  12.   writeln('manual CharIndex returns ', i);
  13. end.
  14.  

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11383
  • FPC developer.
Re: assignments tracking
« Reply #14 on: February 29, 2020, 03:26:46 pm »
high(s)+1 is an expression for the (maximum) length of a static array, so that isn't unlogical. You are confusing high with length, the highest element of a zero based array is one less than the length.

A null based for loop runs from 0 to length-1, so high(arr)+1-1.

What is wrong is your code, that iterates over length+1 characters. Indexchar doesn't have such loop.

« Last Edit: February 29, 2020, 03:32:02 pm by marcov »

 

TinyPortal © 2005-2018