Recent

Author Topic: Improving Appstore stacktraces on fpc_* functions  (Read 6174 times)

BeniBela

  • Hero Member
  • *****
  • Posts: 905
    • homepage
Improving Appstore stacktraces on fpc_* functions
« on: November 12, 2017, 12:13:25 am »
When an app crashes, Google's store provides a stacktrace.

Unfortunately it is often incomplete, especially it seems to stops on fpc_ functions.

Now for example I got a two line stacktrace for my app, which appears to be SYSTEM$_$TOBJECT_$__$$_INHERITSFROM$TCLASS$$BOOLEAN and  fpc_do_is.

Somehow  fpc_do_is, the is operator crashed. Probably I called it on an already freed object, but how can someone find that? Searching and examining  every usage of the is operator would take forever.

So how can you compile everything that it shows the caller of fpc_* functions?

I already compiled my project with -O1. Would it help to compile the rtl  with -O1? Or even  -O-  ?

engkin

  • Hero Member
  • *****
  • Posts: 3112
Re: Improving Appstore stacktraces on fpc_* functions
« Reply #1 on: November 12, 2017, 01:40:25 am »
Compiling with -al will give you the line number where fpc_do_is is used.

BeniBela

  • Hero Member
  • *****
  • Posts: 905
    • homepage
Re: Improving Appstore stacktraces on fpc_* functions
« Reply #2 on: November 12, 2017, 02:58:39 pm »
Compiling with -al will give you the line number where fpc_do_is is used.

Well that lists 196 usages. None with an obvious bug

That does not help to list the caller in the next stacktrace


Perhaps the function needs a better stackframe around it.

Or there is a tool to create new functions fpc_do_is1 ... fpc_do_is196, and the replace the calls to those
« Last Edit: November 12, 2017, 03:26:45 pm by BeniBela »

engkin

  • Hero Member
  • *****
  • Posts: 3112
Re: Improving Appstore stacktraces on fpc_* functions
« Reply #3 on: November 12, 2017, 04:31:29 pm »
Or there is a tool to create new functions fpc_do_is1 ... fpc_do_is196, and the replace the calls to those

I doubt a tool exists, but I like the idea. I tried the following:
  • Add a unit with fpc_do_is1 ... fpc_do_is196. I declared them as:
Code: Pascal  [Select][+][-]
  1. function fpc_do_is1(aclass : tclass;aobject : tobject) : boolean; compilerproc;
  2. function fpc_do_is2(aclass : tclass;aobject : tobject) : boolean; compilerproc;
  3. ...
  4. implementation
  5.  
  6. function fpc_do_is1(aclass : tclass;aobject : tobject) : boolean;[public,alias: 'FPC_DO_IS1']; compilerproc;
  7.   begin
  8.      Result:=assigned(aobject) and assigned(aclass) and
  9.        aobject.inheritsfrom(aclass);
  10.   end;
  11.  
  12. function fpc_do_is2(aclass : tclass;aobject : tobject) : boolean;[public,alias: 'FPC_DO_IS2']; compilerproc;
  13.   begin
  14.      Result:=assigned(aobject) and assigned(aclass) and
  15.        aobject.inheritsfrom(aclass);
  16.   end;
  17. ...
  • Compile your project using custom assembler (I used -al -s -Aas)
  • Modify calls to fpc_do_is in the *.s files to use your numbered calls
  • Run PPAS

Should be easy automate the process with some code.

BeniBela

  • Hero Member
  • *****
  • Posts: 905
    • homepage
Re: Improving Appstore stacktraces on fpc_* functions
« Reply #4 on: January 20, 2018, 11:17:34 am »
I did not want to mess with the assembly, so I removed the "is" operator from my Pascal source. A dozens in every update, till there are no more calls to fpc_do_is anywhere

It is still crashing on fpc_do_is


The caller must be some other fpc function

taazz

  • Hero Member
  • *****
  • Posts: 5368
Re: Improving Appstore stacktraces on fpc_* functions
« Reply #5 on: January 20, 2018, 11:55:57 am »
I did not want to mess with the assembly, so I removed the "is" operator from my Pascal source. A dozens in every update, till there are no more calls to fpc_do_is anywhere

It is still crashing on fpc_do_is


The caller must be some other fpc function
the is operator should never raise an exception. That leads me to believe that its more probable that you have a buffer overrun (or similar) problem.
any chance to unwind the stack to the origin? that might give you a path to follow but at this point I would try to find an emulator/device that has the problem and remote debug it.
Good judgement is the result of experience … Experience is the result of bad judgement.

OS : Windows 7 64 bit
Laz: Lazarus 1.4.4 FPC 2.6.4 i386-win32-win32/win64

Thaddy

  • Hero Member
  • *****
  • Posts: 14157
  • Probably until I exterminate Putin.
Re: Improving Appstore stacktraces on fpc_* functions
« Reply #6 on: January 20, 2018, 12:07:54 pm »
First thing that comes to mind to check is the use of interfaces (uses is internally) may be mix up between interface var en class var?
Specialize a type, not a var.

engkin

  • Hero Member
  • *****
  • Posts: 3112
Re: Improving Appstore stacktraces on fpc_* functions
« Reply #7 on: January 20, 2018, 03:25:53 pm »
the is operator should never raise an exception.
I am not sure I follow you here. is operator ends up calling InheritsFrom which in turn plays with pointers:
Code: Pascal  [Select][+][-]
  1.       class function TObject.InheritsFrom(aclass : TClass) : Boolean;
  2.  
  3.         var
  4.            vmt: PVmt;
  5.  
  6.         begin
  7.            if assigned(aclass) then
  8.              begin
  9.                vmt:=PVmt(self);
  10.                while assigned(vmt) and (vmt <> PVmt(aclass)) do
  11.                  vmt := vmt^.vParent;
  12.                InheritsFrom := (vmt = PVmt(aclass));
  13.              end
  14.            else
  15.              inheritsFrom := False;
  16.         end;

The highlighted line is an evil one. Imagine vmt holds a random number. An invalid address. Actually we can produce it:
Code: Pascal  [Select][+][-]
  1. procedure Test;
  2. var
  3.   sl:TStringList;
  4. begin
  5.   sl := TStringList($12345678);
  6.  
  7.   if sl is TStringList then
  8.     WriteLn('sl is TStringList')
  9.   else
  10.     WriteLn('sl is not TStringList');
  11. end;
This code should crash. It does on my system.

Edit:
Maybe is operator should be protected with try except:
Code: Pascal  [Select][+][-]
  1. function fpc_do_is(aclass : tclass;aobject : tobject) : boolean;[public,alias: 'FPC_DO_IS']; compilerproc;
  2.   begin
  3.     try
  4.      fpc_do_is:=assigned(aobject) and assigned(aclass) and
  5.        aobject.inheritsfrom(aclass);
  6.     except on EAccessViolation do
  7.       Result := False;
  8.     end;
  9.   end;
« Last Edit: January 20, 2018, 03:39:07 pm by engkin »

BeniBela

  • Hero Member
  • *****
  • Posts: 905
    • homepage
Re: Improving Appstore stacktraces on fpc_* functions
« Reply #8 on: January 20, 2018, 04:34:16 pm »
That leads me to believe that its more probable that you have a buffer overrun (or similar) problem.
any chance to unwind the stack to the origin? that might give you a path to follow but at this point I would try to find an emulator/device that has the problem and remote debug it.

The app store does not give the stack and I could never reproduce the issue anywhere...

The app stores tries unwinding to create a backtrace, but clearly it fails after the second function

First thing that comes to mind to check is the use of interfaces (uses is internally) may be mix up between interface var en class var?

Is with interfaces  usually becomes fpc_intf_is


Maybe is operator should be protected with try except:
Code: Pascal  [Select][+][-]
  1. function fpc_do_is(aclass : tclass;aobject : tobject) : boolean;[public,alias: 'FPC_DO_IS']; compilerproc;
  2.   begin
  3.     try
  4.      fpc_do_is:=assigned(aobject) and assigned(aclass) and
  5.        aobject.inheritsfrom(aclass);
  6.     except on EAccessViolation do
  7.       Result := False;
  8.     end;
  9.   end;

Or perhaps try except is the problem.

There is a use of fpc_do_is in fpc_catches.

Code: Pascal  [Select][+][-]
  1.   try
  2.     try
  3.       raise exception.Create('..');
  4.     except on e: Exception do begin
  5.       e.free;
  6.       raise;
  7.     end;
  8.     end;
  9.   except on e: Exception do begin
  10.     writeln('b');
  11.   end;
  12.   end;
  13.  

gives exactly that backtrace (see attachment)


Amd the big problem are the ??, especially the one in the third row

GDB figures out that it should continue to fpc_catches, but the app store can't. It aborts on the first ??


How can you modify fpc_do_is/fpc_catches to not cause ?? in backtraces

BeniBela

  • Hero Member
  • *****
  • Posts: 905
    • homepage
Re: Improving Appstore stacktraces on fpc_* functions
« Reply #9 on: January 21, 2018, 01:46:12 am »
I have found my problem :/

Code: Pascal  [Select][+][-]
  1.   except on e: Exception do begin
  2.     ...
  3.     raise e;
  4.   end;    
  5.  



But still how do you prevent the holes in the stacktrace between fpc_do_is and fpc_catches? If fpc_catches was in the backtrace the issue would have been much easier to find.  (actually the app store handles it even better than fpc by showing inheritsfrom and fpc_do_is. The default backtrace on linux just shows inheritsfrom )
 

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9754
  • Debugger - SynEdit - and more
    • wiki
Re: Improving Appstore stacktraces on fpc_* functions
« Reply #10 on: January 21, 2018, 01:58:09 am »
If you do not want the "holes" then you must compile the rtl and packages without optimization.

Those functions are optimized, and have their stack frame dropped. That is, the code that generates a frame on the stack.

There may be:
{$OPTIMIZATION STACKFRAME}
If this is in the rtl code, you may need to remove it.

taazz

  • Hero Member
  • *****
  • Posts: 5368
Re: Improving Appstore stacktraces on fpc_* functions
« Reply #11 on: January 21, 2018, 02:14:42 am »
the is operator should never raise an exception.
I am not sure I follow you here. is operator ends up calling InheritsFrom which in turn plays with pointers:
Code: Pascal  [Select][+][-]
  1.       class function TObject.InheritsFrom(aclass : TClass) : Boolean;
  2.  
  3.         var
  4.            vmt: PVmt;
  5.  
  6.         begin
  7.            if assigned(aclass) then
  8.              begin
  9.                vmt:=PVmt(self);
  10.                while assigned(vmt) and (vmt <> PVmt(aclass)) do
  11.                  vmt := vmt^.vParent;
  12.                InheritsFrom := (vmt = PVmt(aclass));
  13.              end
  14.            else
  15.              inheritsFrom := False;
  16.         end;

The highlighted line is an evil one. Imagine vmt holds a random number. An invalid address. Actually we can produce it:
Code: Pascal  [Select][+][-]
  1. procedure Test;
  2. var
  3.   sl:TStringList;
  4. begin
  5.   sl := TStringList($12345678);
  6.  
  7.   if sl is TStringList then
  8.     WriteLn('sl is TStringList')
  9.   else
  10.     WriteLn('sl is not TStringList');
  11. end;
This code should crash. It does on my system.

Edit:
Maybe is operator should be protected with try except:
Code: Pascal  [Select][+][-]
  1. function fpc_do_is(aclass : tclass;aobject : tobject) : boolean;[public,alias: 'FPC_DO_IS']; compilerproc;
  2.   begin
  3.     try
  4.      fpc_do_is:=assigned(aobject) and assigned(aclass) and
  5.        aobject.inheritsfrom(aclass);
  6.     except on EAccessViolation do
  7.       Result := False;
  8.     end;
  9.   end;
the idea of the is operator is to safely return a true/false answer with out raising exceptions. It is used on classes, interfaces and pointers. I have no idea how it is implemented in fpc but the idea behind it is something along the lines of
Code: [Select]
default answer is  no;
  if isobject(InputVar) and isclass(InputClass) then result := Tclass(InputVar).InheritsFrom(InputClass)
  else
  if isInterface(Inputvar) and isInterface(inputClass) then result := Supports(InputVar,InputClass)
  else ....
The line shown should never execute if the inputvar has no vmt or the InputClass is an interface. While the compiler can (and should) decide at compile time which xxxx_do_is to call based on the parameters (overload?) there must be a small runtime check for validity as well. for me if it raises an exception the its a bug, as opposed to inheritsfrom which I expect to get an exception in the same case.
then again I'm OT (again) in this thread.

How can you modify fpc_do_is/fpc_catches to not cause ?? in backtraces
you build the rtl/fcl with debug info I guess.
Good judgement is the result of experience … Experience is the result of bad judgement.

OS : Windows 7 64 bit
Laz: Lazarus 1.4.4 FPC 2.6.4 i386-win32-win32/win64

engkin

  • Hero Member
  • *****
  • Posts: 3112
Re: Improving Appstore stacktraces on fpc_* functions
« Reply #12 on: January 21, 2018, 04:04:32 am »
the idea of the is operator is to safely return a true/false answer with out raising exceptions. It is used on classes, interfaces and pointers. I have no idea how it is implemented in fpc but the idea behind it is something along the lines of
Code: [Select]
default answer is  no;
  if isobject(InputVar) and isclass(InputClass) then result := Tclass(InputVar).InheritsFrom(InputClass)
  else
  if isInterface(Inputvar) and isInterface(inputClass) then result := Supports(InputVar,InputClass)
  else ....
FPC for isobject(InputVar) checks that InputVar is not nil. The same applies to isclass(InputClass). That's why both operator is and InheritsFrom could crash when invalid pointers are used.

[snip]for me if it raises an exception the its a bug
That why I suggested catching EAccessViolation.



I have found my problem :/

Congrats!!

taazz

  • Hero Member
  • *****
  • Posts: 5368
Re: Improving Appstore stacktraces on fpc_* functions
« Reply #13 on: January 21, 2018, 07:18:23 pm »
the idea of the is operator is to safely return a true/false answer with out raising exceptions. It is used on classes, interfaces and pointers. I have no idea how it is implemented in fpc but the idea behind it is something along the lines of
Code: [Select]
default answer is  no;
  if isobject(InputVar) and isclass(InputClass) then result := Tclass(InputVar).InheritsFrom(InputClass)
  else
  if isInterface(Inputvar) and isInterface(inputClass) then result := Supports(InputVar,InputClass)
  else ....
FPC for isobject(InputVar) checks that InputVar is not nil. The same applies to isclass(InputClass). That's why both operator is and InheritsFrom could crash when invalid pointers are used.

[snip]for me if it raises an exception the its a bug
That why I suggested catching EAccessViolation.
That would be a blanket statement yes.
Good judgement is the result of experience … Experience is the result of bad judgement.

OS : Windows 7 64 bit
Laz: Lazarus 1.4.4 FPC 2.6.4 i386-win32-win32/win64

BeniBela

  • Hero Member
  • *****
  • Posts: 905
    • homepage
Re: Improving Appstore stacktraces on fpc_* functions
« Reply #14 on: March 03, 2018, 07:33:08 pm »
Compiling with -al will give you the line number where fpc_do_is is used.

I think I always put -al there for debugging and removed  it for the release. Although I am not sure how it was months ago.

But with the previous update I forgot to remove -al.

And then the app store reported another eleven crashes: java.lang.UnsatisfiedLinkError on aarch64 OnePlus OnePlus3.

Now I have removed -al again and the reports stopped coming

Logical conclusion: besides giving numbers -al damages the .so file

 

 

TinyPortal © 2005-2018