How to get the call stack for an exception.

[spenov]

If I use Lazarus' built-in tools, the resulting executable file is too large, and address-to-line decoding is slow.
In mORMot, the .mab file is very compact. However, TSynLog does not output the call stack.
I tried replacing BackTraceStrFunc with a call to mORMot, but the call line is not determined correctly.
I can't figure out how to get the proper call stack from mORMot.


Example of a call stack obtained using Lazarus' built-in tools:

TSynLog 2.2.7641 2025-08-13T11:49:53

20250813 11495314 EXC   Exception {Message:"test"} [Main] at 01000368e1 unit1.pas  (132)

Exception class: Exception
Message: test
FrameCount=16
Stacktrace:
  $00000001000368E1  Button1Click,  line 131 of unit1.pas
  $0000000100134927  CLICK,  line 2974 of include/control.inc
  $0000000100150889  CLICK,  line 55 of include/buttoncontrol.inc
  $0000000100150F79  CLICK,  line 169 of include/buttons.inc
  $0000000100150782  WMDEFAULTCLICKED,  line 21 of include/buttoncontrol.inc
  $000000010000ED61
  $0000000100133621  WNDPROC,  line 2304 of include/control.inc
  $0000000100127CAD  WNDPROC,  line 5434 of include/wincontrol.inc
  $0000000100192E89  DELIVERMESSAGE,  line 114 of lclmessageglue.pas
  $0000000100108F61  DOWINDOWPROC,  line 2642 of win32/win32callback.inc
  $000000010010973F  WINDOWPROC,  line 2807 of win32/win32callback.inc
  $0000000100197BDC  CUSTOMFORMWNDPROC,  line 412 of win32/win32wsforms.pp
  $00007FFC5EFFEF5C
  $00007FFC5EFFDFBB
  $00007FFC5EFFD814
  $00007FFC51762927
  $00007FFC517725B0

Example of a call stack obtained using mORMot procedures

Exception class: Exception
Message: test
FrameCount=16
Stacktrace:

01000368e1 unit1.pas  (132)
0100134927 include\controlactionlink.inc tcontrol.click (110)
0100150889 include\buttoncontrol.inc tbuttoncontrol.click (56)
0100150f79 include\buttons.inc tcustombutton.click (170)
0100150782 include\buttoncontrol.inc tbuttoncontrol.wmdefaultclicked (21)
010000ed61 project1.lpr  (24)
0100133621 include\controlactionlink.inc tcontrol.wndproc (110)
0100127cad include\dragmanager.inc twincontrol.wndproc (902)
0100192e89 win32\win32memostrings.inc delivermessage (199)
0100108f61 win32\win32int.pp twindowprochelper.dowindowproc (521)
010010973f win32\win32int.pp windowproc (521)
0100197bdc win32\win32memostrings.inc customformwndproc (199)
7ffc5effef5c
7ffc5effdfbb
7ffc5effd814
7ffc51762927
7ffc517725b0

As can be seen, the procedure names match, but the line numbers do not.
I've attached a test project that reproduces this data.

Code: Pascal  [Select][+][-]

1. function DumpExceptionCallStack(E: Exception):string;
2. var
3.    I:Integer;
4.    Frames:PPointer;
5.    log: TSynLog;
6. begin
7.    Frames:=ExceptFrames;
8.    Result:='Exception class: '+E.ClassName+LineEnding+
9.            'Message: '+E.Message+LineEnding+
10.            'FrameCount='+IntToStr(ExceptFrameCount)+LineEnding+
11.            'Stacktrace:'+LineEnding+
12.            BackTraceStrFunc(ExceptAddr);
13.    for I:=0 to ExceptFrameCount-1 do
14.       Result:=Result+LineEnding+BackTraceStrFunc(Frames[I]);
15.    //
16.    log := GlobalCurrentHandleExceptionSynLog;
17.    log.Writer.AddCRAndIndent;
18.    log.Writer.AddString(Result);
19.    log.Writer.AddCRAndIndent;
20. end;
21.  
22. function DumpExceptionCallStack_mormot(E: Exception):string;
23. const
24.    MaxDepth=10;
25. var
26.    I, FC:Integer;
27.    Frames:PCodePointer;
28.    log: TSynLog;
29. begin
30.    Frames:=ExceptFrames;
31.    FC:=ExceptFrameCount;
32.    log := GlobalCurrentHandleExceptionSynLog;
33.    log.Writer.AddCRAndIndent;
34.    Result:='Exception class: '+E.ClassName+LineEnding+
35.            'Message: '+E.Message+LineEnding+
36.            'FrameCount='+IntToStr(FC)+LineEnding+
37.            'Stacktrace:'+LineEnding;
38.    log.Writer.AddString(Result);
39.    log.Writer.AddCRAndIndent;
40.    TDebugFile.Log(log.Writer, PtrUInt(ExceptAddr), {notcode=}true, {symbol=}false);
41.    log.Writer.AddCRAndIndent;
42.    {if FC>MaxDepth then
43.       FC:=MaxDepth;}
44.    for I:=0 to FC-1 do begin
45.       TDebugFile.Log(log.Writer, PtrUInt(Frames[I]), {notcode=}true, {symbol=}false);
46.       log.Writer.AddCRAndIndent;
47.    end;
48. end;
49.  

[Martin_fr]

If this is for shipping a release, there is a 3rd option.

- Build the app with debug info -gl => so you get the big one.
- Make a copy of the exe. (you must have an exact copy, a 2nd build may not work, even if from identical sources)
- use strip (strip.exe) to remove debug info on one copy.

Now you can release the stripped version (small).
When it crashes, it prints only addresses.

When someone sends you those addresses, you can in the IDE use Menu View > Leaks and traces => button resolve: select the copy that still has debug info.

And it will show lines and all.

Test it before relying on it.

Make sure, that when you get an address only stack, you also get the exact version of your release, so you can find the correct copy.

You can also use gdb to resolve addresses (one by one). You would need to google that.

[spenov]

Thank you. I've tried this approach—it works. However, getting the call stack directly, without additional manipulations and keeping a copy of the file, would be much more convenient.

[Martin_fr]

Unfortunately I don't know the mormot stack routines. So I can't give advice.


I don't know what your scenario is?

I did guess "user reporting on releases", but it now sounds like a more repeat process?
Then the questions are
- why not run in the debugger directly?
- your stack has win32, so no embedded? Then what limits you on the size? Or is it just the speed?

There isn't much that can be done about the speed of the build in stack decoder. Its meant to work on low mem usage, so it compensates with time (afaik: needs to keep loading...)

Maybe edit the subject, and add "Mormot" so people with that background will be paying attention.


Just some observations => the lines that are way off are in *.inc files.
So maybe that causes a bug (as they are originally encoded together with the pas file that includes them, and the decoder needs to pay attention to file switching)

You may also try using a different FPC => there used to be some issues, but I don't know if they would have affected the files in your stack. (don't think so / but still may be worth a try)...

Also test with -O- (no opt) or -O1 => in case you used any higher opt.


Some of the lines are off by 1. That may not be an error, but a matter of interpretation.
Technically, if you are in a "call" then the return address points to the asm statement after the call. Which can (sometimes / other-times not) be on the next line. Some decoders account for that, and always show the "call" line. Other decoders show "as is".


You may also find that sometimes a single frame gets skipped. That happens due to optimization, if the called code is optimized.
E.g. if your fpc/RTL is -O2, then your code calling certain rtl functions may not be in the stack.
I don't know if any build in decoder deals with that. (they may, but I wouldn't expect it)

[spenov]

The executable file with debug information is over 500 MB, which is quite large. But what bothers me even more is that the BackTraceStrFunc procedure works slowly, causing significant delays in exception handling. That's why I'm trying to find a solution that is both fast and doesn't produce such large file sizes. In Delphi, I use the JCL library—everything works perfectly there.

[Martin_fr]

Well, an experimental idea....

If you are willing to recompile FPC.... (or copy the unit, and get the copy to do the job...)
Find the unit lnfodwrf

It has some buffers and caches....

Code: Pascal  [Select][+][-]

1. const
2.   EBUF_SIZE = 100;
3. ...
4. {$if defined(CPU64)}
5.   LineInfoCacheLength = 2039;
6. {$elseif defined(CPU32)}
7.   LineInfoCacheLength = 251;
8. {$else}
9.   LineInfoCacheLength = 1;
10. {$endif CPU64}
11.  

You could increase those (that memory is statically allocate when the app is loaded. So it will be used ALL THE TIME, even if you don't need it at all.
But especially on 64 bit systems, that should not be a too big concern.

You could try to set BOTH of them to maybe 10000, even more...

But I have no idea. Might make a diff, might not at all....

[spenov]

Apologies, but I don't understand what exactly is supposed to change. Will the built-in procedure become faster at resolving the module name and line number from an address?

[ALLIGATOR]

I made something similar for myself based on FpDebug. I don't know how much faster it reads DWARF, but it seems that not long ago Alexander Bagel and Martin made some improvements to speed it up

Take a look, maybe you'll find something useful for yourself

[Martin_fr]

Apologies, but I don't understand what exactly is supposed to change. Will the built-in procedure become faster at resolving the module name and line number from an address?

Maybe...

I haven't looked in any depth at it. But from what I know:
- it opens the dwarf info, parses it, and shows the result.
- it is intentionally using little memory

E.g. "EBUF_SIZE" from a quick look it the size of the buffer for loading info from disk. If it is increased bigger chunks can be loaded. (If it goes to big, then too much is loaded).
Anyway 100 should mean to constantly load very small pieces...

So, if you want to experiment, this is where to start.

Disk loading will not be the only factor, and other factors may not be as easily tuned. So there is no prediction how much it will get you.
- If you already build your own fpc, then that is changed quite quickly / and a few minutes to compile fpc.
- If you don't, then its gonna be some work (and no guarantees)

[spenov]

Thank you. I'll make the changes to the module tomorrow, recompile FPC, and check whether there are any delays.

[spenov]

I tried increasing the buffer sizes. It became slightly faster, but the delay is still noticeable.

[spenov]

I made something similar for myself based on FpDebug. I don't know how much faster it reads DWARF, but it seems that not long ago Alexander Bagel and Martin made some improvements to speed it up

Take a look, maybe you'll find something useful for yourself

I've tried your approach. It works fast. This is essentially an alternative to the one built into the IDE, but more convenient—since there's no need to keep a separate debug version of the executable with debug information. Thank you.