Access violation with useless stack trace on macOS M1

[gitPr0bit]

Hi!
I'm new in this forum so I hope I'm posting my question in the right place.

I'm experiencing "random" Access Violation crashes with my application on macOS with M1 cpu (the crash never happens on x64).

• the Access Violation always happens when launching the application (when it happens)
• "random" means that a broken version of the app crashes at least 50% of the times it gets launched
• I say "a broken version" because making some changes (wherever) in the source code makes the Access Violation disappear (e.g. I add a couple of log prints, the Access Violation completely disappears, but adding a third one makes it appear again)

I cannot debug the issue, because the Access Violation never happens when launching the app in debug from Lazarus.

Also, the stack trace I get is totally useless, since I don't get lines numbers nor any other useful info that could help me in identifying the real cause of this Access Violations. Here's what I see when launching the app from terminal:

Code: [Select]

[FORMS.PP] ExceptionOccurred
  Sender=EAccessViolation
  Exception=Access violation
  Stack trace:
  $00000001810DFE64
  $00000001060CEBBC
  $000000010603F04C
  $0000000106066600
  $0000000106067E90
  $000000010606850C
  $00000001060E8A34
  $00000001810AFD60
  $9A4C0001810DFF40
  $42188001810AFCF8
  $2E76000106132E38
  $00000001060E85F8
  $00000001045BE2BC
  $00000001045B1870
  $00000001045C436C
  $00000001045B1840
  $0000000180D5BF28
Exception at 00000001810DFE64: EAccessViolation:
Access violation.

I currently have the latest version of FPC, compiled from the main git branch this morning, and the version 2.2.7 of Lazarus, compiled from fixes_2_2 git branch.

Here's how I compiled and installed FPC:

Code: Bash  [Select][+][-]

1. make distclean all FPC=/usr/local/lib/fpc/3.2.2/ppca64 OPT="-ap -FD/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin -XR/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk" OS_TARGET=darwin CPU_TARGET=aarch64 -j 4 FPMAKEOPT="-T 4" > build.log
2. sudo make install FPC=$PWD/compiler/ppca64 OPT="-ap -FD/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin -XR/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk" OS_TARGET=darwin CPU_TARGET=aarch64

Does somebody know how could I get a useful stack trace on crashes?

[SymbolicFrank]

I don't know how to make a stacktrace, but in the Project options, you can turn on:
- Use Heaptrc unit
- Trash variables

If it's an error inside your application, that might find it. If it's external, it should be in the System log.

[Martin_fr]

I don't know if it works on Mac, but you can try.

If you get that trace (and if the app was compiled with debug info / dwarf), then
- copy the trace
- open: Menu: View > Leaks and traces
- paste the trace
- press "Resolve" (not sure if you need to select the app bundle, or the actual app inside).

If you are lucky the IDE will get you some of the line numbers.

If not, you can google if "lldb" can resolve the addresses (gdb used to be able to do that).


You probably use -gl
And normally that is supposed to print units and line numbers.
Only it is currently afaik not working for Mac. This is an issue with the compiler (or the provided units, part of fpc).
There may already be an issue report. Not sure...

[Чебурашка]

Does somebody know how could I get a useful stack trace on crashes?

Maybe you might find this useful:

https://wiki.freepascal.org/Logging_exceptions

[Martin_fr]

- Trash variables

May be a good idea. It can (as in "maybe") provoke the error.

The checkbox is not all of it.
If you leave the checkbox off, and go to "custom options" you can enter either
-gt
-gtt
-gttt
-gtttt

And try to compile with each of them, and see if it forces the error, and you can get it in the debugger. (more that 4 t will not do anything new)

[gitPr0bit]

I don't know how to make a stacktrace, but in the Project options, you can turn on:
- Use Heaptrc unit
- Trash variables

If it's an error inside your application, that might find it. If it's external, it should be in the System log.

Unfortunately, I already have them turned on and they are not helpful in my case... Here I post a screenshot of Debugging's configuration

[gitPr0bit]

I don't know if it works on Mac, but you can try.

If you get that trace (and if the app was compiled with debug info / dwarf), then
- copy the trace
- open: Menu: View > Leaks and traces
- paste the trace
- press "Resolve" (not sure if you need to select the app bundle, or the actual app inside).

If you are lucky the IDE will get you some of the line numbers.

If not, you can google if "lldb" can resolve the addresses (gdb used to be able to do that).


You probably use -gl
And normally that is supposed to print units and line numbers.
Only it is currently afaik not working for Mac. This is an issue with the compiler (or the provided units, part of fpc).
There may already be an issue report. Not sure...

I tried this steps too, since I read them in another recent post, but the only thing I get is an error from Lazarus when trying to resolve. I post the attachment.
Talking about lldb, I tried to launch the application from lldb but in this case the Access Violation doesn't appear (as when I launch it from Lazarus in debug).

Now I'm gonna try each of -gt -gtt ... as you suggest.

[gitPr0bit]

Does somebody know how could I get a useful stack trace on crashes?

Maybe you might find this useful:

https://wiki.freepascal.org/Logging_exceptions

I already gave it a look, but I'm gonna read it carefully again, maybe I missed something. Thank you

[TRon]

I don't know if it works on Mac, but you can try.

As you mentioned it yourself before:

IIRC I did read somewhere that the line info does not work for MacOS. Not sure if that still applies (or ever did, but afaik others had the issue too)

Unfortunately (still) no linenumbers for/on/at Mac (that I am aware of).

[gitPr0bit]

- Trash variables

May be a good idea. It can (as in "maybe") provoke the error.

The checkbox is not all of it.
If you leave the checkbox off, and go to "custom options" you can enter either
-gt
-gtt
-gttt
-gtttt

And try to compile with each of them, and see if it forces the error, and you can get it in the debugger. (more that 4 t will not do anything new)

First of all, thank you for your replies.
I tried each of the custom options but there's no way to get the Access Violation launching the app in debug (apparently, it never happens when the debugger is attached)

[Martin_fr]

- open: Menu: View > Leaks and traces
- paste the trace
- press "Resolve" (not sure if you need to select the app bundle, or the actual app inside).

I tried this steps too, since I read them in another recent post, but the only thing I get is an error from Lazarus when trying to resolve. I post the attachment.

Oh, ouch.

You can't by any chance test, if that still happens with the current git main branch? (FpcUpDeluxe).

Or run the Lazarus IDE in lldb, and when you the IDE crashes, get a trace from lldb? (would be a start, may need more)

Otherwise, how big, and how private is the app (the entire bundle) on your disk? Could it be mailed?

Talking about lldb, I tried to launch the application from lldb but in this case the Access Violation doesn't appear (as when I launch it from Lazarus in debug).

You don't need to launch the app in lldb.

When you got the trace, load the app in lldb (but don't start it).

Then for each address try:

Code: Text  [Select][+][-]

1. (lldb) image lookup -a 0x7fff89e6752e
2.       Address: libsystem_kernel.dylib[0x000000000001152e] (libsystem_kernel.dylib.__TEXT.__text + 68126)
3.       Summary: libsystem_kernel.dylib`mach_msg_trap + 10

[gitPr0bit]

- open: Menu: View > Leaks and traces
- paste the trace
- press "Resolve" (not sure if you need to select the app bundle, or the actual app inside).

I tried this steps too, since I read them in another recent post, but the only thing I get is an error from Lazarus when trying to resolve. I post the attachment.

Oh, ouch.

You can't by any chance test, if that still happens with the current git main branch? (FpcUpDeluxe).

Or run the Lazarus IDE in lldb, and when you the IDE crashes, get a trace from lldb? (would be a start, may need more)

Otherwise, how big, and how private is the app (the entire bundle) on your disk? Could it be mailed?

Talking about lldb, I tried to launch the application from lldb but in this case the Access Violation doesn't appear (as when I launch it from Lazarus in debug).

You don't need to launch the app in lldb.

When you got the trace, load the app in lldb (but don't start it).

Then for each address try:

Code: Text  [Select][+][-]

1. (lldb) image lookup -a 0x7fff89e6752e
2.       Address: libsystem_kernel.dylib[0x000000000001152e] (libsystem_kernel.dylib.__TEXT.__text + 68126)
3.       Summary: libsystem_kernel.dylib`mach_msg_trap + 10

1. Yes I can test the last version, I'm just not sure about which one you are talking about (should I install FPC and Lazarus both on 'trunk' version from FPCUpDeluxe?)
2. While waiting your answer to the first point, I'm gonna try to launch Lazarus in lldb, great idea
3. I tried the lookup of the app in lldb after crash, as you suggested, and I got something for a couple of addresses. Here's the output
   

Code: Bash  [Select][+][-]

1. (lldb) image lookup -a 0x00000001810DFE64
2.       Address: SystemConfiguration[0x00000001810dfe64] (SystemConfiguration.__TEXT.__text + 102628)
3.       Summary: SystemConfiguration`_SC_isInstallEnvironment + 16
4. (lldb) image lookup -a 0x0000000102F16BBC
5. (lldb) image lookup -a 0x0000000102E8704C
6. (lldb) image lookup -a 0x0000000102EAE600
7. (lldb) image lookup -a 0x0000000102EAFE90
8. (lldb) image lookup -a 0x0000000102EB050C
9. (lldb) image lookup -a 0x0000000102F30A34
10. (lldb) image lookup -a 0x00000001810AFD60
11.       Address: libSparse.dylib[0x00000001810afd60] (libSparse.dylib.__TEXT.__text + 329160)
12.       Summary: libSparse.dylib`_SparseRefactorSymmetric_Float + 480
13. (lldb) image lookup -a 0x50588001810DFF40
14. (lldb) image lookup -a 0x933F8001810AFCF8
15. (lldb) image lookup -a 0xCD55800102F7AE38
16. (lldb) image lookup -a 0x0000000102F305F8
17. (lldb) image lookup -a 0x00000001020C22BC
18. (lldb) image lookup -a 0x00000001020B5870
19. (lldb) image lookup -a 0x00000001020C836C
20. (lldb) image lookup -a 0x00000001020B5840
21. (lldb) image lookup -a 0x0000000180D5BF28
22.       Address: libBLAS.dylib[0x0000000180d5bf28] (libBLAS.dylib.__TEXT.__text + 433720)
23.       Summary: libBLAS.dylib`___lldb_unnamed_symbol1947 + 5676

4. Unfortunately the app is not personal and I'm not allowed to share it

[Martin_fr]

1. Yes I can test the last version, I'm just not sure about which one you are talking about (should I install FPC and Lazarus both on 'trunk' version from FPCUpDeluxe?)

You can use the existing FPC.
You only need the "trunk" version of lazarus. It should run fine with the existing fpc.

Then you can do the same "resolve" => And that will at least tell, if the bug in the IDE has been fixed.
(as you likely will have changed/recompiled your work, test in your current Lazarus, that you have a case that crashes)

2. While waiting your answer to the first point, I'm gonna try to launch Lazarus in lldb, great idea

That would be the next step, after the above. If the IDE still crashes (which we really don't want), then a trace of that crash might be the first step to find out how to fix it.

3. I tried the lookup of the app in lldb after crash, as you suggested, and I got something for a couple of addresses. Here's the output

If lldb does not find the addresses, then "resolve" likely wont either. (But still good to test with "trunk", so we know if or if not it crashes)

That is for "resolve" or lldb "image lookup" to work:

* You must have used the exact same compiled exe in which you got the trace. If you later recompiled the exe, the addresses could have changed.

* You must *not* have compiled as "relocatable". Afaik the is default. "relocatable" is usually used as a security feature, so exploits can not predict addresses.

If you use packages, then make sure they all have debug info.
=> Project Options > Additions and Overrides > Add custom option (Target: *):  -gw
(or -gw3 if you use Dwarf 3 / but for line numbers it makes no diff)

If you do use FpcUpDeluxe: You may want to get your self an FPC 3.2.3 (fixes branch).
It has many very valuable bugfixes (including issues where the compiler breaks your app).
You may even want to compile it with -gw (you can mix this with optimizations, if you want an optimized build / it is just for the line info).

FPC/RTL with -gw may slow down linking / so you may want to have 2 installs: with/without.

If you do build an fpc with -gw and the crash is (even if triggered by your code) somewhere in the RTL, then you will get better traces. (assuming any of the lookup techniques start to work, or you can get in the debugger).

If you can build and test your app on Linux...

There is a tool called valgrind.

Code: Text  [Select][+][-]

1. valgrind --tool=memcheck yourapp

is a really great tool to find access violations.
Valgrind very often can report the fault even if it does not cause a crash. (e.g. dangling pointers / double free / ...)

But it is Linux only...

If you are not on Linux => Try heaptrc (again). But set the environment HEAPTRC="keepreleased"
(and then mix that with -gt variants).

[Martin_fr]

I done some testing on Mac.

It seems Mac compiles "relocatable" by default. That means, the same code will be on a new address for each run (without recompile, just starting the same app again).

That means, once the app stopped, the addresses from the stack dump are useless.

I don't know the compiler (or probably linker) settings well enough. So I don't know how to turn that off.

If you can switch it off, then lldb (and hopefully the IDE's "resolve") should be able to translate the addresses.

[Martin_fr]

https://lists.freepascal.org/pipermail/fpc-devel/2023-June/045156.html

How to compile:
https://lists.freepascal.org/pipermail/fpc-devel/2023-June/045158.html
(I have not tested myself, but I expect with that lldb may work, and "resolve" hopefully too)


FPC trunk:
https://lists.freepascal.org/pipermail/fpc-devel/2023-June/045159.html