Recent

Author Topic: Access violation with useless stack trace on macOS M1  (Read 5001 times)

gitPr0bit

  • New Member
  • *
  • Posts: 11
Access violation with useless stack trace on macOS M1
« on: June 05, 2023, 01:20:06 pm »
Hi!
I'm new in this forum so I hope I'm posting my question in the right place.

I'm experiencing "random" Access Violation crashes with my application on macOS with M1 cpu (the crash never happens on x64).
  • the Access Violation always happens when launching the application (when it happens)
  • "random" means that a broken version of the app crashes at least 50% of the times it gets launched
  • I say "a broken version" because making some changes (wherever) in the source code makes the Access Violation disappear (e.g. I add a couple of log prints, the Access Violation completely disappears, but adding a third one makes it appear again)

I cannot debug the issue, because the Access Violation never happens when launching the app in debug from Lazarus.

Also, the stack trace I get is totally useless, since I don't get lines numbers nor any other useful info that could help me in identifying the real cause of this Access Violations. Here's what I see when launching the app from terminal:
Code: [Select]
[FORMS.PP] ExceptionOccurred
  Sender=EAccessViolation
  Exception=Access violation
  Stack trace:
  $00000001810DFE64
  $00000001060CEBBC
  $000000010603F04C
  $0000000106066600
  $0000000106067E90
  $000000010606850C
  $00000001060E8A34
  $00000001810AFD60
  $9A4C0001810DFF40
  $42188001810AFCF8
  $2E76000106132E38
  $00000001060E85F8
  $00000001045BE2BC
  $00000001045B1870
  $00000001045C436C
  $00000001045B1840
  $0000000180D5BF28
Exception at 00000001810DFE64: EAccessViolation:
Access violation.

I currently have the latest version of FPC, compiled from the main git branch this morning, and the version 2.2.7 of Lazarus, compiled from fixes_2_2 git branch.

Here's how I compiled and installed FPC:
Code: Bash  [Select][+][-]
  1. make distclean all FPC=/usr/local/lib/fpc/3.2.2/ppca64 OPT="-ap -FD/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin -XR/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk" OS_TARGET=darwin CPU_TARGET=aarch64 -j 4 FPMAKEOPT="-T 4" > build.log
  2. sudo make install FPC=$PWD/compiler/ppca64 OPT="-ap -FD/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin -XR/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk" OS_TARGET=darwin CPU_TARGET=aarch64

Does somebody know how could I get a useful stack trace on crashes?

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 1313
Re: Access violation with useless stack trace on macOS M1
« Reply #1 on: June 05, 2023, 03:07:09 pm »
I don't know how to make a stacktrace, but in the Project options, you can turn on:
- Use Heaptrc unit
- Trash variables

If it's an error inside your application, that might find it. If it's external, it should be in the System log.

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9754
  • Debugger - SynEdit - and more
    • wiki
Re: Access violation with useless stack trace on macOS M1
« Reply #2 on: June 05, 2023, 03:17:06 pm »
I don't know if it works on Mac, but you can try.

If you get that trace (and if the app was compiled with debug info / dwarf), then
- copy the trace
- open: Menu: View > Leaks and traces
- paste the trace
- press "Resolve" (not sure if you need to select the app bundle, or the actual app inside).

If you are lucky the IDE will get you some of the line numbers.

If not, you can google if "lldb" can resolve the addresses (gdb used to be able to do that).



You probably use -gl
And normally that is supposed to print units and line numbers.
Only it is currently afaik not working for Mac. This is an issue with the compiler (or the provided units, part of fpc).
There may already be an issue report. Not sure...

Чебурашка

  • Hero Member
  • *****
  • Posts: 566
  • СЛАВА УКРАЇНІ! / Slava Ukraïni!
Re: Access violation with useless stack trace on macOS M1
« Reply #3 on: June 05, 2023, 03:18:27 pm »
Does somebody know how could I get a useful stack trace on crashes?

Maybe you might find this useful:

https://wiki.freepascal.org/Logging_exceptions
FPC 3.2.0/Lazarus 2.0.10+dfsg-4+b2 on Debian 11.5
FPC 3.2.2/Lazarus 2.2.0 on Windows 10 Pro 21H2

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9754
  • Debugger - SynEdit - and more
    • wiki
Re: Access violation with useless stack trace on macOS M1
« Reply #4 on: June 05, 2023, 03:19:50 pm »
- Trash variables

May be a good idea. It can (as in "maybe") provoke the error.

The checkbox is not all of it.
If you leave the checkbox off, and go to "custom options" you can enter either
-gt
-gtt
-gttt
-gtttt

And try to compile with each of them, and see if it forces the error, and you can get it in the debugger. (more that 4 t will not do anything new)


gitPr0bit

  • New Member
  • *
  • Posts: 11
Re: Access violation with useless stack trace on macOS M1
« Reply #5 on: June 05, 2023, 03:48:29 pm »
I don't know how to make a stacktrace, but in the Project options, you can turn on:
- Use Heaptrc unit
- Trash variables

If it's an error inside your application, that might find it. If it's external, it should be in the System log.

Unfortunately, I already have them turned on and they are not helpful in my case... Here I post a screenshot of Debugging's configuration

gitPr0bit

  • New Member
  • *
  • Posts: 11
Re: Access violation with useless stack trace on macOS M1
« Reply #6 on: June 05, 2023, 03:54:28 pm »
I don't know if it works on Mac, but you can try.

If you get that trace (and if the app was compiled with debug info / dwarf), then
- copy the trace
- open: Menu: View > Leaks and traces
- paste the trace
- press "Resolve" (not sure if you need to select the app bundle, or the actual app inside).

If you are lucky the IDE will get you some of the line numbers.

If not, you can google if "lldb" can resolve the addresses (gdb used to be able to do that).



You probably use -gl
And normally that is supposed to print units and line numbers.
Only it is currently afaik not working for Mac. This is an issue with the compiler (or the provided units, part of fpc).
There may already be an issue report. Not sure...

I tried this steps too, since I read them in another recent post, but the only thing I get is an error from Lazarus when trying to resolve. I post the attachment.
Talking about lldb, I tried to launch the application from lldb but in this case the Access Violation doesn't appear (as when I launch it from Lazarus in debug).

Now I'm gonna try each of -gt -gtt ... as you suggest.
« Last Edit: June 05, 2023, 04:19:46 pm by gitPr0bit »

gitPr0bit

  • New Member
  • *
  • Posts: 11
Re: Access violation with useless stack trace on macOS M1
« Reply #7 on: June 05, 2023, 03:56:52 pm »
Does somebody know how could I get a useful stack trace on crashes?

Maybe you might find this useful:

https://wiki.freepascal.org/Logging_exceptions

I already gave it a look, but I'm gonna read it carefully again, maybe I missed something. Thank you

TRon

  • Hero Member
  • *****
  • Posts: 2399
Re: Access violation with useless stack trace on macOS M1
« Reply #8 on: June 05, 2023, 04:02:03 pm »
I don't know if it works on Mac, but you can try.

As you mentioned it yourself before:

IIRC I did read somewhere that the line info does not work for MacOS. Not sure if that still applies (or ever did, but afaik others had the issue too)

Unfortunately (still) no linenumbers for/on/at Mac (that I am aware of).

gitPr0bit

  • New Member
  • *
  • Posts: 11
Re: Access violation with useless stack trace on macOS M1
« Reply #9 on: June 05, 2023, 04:41:02 pm »
- Trash variables

May be a good idea. It can (as in "maybe") provoke the error.

The checkbox is not all of it.
If you leave the checkbox off, and go to "custom options" you can enter either
-gt
-gtt
-gttt
-gtttt

And try to compile with each of them, and see if it forces the error, and you can get it in the debugger. (more that 4 t will not do anything new)

First of all, thank you for your replies.
I tried each of the custom options but there's no way to get the Access Violation launching the app in debug (apparently, it never happens when the debugger is attached)

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9754
  • Debugger - SynEdit - and more
    • wiki
Re: Access violation with useless stack trace on macOS M1
« Reply #10 on: June 05, 2023, 04:48:03 pm »
- open: Menu: View > Leaks and traces
- paste the trace
- press "Resolve" (not sure if you need to select the app bundle, or the actual app inside).

I tried this steps too, since I read them in another recent post, but the only thing I get is an error from Lazarus when trying to resolve. I post the attachment.
Oh, ouch.

You can't by any chance test, if that still happens with the current git main branch? (FpcUpDeluxe).

Or run the Lazarus IDE in lldb, and when you the IDE crashes, get a trace from lldb? (would be a start, may need more)

Otherwise, how big, and how private is the app (the entire bundle) on your disk? Could it be mailed?


Quote
Talking about lldb, I tried to launch the application from lldb but in this case the Access Violation doesn't appear (as when I launch it from Lazarus in debug).

You don't need to launch the app in lldb.

When you got the trace, load the app in lldb (but don't start it).

Then for each address try:
Code: Text  [Select][+][-]
  1. (lldb) image lookup -a 0x7fff89e6752e
  2.       Address: libsystem_kernel.dylib[0x000000000001152e] (libsystem_kernel.dylib.__TEXT.__text + 68126)
  3.       Summary: libsystem_kernel.dylib`mach_msg_trap + 10

gitPr0bit

  • New Member
  • *
  • Posts: 11
Re: Access violation with useless stack trace on macOS M1
« Reply #11 on: June 05, 2023, 05:31:27 pm »
- open: Menu: View > Leaks and traces
- paste the trace
- press "Resolve" (not sure if you need to select the app bundle, or the actual app inside).

I tried this steps too, since I read them in another recent post, but the only thing I get is an error from Lazarus when trying to resolve. I post the attachment.
Oh, ouch.

You can't by any chance test, if that still happens with the current git main branch? (FpcUpDeluxe).

Or run the Lazarus IDE in lldb, and when you the IDE crashes, get a trace from lldb? (would be a start, may need more)

Otherwise, how big, and how private is the app (the entire bundle) on your disk? Could it be mailed?


Quote
Talking about lldb, I tried to launch the application from lldb but in this case the Access Violation doesn't appear (as when I launch it from Lazarus in debug).

You don't need to launch the app in lldb.

When you got the trace, load the app in lldb (but don't start it).

Then for each address try:
Code: Text  [Select][+][-]
  1. (lldb) image lookup -a 0x7fff89e6752e
  2.       Address: libsystem_kernel.dylib[0x000000000001152e] (libsystem_kernel.dylib.__TEXT.__text + 68126)
  3.       Summary: libsystem_kernel.dylib`mach_msg_trap + 10


1. Yes I can test the last version, I'm just not sure about which one you are talking about (should I install FPC and Lazarus both on 'trunk' version from FPCUpDeluxe?)
2. While waiting your answer to the first point, I'm gonna try to launch Lazarus in lldb, great idea
3. I tried the lookup of the app in lldb after crash, as you suggested, and I got something for a couple of addresses. Here's the output
   
Code: Bash  [Select][+][-]
  1. (lldb) image lookup -a 0x00000001810DFE64
  2.       Address: SystemConfiguration[0x00000001810dfe64] (SystemConfiguration.__TEXT.__text + 102628)
  3.       Summary: SystemConfiguration`_SC_isInstallEnvironment + 16
  4. (lldb) image lookup -a 0x0000000102F16BBC
  5. (lldb) image lookup -a 0x0000000102E8704C
  6. (lldb) image lookup -a 0x0000000102EAE600
  7. (lldb) image lookup -a 0x0000000102EAFE90
  8. (lldb) image lookup -a 0x0000000102EB050C
  9. (lldb) image lookup -a 0x0000000102F30A34
  10. (lldb) image lookup -a 0x00000001810AFD60
  11.       Address: libSparse.dylib[0x00000001810afd60] (libSparse.dylib.__TEXT.__text + 329160)
  12.       Summary: libSparse.dylib`_SparseRefactorSymmetric_Float + 480
  13. (lldb) image lookup -a 0x50588001810DFF40
  14. (lldb) image lookup -a 0x933F8001810AFCF8
  15. (lldb) image lookup -a 0xCD55800102F7AE38
  16. (lldb) image lookup -a 0x0000000102F305F8
  17. (lldb) image lookup -a 0x00000001020C22BC
  18. (lldb) image lookup -a 0x00000001020B5870
  19. (lldb) image lookup -a 0x00000001020C836C
  20. (lldb) image lookup -a 0x00000001020B5840
  21. (lldb) image lookup -a 0x0000000180D5BF28
  22.       Address: libBLAS.dylib[0x0000000180d5bf28] (libBLAS.dylib.__TEXT.__text + 433720)
  23.       Summary: libBLAS.dylib`___lldb_unnamed_symbol1947 + 5676
4. Unfortunately the app is not personal and I'm not allowed to share it
« Last Edit: June 05, 2023, 05:44:32 pm by gitPr0bit »

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9754
  • Debugger - SynEdit - and more
    • wiki
Re: Access violation with useless stack trace on macOS M1
« Reply #12 on: June 05, 2023, 06:06:51 pm »
1. Yes I can test the last version, I'm just not sure about which one you are talking about (should I install FPC and Lazarus both on 'trunk' version from FPCUpDeluxe?)

You can use the existing FPC.
You only need the "trunk" version of lazarus. It should run fine with the existing fpc.

Then you can do the same "resolve" => And that will at least tell, if the bug in the IDE has been fixed.
(as you likely will have changed/recompiled your work, test in your current Lazarus, that you have a case that crashes)

Quote
2. While waiting your answer to the first point, I'm gonna try to launch Lazarus in lldb, great idea

That would be the next step, after the above. If the IDE still crashes (which we really don't want), then a trace of that crash might be the first step to find out how to fix it.


Quote
3. I tried the lookup of the app in lldb after crash, as you suggested, and I got something for a couple of addresses. Here's the output

If lldb does not find the addresses, then "resolve" likely wont either. (But still good to test with "trunk", so we know if or if not it crashes)

That is for "resolve" or lldb "image lookup" to work:

* You must have used the exact same compiled exe in which you got the trace. If you later recompiled the exe, the addresses could have changed.

* You must *not* have compiled as "relocatable". Afaik the is default. "relocatable" is usually used as a security feature, so exploits can not predict addresses.


If you use packages, then make sure they all have debug info.
=> Project Options > Additions and Overrides > Add custom option (Target: *):  -gw
(or -gw3 if you use Dwarf 3 / but for line numbers it makes no diff)


If you do use FpcUpDeluxe: You may want to get your self an FPC 3.2.3 (fixes branch).
It has many very valuable bugfixes (including issues where the compiler breaks your app).
You may even want to compile it with -gw (you can mix this with optimizations, if you want an optimized build / it is just for the line info).

FPC/RTL with -gw may slow down linking / so you may want to have 2 installs: with/without.

If you do build an fpc with -gw and the crash is (even if triggered by your code) somewhere in the RTL, then you will get better traces. (assuming any of the lookup techniques start to work, or you can get in the debugger).




If you can build and test your app on Linux...

There is a tool called valgrind.
Code: Text  [Select][+][-]
  1. valgrind --tool=memcheck yourapp
is a really great tool to find access violations.
Valgrind very often can report the fault even if it does not cause a crash. (e.g. dangling pointers / double free / ...)

But it is Linux only...

If you are not on Linux => Try heaptrc (again). But set the environment HEAPTRC="keepreleased"
(and then mix that with -gt variants).


Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9754
  • Debugger - SynEdit - and more
    • wiki
Re: Access violation with useless stack trace on macOS M1
« Reply #13 on: June 07, 2023, 12:55:49 pm »
I done some testing on Mac.

It seems Mac compiles "relocatable" by default. That means, the same code will be on a new address for each run (without recompile, just starting the same app again).

That means, once the app stopped, the addresses from the stack dump are useless.


I don't know the compiler (or probably linker) settings well enough. So I don't know how to turn that off.

If you can switch it off, then lldb (and hopefully the IDE's "resolve") should be able to translate the addresses.


Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9754
  • Debugger - SynEdit - and more
    • wiki
« Last Edit: June 08, 2023, 09:09:43 pm by Martin_fr »

 

TinyPortal © 2005-2018