I don't have a Mac for testing myself. But a few "maybe" pointers...
"The debugger" => On Mac consist of 2 parts: LLDB and FpDebug.
For starters I don't know if there is anything affecting LLDB, such as maybe if x86 code needs to be run in emulation... I don't know your setup. Or if that is even still possible, or if only M-CPU code will run.
The only other thing that I can think of, is the amount of debug info. You can check if packages and LCL are compiled with debug info, and if you don't need that, then change it (probably remove from "configure build Lazarus", but maybe other places).
You can check if it is lldb, by trying "LLDB alpha" without FpDebug (it be crappy debugging, but it shows the speed part of LLDB). Or even try LLDB from command line.
If it is FpDebug, then the question is what changed since 4.6 ?? May I assume it worked faster in 4.6? Are the exe-bundles of the same size? I.e. is the amount of debug info roughly the same?
Also you could try running Lazarus from a console/terminal. So you get to see its STDOUT/STDERR. Maybe it prints some error messages that provide more info.