The next question is "which debugger".
The IDE has several. On Windows there are 2 debuggers commonly used.
Menu: Tools > Options > Debugger > Debugger backend.
Check for FpDebug or GDB.
Try the other one, to see if it matters.
In gdb, you can "DisableLoadDebugInfoForLibraries" (in the property grid, on the debugger option page). If you use many libraries, and they have all debug info, then this may speed things up.
Having said that, starting the IDE itself in the debugger (either of the 2) takes less time than you experience. So I wonder how much "fairly complex" is.