Then I changed the build mode to "Release" - but nothing changed.
That wouldn't have changed the relevant libs/so files. => You don't rebuild the gtk.so files, if they had (included, or as separate file) any debug info (such that gdb could find it), then gdb would find it, even if the main exe was build without debug info.
Does the memory footprint of the gdb process increase during the slow down? Because if gdb loads debug info, that requires significant amount of memory.
There are plenty of other reasons for a process to be slow under a debugger. Those are not necessarily a fault of the process itself.
E.g., if the process (any code in the process, including libs/so) performs certain tasks (such as loading libs, triggering signals /sig..., eg sighup etc) then those signals may first be seen by the debugger.
Normally that is only a few ticks... But if the process triggers that a million times.... Well, unlikely ....
You can check
menu: View > Ide internals > Debug output
for any feedback gdb prints.
Maybe it will have a hint.
Anyway, you said gdb is at high cpu usage during this time.
You could try fpdebug instead (search forum / announcements / wiki). Maybe you get better results. Depends what exactly is happening.