OK, so you're on Linux (you should've mention it earlier). On platforms other than Windows, FPC relies on GNU linker for linking (I don't know how's the progress of ELF internal linker).
Smartlinking is indeed time and memory consuming on these platforms because when enabled, FPC creates a small assembly for for EACH variables and routines, which is zipped in a .a file. This is done because GNU linker can only remove unused files, while the term smartlinking in FPC means remove everything that's not used.
Create smartlinkable units (-CX): only affect compiling process, so the difference won't be big.
Link smart (-XX): Heavy step, GNU linker might need to load hundreds to thousands or even more object files to memory. This what's make the difference big.
I still need to cut down 2-3 seconds
Disable both -CX and -XX, recompile LCL (and RTL + FCL) if you've built them with smartlinking on.