128MB is extremely low, I already had more in the nineties.
You don't mention OS, this matters specially when smartlinking, the memory consumption of LD can go through the roof. (1GB+ is possible).
On Windows, the last time I checked for memory requirements, the internal linker eats about 250MB when linking lazarus with smartlinking.
I know it is less without smartlinking, but I don't know those figures by heart.