Does it happen during compile or linking?
If "linking", then check if smartlinking is enabled. Smart-linking an app as big as the IDE can require serious amount of memory. (and in case of the IDE does not win you any noticeable savings).
The flag can be in fpc.cfg, in the IDE "config build Laz" dialog, or maybe in the ide/lazarus.lpi file.
If "compiling", and doing so from inside the ide, check (tools/options) how many compilations run in parallel. Set it to max one.