I can related to that JuhaManninen.
I'm more into retro/SBC platforms and also have a B202 (though I need to replace the fan as unfortunately that died on me). I used to use the B202 as a daily (development) driver till it was becoming unbearable to do so for similar reasons.
The sleek compilation for/of FPC/Lazarus is indeed one of its strengths. There isn't even a comparison possible, (re)compiling things with gcc on the same hardware requires hours instead of minutes when using lazarus/fpc. And yes gcc build system is much more sophisticated and has more things to take into account (but that is their approach).
I test and develop on low hardware specs because if it can run there it can run about anywhere. I guess it is a niche these days to try and (micro)optimize your code. I see more and more people with specs that are ridiculous for development but don't even know how (to begin) to utilize such specs to their full extent.
Though you might find the raise in compilation time and memory use alarming, I am personally not /that/ concerned. We are talking about a development cycle of years for all these versions to appear and it does not compare to the usual bloat of other software that is released these days. If you compare these softwares using the same (overlapping) time-period then you can see that percentage wise fpc/lazarus is doing pretty well. Don't forget that the rtl and components library get extended as well (I don't believe I have seen the count of API calls and component numbers decrease in all these years).
One thing that definitely impacts compilation/build time is the use of generics (takes up a fair amount of memory during compilation as well). I am ware it is a trade off but it is one that is reasonably fair (see also c and their macros/templates).
Note, there are 2 different build systems for the Lazarus IDE. Makefiles and a built-in system in the IDE. Did the slowdown happen in both?
The situation as described by me earlier is about the IDE. E.g. the normal daily use someone would have on a pi: Develop a program, decide that you need a 3th party component and install it (which invokes the IDE rebuild). If that is your use of/for the SBC such as a pi3 (or similar specs) then I strongly recommend against doing so. It is no problem using Lazarus as a 'normal' editor and compile/build suitable projects.