I’ve only recently switched to FPC x86_64. Until now, I’d been using the 32-bit version, though I occasionally did cross-compilation to 64-bit for certain projects. Until the switch, I didn’t notice any shortcomings with the 32-bit version, but eventually, due to the large number of components, the Lazarus IDE stopped compiling for me, and I had to switch to x86_64.
If you use 32-bit libraries in your projects, I wouldn’t switch. If you use resources from rc files in your projects, I wouldn’t switch either - there’s a bug in stable version, that’s fixed in the fixes and trunk branches.