For ARM, it's rather complicated. The resulting compiler doesn't carry a bunch of CPU/ABI/FPU mode used to build it, instead it has its own default. Therefore, you have to use the same set of options when building your project.
On a side note, I belive on RPi, if you use Raspbian, you will need to target ARMHF instead of ARMEL.