A very simple code test I just did, adding integers together, is sometimes faster in the 32bit compiler, and sometimes faster in the 64bit compiler.
Well, there are lots of factors. Both code and data may change in size and by that may be differently aligned, and that can make surprising speed differences.
Say you have a loop in your code, then its alignment depends on how much other code is in front => but depending on its size, if the loop has a specific alignment, it may run faster.
In the end it may happen more likely under 32 bits.
Similar stuff can happen with data, if data that is used together ends up in a single cache line.
There are some directives to set up alignment for those things. You can play around with that. In some cases alignment to boundaries up to 32byte bounds can change speed. But its not a guarantee. It can go the other way to.