Well, correct, it seems that you're the first one trying to do this with FreeVision... I suggest that you file a bug report, because the current implementation of TProgram SetScreenMode can indeed never change the resolution.
To be honest, usefulness of this compatibility method is very limited in a world without fixed screen mode numbers, but it should work at least for the modes for which predefined constants are supplied, which is exactly what you tried to do.
In the meantime, use TProgram.SetScreenVideoMode instead - it expects a TVideoMode record (see unit Video for its definition) which allows you to select the wanted number of rows and columns. Note that it does not support any arbitrary resolution - it knows some predefined modes available on all IBM PC machines (although even that is obviously not guaranteed to be available on all FPC supported platforms, which is the reason why the FP IDE code asks for confirmation whether it works correctly after trying to switch to the selected mode) plus it adds the resolution available on startup if it can detect it on the used platform and it is different from the predefined ones (this is obviously based on the assumption that if the resolution worked on startup of the FPC compiled program, it should be possible to change to this resolution later too). See unit Video for further information on how to walk through the recognized video modes / resolutions.