Nope, it is the way I develop at home. My main development machine is XP running xwin32 and lazarus running on a FC2 machine.
However if you have a slow network connection, you may get better results with (tight)vnc (I develop remote on sparc and amd64 on a 128kbit link)
Mousemovements cause a lot of X events, which can slow down your app when running X remotely.