Note that in general for remote X machines, you installed no X, but then manually only the parts needed for remote use and a simple manager like window maker.
I did this for quite a while when working at the university that had Hummingbird Xserver licenses in the early 2000s. I Xed home over putty's X forwarding to mostly FreeBSD machines.
Iirc it mostly worked fine, but managing Lazarus multiple windows sometimes was a chore. Starting mozilla (then just released "netscape 6") was very slow though, probably because mostly owner drawn.