The main Lazarus *nix widgetsets only use X for some window management. The bulk of the work goes through either GTK or QT.
Wayland (and I assume Mir too) have a base X emulation, but even when that eventually would (a very big IF) disappear then it should be relative benign.
Maybe customdrawn needs some work, otoh it probably doesn't really work with detail X primitives and just cranks out bitmaps.
So nothing to worry about.