I only used the framebuffer directly with FPC on Raspberry Pie's.
(I wrote some demo's that are possibly still available from the RPI websites somewhere, translated C code to Freepascal.)
But, probably, yes, although it might prove to be a mammoth task.
In effect writing a kind of X. But Possible.
A good starting point is possibly the custom drawn widget set.