I've recently done the same with BGRABitmap, and before with VCL/LCL.
While this works, increasing the font by just 1 is hitting your performance massively. I would recommend to use bigger steps, then trying to get closer by reducing step size.
Didn't do that improvement myself yet though, instead I implemented a cache for optimal sizes, since my texts are mostly static.