What you want to achieve needs the font to be scaled in pixels, not points. That means the font needs to have a negative value (pixels) instead of points (positive value). That is the ONLY way to be accurate.
Formula's are like this:
(Given LOGFONT structure on Windows)
lfHeight := -MulDiv(PointSize, GetDeviceCaps(hDC, LOGPIXELSY), 72); // pixels
PointSize := MulDiv(-lfHeight, 72, GetDeviceCaps(hDC, LogPixelsY);// points
But note that for high resolution screens you should call GetDeviceCaps first. The above formulae are for Windows only, but may be Lazarus supports it cross-platform.