It has nothing to do with OnGetEditMask. StringGrid contains strings.
I assune you have some routine where you fill the grid with data. You can do:
var f: Double;
...
if f>99.9 then f:=99.9;
Cells[2, y]:=FloatToStrF[f, ffFixed, 0, 1];
If user inserts data directly to cell (i.e. grid is not read-only), you can use any cell editor (TFloatSpinEdit should be fine). Or use OnEditingDone to correct inserted value immediately after editing.