I had no idea setting tabstop=false had so little effect.
It can be solved by set the Key to 0 in the OnKeyDown event. Tested on Linux GTK2, it works.
procedure TForm1.FormKeyDown(Sender: TObject; var Key: Word; Shift: TShiftState
);
begin
if (Key in [37..40, 49..57, 97..105]) then Key := 0;
end;