tIniFile Float handling error - probably much deeper

[jollytall]

I have the following simple program:

Code: Pascal  [Select][+][-]

1. var
2.   IniFile : tIniFile;
3.   MaxFloat : double;
4. begin
5. IniFile := tIniFile.Create('test.ini');
6. MaxFloat := IniFile.ReadFloat('test', 'maxfloat', tDoubleHelper.MaxValue);
7. IniFile.WriteFloat('test', 'maxfloat', MaxFloat);
8. IniFile.Free;
9. end.

If you run it once, it is OK, it creates the ini file and in it is the max value of double. However if you run it again, you get a floating point error (an error 8 and an error code 205). Clearly ReadFloat cannot read back what WriteFloat happily wrote out.

I looked into the source code, and WriteFloat uses FloatToStrFIntl and ReadFloat uses TextToFloat and in it Val. There inside it gets more complicated so I could not figure out whether the write writes out something illegal (actually in the file it seems OK) or the read cannot handle it (this is more probable). Also I used tDoubleHelper.MaxValue many times in writeln instructions and that also works (I guess it uses the same float to text conversion somewhere deep in the compiler). So to me it seems that the TextToFloat has a problem (cannot handle this number).

Any easy way to fix it?

I have FPC 3.2.2

[jamie]

That's what you get for calling a helper directly, its not intended to be used that way.

Create a dummy Double or something like.

0.0.maxValue;

At least that has an instance.

[jamie]

I just tried your method over here and got what appears to be correct results but, I did this from the 32 bt compiler.

In any case, the helper isn't a Class or Record so I can't see how the Compiler even allowed that?

Try this directly instead.

Code: Pascal  [Select][+][-]

1. Double(0.0).MaxValue;
2.  

;

Or maybe the IINIFILE is using single floats?

[jollytall]

Thanks Jamie. In the meantime I could localize the bug, but before to your points.

I tried:

Code: Pascal  [Select][+][-]

1. var
2.   IniFile : tIniFile;
3.   MaxFloat : double;
4. begin
5. MaxFloat := 0.0.MaxValue;
6. IniFile := tIniFile.Create('test.ini');
7. MaxFloat := IniFile.ReadFloat('test', 'maxfloat', MaxFloat);
8. IniFile.WriteFloat('test', 'maxfloat', MaxFloat);
9. IniFile.Free;
10. end.

and it does not crash, but not correct either. In the file I get a max single.

See the next code:

Code: Pascal  [Select][+][-]

1. var
2.   MaxFloat : double;
3. begin
4. writeln(tDoubleHelper.MaxValue);
5. MaxFloat := tDoubleHelper.MaxValue;
6. writeln(MaxFloat);
7. MaxFloat := 0.0.MaxValue;
8. writeln(MaxFloat);
9. MaxFloat := 1;
10. MaxFloat := MaxFloat.MaxValue;
11. writeln(MaxFloat);
12. end.

The result is (on a 64 bit Linux):

Code: [Select]

1.7976931348623157E+308
1.7976931348623157E+308
3.4028234663852886E+038
1.7976931348623157E+308
So, when tDoubleHelper.MaxValue or DoubleVariable.MaxValue is used it is indeed a max double, but 0.0.MaxValue calls probably a tSingleHelper.MaxValue.

Going back to the original problem

Code: Pascal  [Select][+][-]

1. var
2.   IniFile : tIniFile;
3.   MaxFloat : double;
4. begin
5. MaxFloat := 1;
6. MaxFloat := MaxFloat.MaxValue;
7. IniFile := tIniFile.Create('test.ini');
8. MaxFloat := IniFile.ReadFloat('test', 'maxfloat', MaxFloat);
9. IniFile.WriteFloat('test', 'maxfloat', MaxFloat);
10. IniFile.Free;
11. end.

crashes the same way, so clearly the problem is not with using tDoubleHelper directly, but with the fact that ReadFloat cannot read back the value WriteFloat wrote. In the Ini File I see

Code: [Select]

[test]
maxfloat=1.79769313486232E308and that is indeed more than the max double and hence a floating point error. The problem is that when WriteFloat writes out the number it rounds it to 14 digits, but so unfortunate that the last digit is rounded upwards (compare it to the writeln result above). If I manually change in the IniFile the last digit, then it reads it back as expected.
So the bug is in the FloatToStrFIntl, it simply should never round a number above the range.

[jamie]

Ok, I looked at the source for the IniFile, looks like its using a function that is within the system.inc file.

Oh well.

[Sieben]

MaxValue is a public constant of TDoubleHelper, so no problem there (and those helpers usually have a set of class functions/procedures too that can also be used directly without an instance).

[Bart]

It seems that Ini.ReadFloat can read a maximum value of appr. 1.7976931348623155e+308 (which is appr. TDoubleHelper.MaxValue / 1.000000000000001).

Bart

[Bart]

Reported as Issue 40972.

Bart

[jollytall]

Bart,

Thank you for opening a ticket, but I think the title and the "workaround" are misleading. Actually it is the other way round. WriteFloat canNOT write MaxValue out correctly. It is rounded to 14 decimals, i.e. 1.79769313486232E308, what is really more than MaxValue and hence the ReadFloat has all the right to refuse it in the form of Float Overflow.
Writeln(tDoubleHelper.MaxValue) writes out 16 decimals, i.e. 1.7976931348623157E+308 and basically that is the maximum precision driven by the 52 bits used to indicate the decimals for a double value.

What can be done to test it, is to run my program and as you also saw, it writes out the 14 digit version.
Now we can play with the ini file with any text editor. If I change it to the number you suggest in the workaround, it will work but incorrectly. Seemingly it works, because the written number can be read back, but it will be incorrect, not being equal to MaxValue (actually the reason, why people often use MaxValue as a synonym to invalid value). Your workaround only works, because the number you suggested has a last digit of 1, when written to the ini file rounded to 14 decimals. And that number is indeed smaller than MaxValue, so can be read (but not equal).
However if we manually change the ini file to the 16 decimal version, then ReadFloat can still read it AND it also finds it equal to MaxValue, as this number is binary converted to the same 64 bit representation as MaxValue is stored internally.

So the only error I think is that WriteFloat writes 14 digits and not the necessary 16 bits. I think I even found the error. It is in sysstr.inc line number 1734, where there is a hard coded 15 precision (1. + 14 decimals?) and if it was 17 then probably the number would be written with enough decimals. However that is a read only include file and FloatToStrFIntl cannot be called directly, so I could not call it with the right precision, but could mimic it with the str procedure used inside FloatToStrIntl and it works. So the bug is extremely easy to fix, just change line 1734 from 15 to 17.
Also, WriteFloat might write an Extended value, in which case also 15 is hardcoded in line 1704. There probably an even higher number should replace the 15. Sorry, WriteFloat can only handle Double (os it probably should be called WriteDouble(), but for other uses of FloatToStr the crossed-out sentence is still correct. There 14 decimals (Precision 15) is far too low.

I used the very simple program to test it:

Code: Pascal  [Select][+][-]

1. var
2.   IniFile : tIniFile;
3.   MaxFloat : double;
4.   s : string;
5. begin
6. writeln(tDoubleHelper.MaxValue);
7. IniFile := tIniFile.Create('test.ini');
8. MaxFloat := IniFile.ReadFloat('test', 'maxfloatasstring', tDoubleHelper.MaxValue);
9. writeln(MaxFloat, ' ', tDoubleHelper.MaxValue, ' ', MaxFloat=tDoubleHelper.MaxValue);
10. MaxFloat := IniFile.ReadFloat('test', 'maxfloat', tDoubleHelper.MaxValue); // << crash here at the second run
11. writeln(MaxFloat, ' ', tDoubleHelper.MaxValue, ' ', MaxFloat=tDoubleHelper.MaxValue);
12. Str(Double(Extended(Aligned(tDoubleHelper.MaxValue))):17+7, s);
13. IniFile.WriteString('test', 'maxfloatasstring', s);
14. IniFile.WriteFloat('test', 'maxfloat', tDoubleHelper.MaxValue);
15. IniFile.Free;
16. end.
17.  

[jamie]

After playing with it myself, it seems the actual problem is what you said before, rounding issues.


 It's not the Inifile code for that is just a victim of the problem.

I don't know where the rounding comes from when it generates or decodes from a string but in this case, it shouldn't be rounding down not up.

 That's just my 2 cents worth, I was able to reproduce the issues simply.

Code: Pascal  [Select][+][-]

1. procedure TForm1.Button1Click(Sender: TObject);
2. Var
3.   F:Double;
4.   S:String;
5. begin
6.    f:= tdoublehelper.MaxValue ;
7.    S:= FloatToStr(F);
8.    F:= StrToFloat(S);
9.    caption := S;
10. end;
11.  

That will crash with an overflow.

[jollytall]

Jamie,
(I guessed that it is not in the ini code from the beginning, hence the title.)
Please check my last post above. As written there, I found clearly the location of the error. It is the Double variant of FloatToStr using too few decimals. Even there is a workaround if I convert manually the number to string with the same routine used deep under WriteFloat and FloatToStr (i.e. Str()), but with a larger "precision" value.
So, the fix of the bug would be to change a value 15 to 17 in the code at the indicated line. As the FloatToStr conversion routine has other variants (e.g. for Extended too), although I do not know where that is used, I would propose to correct that one to an even higher precision number as well.
Btw. that is not a solution, what you suggest, i.e. that max double should not be rounded up, but down. Not only because I guess that would be a much more complex task (how to decide what is rounded and what is not) to correct and test than a simple formatting length. It would also be incorrect because then that rounded-down number would be another number (I already shown it earlier regarding Bart's workaround) and if you read it back then it will not agree to tDoubleHelper.MaxValue. So if one uses MaxValue to indicate an incorrect/unset value (what I do) then the read back value will not be found the "incorrect value", but will be a correct, albeit very large number.

[Thaddy]

floattostr seems to be wrong since

Code: Pascal  [Select][+][-]

1. {$mode objfpc}
2. uses sysutils;
3. begin
4.  writeln(f.maxvalue.tostring);
5. end.

works.

[jollytall]

Thaddy,

I do not get what you say. Is your code using FloatToStr under the hood, or not?
To me it seems that it does use it, and actually your code proves it, that it is wrong. If I run your code then my result is:

Code: [Select]

1.79769313486232E308what is again the 14 decimal rounded number that is actually more than Double.MaxValue. So if you would want to convert it back, it would fail.

[MathMan]

I do not get what you say. Is your code using FloatToStr under the hood, or not?
To me it seems that it does use it, and actually your code proves it, that it is wrong. If I run your code then my result is:

Code: [Select]

1.79769313486232E308what is again the 14 decimal rounded number that is actually more than Double.MaxValue. So if you would want to convert it back, it would fail.

You are correct - and I find that worrying. Iirc - and that is a big /if/, as I haven't read the 2018 version of the relevant IEEE std - IEEE even recommends 18 decimal digits.

Related to your discovery - we seem to have a general issue with FloatToStr for numbers in close proximity of MAX_SINGLE / MAX_DOUBLE. The point is how this works. Usually the value gets converted to the maximal required decimal digit number (17 in this case). After this the decimal representation then gets rounded /in decimal/ to the requested target precision (15 in this case). This must not be done for numbers in proximity of the maximum, where proximity is dependent on the target precision in decimal.

As said this is how it usually works. I don't know if the FPC RTL follows this approach, but your discovery seems to indicate as much.

Cheers,
MathMan