Recent

Author Topic: Rounding from StrToFloat  (Read 2230 times)

Weitentaaal

  • Sr. Member
  • ****
  • Posts: 328
  • Weitental is a very beautiful garbage depot.
Rounding from StrToFloat
« on: October 12, 2021, 01:54:39 pm »
Hello Guys,

I have recently often had the problem that I have received values that I would not have expected. I imported the same code from my old project (VB6) and got different results.

Here the code in VB6 :
Code: Pascal  [Select][+][-]
  1.     If VAl(Edit1.Text) > 0 Then
  2.        SomeVar = Int(VAl(Edit1.Text) * VAl(Edit2.Text) * 1.2 * 2.5 / 1000)
  3.       Else
  4.        If InStr(Combo1.Text, "100") > 1 Then
  5.           WMenge.Text = Int(6 * VAl(Edit2.Text) * 1.2 * 2.5 / 1000)
  6.          Else
  7.           WMenge.Text = Int(7.5 * VAl(Edit2.Text) * 1.2 * 2.5 / 1000)
  8.        End If
  9.     End If
  10.  

and here in Lazarus:
Code: Pascal  [Select][+][-]
  1.       If StrToFloatDef(Edit1.Text, 0) > 0 then Begin
  2.          SomeVar := int(StrToFloatDef(Edit1.Text, 0) * StrToFloatDef(Edit2.Text, 0) * 1.2 * 2.5 / 1000);
  3.       End Else Begin
  4.           If Pos('100', Combo1.Text) > 1 then Begin
  5.               SomeVar  := int(6 * StrToFloatDef(Edit2.Text, 0) * 1.2 * 2.5 / 1000);
  6.           End Else Begin
  7.               SomeVar  := int(7.5 * StrToFloatDef(Edit2.Text, 0) * 1.2 * 2.5 / 1000);
  8.           End;
  9.       End;
  10.  

Edit1 --> 0
Edit2 --> 3000
Combo1 --> "KA100"

Results: VB6 --> 53 Lazarus --> 54

Thanks in advice :)
« Last Edit: October 12, 2021, 02:03:36 pm by Weitentaaal »
Lazarus: 2.0.12 x86_64-win64-win32/win64
Compiler Version: 3.2.0

wp

  • Hero Member
  • *****
  • Posts: 9051
Re: Rounding from StrToFloat
« Reply #1 on: October 12, 2021, 02:54:02 pm »
I dont't know about the numerical routines used by VB6, but it could be that their result is 53.9999999999 while the Lazarus result is 54.000000001. Since you take the integer part of the floatingpoint result you get 53 for VB and 54 for Lazarus. If you use round() in both cases, rather than int(), you will get the same result, 54.

I cannot remember whether VB has a round() function. If not, add a small number (say: 0.01) to the result of the calculation and apply int() then:
Code: [Select]
// VB6
    If VAl(Edit1.Text) > 0 Then
       SomeVar = Int(VAl(Edit1.Text) * VAl(Edit2.Text) * 1.2 * 2.5 / 1000 + 0.01)
      Else
       If InStr(Combo1.Text, "100") > 1 Then
          WMenge.Text = Int(6 * VAl(Edit2.Text) * 1.2 * 2.5 / 1000 + 0.01)
         Else
          WMenge.Text = Int(7.5 * VAl(Edit2.Text) * 1.2 * 2.5 / 1000 + 0.01)
       End If
    End If
Mainly Lazarus trunk / fpc 3.2.0 / all 32-bit on Win-10, but many more...

Weitentaaal

  • Sr. Member
  • ****
  • Posts: 328
  • Weitental is a very beautiful garbage depot.
Re: Rounding from StrToFloat
« Reply #2 on: October 13, 2021, 09:04:35 am »
so i guess its not my fault then.

"If not, add a small number (say: 0.01) to the result of the calculation and apply int()"

-0.01 will fix it in Lazarus then i guess :) Anyway i understand where the problem is.

 just wanted to make sure i did not mistake something, thank you ! :)
Lazarus: 2.0.12 x86_64-win64-win32/win64
Compiler Version: 3.2.0

jamie

  • Hero Member
  • *****
  • Posts: 5052
Re: Rounding from StrToFloat
« Reply #3 on: October 13, 2021, 03:50:05 pm »
I just tested an old delphi D3 and it too also displays 54.

it could be intentional to help with currency types.

The only true wisdom is knowing you know nothing

wp

  • Hero Member
  • *****
  • Posts: 9051
Re: Rounding from StrToFloat
« Reply #4 on: October 13, 2021, 04:43:33 pm »
-0.01 will fix it in Lazarus then i guess :) Anyway i understand where the problem is.
Seriously? If yes, then you did not understand the idea. The point is that a floating point calculation which is supposed to return an integer may be slightly off by a tiny amount due to rounding errors
Code: Pascal  [Select][+][-]
  1. program Project1;
  2. const
  3.   sqrt3: double = sqrt(3.0);
  4. var
  5.   x: Double;
  6. begin
  7.   x := sqr(sqrt3);
  8.   WriteLn(x);                // ---> output: 2.9999999999999997E+000
  9.   WriteLn(int(x));
  10.   ReadLn;
  11. end.
  12.  
The result can either be a bit too small, or a bit too large. When the result is too large - fine. But there is a problem with the int() or trunc() function when the value is too small: The above example outputs the value  2.9999999999999997E+000, rather than 3.0E+000. When I call int(), or trunc() for this number the result will be 2, very wrong compared with the expected 3. Now, by adding a tiny number to  2.9999999999999997E+000 brings the sum over 3.0 and the int() will return the correct result. This does not depend on programming language.

BTW, be careful with negative values. Suppose we want to WriteLn(-sqr(sqrt3)). Do we have to ADD a small number here, too?
Mainly Lazarus trunk / fpc 3.2.0 / all 32-bit on Win-10, but many more...

Weitentaaal

  • Sr. Member
  • ****
  • Posts: 328
  • Weitental is a very beautiful garbage depot.
Re: Rounding from StrToFloat
« Reply #5 on: October 14, 2021, 09:51:06 am »
Hey Thanks :)

"BTW, be careful with negative values. Suppose we want to WriteLn(-sqr(sqrt3)). Do we have to ADD a small number here, too?"

I guess not bc its going backwards ?  :-[

"Seriously? If yes, then you did not understand the idea. The point is that a floating point calculation which is supposed to return an integer may be slightly off by a tiny amount due to rounding erro"

was just in a hurry and didn't fully think of what i wrote, Sorry.

i do this now before every int operation:

int(X * 100 + 0.001) / 100

i hope thats not wrong
Lazarus: 2.0.12 x86_64-win64-win32/win64
Compiler Version: 3.2.0

wp

  • Hero Member
  • *****
  • Posts: 9051
Re: Rounding from StrToFloat
« Reply #6 on: October 14, 2021, 10:28:22 am »
i do this now before every int operation:

int(X * 100 + 0.001) / 100

i hope thats not wrong
Only when X is > 0. When X < 0 you must SUBTRACT 0.001.

There is also the question what you want to do when the result of the calculation (X) is not supposed to be an integer, but, maybe, 2.6. Should the result, after the int(), be 2 or 3? Depending on the answer you'll have to follow a different strategy.

In Pascal you certainly have several rounding/truncation functions that you can use straight-away:
- trunc(): cuts off the fractional part, like int(): trunc(2.999997) = 2, trunc(-2.999997) = -2
- round(): rounds up to the next larger integer, or down to the next smaller integer, depending on which is closer:
    round(2.6) = 3, round(-2.6) = -3, round(2.4) = 2, round(-2.4) = -2
- ceil(): always rounds to the next larger integer (ceil(2.999997) = 3, ceil(-2.9999997) = -2)
- floor(): always rounds to the next smaller integer (floor(2.999997) = 2, floor(-2.9999997) = -3)

In VB6 you maybe should write these functions yourself (but I don't know, maybe they already exist).
Mainly Lazarus trunk / fpc 3.2.0 / all 32-bit on Win-10, but many more...

 

TinyPortal © 2005-2018