Recent

Author Topic: Precision of COMP  (Read 3149 times)

Elap

  • Jr. Member
  • **
  • Posts: 52
Precision of COMP
« on: February 22, 2016, 11:59:43 am »
When I run the program below in Turbo Pascal, I get the fifth power as 30973432405238707, but in FP 2.6.4 I get 30973432405238700.

In general I don't seem to get the number of significant digits that I would expect for large numbers like these.

Have you any idea why this discrepancy occurs?  I am using Windows 7 Professional and use the console IDE.

Code: Pascal  [Select]
  1. {$R+}
  2. PROGRAM BUG3;
  3.  
  4. USES
  5.     CRT;
  6.  
  7. VAR
  8.     I_comp:      COMP;
  9.     I_comp5:     COMP;
  10.  
  11. BEGIN
  12.     CLRSCR;
  13.  
  14.     I_comp:=1987;
  15.     I_comp5:=I_comp*I_comp*I_comp*I_comp*I_comp;
  16.  
  17.     WRITELN(I_comp:4:0,' to the fifth power is ',I_comp5:17:0);
  18.     READLN;
  19. END.
  20.  

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 6628
Re: Precision of COMP
« Reply #1 on: February 22, 2016, 12:14:47 pm »
I use FPC 3.0 and 3.1.1 both show  *707, the value according to calculator.


BeniBela

  • Hero Member
  • *****
  • Posts: 634
    • homepage
Re: Precision of COMP
« Reply #2 on: February 22, 2016, 12:51:59 pm »
Floating point conversions were completely broken in 2.6.x

Still not working optimally in 3.x, but now it is too precise

Elap

  • Jr. Member
  • **
  • Posts: 52
Re: Precision of COMP
« Reply #3 on: February 22, 2016, 01:06:17 pm »
Thank you for that.

I'm using Qword which seems to be fine.

BeniBela

  • Hero Member
  • *****
  • Posts: 634
    • homepage
Re: Precision of COMP
« Reply #4 on: February 22, 2016, 01:07:56 pm »
That becomes buggy if you do range checked calculations over high(int64) ...