* * *

Author Topic: Precision of COMP  (Read 1563 times)

Elap

  • Jr. Member
  • **
  • Posts: 52
Precision of COMP
« on: February 22, 2016, 11:59:43 am »
When I run the program below in Turbo Pascal, I get the fifth power as 30973432405238707, but in FP 2.6.4 I get 30973432405238700.

In general I don't seem to get the number of significant digits that I would expect for large numbers like these.

Have you any idea why this discrepancy occurs?  I am using Windows 7 Professional and use the console IDE.

Code: Pascal  [Select]
  1. {$R+}
  2. PROGRAM BUG3;
  3.  
  4. USES
  5.     CRT;
  6.  
  7. VAR
  8.     I_comp:      COMP;
  9.     I_comp5:     COMP;
  10.  
  11. BEGIN
  12.     CLRSCR;
  13.  
  14.     I_comp:=1987;
  15.     I_comp5:=I_comp*I_comp*I_comp*I_comp*I_comp;
  16.  
  17.     WRITELN(I_comp:4:0,' to the fifth power is ',I_comp5:17:0);
  18.     READLN;
  19. END.
  20.  

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 5641
Re: Precision of COMP
« Reply #1 on: February 22, 2016, 12:14:47 pm »
I use FPC 3.0 and 3.1.1 both show  *707, the value according to calculator.


BeniBela

  • Hero Member
  • *****
  • Posts: 570
    • homepage
Re: Precision of COMP
« Reply #2 on: February 22, 2016, 12:51:59 pm »
Floating point conversions were completely broken in 2.6.x

Still not working optimally in 3.x, but now it is too precise

Elap

  • Jr. Member
  • **
  • Posts: 52
Re: Precision of COMP
« Reply #3 on: February 22, 2016, 01:06:17 pm »
Thank you for that.

I'm using Qword which seems to be fine.

BeniBela

  • Hero Member
  • *****
  • Posts: 570
    • homepage
Re: Precision of COMP
« Reply #4 on: February 22, 2016, 01:07:56 pm »
That becomes buggy if you do range checked calculations over high(int64) ...

 

Recent

Get Lazarus at SourceForge.net. Fast, secure and Free Open Source software downloads Open Hub project report for Lazarus