Lazarus

Free Pascal => General => Topic started by: rossh_lz on November 21, 2020, 01:56:56 pm

Title: static array [0..0] now fail at run time
Post by: rossh_lz on November 21, 2020, 01:56:56 pm
The old method of defining a static type of array with an open ended size, now fails a runtime. 

sample:
  PCells = array [0..3] of Byte;
  TPArray = array [0..0] of PCells;
  PTPArray = ^TPArray;

The TPArray array can be sized to suit with GetMem, or assigning existing mem to PTPArray.


Now, as of FPC 3.2.0, I get range check fatal error on every attempt to access an array index > 0.  Code that still runs OK in Delphi, and for the last 12 years in Lazarus, now fails.

Range checking is not active (and never affected this anyway).

Why the implied range checking now here?  How to get rid of it?

Thanks.
Title: Re: static array [0..0] now fail at run time
Post by: Jonas Maebe on November 21, 2020, 02:15:42 pm
The range check at compile time was added because it has always generated code that gave unexpected results in some cases (even when range checking was inactive). You can find a very long thread about this at https://forum.lazarus.freepascal.org/index.php/topic,44655.0.html . It seemed more prudent to give a compile time error than to silently generate unexpected code.

Since both FPC and Delphi support indexing pointers as arrays, the portable and safe way to implement what you want is to declare PTPArray as ^PCells instead. If you're using {$mode delphi}, and for Delphi proper, you also need to enable the pointermath switch (http://docwiki.embarcadero.com/RADStudio/Sydney/en/Pointer_Math_(Delphi)). You should not need to make any other changes to your code.
Title: Re: static array [0..0] now fail at run time
Post by: marcov on November 21, 2020, 02:21:13 pm
Quote from: rossh_lz on November 21, 2020, 01:56:56 pm
The old method of defining a static type of array with an open ended size, now fails a runtime. 

You are comparing with Delphi, I assume you have enabled delphi mode?
Title: Re: static array [0..0] now fail at run time
Post by: nanobit on November 21, 2020, 02:22:25 pm
Quote from: rossh_lz on November 21, 2020, 01:56:56 pm
sample:
  PCells = array [0..3] of Byte;
  TPArray = array [0..0] of PCells;
  PTPArray = ^TPArray;

You can define a larger range:
type TPArray = array[0..(high(sizeint) div sizeof(PCells))-1] of PCells;

if you need also negative indices, then you need pointermath instead.
type {$PointerMath on} PTPArray = ^PCells; {$PointerMath off}
But FPC has a known bug here (if item is static array, issue 35745) under {$modeswitch autoderef} and delphi mode:

Then "var pArrays: PTPArray;" has this pointermath bug:
pArrays[1] is next byte, but should be next array (PCells).
( pArrays^[1] (non-pointermath expression) is correct for returning a byte)
Workaround: (pArrays + 1)^ is next array.

@pArrays[1] refers to next byte, but should refer to next array.
Workaround: (pArrays + 1) refers to next array.
Title: Re: static array [0..0] now fail at run time
Post by: PascalDragon on November 21, 2020, 06:18:09 pm
Quote from: rossh_lz on November 21, 2020, 01:56:56 pm
Now, as of FPC 3.2.0, I get range check fatal error on every attempt to access an array index > 0.  Code that still runs OK in Delphi, and for the last 12 years in Lazarus, now fails.

This code fails to compile in current Delphi versions as well (also with range checks disabled):

Code: Pascal  [Select][+][-]
  1. type
  2.   PCells = array [0..3] of Byte;
  3.   TPArray = array [0..0] of PCells;
  4.   PTPArray = ^TPArray;
  5.  
  6. var
  7.   p: PTPArray;
  8. begin
  9.   GetMem(p, 10 * SizeOf(PCells));
  10.   Writeln(p^[2]^);
  11. end.

The error generated is E1012: constant expression violates subrange bounds.

@Jonas: maybe this should be mentioned in User Changes 3.2.0 (https://wiki.freepascal.org/User_Changes_3.2.0#Dynamic_array_parameters_are_passed_like_pointers)?
Title: Re: static array [0..0] now fail at run time
Post by: 440bx on November 21, 2020, 06:46:54 pm
Quote from: PascalDragon on November 21, 2020, 06:18:09 pm
The error generated is E1012: constant expression violates subrange bounds.
Does it generate correct code in 64bit mode when the index is a 32bit integer _variable_ less than zero ?
Title: Re: static array [0..0] now fail at run time
Post by: rossh_lz on November 21, 2020, 07:56:37 pm
Quote from: PascalDragon on November 21, 2020, 06:18:09 pm
Quote from: rossh_lz on November 21, 2020, 01:56:56 pm
Now, as of FPC 3.2.0, I get range check fatal error on every attempt to access an array index > 0.  Code that still runs OK in Delphi, and for the last 12 years in Lazarus, now fails.

This code fails to compile in current Delphi versions as well (also with range checks disabled):

Code: Pascal  [Select][+][-]
  1. type
  2.   PCells = array [0..3] of Byte;
  3.   TPArray = array [0..0] of PCells;
  4.   PTPArray = ^TPArray;
  5.  
  6. var
  7.   p: PTPArray;
  8. begin
  9.   GetMem(p, 10 * SizeOf(PCells));
  10.   Writeln(p^[2]^);
  11. end.

The error generated is E1012: constant expression violates subrange bounds.

@Jonas: maybe this should be mentioned in User Changes 3.2.0 (https://wiki.freepascal.org/User_Changes_3.2.0#Dynamic_array_parameters_are_passed_like_pointers)?

Yes, your sample above fails, because the hard coded index of 2, is beyond range at compile time.

But this code below works and it's the beauty of using static 0..0 arrays.  This was a way to get dynamic arrays, before real dynamic array were available.

Code: Pascal  [Select][+][-]
  1.     var
  2.       i, j: Integer;
  3.       p: PTPArray;
  4.     begin
  5.       GetMem(p, 10 * SizeOf(PCells));
  6.       i := 7;
  7.       j := 1;
  8.       p^[i][j] := 4;
  9.       for i := 1 to 9 do
  10.         p^[i] := p^[i-1];
  11.     end;

The issue is, the above works just fine in Delphi, and all previous FPC, except the latest 3.2.0, where it seems that runtime range checking of static arrays has been added.  This is not mentioned in the notes (and not welcome either).
Title: Re: static array [0..0] now fail at run time
Post by: nanobit on November 21, 2020, 08:11:38 pm
Quote from: rossh_lz on November 21, 2020, 07:56:37 pm
But this code below works and it's the beauty of using static 0..0 arrays. 

array [0..0] means single element,
this is less than beautiful for larger arrays.
Title: Re: static array [0..0] now fail at run time
Post by: jamie on November 21, 2020, 08:29:21 pm
the easy fix which I am sure will work in all versions is this.

Code: Pascal  [Select][+][-]
  1. Type
  2.   PCells = array [0..3] of Byte;
  3.   PTPArray = ^PCells;  
  4.  ///---- Code land ---//
  5. Var
  6.   p:PTpArray;
  7. begin
  8.   getmem(p, sizeof(Pcells)*10);
  9.   P[5][0] := 10;
  10. end;                    
  11.              
  12.  

ect.
Title: Re: static array [0..0] now fail at run time
Post by: rossh_lz on November 21, 2020, 08:32:00 pm
Quote from: nanobit on November 21, 2020, 08:11:38 pm
Quote from: rossh_lz on November 21, 2020, 07:56:37 pm
But this code below works and it's the beauty of using static 0..0 arrays. 

array [0..0] means single element,
this is less than beautiful for larger arrays.

No.   The beauty is that it's open ended for size, and the use of indexes at runtime, mean you are not constrained to declaring array sizes at design time.   This was how you had to do it before dynamic arrays were available.
Title: Re: static array [0..0] now fail at run time
Post by: nanobit on November 21, 2020, 08:36:30 pm
Quote from: jamie on November 21, 2020, 08:29:21 pm
the easy fix which I am sure will work in all versions is this.

Code: Pascal  [Select][+][-]
  1. Type
  2.   PCells = array [0..3] of Byte;
  3.   PTPArray = ^PCells;  
  4.  

Not all versions. As stated earlier, works only with {$PointerMath on},
and if none of the following is true: {$modeswitch autoderef}, {$mode delphi}
Title: Re: static array [0..0] now fail at run time
Post by: PascalDragon on November 21, 2020, 08:43:32 pm
Quote from: 440bx on November 21, 2020, 06:46:54 pm
Quote from: PascalDragon on November 21, 2020, 06:18:09 pm
The error generated is E1012: constant expression violates subrange bounds.
Does it generate correct code in 64bit mode when the index is a 32bit integer _variable_ less than zero ?

I don't have a 64-bit Delphi.

Quote from: rossh_lz on November 21, 2020, 07:56:37 pm
The issue is, the above works just fine in Delphi, and all previous FPC, except the latest 3.2.0, where it seems that runtime range checking of static arrays has been added.  This is not mentioned in the notes (and not welcome either).

As long as I don't enable range checking the following code compiles and runs without problems on FPC 3.2.0 and newer (and if I enable $R there'll be a runtime error in both FPC and Delphi):

Code: Pascal  [Select][+][-]
  1. program tarrtest;
  2.  
  3. {$mode objfpc}
  4.  
  5. type
  6.   PCells = array [0..3] of Byte;
  7.   TPArray = array [0..0] of PCells;
  8.   PTPArray = ^TPArray;
  9.  
  10. var
  11.  i, j: Integer;
  12.  p: PTPArray;
  13. begin
  14.  GetMem(p, 10 * SizeOf(PCells));
  15.  i := 7;
  16.  j := 1;
  17.  p^[i][j] := 4;
  18.  for i := 1 to 9 do
  19.    p^[i] := p^[i-1];
  20. end.

So please provide a full example including compiler parameters that fails for you that you think shouldn't fail.
Title: Re: static array [0..0] now fail at run time
Post by: 440bx on November 21, 2020, 08:48:05 pm
Quote from: PascalDragon on November 21, 2020, 08:43:32 pm
Quote from: 440bx on November 21, 2020, 06:46:54 pm
Quote from: PascalDragon on November 21, 2020, 06:18:09 pm
The error generated is E1012: constant expression violates subrange bounds.
Does it generate correct code in 64bit mode when the index is a 32bit integer _variable_ less than zero ?

I don't have a 64-bit Delphi.
I wasn't asking about Delphi.  It works in Delphi.  I was asking if FPC 3.2.0 generates correct code when in 64bit mode with an index that is a 32bit integer variable with a value less than zero.  (FPC 3.0.4 does not because it fails to sign extend the 32bit integer variable.)
Title: Re: static array [0..0] now fail at run time
Post by: PascalDragon on November 21, 2020, 09:07:59 pm
Quote from: 440bx on November 21, 2020, 08:48:05 pm
Quote from: PascalDragon on November 21, 2020, 08:43:32 pm
Quote from: 440bx on November 21, 2020, 06:46:54 pm
Quote from: PascalDragon on November 21, 2020, 06:18:09 pm
The error generated is E1012: constant expression violates subrange bounds.
Does it generate correct code in 64bit mode when the index is a 32bit integer _variable_ less than zero ?

I don't have a 64-bit Delphi.
I wasn't asking about Delphi.  It works in Delphi.  I was asking if FPC 3.2.0 generates correct code when in 64bit mode with an index that is a 32bit integer variable with a value less than zero.  (FPC 3.0.4 does not because it fails to sign extend the 32bit integer variable.)

Nothing was changed in this regard compared to the previous thread that was already linked here.
Title: Re: static array [0..0] now fail at run time
Post by: nanobit on November 22, 2020, 08:57:11 am
Quote from: rossh_lz on November 21, 2020, 08:32:00 pm
Quote from: nanobit on November 21, 2020, 08:11:38 pm
array [0..0] means single element, this is less than beautiful for larger arrays.

No. The beauty is that it's open ended for size, and the use of indexes at runtime, mean you are not constrained to declaring array sizes at design time.   This was how you had to do it before dynamic arrays were available.

Truly open-ended (unchecked) is only pointermath (see previous comments).
array[0..max] is open-ended with maximum, which is range-checked {$R}.
Open-ended because you use PTPArray only with dynamic array-size, but not: var a: TPArray; sizeof(TPArray); high(TPArray).
array[0..0] is just one example of non-empty ranges.
 
(max+1)*sizeOf(element) must lie within the maximum dynamic allocation size of your array
(constrained by memory manager or api function if you don't know a lower maximum).
Index (0 or larger) can still point to invalid dynamic memory, regardless of range declaration.
If your variables use only type PTPArray, you can change your example to:
type TPArray = array[0..(high(sizeint) div sizeof(PCells))-1] of PCells;
type PTPArray = ^TPArray; var p: PTPArray;
Title: Re: static array [0..0] now fail at run time
Post by: rossh_lz on November 22, 2020, 03:46:53 pm
The underlying problem I have is this error below.  It shows at most places that the compiler has to do indexing of underlying array pointers.   As I said before this code is many years old, and runs fine in all versions prior to the current 3.2.0.

I started modify the code to replace code like   p^.data^[j].something , where data is a pointer to a static array.  I changed this to a wrapper function that does the pointer incrementing:  inc(data_ptr, j).  Of course this is the compilers job.   The fix works, to force the code to be more specific.  But this project is too big to be fiddled with, so I'm going back a version of the FPC and see how that works.

I can't give you a simple example to test - simple examples don't crash.   But a complex app with large mem usage is another story.

The 3.2.0 notes talk about changes to array type parameter passing.  My instinct says it seems to be some error here that is conflicting.


*****

"Project xyz raised exception class 'ERangeError' with message with message range check error.  address 1000013FA0.


'/usr/local/bin/fpc'  -MDelphi -Scghi -O1 -gw3 -gl -k-framework -kCocoa -l -vewnhibq -Filib/x86_64-darwin -Fu../codeadd -Fu../codeadd/registry -Fu../codeadd/Abbrevia_5.2/source -Fu../codeadd/synapse_tcp-ip_209 -Fu../../.lazarus/lib/Printer4Lazarus/lib/x86_64-darwin/cocoa -Fu../../.lazarus/lib/ImagesForLazarus/lib/x86_64-darwin/cocoa -Fu../../.lazarus/lib/cairocanvas_pkg/lib/x86_64-darwin/cocoa -Fu../../.lazarus/lib/units/x86_64-darwin/cocoa -Fu../../.lazarus/lib/LCLBase/units/x86_64-darwin -Fu/Library/Lazarus/components/lazutils/lib/x86_64-darwin -Fu/Library/Lazarus/packager/units/x86_64-darwin -Fu. -FElib/x86_64-darwin -olib/x86_64-darwin/xyz -dLCL -dLCLcocoa -dCURSOR_ERROR -dUseCThreads -dDEBUG
Title: Re: static array [0..0] now fail at run time
Post by: nanobit on November 22, 2020, 05:53:24 pm
Take your old code and just change your static-array type (used only by array pointer)
from array[0..0] to array[0..max]. Max (some constant) will suppress range-check errors
if max is large enough to hold all actual index values.
Title: Re: static array [0..0] now fail at run time
Post by: Thaddy on November 22, 2020, 06:03:15 pm
range checks are local, so this will also hide the error
Code: Pascal  [Select][+][-]
  1. program untitled;
  2.  {$mode delphi}{$rangechecks on}{$pointermath on}
  3. var
  4. {$push}{$rangechecks off}
  5.   arr:array[0..0] of byte;
  6. {$pop} // restore on.
  7. begin
  8. end.
Now the arr variable will never be range checked. Basically it restores the old, less restrictive behavior.
Title: Re: static array [0..0] now fail at run time
Post by: Jonas Maebe on November 22, 2020, 06:07:16 pm
Quote from: Thaddy on November 22, 2020, 06:03:15 pm
range checks are local, so this will also hide the error
Code: Pascal  [Select][+][-]
  1. program untitled;
  2.  {$mode delphi}{$rangechecks on}{$pointermath on}
  3. var
  4. {$push}{$rangechecks off}
  5.   arr:array[0..0] of byte;
  6. {$pop} // restore on.
  7. begin
  8. end.
Now the arr variable will never be range checked.
That is completely wrong.
Title: Re: static array [0..0] now fail at run time
Post by: PascalDragon on November 22, 2020, 06:08:25 pm
Quote from: rossh_lz on November 22, 2020, 03:46:53 pm
I can't give you a simple example to test - simple examples don't crash.   But a complex app with large mem usage is another story.

Without an example we can't help much.
Title: Re: static array [0..0] now fail at run time
Post by: Thaddy on November 22, 2020, 06:14:04 pm
Quote from: Jonas Maebe on November 22, 2020, 06:07:16 pm
That is completely wrong.
I saw it was wrong after testing. IMO it should work.
Title: Re: static array [0..0] now fail at run time
Post by: Jonas Maebe on November 22, 2020, 06:16:23 pm
Quote from: Thaddy on November 22, 2020, 06:14:04 pm
Quote from: Jonas Maebe on November 22, 2020, 06:07:16 pm
That is completely wrong.
I saw it was wrong after testing. IMO it should work.
Range checking is a setting that applies to code, not to data.
Title: Re: static array [0..0] now fail at run time
Post by: rossh_lz on November 23, 2020, 12:42:34 pm
Quote from: PascalDragon on November 22, 2020, 06:08:25 pm
Without an example we can't help much.

Yes, understood.   Here is the issue in assembler.  Code is made with O1., range check off.   The code is doing unwanted range checking on the array size.  Hence the fatal failure.


Code: Pascal  [Select][+][-]
  1.  
  2.   PTData = ^TData;
  3.   TData = packed record
  4.     Info: TDataInfo;
  5.     Segment: array [0..0] of TDataSeg;
  6.   end;
  7.  
  8. var
  9.   Data: PTData;
  10.  
  11.  
  12. /Users/rossh/Lazarus/Datalog/Utils.pas:1017  i := 2;
  13. 00000001003082CA                          : c7 85 6c ff ff ff 02 00 00 00     movl   $0x2, -0x94(%rbp)
  14. /Users/rossh/Lazarus/Datalog/Utils.pas:1018  if Data^.Segment[i].First = $CC then
  15. 00000001003082D4                          : 48 8b 5d a8                       movq   -0x58(%rbp), %rbx
  16. 00000001003082D8                          : 8b 85 6c ff ff ff                 movl   -0x94(%rbp), %eax
  17. 00000001003082DE                          : 85 c0                             testl  %eax, %eax
  18. 00000001003082E0                          : 76 05                             jbe    0x1003082e7               ; <+6719> at Utils.pas:1018:17
  19. 00000001003082E2                          : e8 89 e6 d0 ff                    callq  0x100016970               ; FPC_RANGEERROR
  20. 00000001003082E7                          : 8b 85 6c ff ff ff                 movl   -0x94(%rbp), %eax
  21. 00000001003082ED                          : 48 6b c0 5c                       imulq  $0x5c, %rax, %rax
  22. 00000001003082F1                          : 81 bc 03 b9 01 00 00 cc 00 00 00  cmpl   $0xcc, 0x1b9(%rbx,%rax)
  23. 00000001003082FC                          : 0f 84 bf 1c 00 00                 je     0x100309fc1               ; <+14105> at Utils.pas
  24.  
  25.  
  26.  
  27.     Segment: array [0..$FFFF] of TDataSeg;
  28.  
  29.  
  30. /Users/rossh/Lazarus/Datalog/Utils.pas:1017  i := 2;
  31. 00000001003081DA                          : c7 85 6c ff ff ff 02 00 00 00     movl   $0x2, -0x94(%rbp)
  32. /Users/rossh/Lazarus/Datalog/Utils.pas:1018  if Data^.Segment[i].First = $CC then
  33. 00000001003081E4                          : 48 8b 5d a8                       movq   -0x58(%rbp), %rbx
  34. 00000001003081E8                          : 8b 85 6c ff ff ff                 movl   -0x94(%rbp), %eax
  35. 00000001003081EE                          : 3d ff ff 00 00                    cmpl   $0xffff, %eax             ; imm = 0xFFFF
  36. 00000001003081F3                          : 76 05                             jbe    0x1003081fa               ; <+6722> at Utils.pas:1018:17
  37. 00000001003081F5                          : e8 86 e6 d0 ff                    callq  0x100016880               ; FPC_RANGEERROR
  38. 00000001003081FA                          : 8b 85 6c ff ff ff                 movl   -0x94(%rbp), %eax
  39. 0000000100308200                          : 48 6b c0 5c                       imulq  $0x5c, %rax, %rax
  40. 0000000100308204                          : 81 bc 03 b9 01 00 00 cc 00 00 00  cmpl   $0xcc, 0x1b9(%rbx,%rax)
  41. 000000010030820F                          : 0f 84 c0 1c 00 00                 je     0x100309ed5               ; <+14109> at Utils.pas
  42.  
Title: Re: static array [0..0] now fail at run time
Post by: nanobit on November 23, 2020, 02:18:14 pm
Such code works for me with {$R-}.
And your code is different from your previous descriptions.
Title: Re: static array [0..0] now fail at run time
Post by: rossh_lz on November 23, 2020, 02:53:42 pm
Quote from: nanobit on November 23, 2020, 02:18:14 pm
Such code works for me with {$R-}.
And your code is different from your previous descriptions.

Yes I have various different constructs in use, but all end up with the same error.

I thought $R- was the default?  Seemed to be so before.  You mean I have to force $R-, in every unit?

Title: Re: static array [0..0] now fail at run time
Post by: nanobit on November 23, 2020, 04:01:25 pm
{$R-} before index-use would be needed technically, but {$R-} is not best practice.
And working under {$R+} would need changes in your types.

Here is one potential idea, but I cannot say which compilers support this efficiently:
This still uses the classic array type TDataSegs, because TDataSeg itself
might be array type and then pointermath has issues as said earlier.

type TDataSeg = record first, last: byte; end;
type TDataSegs = array[0..high(sizeInt) div sizeof(TDataSeg)-1] of TDataSeg;
type PDataSegs = ^TDataSegs; 

  PTData = ^TData;
  TData = packed record
    Info: TDataInfo;
    function Segment: PDataSegs; inline;
  end;

   function TData.Segment: PDataSegs; inline;
   begin result := PDataSegs(PByte(@self) + sizeof(Info)); end;
Title: Re: static array [0..0] now fail at run time
Post by: PascalDragon on November 23, 2020, 04:29:08 pm
Quote from: rossh_lz on November 23, 2020, 02:53:42 pm
I thought $R- was the default?  Seemed to be so before.  You mean I have to force $R-, in every unit?

$R- indeed is the default. Maybe something resulted in it being enabled? You can use {$ifopt R+}{$warning Range checks enabled}{$endif} to warn at compile time whether the option is enabled at a specific code point. Additionally you could check the compiler command line as well as the used fpc.cfg whether something passes -Cr while it shouldn't.
Title: Re: static array [0..0] now fail at run time
Post by: Almir.Bispo on November 23, 2020, 07:30:14 pm
Code: Pascal  [Select][+][-]
  1. procedure TForm1.Button1Click(Sender: TObject);
  2. var  S :array  of byte;   //dynamic array declared as byte
  3.     i :integer;
  4. begin
  5. setlength(S,12); //dynamic dimension declared
  6.  for i:=0 to 10 do
  7.  begin
  8.    s[i]:=i;
  9.    showmessage('Test: '+inttostr(S[i]));  //test
  10.    showmessage('Test: '+inttostr( length(S)) );  //test
  11.  end;
  12.  
  13. end;    

Visit my Blog: http://adltecnologia.blogspot.com (http://adltecnologia.blogspot.com)
TinyPortal © 2005-2018