Negative array indices produce incorrect pointers

[Remy Lebeau]

Mode Delphi allows index based pointer access only for a few base types like PByte and not for arbitrary types.

Does FPC have something similar to Delphi's {$POINTERMATH ON} directive for arbitrary types?

[440bx]

What's ptrint? That type doesn't exist in Delphi, Is that a nativeint?

Yes, a ptrint is a nativeint.

[Thaddy]

Yes. to be more precise it is the width op the CPU registers which varies between platforms.

[Madoc]

Yes. to be more precise it is the width op the CPU registers which varies between platforms.

All 64 bit registers still have 32 bit, 16 bit and 8 bit versions that work exactly as they did in 32 bit, and 16 bit before then. They are still useful in those reduced formats, obviously. They've simply been extended with an additional 32 bits, to be able hold 64 bit values as well. e.g. al -> ax -> eax -> rax. You absolutely cannot put a 32 bit signed value into eax and then treat rax as that signed value, this is a spectacularly epic fail. As far as I can tell this is exactly what the compiler is doing here.

[Thaddy]

That is missing the point, not by a mile but by ion's.

[Martin_fr]

You absolutely cannot put a 32 bit signed value into eax and then treat rax as that signed value

Only, that is not what is happening. Somewhere (either in this thread or the other thread( it was explained.

You have

Code: Pascal  [Select][+][-]

1.  TAInt = array[0..255] of integer;

Your index is defined as

Code: Pascal  [Select][+][-]

1. type index = 0..255;

A sub-range, that can only hold value from 0 to 255.

And you can put any of those values into a 32 bit register, and use it as 64 bit without sign extension. (so long as the upper 32 bit are zero).


And yes, your variable that you use for the index may be declared integer. Or actually in your example the constant

Code: Pascal  [Select][+][-]

1. v := b[-1]

But then if you have

Code: Pascal  [Select][+][-]

1. procedure foo(a: cardinal);
2. ...
3. {$R-} foo(-1);
4.  

You end up getting a cardinal with the same bit pattern as integer(-1).

-1 being used in a place that allows 0..255 gets cast/converted too.


Good or bad idea is a different question, but technically it is correct.

[Madoc]

Only, that is not what is happening. Somewhere (either in this thread or the other thread( it was explained.

I honestly have no idea what you're talking about now. That was a discussion about implicit disabling of range checking for an array pointer. It is not related.

The error I describe here occurs even if I declare the array type as having a range of [-100..100] or whatever. It doesn't matter.

For example:

Code: Pascal  [Select][+][-]

1. type
2. type
3.   PAByte = ^TAByte;
4.   TAByte = packed array [-100..100] of byte;
5.  
6. var
7.   i: integer;
8.   j: nativeint;
9.   a: PAbyte;
10.  
11.   a := some valid pointer
12.  
13.   for i := -1 to 1 do if a = $FF then  // Access violation
14.  
15.   for i := 0 to 2 do if a = $FF then  // fine!
16.  
17.   for j := -1 to 1 do if a = $FF then  // fine!
18.  
19.  

The pointer the compiler generates for integer -1 is not being sign extended and is trying to address a + 4 gygabites.
There's lots and lots of ways this erroneous behaviour could be avoided, but here is the utterly bizzare code the compiler is generating for the first step of that loop:

Code: Pascal  [Select][+][-]

1. mov [rbp-$2C],$FFFFFFFE
2. nop word ptr [rax+rax+$00]
3. add dword ptr [rbp-$2C],$01
4. mov r8,[rbp-$48]
5. mov r9d,[rbp-$2C]
6. cmp [r9+r8],$FF
7.  

The example is part of a very tight nested loop, and all of that code could be a single instruction with no memory access and certainly no utterly broken pointers.

I don't understand why we're even having a discussion about this and I really do not have the time to argue like this. This is absolutely without shadow of a doubt a bug and I'm not the first to bring it up.

[Martin_fr]

Only, that is not what is happening. Somewhere (either in this thread or the other thread( it was explained.

I honestly have no idea what you're talking about now. That was a discussion about implicit disabling of range checking for an array pointer. It is not related.

Well, I replied to your part about using an 32bit register, as a signed 64bit without sign extending.
This may not be the original topic, and you may not have introduced it, but you did entertain it.  And I replied to the statement that you made about it.

So my reply is very related to the statement you made.

The error I describe here occurs even if I declare the array type as having a range of [-100..100] or whatever. It doesn't matter.

For example:

Code: Pascal  [Select][+][-]

1. type
2. type
3.   PAByte = ^TAByte;
4.   TAByte = packed array [-100..100] of byte;
5.  

Unfortunately your code sample does not compile... So I can't do my own tests on it.

I took the code from your original post, and modified it accordingly

Code: Pascal  [Select][+][-]

1. program Project1;
2. {$mode objfpc}{$R-}
3.     type
4.       PAInt = ^TAint;
5. //      TAInt = array[0..255] of integer;
6.       TAInt = array[-10..255] of integer;
7.     var
8.       a, b: PAint;
9.       v: integer;
10.  
11. begin
12.     GetMem(a, 200 * SizeOf(integer));
13.     b := @a[100];
14.     v := b^[-1];
15.     v := b^[-2];
16.     v := b^[-20];
17. end.
18.  

I need the "^" operator since it's not mode Delphi.

Now, different fpc versions used with different settings may produce different results.
My tests were done with
  FPC 3.3.1 (from earlier this month)
  -O-

In the original 0..255 array, accessing b^[-1] produced this asm

Code: ASM  [Select][+][-]

1.   mov rax,[rip+$00014990]
2.   mov eax,[rax+$0003FFFC]
3.   mov [rip+$00014994],eax
4.  

The importance here is $3FFFC
And that is indeed strange, but appears somehow $40000 - 1 * sizeof(integer).
I don't know the compiler source code, but I would assume
# the compiler treated this as an unsigned value*
# Calculated cardinal(0) - sizeof(integer) = $FFFFFFFC
# For unknown reason masked that value to 20 bits.

And yes: b^[-2]  produces    mov eax,[rax+$0003FFF8]
So the value goes down in steps of sizeof(integer).

Then if the array is declare  [-10..255]
I get

Code: ASM  [Select][+][-]

1. # [15] v := b^[-1];
2.         movq    U_$P$PROJECT1_$$_B(%rip),%rax
3.         movl    36(%rax),%eax
4.         movl    %eax,U_$P$PROJECT1_$$_V(%rip)
5. # [16] v := b^[-2];
6.         movq    U_$P$PROJECT1_$$_B(%rip),%rax
7.         movl    32(%rax),%eax
8.         movl    %eax,U_$P$PROJECT1_$$_V(%rip)
9. # [17] v := b^[-20];
10.         movq    U_$P$PROJECT1_$$_B(%rip),%rax
11.         movl    -40(%rax),%eax
12.         movl    %eax,U_$P$PROJECT1_$$_V(%rip)
13.  

The index -1 is 9 above the starting point of -10 => So the offset in memory is 36
The index -20 is 10 below -10 so the offset is -40

That looks correct to me.

Mind however, that it is not guaranteed for -20 (or other out of range values) to work. 
If the compiler decided to use a signed-byte to do the calculations, then -300 would fail.


As I said, I couldn't check your latest example. It is not complete code.

But if I do (in above example / with "j:integer")

Code: Pascal  [Select][+][-]

1.     for j:= -1 to 1 do
2.       v := b^[j];
3.  

then I get

Code: ASM  [Select][+][-]

1. # [15] for j:= -1 to 1 do
2.         movl    $-1,U_$P$PROJECT1_$$_J(%rip)
3.         .p2align 4,,10
4.         .p2align 3
5. .Lj3:
6. .Ll5:
7. # [16] v := b^[j];
8.         movq    U_$P$PROJECT1_$$_B(%rip),%rax
9.         movslq  U_$P$PROJECT1_$$_J(%rip),%rdx
10.         movl    40(%rax,%rdx,4),%eax
11.         movl    %eax,U_$P$PROJECT1_$$_V(%rip)
12. .Ll6:
13.         movl    U_$P$PROJECT1_$$_J(%rip),%eax
14.         addl    $1,%eax
15.         movl    %eax,U_$P$PROJECT1_$$_J(%rip)
16.         cmpl    $1,U_$P$PROJECT1_$$_J(%rip)
17.         jg      .Lj5
18.         jmp     .Lj3
19. .Lj5:
20. .Ll7:
21.  

movslq is sign extended.

FPC 3.2.2 at -O- produces a bit different asm, but also use movslq.


Please indicate how you got the assemble code?
If you used the debugger to disassemble, maybe there is an issue there (you can compile with -al to get an assemble file).

Also compare the result you get at different optimization settings. Maybe there is a bug in the optimizer.

[marcov]

Mode Delphi allows index based pointer access only for a few base types like PByte and not for arbitrary types.

Does FPC have something similar to Delphi's {$POINTERMATH ON} directive for arbitrary types?

That directive works in Delphi mode. in FPC/Objfpc modes it is already the default since before 2000.

Overindexing pointers without ^ is however again something more from the Delphi side of things

[jamie]

program Project1;
{$mode objfpc}{$R-}
    type
      PAInt = ^TAint;
//      TAInt = array[0..255] of integer;
      TAInt = array[-10..255] of integer;
    var
      a, b: PAint;
      v: integer;
 
begin
    GetMem(a, 200 * SizeOf(integer));
    b := @a[100];
    v := b^[-1];
    v := b^[-2];
    v := b^[-20];
end.
 

My understanding must be broke or the compiler is after following this thread.

Code: Pascal  [Select][+][-]

1.  b:= @a[100]

equates to the address of 100 th  of TAint arrays.
considering that you can index that phrase V:= a[xArray, Yinteger];
so the above code is actually generating an address that is way outside the memory allocated via the GetMem call.

so all the remaining code is accessing integers that exist in garbage memory because its way outside.

did I miss something?

[Thaddy]

101, since the start of the index is zero....

[jamie]

This place needs a better noise attenuation system, maybe a better designed FFT protocol to blank it out!