Lazarus

Programming => General => Topic started by: Gammatester on June 13, 2018, 09:28:47 am

Title: Inconsistent behaviour of shr operator
Post by: Gammatester on June 13, 2018, 09:28:47 am
I found a inconsistent behaviour for the shr operator: The output of the following program
Code: Pascal  [Select][+][-]
  1. const
  2.   cs:  smallint = -1;
  3.   cl:  longint  = -1;
  4.   c64: int64  = -1;
  5. var
  6.   vs:  smallint;
  7.   vl:  longint;
  8.   v64: int64;
  9. begin
  10.   vs  := cs shr 1;
  11.   vl  := cl shr 1;
  12.   v64 := c64 shr 1;
  13.   writeln('Smallint: ',vs, ' Longint: ',vl, ' Int64: ',v64);
  14. end.
  15.  
is (using FPC304 target Win32 for i386, identical output for FPC304 target Win64 for x64)

Code: [Select]
Smallint: -1 Longint: 2147483647 Int64: 9223372036854775807

Since shr is a logical shift right
(https://www.freepascal.org/docs-html/current/ref/refsu46.html#x149-17100012.8.2) and not a arithmetic shift right I would expect the output for smallint to be 32767.

OK, for 32-bit you can argue that the smallint -1 is promoted to longint -1 (https://www.freepascal.org/docs-html/current/ref/refsu4.html, remark after Table 3.2), but this does not explain the shift result -1. And if this argument would be correct, the shifted longint under 64-bit should be -1 too.

Is this a bug which should be filed to the bug-tracker?
Title: Re: Inconsistent behaviour of shr operator
Post by: RayoGlauco on June 13, 2018, 10:06:24 am
I think the result should be 32767

                  shr
1111111111111111 ----> 0111111111111111 = 32767

I see no reason for the calculation to be different than in the other cases  :-\.
But maybe someone knows best and can explain it :).
Title: Re: Inconsistent behaviour of shr operator
Post by: Phemtik on June 13, 2018, 10:29:50 am
If you use that code

Code: Pascal  [Select][+][-]
  1. var
  2.   vs:  smallint = -1;
  3.   vl:  longint = -1;
  4.   v64: int64 = -1;
  5. begin
  6.   vs  := vs shr 1;
  7.   vl  := vl shr 1;
  8.   v64 := v64 shr 1;
  9.   writeln('Smallint: ',vs, ' Longint: ',vl, ' Int64: ',v64);
  10. end.

You get the corrct results of:
Code: [Select]
Smallint: 32767 Longint: 2147483647 Int64: 9223372036854775807
i think that have something to do with typed constants.
Title: Re: Inconsistent behaviour of shr operator
Post by: Gammatester on June 13, 2018, 10:46:32 am
Quote from: Phemtik on June 13, 2018, 10:29:50 am
i think that have something to do with typed constants.
If I ran your code I still get -1, even the sequence 
Code: Pascal  [Select][+][-]
  1. vs := -1;
  2. vs := vs shr 1;
gives -1 for both 32- and 64-bit.

What compiler/target are you using?
Title: Re: Inconsistent behaviour of shr operator
Post by: rvk on June 13, 2018, 11:11:23 am
For both trunk 32 bit and Lazarus 1.8.2 64 bit (with FPC 3.0.4) it worked fine here.

Could you create a small project (with project options) and place it here?
Maybe it's something in the options.

What version (and IDE) are your using?
Title: Re: Inconsistent behaviour of shr operator
Post by: RayoGlauco on June 13, 2018, 11:37:37 am
Phemtik version works fine for me (Windows 8, Lazarus 1.8.4 FPC 3.0.4 win32)
Title: Re: Inconsistent behaviour of shr operator
Post by: Nitorami on June 13, 2018, 11:43:21 am
Very odd, I always get -1 (windows 7, fpc 3.0.4, 32 bit).

From http://wiki.freepascal.org/Shr: "unlike the >> operator in the C language, the shr operator is a logical (not arithmetic) bit shift, even if the left operand is a signed integer. An implicit typecast and extension to a larger unsigned type may be performed before the shift operation."

I guess the shr is in fact done on the native integer size 32 bit:
Code: Pascal  [Select][+][-]
  1. vs := -1;
  2. writeln (vs shr 1);
results in 2147483647 ($7FFFFFFF)

The code
Code: Pascal  [Select][+][-]
  1. cs := -1;
  2. vs := cs shr 1;
hence tries to assign 2147483647 to a smallint which is not possible. When turning on the range checks {$R+}, which is always a good idea, the code throws a range check error, otherwise vs will be -1. I am rather surprised that some get a different result than that.
Title: Re: Inconsistent behaviour of shr operator
Post by: Gammatester on June 13, 2018, 11:43:36 am
This is the complete Pascal source I used for the last test
Code: Pascal  [Select][+][-]
  1. var
  2.   vs:  smallint = -1;
  3.   vl:  longint = -1;
  4.   v64: int64 = -1;
  5. begin
  6.   vs  := vs shr 1;
  7.   vl  := vl shr 1;
  8.   v64 := v64 shr 1;
  9.   writeln('Smallint: ',vs, ' Longint: ',vl, ' Int64: ',v64);
  10. end.
  11.  
As already written, I am using 32- and 64-bit FPC304 under Win7/64-bit. Here compilations and outputs.
Code: [Select]
D:\Work\Dev>D:\FPC304\bin\i386-win32\fpc.exe shr1.pas
Free Pascal Compiler version 3.0.4 [2017/10/06] for i386
Copyright (c) 1993-2017 by Florian Klaempfl and others
Target OS: Win32 for i386
Compiling shr1.pas
Linking shr1.exe
10 lines compiled, 0.1 sec, 26560 bytes code, 1300 bytes data

D:\Work\Dev>shr1.exe
Smallint: -1 Longint: 2147483647 Int64: 9223372036854775807

D:\Work\Dev>fpc64304 shr1.pas

D:\Work\Dev>D:\FPC304\bin\i386-win32\ppcrossx64.exe shr1.pas
Free Pascal Compiler version 3.0.4 [2017/10/06] for x86_64
Copyright (c) 1993-2017 by Florian Klaempfl and others
Target OS: Win64 for x64
Compiling shr1.pas
Linking shr1.exe
10 lines compiled, 0.1 sec, 31072 bytes code, 1348 bytes data

D:\Work\Dev>shr1.exe
Smallint: -1 Longint: 2147483647 Int64: 9223372036854775807and attached the FPC.CFG file.
Title: Re: Inconsistent behaviour of shr operator
Post by: rvk on June 13, 2018, 11:49:33 am
Okay, what does this give you?

Quote
D:\FPC304\bin\i386-win32\fpc.exe shr1.pas -O1

(-O1 optimization level is default in Lazarus)
Title: Re: Inconsistent behaviour of shr operator
Post by: circular on June 13, 2018, 11:51:33 am
I guess it is a bit of a hole in the pascal language. There is no way to tell you want an arithmetic shift or a logical shift unless using assembler.
Title: Re: Inconsistent behaviour of shr operator
Post by: Gammatester on June 13, 2018, 12:01:28 pm
Quote from: rvk on June 13, 2018, 11:49:33 am
Okay, what does this give you?

Quote
D:\FPC304\bin\i386-win32\fpc.exe shr1.pas -O1

(-O1 optimization level is default in Lazarus)
Yes, with -O1 I get 32767 with var vs:  smallint = -1.  But with my original program using typed constants I still get -1.

And another observation. If I use the http://wiki.freepascal.org/Shr test example I get the output 9223372036854775806, i.e. the smallint -3 is promoted to int64 -3, then why the longint -1 is not promoted to int64 -1 ?
Title: Re: Inconsistent behaviour of shr operator
Post by: rvk on June 13, 2018, 12:15:46 pm
Is says
Quote
An implicit typecast and extension to a larger unsigned type may be performed before the shift operation.
So, it's not always promoted to a larger unsigned type.

And these seem to give the same result:
Code: Pascal  [Select][+][-]
  1. WriteLn(ShortInt(-2) shr 1); // 9223372036854775807
  2. WriteLn(ShortInt(-1) shr 1); // 9223372036854775807
Title: Re: Inconsistent behaviour of shr operator
Post by: circular on June 13, 2018, 04:01:57 pm
I would suggest to avoid doing shr on a negative number.

EDIT: There is a system function for the arithmetic shift
http://lazarus-ccr.sourceforge.net/docs/rtl/system/sarlongint.html

You could emulate it with something like that:
Code: Delphi  [Select][+][-]
  1. function ASR(AValue: LongInt; AShift: Byte): LongInt;
  2. begin
  3.   if AValue < 0 then
  4.     result := -1-((-1-AValue) shr AShift)
  5.   else
  6.     result := AValue shr AShift;
  7. end;
  8.  
Title: Re: Inconsistent behaviour of shr operator
Post by: Phemtik on June 13, 2018, 04:30:27 pm
normally you can make a arithmetic right shift (SAR) with x div 2.
free pascal do that even. The problem is, the compiler is not optimized for that particular example.
It make the shift in a 64 / 32 -bit register and doesn't make a shift in the register size of the variable.

But to have SAR and SAL, would be really nice.
Title: Re: Inconsistent behaviour of shr operator
Post by: howardpc on June 13, 2018, 04:46:58 pm
Quote from: Phemtik on June 13, 2018, 04:30:27 pm
But to have SAR and SAL, would be really nice.
Are you aware that 2.6.0 introduced the intrinsics:
SARShortInt
SARSmallInt
SARLongInt
SARInt64
?
Title: Re: Inconsistent behaviour of shr operator
Post by: Phemtik on June 13, 2018, 04:55:17 pm
Hm nice, I did't know.
Is it somewhere in the wiki/documentation? i hadn't found.
Title: Re: Inconsistent behaviour of shr operator
Post by: howardpc on June 13, 2018, 06:45:15 pm
See here (http://wiki.freepascal.org/FPC_New_Features_2.6.0#SAR_intrinsics)
Title: Re: Inconsistent behaviour of shr operator
Post by: circular on June 13, 2018, 07:51:02 pm
Oh that's great. It could be added to the doc then to make people aware of it:
http://wiki.lazarus.freepascal.org/Shr
https://freepascal.org/docs-html/ref/refsu46.html
Title: Re: Inconsistent behaviour of shr operator
Post by: ASerge on June 14, 2018, 06:37:47 pm
Quote from: Gammatester on June 13, 2018, 09:28:47 am
I found a inconsistent behaviour for the shr operator: The output of the following program
Code: [Select]
Smallint: -1 Longint: 2147483647 Int64: 9223372036854775807Since shr is a logical shift right
Rules:
1. shr operation is always a logical shift (unsigned).
2. shr operation is 32 bits for the size of the operands <= 32 bits and 64 bits otherwise.

Now details:
vs := cs shr 1;
cs is SmallInt ($FFFF) are sign extended to 32 bit ($FFFFFFFF). Now it shifted without sign ($7FFFFFFF). Now it's needed store to SmallInt. If {$RANGECHECKS ON} will be range error (very big for SmallInt), otherwize only last 16 bits saved ($FFFF). Result is -1.

vl := cl shr 1;
cl is not changed (already 32), signed, so simple unsigned shift ($7FFFFFFF). Because result is fit in vl, so it's simple 2147483647.
Same behaviour for 64 bit integer.

It's all Delphi compatible (including range error).

What about vs := vs shr 1? With optimization, compiler coded this as in-place operation (which ignore rule 2), but do range check as by rule 2 (range error). It's NOT Delphi compatible. Delphi always convert to 32 bit (and result is -1). Without optimization FPC result is -1.

PS: for constants shr operation make by compiler itself, and result can be arithmetic shift or using large size operand (not documented and version depended).
Title: Re: Inconsistent behaviour of shr operator
Post by: Gammatester on June 14, 2018, 07:33:07 pm
Quote from: ASerge on June 14, 2018, 06:37:47 pm
Rules:
1. shr operation is always a logical shift (unsigned).
2. shr operation is 32 bits for the size of the operands <= 32 bits and 64 bits otherwise.
If rule 2 were correct, how do you explain
Quote
WriteLn(ShortInt(-1) shr 1); // 9223372036854775807
Title: Re: Inconsistent behaviour of shr operator
Post by: ASerge on June 14, 2018, 07:39:44 pm
Quote from: Gammatester on June 14, 2018, 07:33:07 pm
If rule 2 were correct, how do you explain
Quote
WriteLn(ShortInt(-1) shr 1); // 9223372036854775807
Did you read my PS: ?
Title: Re: Inconsistent behaviour of shr operator
Post by: Gammatester on June 14, 2018, 08:03:38 pm
Quote from: ASerge on June 14, 2018, 07:39:44 pm
Quote from: Gammatester on June 14, 2018, 07:33:07 pm
If rule 2 were correct, how do you explain
Quote
WriteLn(ShortInt(-1) shr 1); // 9223372036854775807
Did you read my PS: ?
Yes, but then the rules are not very clear. It looks like the same disclaimer applies as with
Quote
An implicit typecast and extension to a larger unsigned type may be performed before the shift operation.
I would like do have definite rules.
Title: Re: Inconsistent behaviour of shr operator
Post by: Nitorami on June 14, 2018, 09:36:13 pm
I understand that shr is normally done on nativeinteger, i.e. 32 bit on 32 bit platforms, 64 on 64 bit platforms. EXCEPT if evaluation can already be done by the parser, which consistently uses 65bit (yes) arithmetic, hence

Code: Pascal  [Select][+][-]
  1. WriteLn(ShortInt(-1) shr 1); // 9223372036854775807

But whatever the rules are, they should be consistent. What still puzzles me is that some seem to get 32767 as result of vs := cs shr 1;
I cannot confirm that; with FPC 3.0.4, Win 7 and win 10, with and without optimisation, and whatever other settings,  I always get -1, but never 32767. !?!
Title: Re: Inconsistent behaviour of shr operator
Post by: jamie on June 15, 2018, 12:21:38 am
First , Shortint are byte size so you won't get 32767, you'll get 127 or what ever it is >:(

smallInt are word size.

when shifting a signed integer the smallest value is -1, so shifting -1 will result in -1.

 0 and up are positive numbers which is why you can shift all the way to 0.

 and if you were to shl 7 with a -1 you would get -128, one more you get 0

 You could almost look at it like SHR is actually a SHL and SHL is actually a SHR with - numbers

 with the exception that -1 is the smallest number.

This is how I understand it...

P.S.
 
The compiler always tries to elevate the type to an integer, but if you cast around the whole equation it
will do what you ask for.




Title: Re: Inconsistent behaviour of shr operator
Post by: Gammatester on June 15, 2018, 08:52:50 am
Quote from: jamie on June 15, 2018, 12:21:38 am
when shifting a signed integer the smallest value is -1, so shifting -1 will result in -1.

 0 and up are positive numbers which is why you can shift all the way to 0.

 and if you were to shl 7 with a -1 you would get -128, one more you get 0
...
This is how I understand it...
Your understanding is wrong. shr is a logical right shift, that  means the lowest bits go to nirvana and zero  bits are inserted at the highest positions instead. As shown by
Quote from: RayoGlauco on June 13, 2018, 10:06:24 am
                  shr
1111111111111111 ----> 0111111111111111 = 32767
Theoretically the results are clearly defined by the bit-size of the variables and the shift count. The problems in practice come from the promotion of small-bit-sized types to signed longint or int64, to which the logical shift is applied (and constant expressions are handled by the compiler with int64). Your `shifting -1 will result in -1` is seen to be completely wrong by looking at the results for longint and int64.
TinyPortal © 2005-2018