Strange bugs with Advanced Records - SOLVED

[Warfley]

The Wiki, neither the french nor english pages, are documentation. They are use provided content. The freepascal website is down right now, so I cannot look up the actual documentation, but if I remember correctly the idea behind constref is that in some situations you must ensure that a variable is passed as reference. Because "const" always tries to do the most efficient way of passing (as for types that fit in registers, it's less efficient to pass by reference than by register), it cannot be guaranteed that it is passed by reference. Var which has this guarantee on the other hand makes optimizations more difficult, because if the variable can or will be changed, some optilizations like changing order of execution, may impact the result.

So constref is a pass by reference, where you as a programmer promise the compiler that you won't change it. It's somewhat semantically enforced, but not fully, e.g.

Code: Pascal  [Select][+][-]

1. procedure Foo(constref A: Integer);
2. begin
3.   PInteger(@A)^ := 42;
4. end;

Because it's constref you can take the pointer of it (which may be required for interfacing C style functions), but because the FPCs typesystem has (unlike e.g. C's type system) no concept of constness outside of parameters, the constness is gone.

The reason why you need constref for the copy operator is very simple. Imagine the parameter would be passed by value, then it would needed to be addref'd or copied first, which would interfere with the implementation of the copy operator. So in order to ensure that it will never be copied when calling copy, you enforce a reference by using constref

[Thaddy]

That is my hack, but here that is not the case: S has outer scope, so priority scope over T.
That is not only expected, but correct, since the contract between the programmer and the compiler is not touching T within the scope. It is also simply bad programming.

[440bx]

I see there is a lot of misunderstanding as to what "constref" means and does...

Consider the following code:

Code: Pascal  [Select][+][-]

1. type
2.   TSOMETYPE = record
3.     Field1    : integer;
4.     Field2    : integer;
5.     Field3    : integer;
6.     Field4    : integer;
7.   end;
8.  
9. var
10.   MyVar : TSOMETYPE;
11.  
12. function TheFunction(constref TheParameter : TSOMETYPE; var ANumber : integer) : boolean;
13. var
14.   LocalVar : integer;
15. begin
16.   LocalVar := TheParameter.Field3;
17.  
18.   ANumber := 5;  { any value will do }
19.  
20.   result := TRUE;
21. end;
22.  
23. begin
24.   { note that "MyVar" being "constref" does NOT prevent passing }
25.   { MyVar.Field3 as a "var" parameter that IS modified.         }
26.  
27.   { "constref" only determines what can be done in the scope in }
28.   { which it is active and does NOT prevent modifying the record}
29.   { or its fields by other means and, that's LEGAL.             }
30.  
31.   TheFunction(MyVar; MyVar.Field3);
32. end.
33.  

Note that "MyVar" is constref, yet the field "MyVar.Field3" can be modified by the function because that field is passed by value reference.  It may seem strange but, that's perfectly legal (not to mention very useful.)

HTH.

ETA:

Corrected "value" to "reference"  (thank you Martin)

[Martin_fr]

"var ANumber" => that is passed by reference.

So this is actually breaking the const rule. The compiler just can't detect it. And currently no version of fpc does any optimization (that I know of) that would lead to an error as result of this violation. However, a future version may have such optimizations.

"constref" is described on this page. And - as far as I know - falls under the same conditions
https://www.freepascal.org/docs-html/ref/refsu67.html

It is the programmer who tells the compiler that the contents of the const parameter will not be changed when the routine is executed

And that does NOT say "by code inside the routine directly accessing the field".

[Warfley]

And currently no version of fpc does any optimization (that I know of) that would lead to an error as result of this violation. However, a future version may have such optimizations.

A bit curious, is the constness information compiled into llvm IR when the llvm backend is used? If so, it might be relevant because llvm has such optimizations that rely on constness.
To be fair my experience with llvm dates back to llvm 3.8 and I never used FPC with llvm (even though it's definitely on my list to try out)

[440bx]

So this is actually breaking the const rule.

I've come to interpret the "const rule" a little bit differently than what the FPC documentation states.

For me, the promise is that the parameter or, in the case of a record, parameter fields will _not_ be modified using that parameter.  IOW, it does not promise the parameter isn't changed using an alias (which is what the "var" parameter in the example I provided does.)

I believe that when an alias is used, as in the example I gave then, it is entirely the programmer's responsibility to ensure no undesirable side effects occur no matter what the compiler decides to do (as long as what the compiler does is correct.) 

Some compilers have optimization levels that explicitly preclude the use of pointer aliasing and offer ways for the programmer to declare that a specific sequence of code uses pointer aliasing to inform the compiler some optimizations cannot be applied to that sequence of code.

The example I provided in my previous post is a bit too simplified.  Consider the record as being made of other records (instead of just integers.)  It is not uncommon for code to need read access to many of the records and read-write to only one or two.  That's when passing the entire record as "constref" and one or two of its component records as "var" makes sense and is very useful.

[Martin_fr]

Run this code

Code: Pascal  [Select][+][-]

1. program Project1;
2.  
3. uses SysUtils;
4.  
5. //procedure Foo(constref s1: Ansistring; var s2: Ansistring);
6. procedure Foo(const s1: Ansistring; var s2: Ansistring);
7. var
8.   m: Pointer;
9. begin
10.   writeln(s1);
11.   s2 := StringOfChar('x', 100+ Random(99));
12.   m := AllocMem(10);
13.   FillChar(m^, 10 , 1);
14.   writeln(Length(s1), s1);
15.   writeln(Length(s2), s2);
16. end;
17.  
18.  
19. var a,b: Ansistring;
20. begin
21.   a := 'abc ' + IntToStr(Random(99));
22.   b := 'abc ' + IntToStr(Random(99));
23.  
24.   Foo(a, a);
25.   readln;
26. end.
27.  

It is essentially the same you have, just with ansistring.

And in more complex apps, with more memory used and freed over time, you can get crashes and all.

The constref behaves a bit more forgiving.

This is because in todays FPC I don't know any optimization that would allow me to exploit (in a negative way) the broken promise. But a future FPC may bring such an option.

Mind: "I don't know any" doesn't mean that there isn't.

[440bx]

That's why I said that it is ultimately the programmer's responsibility to ensure there are no undesirable side effects.

For instance, the compiler does not know if some area of memory being referenced in some function/procedure is also being referenced by a different function/procedure in another thread.  It is the programmer who must know and account for situations like that.

[Joanna from IRC]

Is there any way to prevent the fields of a record Or object passed as a constant from being modified?

[440bx]

Is there any way to prevent the fields of a record Or object passed as a constant from being modified?

Depends...

if what you're asking for is some way to tell the compiler to prevent modification in all circumstances then the answer is _no_ because there are plenty of dirty tricks that can be used to sidestep the compiler's guard rails.

That said, if your data is structured the right way then the answer can be _yes_.  Here is an example, say you have a variable of type record TMYRECORD and you want to make certain that it cannot be changed by any other code then, what you do is use the O/S facility VirtualProtect to change the memory attributes to read-only.  Important to know that to do that the variable _must_ have been allocated using VirtualAlloc, otherwise using VirtualProtect on it can cause all kinds of problems.  For something like this to work well, upfront design is most definitely necessary but, well worth it.

The bottom line is always the same: the programmer has to know what he/she is doing.  The compiler can help catch mistakes but cannot catch all mistakes.  This has always been true but, it is particularly true in multi-tasking systems.

[Thaddy]

Is there any way to prevent the fields of a record Or object passed as a constant from being modified?

If the record is compiled in {$J-} state, you can not modify it. (It usually ends up in .rodata, read-only data.)
{$J+/-} is a local directive, so you can use {$push}{$J-}record declaration{$pop}. Only in {$J+} state the record is writable and that is the default.
Small demo:

Code: Pascal  [Select][+][-]

1. {$mode objfpc}
2. type
3.   trec1 = record
4.   a:integer;
5.   end;
6.      
7. {$push}{$J-}
8. const
9.   rec1:trec1 = (a:100);
10. {$pop}
11. begin
12.   writeln(rec1.a);
13.   { in J- state this will not compile
14.     in J+ state it will compile }
15.   rec1.a:= 101;
16.   writeln(rec1.a);
17. end.

This is applicable to any typed const, not only records, everything.
You can even pass it as a const parameter to a procedure, does not matter, you can't modify it at all anyway.
But in J+ state, even if you pass it as a const parameter, depending on scope there are ways to modify it if you are not careful. See the example in the documentation for const (the one with T and S mentioned above)

But in the above example the record is not modifiable anywhere.
To show that effect:

Code: Pascal  [Select][+][-]

1. {$mode objfpc}
2. type
3.   trec1 = record
4.   a:integer;
5.   end;
6.      
7. {$push}{$J+} // + state
8. const
9.   rec1:trec1 = (a:100);
10. {$pop}
11. procedure dosomething(const value:trec1);
12. begin
13.   writeln(value.a);
14.   // value.a :=104;     // this will not compile, but...
15.   rec1.a := 102;        // rec1 has outer scope
16.   writeln(value.a);     // so this ends up modified too..but only in J+ state.
17. end;
18.  
19. begin
20.   writeln(rec1.a);
21.   { in J- state this will not compile
22.     in J+ state it will compile }
23.   rec1.a:= 101;
24.   writeln(rec1.a);
25.   dosomething(rec1);
26. end.

Play with the +/- switch.

[Warfley]

That's why I said that it is ultimately the programmer's responsibility to ensure there are no undesirable side effects.

For instance, the compiler does not know if some area of memory being referenced in some function/procedure is also being referenced by a different function/procedure in another thread.  It is the programmer who must know and account for situations like that.

Yes and no. It's the responsibility of the programmer to follow the defined semantics of the language. And in Pascal the defined semantic for a const(ref) parameter is that it will not change during the rumtime, so you as a programmer must make sure that this doesn't happen. It's a two way contract so to speak. You promis to make sure the parameter doesn't change, and in exchange the compiler promises that it generates correct code.

For example, if I compile the following code for 32 and 64 bit I get differing results:

Code: Pascal  [Select][+][-]

1. type
2.   TMyRec = record
3.     A: Integer;
4.   end;
5.  
6. procedure Foo(const r: TMyRec; var A: Integer);
7. begin
8.   A:=42;
9.   WriteLn(r.A);
10. end;
11.  
12. var
13.   r: TMyRec;
14. begin
15.   r.A:=32;
16.   Foo(r, r.A);
17. end.

The reason is that in one case the parameter is passed by register in the other it's passed by reference. This means this code has undefined semantics, which only happens because I broke the promise I made the compiler through passing a "const".

There are just things that you as a programmer must never make assumptions about. It's called undefined behavior and no matter how good you know the language and the compiler, you should never think you know whats happening there.

I had a great experience with C++, because prior to C++20 bit representation (and thereby overflow) for the int types was not defined. So I had a for loop like this:

Code: C  [Select][+][-]

1. for (;myvar>0;++myvar) {
2.   ...
3. }

Which should loop until overflow happend, and the compiler just turned it into:

Code: C  [Select][+][-]

1. while (true) {
2.   ...
3. }

Because overflow is not defined, so the compiler reasoned, a value >0 that will only ever be incremented will never get smaller and therefore this condition can never be met. Mathematically a sound argumentation, but of course vary nasty to figure out in my situation.

Long story short, you as a programmer only ever are responsible to follow the semantics of the language. If something falls outside of the defined behavior of the language, you should not make any assumptions about it

[Joanna from IRC]

Interesting tricks but still it’s kind of misleading declaring a record Parameterconst if it in fact is not. 

[Thaddy]

It is const within the scope. Examples that show you otherwise is imo bad programming and that includes my own second example above. That is merely an illustration, as is the anti-pattern in the documentation, which is meant to be an ...anti-pattern.

@Warfley forgot another one: the calling convention also impacts the behavior of how const parameters are passed.

[Warfley]

Interesting tricks but still it’s kind of misleading declaring a record Parameterconst if it in fact is not. 

This is the asymmetry of the definition. By declaring a parameter const you promise that your function will not change that parameter (mostly enforced by the compiler). But thats just one side of the equation, the other side is that the caller must also respect that contract. If the function is declared with one const parameter and one var parameter, i.e. one parameter where my function promises that it will not be changed, and one where my function says it will change it, then passing the same parameter in both, "tricks" the function into violating it's own contract. So both parties must agree to not change the variable.
And the thing is, the compiler can only enforce the former, not the latter (well you could also enforce the latter for most cases, by doing a bit of data flow analysis, but I guess thats not yet implemented in -Oodfa).

Note that the function may be provided externally, e.g. I write a lot of libraries, so I provide a lot of functions where I have absolutely no idea who's going to use them in which context. On the other hand if you use a library, you have no influence on the functions you are calling. So I as a library writer can put const everywhere where I don't intend to change the value of the parameter, but thats only half the story. The user of that library also must respect that "contract" when using the function