[Solved] Is it possible to const an argument passed by reference in FPC?

[440bx]

untyped parameters alway pass the address of what you passed in

In the example I posted above, it didn't pass the address, it passed the value.

Please, explain why in the above example it passes the value but, when used in the definition of the WriteFile API, it passes the address.

I cannot see what causes the parameter to be passed one way in one case and another way in the other case.

[Zoran]

untyped parameters alway pass the address of what you passed in

In the example I posted above, it didn't pass the address, it passed the value.

Please, explain why in the above example it passes the value but, when used in the definition of the WriteFile API, it passes the address.

I cannot see what causes the parameter to be passed one way in one case and another way in the other case.

As I understand it (sorry if I am wrong, but I would also want this clarifyed), in case of const untyped parameters there are two things which can switch to passing the reference to value istead of the value itself:

1. using const - this may or may not decide to do so. Then we have
  1a - const decides to make the reference (so now we have refernce to "buffer") OR
  1b - const decides not to make the reference (so we still have "buffer" itself)
2. the param is untyped - this will always make the reference (pointer) to what we got from 1, so what is actually passed may be:
  2a - the reference to reference (pointer to pointer) to buffer (in case we got 1a in step 1.) OR
  2b - the reference (pointer) to buffer (in case we had 1b in step 1.)

* Note that, the actual order of 1 and 2 have no significance for this example (if step 2 actually happens before what I put under step 1, we end up with same after both steps).

So, it is not strange that you get different behaviour with the two routines, once the compiler took aproach 2a, and the other time, it took 2b.

And the bottom line is -- it is unpredictable how parameters are passed, so:
 - inside your program (when you don't use external routines), be free to use const untyped variable and just use it normally within the called function's body (the compiler knows its decision and handles it correctly, you needn't care).
 - with external routines, don't use it.

Actually, it falls down to the same approach which I think should be taken with typed parameters as well, so the general rule:
 - don't use const with external routines, as there you have to know how parameters are passed (must be same as external routine expects).
 - use const freely with routines from your program, as there you need not know if they are passed by reference or by value.

[440bx]

And the bottom line is -- it is unpredictable how parameters are passed,

How parameters are passed must _always_ be completely predictable.

That said,. I looked at the assembly code generated for the example I posted and, it is true that the parameter is passed by reference.  That much I can state because it is a fact reflected in the assembler (and disassembly) code.

After that, it's hard to make sense why the code is the way it is.  It _seems_ that the Buffer pointer is automatically dereferenced (which is completely inexplicable since there is no type associated with the pointer) and for that reason it outputs zero, thereby giving the _impression_ that the parameter is passed by value.

There is no reason to require "const" (provided its meaning is restricted to "constant") not to be used in the declaration of an external routine.  In that case "const" is mostly ornamental since the compiler has no way to enforce the value remaining constant (it doesn't generate the called routine.)

There is something in that call which is not working as it should, because zero (0) is not the value that should be output unless the compiler has become clairvoyant.


ETA:

Figured it out, finally. 

The compiler passes a pointer to the parameter, when the parameter is referenced, the compiler automatically dereferences it (something which is Pascal specific), which produces the value of the parameter that was passed to the routine.

In the case of WriteFile, defining that function as taking an untyped parameter is _incorrect_ because C never does automatic dereferencing.  The problem is the untyped parameter not the "const" modifier.

The "const" modifier just further muddies the waters, because in Delphi it does cause a parameter to be passed by reference (_except_ in the case of an untyped parameter, in which case, it simply means "constant", just as it does in FPC.)

conclusion: in FPC (not Delphi), const can be used with API functions the same way it can be used with any Pascal function/procedure.  Untyped parameters should _never_ be used in an API definition because neither C nor C++ (the two languages used to implement the O/S) do automatic dereferencing.

[Zoran]

And the bottom line is -- it is unpredictable how parameters are passed,

How parameters are passed must _always_ be completely predictable.

But no, it is clearly documented as out of programmers control.
Khm... please read again the topic, at least that was completely clarified so far.

Let's summarize what has been clarified -
The parameter declared with const -
- is guaranteed to be constant. It will not change in future versions of FPC.
- has nothing to do with how the parameter is passed -- the compiler is free to decide to pass it by value or by reference. Programmer must not make assumptions about it.

Therefore:

- you can freely use const with parameters declared in procedures and functions header in your programme. And this programme is portable to Delphi, although in Delphi it will always be passed by reference, but there both sides (the caller and the called function) will be controlled by Delphi compiler, so everything is okay.

- you should however not use const when you declare external routine, as this routine must pass the parameters the same way as receiving function (written outside of your programme). The const can (and soon it will) break the call.
It can happen even if the original function is from the library written in FPC, as (different version of) compiler might have decided differently when it was compiling the library where the function is implemented!

- never use const when writing routines which are intended to be exported from shared library. You will not know later how to import this in the program, as you will not know how the original function in library handles it.

Don't hasitate to use const everywhere, except in special cases (not to be used for communication with outside world, as when creating libraries, using external routines, translating C headers).

I don't think there is much left to be said. Thank you very much, 440bx, for making me think about it. Now I am aware that there really are cases where I must not use const parameters, so I really learned from this conversation.

[Zoran]

...
Figured it out, finally. 
 ...
conclusion: in FPC (not Delphi), const can be used with API functions the same way it can be used with any Pascal function/procedure.

No! It just happens to pass it the same way, and it is in no way guaranteed.
This is exactly the case where you should either use var/constref or nothing, depending of what is needed!

[440bx]

No! It just happens to pass it the same way, and it is in no way guaranteed.
This is exactly the case where you should use var or constref!

You are mistaken.  As PascalDragon said, in FPC, "const" only means "constant" it does not say anything about how the parameter is passed, it only restricts what can be done to it, which is, read it, not write to it.

[Zoran]

No! It just happens to pass it the same way, and it is in no way guaranteed.
This is exactly the case where you should use var or constref!

You are mistaken.  As PascalDragon said, in FPC, "const" only means "constant" it does not say anything about how the parameter is passed, it only restricts what can be done to it, which is, read it, not write to it.

But, the program which uses the function has to pass it the same way as the original function expects it. Right?

[440bx]

But, the program which uses the function has to pass it the same way as the original function expects it. Right?

Yes, definitely and "const" does not alter that.  It only informs the compiler that the parameter cannot be assigned to, that's all it does (in FPC, in Delphi, it's another story.)

I attached a modified version of the SortCount API implementation with a _correct_ WriteFile API definition (compile and run the InputGenerator program before running SortCount.)  Note that, "const" is used to specify that the buffer pointer is not modified and, since the buffer pointer is now typed (as it should be), the call works as it should.   "const" in that case is mostly ornamental since FPC couldn't stop WriteFile from changing the pointer to the buffer anyway.  The reason for the "mostly" is because the compiler could (and should) use that information to generate more efficient code.

The generated code for the WriteFile call is:

Code: Text  [Select][+][-]

1.  
2. SortCount.lpr:318                 Ok := WriteFile(FileHandle,
3. 00401967 6a00                     push   $0x0                 ; overlapped = nil
4. 00401969 8d45e4                   lea    -0x1c(%ebp),%eax    
5. 0040196C 50                       push   %eax                 ; address of "NumberOfBytesWritten" since it's a var
6. 0040196D ff75ec                   pushl  -0x14(%ebp)          ; number of bytes to write - passed by value
7. 00401970 ff75f8                   pushl  -0x8(%ebp)           ; buffer address - passed by value, as it should be
8. 00401973 ff75f0                   pushl  -0x10(%ebp)          ; file handle passed by value
9. 00401976 e885f6ffff               call   0x401000 <_$dll$kernel32$WriteFile>
10. 0040197B 8945e8                   mov    %eax,-0x18(%ebp)
11. SortCount.lpr:324                 if not Ok then Error(3);
12. 0040197E 85c0                     test   %eax,%eax
13. ...
14. ... etc
15.  

"const" has no effect on how the parameter is passed to the API function.

HTH.

ETA:

@Zoran,

I'm pleased you found the conversation useful and you're most welcome.

[Zoran]

But, the program which uses the function has to pass it the same way as the original function expects it. Right?

Yes, definitely and "const" does not alter that.  It only informs the compiler that the parameter cannot be assigned to, that's all it does (in FPC, in Delphi, it's another story.)

Does not alter what? When compiling your external function declaration, the compiler has no way of knowing what the original function expects.

If the compiler could take look into original function, we would not need to write external declarations at all.

The programmer should instruct the compiler what to use - by using var or constref if "by reference" is expected, or nothing if "by value" is expected.
When you put const, you are telling the compiler to use whatever. And then the compiler will have to guess wheter to pass by value or by reference, and it's decision has fifty percent to be what the original function expects.

You just had luck with your test. You might not with next fpc version. Or with another function with the same compiler version.

Maybe a CONSTREF for any exported functions should be forced when doing LIB code?

At least emit a warning. And same with external declarations.

[440bx]

Does not alter what?

Does not alter the way the parameter is passed.  That's what "const" does not alter (true in FPC, not in Delphi.)

When compiling your external function declaration, the compiler has no way of knowing what the original function expects.

That's correct.  It relies entirely on the external function declaration.  That's why the declaration is required.

If the compiler could take look into original function, we would not need to write external declarations at all.

Correct but, that doesn't have anything to do with the use of "const".

When you put const, you are telling the compiler to use whatever. And then the compiler will have to guess wheter to pass by value or by reference, and it's decision has fifty percent to be what the original function expects.

No.  That is incorrect.  When specifying "const", the compiler is being told that the value of the parameter cannot be changed by the function/procedure, that's all it does.  Nothing else.

The programmer should instruct the compiler what to use - by using var or constref if "by reference" is expected, or nothing if "by value" is expected.

Correct but, in addition to that, in the case of "nothing or by value", the modifier "const" can be added to inform the compiler that the parameter cannot be assigned to.

You just had luck with your test. You might not with next fpc version. Or with another function with the same compiler version.

There was no luck involved at all.  You could change the declaration, which in the example I attached is currently

Code: Pascal  [Select][+][-]

1. function WriteFile(FileHandle           : THANDLE;
2.              const Buffer               : pointer;
3.                    NumberOfBytesToWrite : DWORD;
4.                var NumberOfBytesWritten : DWORD;
5.                    Overlapped           : POverlapped)
6.          : BOOL; stdcall; external 'kernel32';
7.  

to:

Code: Pascal  [Select][+][-]

1. function WriteFile(const FileHandle           : THANDLE;
2.                    const Buffer               : pointer;
3.                    const NumberOfBytesToWrite : DWORD;
4.                      var NumberOfBytesWritten : DWORD;    // DON'T change "var" to "const", that would make the definition incorrect.
5.                    const Overlapped           : POverlapped)
6.          : BOOL; stdcall; external 'kernel32';
7.  

and the code generated would be identical and the API call would work fine because "const" does not alter the way the parameter is passed.

Just to save you the hassle, I did it and, here is the result (feel free to change the declaration to match the one above and run the code to convince yourself.)

Code: ASM  [Select][+][-]

1. SortCount.lpr:328                 Ok := WriteFile(FileHandle,
2. 00401967 6a00                     push   $0x0
3. 00401969 8d45e4                   lea    -0x1c(%ebp),%eax
4. 0040196C 50                       push   %eax
5. 0040196D ff75ec                   pushl  -0x14(%ebp)
6. 00401970 ff75f8                   pushl  -0x8(%ebp)
7. 00401973 ff75f0                   pushl  -0x10(%ebp)
8. 00401976 e885f6ffff               call   0x401000 <_$dll$kernel32$WriteFile>
9. 0040197B 8945e8                   mov    %eax,-0x18(%ebp)
10. SortCount.lpr:334                 if not Ok then Error(3);
11. 0040197E 85c0                     test   %eax,%eax
12.  

You can compare with the code in my previous post and, you'll see it is identical.

Maybe a CONSTREF for any exported functions should be forced when doing LIB code?

For any external declaration, the presence of "const" is superfluous because it can only be used for parameters passed by value and, the compiler has no way to prevent the external function from locally altering their values. 

Whenever you see var, in an external declaration, you could change it to "constref" and it would make no difference, because what really matters is the "ref" part, which informs the compiler the parameter is being passed by reference.  In the case of an external declaration, the compiler has no way of enforcing the "const" part.

For instance, the WriteFile API definition could be

Code: Pascal  [Select][+][-]

1. function WriteFile(const    FileHandle           : THANDLE;
2.                    const    Buffer               : pointer;
3.                    const    NumberOfBytesToWrite : DWORD;
4.                    constref NumberOfBytesWritten : DWORD;
5.                    const    Overlapped           : POverlapped)
6.          : BOOL; stdcall; external 'kernel32';
7.  

and it would work just fine too, even though specifying "constref" for the NumberOfBytesWritten is a lie since its value is changed by the API call. 

That said, even though specifying NumberOfBytesWritten as constref is a bit of a lie, doing so could prevent a programmer inadvertently overwriting the value returned by the API call in the remainder of the function containing the call, which would most likely be a programming mistake.


The important part is the "ref" which causes the parameter to be passed by reference just as "var" does.


ETA:

NOTE: declaring parameters as "const" which are not really constant will work fine in 32bit but will likely cause problems in 64bit.

[Zoran]

I don't get it, isn't the sentence from documentation quite clear:

Remark: Contrary to Delphi, no assumptions should be made about how const parameters are passed to the underlying routine.
...

It is obvious that in your tests, the compiler decides to pass this by value.
But, as clearly documented, it is not guaranteed. You must not rely on that.

That is why PascalDragon said - this might change in future version.

It is normal that any undocumented behaviour can always change.

Did you try the same with parameters of larger data types?
For instance,

TInt100000 = Array[1..100000] of Integer;
...
function DoSomething(const N: TInt100000): Boolean;

I just doubt that compiler will decide to pass it by value.
As compiler is free to make a choice when it sees const, I expect it to be clever enough to pass it by reference.

Anyway, it is just implementation detail -- instead of testing what it actually does, better rely on documentation. So const parameters, which are now in your tests passed by value, could also be passed by reference.

[PascalDragon]

conclusion: in FPC (not Delphi), const can be used with API functions the same way it can be used with any Pascal function/procedure.  Untyped parameters should _never_ be used in an API definition because neither C nor C++ (the two languages used to implement the O/S) do automatic dereferencing.

Untyped parameters can be used with C/C++ functions, because for them it's simply a pointer parameter.
This Pascal declaration:

Code: Pascal  [Select][+][-]

1. procedure Something(const Data; Len: cint);

Is equivalent to this C/C++ declaration:

Code: C  [Select][+][-]

1. void Something(void* Data; int Len);

And you can see that it works correctly if you don't use a Pointer type variable as input parameter:

Code: Pascal  [Select][+][-]

1. Something(SomeIntVar, SizeOf(SomeIntVar));

This will result in @SomeIntVar being passed to Something and the C code will happily work with it.
Now if you use a Pointer you need to dereference it so that the correct value is passed:

Code: Pascal  [Select][+][-]

1. Something(SomePointerVar^, MyDataSize));

Cause this way SomePointerVar will be passed, otherwise @SomePointerVar will be passed. Again that's the same as with the Move and FillChar procedures.

TL;DR: Importing C/C++ functions using untyped pointers is no problem, you just need to pass the parameters correctly.

untyped parameters alway pass the address of what you passed in

In the example I posted above, it didn't pass the address, it passed the value.

Please, explain why in the above example it passes the value but, when used in the definition of the WriteFile API, it passes the address.

I cannot see what causes the parameter to be passed one way in one case and another way in the other case.

As I understand it (sorry if I am wrong, but I would also want this clarifyed), in case of const untyped parameters there are two things which can switch to passing the reference to value istead of the value itself:

1. using const - this may or may not decide to do so. Then we have
  1a - const decides to make the reference (so now we have refernce to "buffer") OR
  1b - const decides not to make the reference (so we still have "buffer" itself)
2. the param is untyped - this will always make the reference (pointer) to what we got from 1, so what is actually passed may be:
  2a - the reference to reference (pointer to pointer) to buffer (in case we got 1a in step 1.) OR
  2b - the reference (pointer) to buffer (in case we had 1b in step 1.)

* Note that, the actual order of 1 and 2 have no significance for this example (if step 2 actually happens before what I put under step 1, we end up with same after both steps).

So, it is not strange that you get different behaviour with the two routines, once the compiler took aproach 2a, and the other time, it took 2b.

Untyped parameters always take the address of the expression you pass in. It's always been this way and it will always be this way and I've already filed a bug report that this gets documented properly. If you pass in a pointer variable the address to the pointer variable will be used except if you pass it as PointerVar^. Again: this is the same as when calling Move or FillChar.

I don't get it, isn't the sentence from documentation quite clear:

Remark: Contrary to Delphi, no assumptions should be made about how const parameters are passed to the underlying routine.
...

It is obvious that in your tests, the compiler decides to pass this by value.
But, as clearly documented, it is not guaranteed. You must not rely on that.

That is why PascalDragon said - this might change in future version.

It is normal that any undocumented behaviour can always change.

Don't forget the following paragraph in the documentation:

An exception is the stdcall calling convention: for compatibility with COM standards, large const parameters are passed by reference.

If you use stdcall then const behaves as would be expected by COM/Windows compatibility.

[440bx]

I don't get it, isn't the sentence from documentation quite clear:

Remark: Contrary to Delphi, no assumptions should be made about how const parameters are passed to the underlying routine.
...

It is obvious that in your tests, the compiler decides to pass this by value.
But, as clearly documented, it is not guaranteed. You must not rely on that.

You can rely on that as long as the parameter(s) fit in a register.  With Delphi, the problem is that Borland associated behavior with "const" that it should not have.  "const" should mean constant, nothing else, Delphi breaks that rule, FPC does not.

That is why PascalDragon said - this might change in future version.

It is normal that any undocumented behaviour can always change.

Yes and, that leads to potential problems when interfacing with a different language - usually C - which may not implement the semantics of "const" the same way.

Did you try the same with parameters of larger data types?
For instance,

TInt100000 = Array[1..100000] of Integer;
...
function DoSomething(const N: TInt100000): Boolean;

I just doubt that compiler will decide to pass it by value.
As compiler is free to make a choice when it sees const, I expect it to be clever enough to pass it by reference.

The compiler will not pass a data structure that does not fit in a register (or two sometimes) by value.  It will pass a reference to a copy.  With FPC, when compiling for 32bit, the callee will use the reference to make a copy of the data structure no matter how large it is.  In 64bit, the compiler creates the copy first then, passes a reference to the copy.

What "const" changes in that case is, since the compiler has been told the parameter values will not change (and hopefully, the programmer didn't lie to the compiler when specifying "const") the compiler does not have to make a copy of the data structure but, the important thing is, whether "const" is specified or not, the compiler will pass the parameter the same way.

Anyway, it is just implementation detail -- instead of testing what it actually does, better rely on documentation. So const parameters, which are now in your tests passed by value, could also be passed by reference.

The part you are misunderstanding is that "const" will not change how the compiler passes the parameter.  What it will change is whether or not the compiler decides to create a copy of the parameter values.

The problem with being "cavalier" with "const" (as I was in the examples I provided) is that when interfacing with other languages - such as C - the way the different compilers will treat the parameter may vary (they usually will, since as you stated, it's an implementation detail.)

Personally, I'll never use "const" with parameters because its semantics are different between Delphi and FPC and, I try to make the _same_ code compile with both compilers, that precludes using "const".


ETA:

Untyped parameters can be used with C/C++ functions, because for them it's simply a pointer parameter.

NO.  That is incorrect.  C, unlike FPC/Delphi does NOT do automatic dereferencing.   That problem is clearly evidenced in the _incorrect_ definition of the WriteFile API.  As currently defined, when the programmer passes the pointer to the buffer, FPC passes a pointer to the buffer pointer which causes the call not to work as it should.

As you pointed out, that behavior is documented but, specifying an untyped parameter for the Buffer is incorrect because C, unlike FPC, will NOT dereference the pointer.

Don't forget the following paragraph in the documentation:

An exception is the stdcall calling convention: for compatibility with COM standards, large const parameters are passed by reference.

If you use stdcall then const behaves as would be expected by COM/Windows compatibility.

I realize there are exceptions to how "const" is going to behave. 

[PascalDragon]

Untyped parameters can be used with C/C++ functions, because for them it's simply a pointer parameter.

NO.  That is incorrect.  C, unlike FPC/Delphi does NOT do automatic dereferencing.   That problem is clearly evidenced in the _incorrect_ definition of the WriteFile API.  As currently defined, when the programmer passes the pointer to the buffer, FPC passes a pointer to the buffer pointer which causes the call not to work as it should.

As you pointed out, that behavior is documented but, specifying an untyped parameter for the Buffer is incorrect because C, unlike FPC, will NOT dereference the pointer.

THE AUTOMATIC DEREFERENCING IS NOT RELEVANT. On the ABI level the function simply receives a pointer to the parameter. On the C/C++ callee side you use it as such. That the FPC callee side dereferences it for you is of NO consequence for a C/C++ calllee.
Thus if you have a void* x parameter in C/C++ it's perfectly valid to use const X in FPC/Delphi as long as it's not necessary for the parameter to be NULL/Nil.

[440bx]

THE AUTOMATIC DEREFERENCING IS NOT RELEVANT.

The pointer dereferencing is VERY relevant.  The C function isn't going to dereference its parameter _twice_ as is required if the buffer is declared as untyped.  It will only dereference once because it is declared as a pointer, NOT as a pointer to a pointer.

Thus if you have a void* x parameter in C/C++ it's perfectly valid to use const X in FPC/Delphi as long as it's not necessary for the parameter to be NULL/Nil.

No, it is NOT.  To drive the point home, if you have a declaration "void* p" and you pass p to an API that takes a "void *" parameter, the VALUE of p is what is passed to the function, NOT the address of p.  The function is going to dereference the pointer ONCE to access whatever p points to.

In FPC, if you have a declaration "p : pointer" and you pass p to an API that is declared as taking an untyped parameter, what FPC passes is the ADDRESS OF P, not the value of p.  BIG DIFFERENCE.  Dereferencing ONCE which is what WriteFile does to access the buffer, does not yield a pointer to the buffer, it yields a pointer to the value of the pointer instead.

The definition of WriteFile is currently _incorrect_.  To boot, the definition of WriteFileEx which follows it immediately is correct (the parameter is declared as "Buffer : pointer" - as it should be), if a programmer simply changes WriteFile to WriteFileEx and adds a pointer to a completion routine, the result will either be garbage or an access violation because the Buffer pointer parameter is now incorrect.