Recent

Author Topic: Strange capture denial and acceptance for Anonymous Functions and Function refs  (Read 789 times)

Bogen85

  • Full Member
  • ***
  • Posts: 146
This does not work, a direct capture of nested procedure.
Code: Pascal  [Select][+][-]
  1. {$mode delphi}
  2. {$modeswitch anonymousfunctions}
  3. {$modeswitch functionreferences}
  4.  
  5. program nocapture;
  6.  
  7. type
  8.   tSimpleProc = reference to procedure;
  9.  
  10. procedure main;
  11.  
  12.   var
  13.     tcf: tSimpleProc;
  14.     x: integer;
  15.  
  16.   procedure nested;
  17.     begin
  18.       writeln ('Hello from nested! x: ', x);
  19.       inc(x);
  20.     end;
  21.  
  22.   begin
  23.     x := 100;
  24.  
  25.     tcf := procedure begin nested; end;
  26.     tcf();
  27.     tcf();
  28.  
  29.   end;
  30.  
  31. begin
  32.   main;
  33. end.
  34.  

The above results in:

Free Pascal Compiler version 3.3.1 [2022/09/26] for x86_64
Copyright (c) 1993-2022 by Florian Klaempfl and others
Target OS: Linux for x86-64
Compiling src/nocapture.pas
nocapture.pas(25,28) Error: Symbol "nested" can not be captured
nocapture.pas(34) Fatal: There were 1 errors compiling module, stopping
Fatal: Compilation aborted
Error: /home/shared-development/fpc_usr/lib/fpc/3.3.1/ppcx64 returned an error exitcode


This does work, example includes indirect capture to get the nested procedure to be captured.
Code: Pascal  [Select][+][-]
  1. {$mode delphi}
  2. {$modeswitch anonymousfunctions}
  3. {$modeswitch functionreferences}
  4.  
  5. program doescapture;
  6.  
  7. type
  8.   tSimpleProc = reference to procedure(const n: integer);
  9.  
  10. procedure main0;
  11.  
  12.   var
  13.     tcf: tSimpleProc;
  14.     tcf2: tSimpleProc;
  15.     x: integer;
  16.  
  17.   procedure nested(const n: integer);
  18.     begin
  19.       writeln ('Hello from nested! ', n, ' x: ', x);
  20.       inc(x);
  21.     end;
  22.  
  23.   begin
  24.     tcf := nested;
  25.     x := 100;
  26.  
  27.     procedure(const aArg: String) begin Writeln(aArg); end('Hello World');
  28.     procedure begin nested(1); end();
  29.     tcf2 := procedure(const n: integer) begin tcf(n); end;
  30.     tcf2(2);
  31.   end;
  32.  
  33.  
  34. procedure main1;
  35.   var
  36.     x: integer;
  37.  
  38.   begin
  39.     x := 132;
  40.     procedure begin writeln ('x = ', x) end();
  41.   end;
  42.  
  43. begin
  44.   main0;
  45.   main1;
  46. end.
  47.  

Produces this output:


Hello World
Hello from nested! 1 x: 100
Hello from nested! 2 x: 101
x = 132
« Last Edit: September 27, 2022, 02:06:46 am by Bogen85 »

Mr.Madguy

  • Hero Member
  • *****
  • Posts: 782
First of all, I'm not sure if FPC fully supports closures now.

Second thing - Delphi is considered to be standard and it throws the same error. Because, I guess, Nested relies on Main's stack frame, so it can be called from Main only.

Third thing - Delphi throws the same error for your second program too. I don't know, why it works for you. It shouldn't. Because it's exactly the same situation.

Code: Pascal  [Select][+][-]
  1. tcf := nested;
  2.  
This should be internally compiled into exactly the same:
Code: Pascal  [Select][+][-]
  1. tcf := procedure(const n: integer)
  2.     begin
  3.          nested(n);
  4.     end;
  5.  

May be it works due to some optimizations, so code is simply unwrapped?

You should understand, how closures work. Your program is turned into:
Code: Pascal  [Select][+][-]
  1. type
  2.   //There is no such thing, but, I guess, you get this idea
  3.   //It's:
  4.   //tSimpleProc = record
  5.   //  Intf:IInterface
  6.   //  Proc:procedure;
  7.   //end;
  8.   //When it's called - it's something like tcf.Proc(tcf.Intf, ...);
  9.   //I.e. tcf.Intf is passed as Self
  10.   tSimpleProc = procedure of interface;
  11.  
  12.   ITemp = interface
  13.     procedure Temp;
  14.   end;
  15.  
  16.   TTemp = class(TInterfacedObject, ITemp)
  17.     procedure Temp;
  18.   end;
  19.  
  20. procedure TTemp.Temp;
  21. begin
  22.   //nested is inaccessible from here
  23.   //How can local variable be accessible from here?
  24.   //It's added as field to TTemp class
  25.   //I.e. captured
  26.   //Isn't possible for nested procedure
  27.   nested;
  28. end;
  29.  
  30. procedure main;
  31.  
  32. var
  33.   tcf: tSimpleProc;
  34.   x: integer;
  35.  
  36.   procedure nested;
  37.   begin
  38.     WriteLn ('Hello from nested! x: ', x);
  39.     Inc(x);
  40.   end;
  41.  
  42. begin
  43.   tcf := TTemp.Create.Temp;
  44.   tcf();
  45.   tcf();
  46. end;
  47.  

P.S. Such behavior can actually be achieved via turning nested into TTemp's method. But Delphi's docs don't mention such behavior and I'm not sure, what side effects it can have.
« Last Edit: September 27, 2022, 06:36:49 pm by Mr.Madguy »
29.12.2021 - migration to DynamicData 4.1 is completed - complete overhaul of data access driver.
My project still requires full Delphi 2009 support to be ported to Lazarus.
It's time to finally do it, because Delphi 2009 is 13 years old.

Bogen85

  • Full Member
  • ***
  • Posts: 146
First of all, I'm not sure if FPC fully supports closures now.

@Mr.Madguy:

From what I can tell, FPC (built from latest git, does support closures).
I was playing around them and they definitely work as if they are closures (I did various things where I passed them off into various contexts and they worked as expected for closures), which is how I discovered the above behavior.

See this: https://forum.lazarus.freepascal.org/index.php/topic,59468.0.html Topic: Feature announcement: Function References and Anonymous Functions

Specifically this reply:

@VTwin
Function references encapsulates the function together with it's lexical context (at least - part of) which is also knows as a "closure" (C#, C++). May be the  most familiar example in FPC is the method delegate (procedure of object), which keeps together the method address and the Self pointer of the instance it belongs. Thus, the receiver object can execute the method in the context of the original object.

Function references just broadens the context by being able to keep it bigger - including variables from the current scope, etc.

But the devil is into the details, and I wonder how useful it will be in practice considering the dynamic nature of the class instances in FPC.

I don't have access to Delphi (because I don't have any reasonable access to any Windows system), so I can't compare the two (FPC and Delphi)

Code: Bash  [Select][+][-]
  1. $ fpc i-want-the-version
  2. Free Pascal Compiler version 3.3.1 [2022/09/26] for x86_64
  3. Copyright (c) 1993-2022 by Florian Klaempfl and others
  4. Target OS: Linux for x86-64
  5. Compiling i-want-the-version
  6. Fatal: Cannot open file "i-want-the-version"
  7. Fatal: Compilation aborted
  8. Error: /home/shared-development/fpc_usr/lib/fpc/3.3.1/ppcx64 returned an error exitcode
  9.  

I built FPC yesterday from git to play around with function references and anonymous functions
« Last Edit: September 27, 2022, 09:42:58 pm by Bogen85 »

Mr.Madguy

  • Hero Member
  • *****
  • Posts: 782
From what I can tell, FPC (built from latest git, does support closures).
I was playing around them and they definitely work as if they are closures (I did various things where I passed them off into various contexts and they worked as expected for closures), which is how I discovered the above behavior.
Great news then. My Delphi project relies on this feature. I guess, it has been last FPC's incompatibility with modern Delphi versions. I'll try to compile my project via FPC, when I'll have time to do it.

But again, while closures are treated as nested procedures/functions, nested procedures/functions are accessible from each other and in theory capturing nested procedures/functions is possible - such behavior isn't described in Delphi's docs, so it shouldn't be supported. Only variables can be captured.

According to announce your second program works because:
1) Temp variables are created implicitly, when procedure/function types are used instead of closures - they're captured instead of procedures/functions themselves (Delphi incompatible)
2) Unassigned closure is treated as simple nested procedure/function
« Last Edit: September 28, 2022, 08:48:09 am by Mr.Madguy »
29.12.2021 - migration to DynamicData 4.1 is completed - complete overhaul of data access driver.
My project still requires full Delphi 2009 support to be ported to Lazarus.
It's time to finally do it, because Delphi 2009 is 13 years old.

PascalDragon

  • Hero Member
  • *****
  • Posts: 4756
  • Compiler Developer
First of all, I'm not sure if FPC fully supports closures now.

As mentioned in the link provided by Bogen85, yes, it does since end of May.

Second thing - Delphi is considered to be standard and it throws the same error. Because, I guess, Nested relies on Main's stack frame, so it can be called from Main only.

Third thing - Delphi throws the same error for your second program too. I don't know, why it works for you. It shouldn't. Because it's exactly the same situation.

FPC supports a bit more functionality than Delphi does. E.g. assigning a nested function to a function reference is perfectly fine, because the compiler can rework the code so that it will work.

Code: Pascal  [Select][+][-]
  1. tcf := nested;
  2.  
This should be internally compiled into exactly the same:
Code: Pascal  [Select][+][-]
  1. tcf := procedure(const n: integer)
  2.     begin
  3.          nested(n);
  4.     end;
  5.  

Not quite. If you assign a nested function to a function reference the end result will essentially be this:

Code: Pascal  [Select][+][-]
  1. procedure main;
  2. type
  3.   tSimpleProc = interface
  4.     procedure Invoke(const n: integer);
  5.   end;
  6.  
  7.   TCapturer = class(TInterfacedObject, tSimpleProc)
  8.     x: integer;
  9.     procedure nested(const n: integer);
  10.  
  11.     procedure tSimpleProc.Invoke = nested;
  12.   end;
  13.  
  14.   procedure tSimpleProc.Invoke(const n: integer)
  15.   begin
  16.     writeln ('Hello from nested! ', n, ' x: ', x);
  17.     inc(x);
  18.   end;
  19.  
  20. var
  21.   capturer: TCapturer;
  22.   capturer_keepalive: IUnknown;
  23.  
  24.   procedure nested(const n: integer);
  25.   begin
  26.     capturer.nested(n);
  27.   end;
  28.  
  29. var
  30.   tcf: TSimpleProc;
  31. begin
  32.   capturer := TCapturer.Create;
  33.   capturer_keepalive := capturer;
  34.   tcf := capturer as tSimpleProc;
  35. end;

Essentially it moves the code of the original nested function into a method of the capturer class and has the nested function call that instead. This allows the nested function to be used as a function reference and as a nested function at the same time.

You should understand, how closures work. Your program is turned into:
Code: Pascal  [Select][+][-]
  1. type
  2.   //There is no such thing, but, I guess, you get this idea
  3.   //It's:
  4.   //tSimpleProc = record
  5.   //  Intf:IInterface
  6.   //  Proc:procedure;
  7.   //end;
  8.   //When it's called - it's something like tcf.Proc(tcf.Intf, ...);
  9.   //I.e. tcf.Intf is passed as Self
  10.   tSimpleProc = procedure of interface;
  11.  

Not quite. There is no concept of “procedure of interface”, instead a “reference to procedure” is turned into an interface with a single method Invoke. You can see this, because you can manually implement a function reference in a custom class just like an ordinary interface or you can even inherit from it.

1) Temp variables are created implicitly, when procedure/function types are used instead of closures - they're captured instead of procedures/functions themselves (Delphi incompatible)

It's not “Delphi incompatible”, because Delphi-compatible code still works. However it's an enhancement to what Delphi allows.

2) Unassigned closure is treated as simple nested procedure/function

An unassigned anonymous method works in Delphi as well (as long as you only capture stuff that Delphi supports ;) ). Delphi however converts that to a method of the capturer object nevertheless and calls it. FPC is a bit more cheap here and simply calls it like a nested function, cause for the user it will have the same effect.

This does not work, a direct capture of nested procedure.

I had not thought about that use case, but in essence it should be possible as well, considering that it works when assigning a nested function. Please do a feature request on our bug tracker.

Mr.Madguy

  • Hero Member
  • *****
  • Posts: 782
Not quite. There is no concept of “procedure of interface”, instead a “reference to procedure” is turned into an interface with a single method Invoke. You can see this, because you can manually implement a function reference in a custom class just like an ordinary interface or you can even inherit from it.
Problem is - it isn't right. Why? Because TCapturer can have several Invoke methods. Because closures aren't only about making callbacks with data bound to them. They're about declaring classes without actually declaring classes. Because they're thing from scripted languages, that lack explicit class declaration syntax. And therefore reference to procedure/function should also store pointer to specific method - not pointer to interface only. Therefore it's something like "procedure/function of interface".

Sorry, checked it in Delphi - indeed each reference to procedure/function is separate interface, so no pointer to specific procedure/function is needed.

Example:
Code: Pascal  [Select][+][-]
  1. type
  2.   TMyMethod = reference to procedure;
  3.   TMyClass = record
  4.     MyInc:TMyMethod;
  5.     MyDec:TMyMethod;
  6.     MyWrite:TMyMethod;
  7.   end;
  8.  
  9. function ConstructClass:TMyClass;
  10.   var X:Integer;
  11. begin
  12.   X := 0;
  13.   Result.MyInc := procedure begin Inc(X); end;
  14.   Result.MyDec := procedure begin Dec(X); end;
  15.   Result.MyWrite := procedure begin WriteLn(X); end;
  16. end;
  17.  
« Last Edit: September 28, 2022, 10:04:07 am by Mr.Madguy »
29.12.2021 - migration to DynamicData 4.1 is completed - complete overhaul of data access driver.
My project still requires full Delphi 2009 support to be ported to Lazarus.
It's time to finally do it, because Delphi 2009 is 13 years old.

PascalDragon

  • Hero Member
  • *****
  • Posts: 4756
  • Compiler Developer
Not quite. There is no concept of “procedure of interface”, instead a “reference to procedure” is turned into an interface with a single method Invoke. You can see this, because you can manually implement a function reference in a custom class just like an ordinary interface or you can even inherit from it.
Problem is - it isn't right. Why? Because TCapturer can have several Invoke methods. Because closures aren't only about making callbacks with data bound to them. They're about declaring classes without actually declaring classes. Because they're thing from scripted languages, that lack explicit class declaration syntax. And therefore reference to procedure/function should also store pointer to specific method - not pointer to interface only. Therefore it's something like "procedure/function of interface".

Sorry, checked it in Delphi - indeed each reference to procedure/function is separate interface, so no pointer to specific procedure/function is needed.

You think we - or mainly the one who did most of the work regarding function references - didn't do our homework to find out what's going on behind the scenes? ;D

Mr.Madguy

  • Hero Member
  • *****
  • Posts: 782
You think we - or mainly the one who did most of the work regarding function references - didn't do our homework to find out what's going on behind the scenes? ;D
May be my memory fools me or may be it's from older Delphi versions, like 2009. But I always thought, that I investigated it long time ago and that it worked exactly this way.
29.12.2021 - migration to DynamicData 4.1 is completed - complete overhaul of data access driver.
My project still requires full Delphi 2009 support to be ported to Lazarus.
It's time to finally do it, because Delphi 2009 is 13 years old.

Bogen85

  • Full Member
  • ***
  • Posts: 146
I had not thought about that use case, but in essence it should be possible as well, considering that it works when assigning a nested function. Please do a feature request on our bug tracker.

Done. See https://gitlab.com/freepascal.org/fpc/source/-/issues/39926

PascalDragon

  • Hero Member
  • *****
  • Posts: 4756
  • Compiler Developer
You think we - or mainly the one who did most of the work regarding function references - didn't do our homework to find out what's going on behind the scenes? ;D
May be my memory fools me or may be it's from older Delphi versions, like 2009. But I always thought, that I investigated it long time ago and that it worked exactly this way.

More likely your memory fools you, because the Delphi behaviour didn't change between versions. ;)

I had not thought about that use case, but in essence it should be possible as well, considering that it works when assigning a nested function. Please do a feature request on our bug tracker.

Done. See https://gitlab.com/freepascal.org/fpc/source/-/issues/39926

Thank you. At least it won't be forgotten now. :)

 

TinyPortal © 2005-2018