Recent

Author Topic: Feature announcement: Function References and Anonymous Functions  (Read 47347 times)

PascalDragon

  • Hero Member
  • *****
  • Posts: 5446
  • Compiler Developer
Dear Free Pascal Community,

The Free Pascal Developer team is pleased to finally announce the addition of a long awaited feature, though to be precise it's two different, but very much related features: Function References and Anonymous Functions. These two features can be used independantly of each other, but their greatest power they unfold when used together.

These features are based on the work by Blaise.ru, so thank you very much and I hope you're doing well considering the current situation.

In the following we'll highlight both features separately and then we'll take a look at using them together.

Function References

Function References (also applicable names are Procedure References and Routine References, in the following only Function References will be used) are types that can take a function (or procedure or routine), method, function variable (or procedure variable or routine variable), method variable, nested function (or nested procedure or nested routine) or an anonymous function (or anonymous procedure or anonymous routine) as a value. The function reference can then be used to call the provided function just like other similar routine pointer types. In contrast to these other types nearly all function-like constructs can be assigned to it (the only exception are nested function variables (or nested procedure variables or nested routine variables), more about that later on) and then used or stored.

Function references are enabled with the modeswitch FUNCTIONREFERENCES (the following examples will assume that this modeswitch is provided).

A function reference is declared as follows:

REFERENCE TO FUNCTION|PROCEDURE [(argumentlist)][: resulttype;] [directives;]

Examples:

Code: Pascal  [Select][+][-]
  1. type
  2.   TProcLongInt = reference to procedure(aArg: LongInt); stdcall;
  3.   TFuncTObject = reference to function(aArg: TObject): TObject;

Like other function pointer types function references can also be declared as generic:

Code: Pascal  [Select][+][-]
  1. type
  2.   generic TGenericProc<T> = reference to procedure(aArg: T);

As you can see, once function references are enabled you can't use the identifier "REFERENCE" as part of an alias declaration without using "&":

Code: Pascal  [Select][+][-]
  1. type
  2.   someref = reference; // will fail
  3.   someref = &reference; // correct fix
  4.  
  5. var
  6.   somevar: reference; // will fail
  7.   somevar: &reference; // correct fix

A function reference variable can then be called like any other function pointer type:

Code: Pascal  [Select][+][-]
  1. var
  2.   p: TProcLongInt;
  3. begin
  4.   p := @SomeLongIntProc;
  5.   p(42);
  6. end.

If a function reference has no parameters then you need to use "()" nevertheless in the FPC/ObjFPC modes like for other function pointer types:

Code: Pascal  [Select][+][-]
  1. type
  2.   TProc = reference to procedure;
  3. var
  4.   p: TProc;
  5. begin
  6.   p := @SomeProcedure;
  7.   p(); // required
  8.   p; // this can be used e.g. in mode Delphi
  9. end.

Like other function pointer types they can also be declared anonymously as part of a variable, field declaration (but not as part of a paramater declaration):

Code: Pascal  [Select][+][-]
  1. var
  2.   f: reference to function: LongInt;
  3.  
  4. type
  5.   TTest = class
  6.     f: reference to procedure;
  7.   end;

They get their great power from a point that is for once not considered an implementation detail: function references are in fact internally declared as reference counted interfaces with a single Invoke() method of the provided signature. So the above examples are in fact declared like this:

Code: Pascal  [Select][+][-]
  1. type
  2.   TProcLongInt = interface(IInterface)
  3.     procedure Invoke(aArg: LongInt); stdcall; overload;
  4.   end;
  5.  
  6.   TFuncTObject = interface(IInterface)
  7.     procedure Invoke(aArg: TObject): TObject; overload;
  8.   end;
  9.  
  10.   generic TGenericProc<T> = interface(IInterface)
  11.     procedure Invoke(aArg: T); overload;
  12.   end;

This has a few implications:
  • in the RTTI this will appear like a normal interface
  • it reacts to the $M directive like a normal interface
  • it is a managed type
  • it has no valid GUID
  • it can be implemented by a class
  • it can be inherited from

Especially the last two points are important.

That the interface can be implemented means that much more functionality and state can be added to a function reference:

Code: Pascal  [Select][+][-]
  1. type
  2.   TFunc = reference to function: LongInt;
  3.  
  4.   TSomeImpl = class(TInterfacedObject, TFunc)
  5.     f: LongInt;
  6.     function Invoke: LongInt;
  7.   end;
  8.  
  9. function TSomeImpl.Invoke: LongInt;
  10. begin
  11.   Result := f;
  12. end;
  13.  
  14. var
  15.   t: TSomeImpl;
  16.   f: TFunc;  
  17. begin
  18.   t := TSomeImpl.Create;
  19.   f := t;
  20.   Writeln(f()); // will write 0
  21.   t.f := 42;
  22.   Writeln(f()); // will write 42
  23.   f := Nil; // the usual warnings about mixing classes and interface apply!
  24. end.

As function references don't have valid GUIDs you can't however use QueryInterface() or the as-operator to retrieve it. Using the low level interface related functions of TObject however will work.

An interface that inherits from a function reference is still considered invokable by the compiler, so it can still be used like an ordinary function reference could, but you can also add additional methods including overloads for Invoke itself:

Code: Pascal  [Select][+][-]
  1. type
  2.   TTest = reference to procedure(aArg: TObject);
  3.  
  4.   TTestEx = interface(TTest)
  5.     function Invoke: TObject; overload;
  6.   end;
  7.  
  8. var
  9.   f: TTestEx;
  10.   o: TObject;
  11. begin
  12.   f := TSomeImplEx.Create;
  13.   o := f();
  14.   f(o);
  15. end.

This is for example described by Stefan Glienke on his blog. His linked example won't work as-is however due to missing functionality in Rtti.TValue.

As mentioned initially you can assign a nested function to a function reference, but not a nested function variable. There is no real technical reason for this, but it's instead a design choice based on how function references are assumed to behave: they are assumed to be valid beyond their scope (this will become clearer when combined with anonymous functions in the third part), so they can for example be returned from a function or stored in some class instance and can still be considered valid. However a nested function variable is no longer useable once the function frame it was retrieved has been left (for a nested function the compiler can safely convert it in a way that this is no problem, but for a nested function variable it simply can't).
One could argue that the same is true for method pointers and method variables as they aren't callable anymore once their class instance is freed however these are much more common in the Object Pascal world while nested function variables are very seldom used, thus the dangers of the former are much more apparent than the dangers of the later.
For this reason assigning nested function variables to function references is prohibited.

Anonymous Functions

Anonymous Functions (or Anonymous Procedures or Anonymous Routines, in the following simply Anonymous Functions) are routines that have no name associated with them and are declared in the middle of a code block (for example on the right side of an expression or as a parameter for a function call). However they can just as well be called directly like a nested function (or nested procedure or nested routine) would.

Anonymous functions are enabled with the modeswitch ANONYMOUSFUNCTIONS (the following examples will assume that this modeswitch is provided).

An anonymous function is declared as follows:

FUNCTION|PROCEDURE [(argumentlist)][[resultname]: resulttype;] [directives;]
[[VAR|TYPE|CONST section]|[nested routine]]*
BEGIN
[STATEMENTS]
END


As can be seen an anonymous function looks like a regular function (or procedure or routine) with the most important differences being that it does not have a name and that it isn't terminated by a semicolon (because it's essentially an expression). Because it doesn't have a name for modes that don't have the implicit RESULTvariable it's allowed to explicitely name the result variable (even in modes that do have the RESULT variable) like is the case with operator overloads.

It's possible to directly call an anonymous function in which case it essentially behaves like a nested function.

Like nested functions anonymous functions have access to the symbols (variables, functions, etc.) of the surrounding scope including Self if the surrounding scope is a method. Accessing such a symbol is named “capturing” and is one of the core concepts of anonymous functions.

Their main use however is when assigning them to one of the various function pointer types: function variables, method variables, nested function variables and function references. However not every anonymous function is assignable to every function pointer type as it depends on which symbols (if any) are captured from the surrounding scope. Unlike for non-anonymous function or method identifiers this assignment is however always done without the "@"-operator, because aside from calling one can't do much else with anonymous functions.
An anonymous function that captures no symbols at all (except for global symbols or static symbols) is assignable to all four function pointer types. If the anonymous function captures Self then it is no longer assignable to function variables, but still to the other three. And if it captures any local symbol then it's only assignable to nested function variables or function references.
In case of function variables, method variables and nested function variables anonymous functions behave just like their non-anonymous counterparts. The differences appear when they're used with function references which will be highlighted in the next part.

But first some examples:

Code: Pascal  [Select][+][-]
  1. type
  2.   TFunc = function: LongInt;
  3.  
  4. var
  5.   p: TProcedure;
  6.   f: TFunc;
  7.   n: TNotifyEvent;
  8. begin
  9.   procedure(const aArg: String)
  10.   begin
  11.     Writeln(aArg);
  12.   end('Hello World');
  13.  
  14.   p := procedure
  15.        begin
  16.              Writeln('Foobar');
  17.            end;
  18.   p();
  19.  
  20.   n := procedure(aSender: TObject);
  21.        begin
  22.              Writeln(HexStr(Pointer(aSender));
  23.            end;
  24.   n(Nil);
  25.  
  26.   f := function MyRes : LongInt;
  27.        begin
  28.              MyRes := 42;
  29.            end;
  30.   Writeln(f());
  31. end.

Anonymous Function References

As mentioned above the greatest power of the two new features comes when the two are combined: like a nested function an anonymous function can access symbols from the surrounding scope, however unlike for nested functions a anonymous function that has been assigned to a function reference can leave the scope where it has been declared in and it will then take the captured symbols with it.
For this purpose any variable or parameter that is captured by an anonymous function will become part of the implicitely created object instance (which shall be considered opaque) that will be assigned to the function reference instead of belonging to the original function. The original function will then reference these symbols using the object instance instead of its stack frame. This has the implication that changes to the symobls will be reflected in all anonymous function that capture that symbol.

For example:

Code: Pascal  [Select][+][-]
  1. type
  2.   TProc = reference to procedure;
  3.  
  4. procedure Test;
  5. var
  6.   i: LongInt;
  7.   p: TProc;
  8. begin
  9.   i := 42;
  10.   p := procedure
  11.        begin
  12.              Writeln(i);
  13.            end;
  14.            
  15.   p(); // will print 42
  16.  
  17.   i := 21;
  18.  
  19.   p(); // will print 21
  20. end;

Changes will those also be persistent across calls and different anonymous functions as long as they capture the same symbols:

Code: Pascal  [Select][+][-]
  1. type
  2.   TProc = reference to procedure;
  3.  
  4. procedure Test;
  5. var
  6.   i: LongInt;
  7.   p1, p2: TProc;
  8. begin
  9.   i := 42;
  10.   p1 := procedure
  11.         begin
  12.               Writeln(i);
  13.                   i := i * 2;
  14.             end;
  15.            
  16.   p1(); // will print 42
  17.  
  18.   p2 := procedure
  19.         begin
  20.                   Writeln(i);
  21.                 end;
  22.  
  23.   p1(); // will print 84
  24.   p2(); // will print 168
  25. end;

The lifetime of managed types captured by anonymous function references will be handled accordingly (they will stay alive as long as at least one anonymous function that has captured them is alive as well), however special care needs to be taken regarding manual memory management:

Code: Pascal  [Select][+][-]
  1. type
  2.   TProc = reference to procedure;
  3.  
  4. function Test: TProc;
  5. var
  6.   o: TObject;
  7. begin
  8.   o := TObject.Create;
  9.   Result := procedure
  10.             begin
  11.                           Writeln(o.ClassName);
  12.                         end;
  13.   o.Free;
  14. end;

Calling the function reference returned by Test will essentially result in use-after-free. And not freeing “o” at all will result in a memory leak.

Compatibility

The two features are by and large compatible to Delphi's Anonymous Methods. However FPC allows the assignment of anonymous functions to various function pointer types while Delphi restricts them to function references.
Also FPC handles the assignment of function, method and nested function variables to function variables slightly differently. Take the following code:

Code: Pascal  [Select][+][-]
  1. procedure Foo;
  2. begin
  3.   Writeln('Foo');
  4. end;
  5.  
  6. procedure Bar;
  7. begin
  8.   Writeln('Bar');
  9. end;
  10.  
  11. procedure Test;
  12. var
  13.   p: reference to procedure;
  14.   p2: procedure;
  15. begin
  16.   p2 := Foo;
  17.   p := p2;
  18.   p();
  19.   p2 := Bar;
  20.   p();
  21. end;

Delphi essentially generates the following:

Code: Pascal  [Select][+][-]
  1. procedure Test;
  2. var
  3.   p: reference to procedure;
  4.   p2: procedure;
  5. begin
  6.   p2 := Foo;
  7.   p := procedure
  8.        begin
  9.              p2();
  10.            end;
  11.   p();
  12.   p2 := Bar;
  13.   p();
  14. end;

This will result in the following output:

Code: [Select]
Foo
Bar

However FPC will generate the following:

Code: Pascal  [Select][+][-]
  1. procedure Test;
  2. var
  3.   p: reference to procedure;
  4.   p2, tmp: procedure;
  5. begin
  6.   p2 := Foo;
  7.   tmp := p2;
  8.   p := procedure
  9.        begin
  10.              tmp();
  11.            end;
  12.   p();
  13.   p2 := Bar;
  14.   p();
  15. end;

This will result in the following output:

Code: [Select]
Foo
Foo

This is more consistent with assignments of other function pointer types to function pointer types.

The Function References feature is available on all platforms which have the Classes feature available (so essentially everything except AVR) and Anonymous Functions themselves are available on all platforms (excluding the assignment to function references on platforms where these are missing). Yes, this includes platform like DOS where directives like “far” and “near” are handled accordingly (which means that these need to be compatible as well when assigning).

As these two features are rather complicated there might still be a huge bundle of bugs lurking around so I ask you to test them to year heart's content and report found bugs to the issues on GitLab so that we can fix as many of them as possible before the next major version (which is not yet planned, so don't worry ;) ).

Further RTL enhancements like the declaration of TProc<> or the addition of a TThread.Queue() that takes a function reference will come in the near future now that the basics on the compiler side are done. Maybe we can now also tackle ports of libraries like Spring4D and OmniThreadLibrary. There's also the idea to introduce a syntax to control whether symbols are captured by-reference (as currently) or by-value.

Enjoy!

Okoba

  • Hero Member
  • *****
  • Posts: 528
Re: Feature announcement: Function References and Anonymous Functions
« Reply #1 on: May 26, 2022, 09:52:59 pm »
Thank you so much to the team!

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11382
  • FPC developer.
Re: Feature announcement: Function References and Anonymous Functions
« Reply #2 on: May 26, 2022, 10:09:52 pm »
Are the modeswitches already always on for $mode Delphi ?

PascalDragon

  • Hero Member
  • *****
  • Posts: 5446
  • Compiler Developer
Re: Feature announcement: Function References and Anonymous Functions
« Reply #3 on: May 26, 2022, 10:13:07 pm »
Are the modeswitches already always on for $mode Delphi ?

Not yet, I first want to spot any mature hurdles with more testers and then we can make them default before the next major release (though at least they shouldn't affect exisiting code except for the point I had mentioned in my announcement post regarding reference, so we can make them default sooner as well *shrugs*).

bytebites

  • Hero Member
  • *****
  • Posts: 632
Re: Feature announcement: Function References and Anonymous Functions
« Reply #4 on: May 27, 2022, 08:02:40 am »
Oh, it happened in our lifetime.  :D  Thank you.

Thaddy

  • Hero Member
  • *****
  • Posts: 14197
  • Probably until I exterminate Putin.
Re: Feature announcement: Function References and Anonymous Functions
« Reply #5 on: May 27, 2022, 01:20:11 pm »
Nice toy! Well done you all.
Here's an old Barry Kelly smartpointer example for Delphi that now compiles in FPC:
Code: Pascal  [Select][+][-]
  1. {$ifdef windows}{$apptype console}{$endif}
  2. {$mode delphi}{$modeswitch functionreferences}{$modeswitch anonymousfunctions}
  3. {$warn 5036 off}
  4. uses
  5.   SysUtils;
  6.  
  7. type
  8.   Tproc = reference to procedure;
  9.   TLifetimeWatcher = class(TInterfacedObject)
  10.   private
  11.     FWhenDone: TProc;
  12.   public
  13.     constructor Create(const AWhenDone: TProc);
  14.     destructor Destroy; override;
  15.   end;
  16.  
  17. { TLifetimeWatcher }
  18.  
  19. constructor TLifetimeWatcher.Create(const AWhenDone: TProc);
  20. begin
  21.   FWhenDone := AWhenDone;
  22. end;
  23.  
  24. destructor TLifetimeWatcher.Destroy;
  25. begin
  26.   if Assigned(FWhenDone) then
  27.     FWhenDone;
  28.   inherited;
  29. end;
  30.  
  31. type
  32.   TSmartPointer<T: class> = record
  33.   strict private
  34.     FValue: T;
  35.     FLifetime: IInterface;
  36.   public
  37.     constructor Create(const AValue: T); overload;
  38.     class operator Implicit(const AValue: T): TSmartPointer<T>;
  39.     property Value: T read FValue;
  40.   end;
  41.  
  42. { TSmartPointer<T> }
  43.  
  44. constructor TSmartPointer<T>.Create(const AValue: T);
  45. begin
  46.   FValue := AValue;
  47.   FLifetime := TLifetimeWatcher.Create(procedure
  48.   begin
  49.     AValue.Free;
  50.   end);
  51. end;
  52.  
  53. class operator TSmartPointer<T>.Implicit(const AValue: T): TSmartPointer<T>;
  54. begin
  55.   Result := TSmartPointer<T>.Create(AValue);
  56. end;
  57.  
  58. procedure UseIt;
  59. var
  60.   x: TSmartPointer<TLifetimeWatcher>;
  61. begin
  62.   x := TLifetimeWatcher.Create(procedure
  63.   begin
  64.     Writeln('I died.');
  65.   end);
  66. end;
  67.  
  68. begin
  69.   try
  70.     UseIt;
  71.     Readln;
  72.   except
  73.     on E:Exception do
  74.       Writeln(E.Classname, ': ', E.Message);
  75.   end;
  76. end.
« Last Edit: May 27, 2022, 01:22:34 pm by Thaddy »
Specialize a type, not a var.

simone

  • Hero Member
  • *****
  • Posts: 573
Re: Feature announcement: Function References and Anonymous Functions
« Reply #6 on: May 27, 2022, 03:05:29 pm »
Thanks to the development team. Will this new feature be available in the next release version of fpc?
Microsoft Windows 10 64 bit - Lazarus 3.0 FPC 3.2.2 x86_64-win64-win32/win64

PascalDragon

  • Hero Member
  • *****
  • Posts: 5446
  • Compiler Developer
Re: Feature announcement: Function References and Anonymous Functions
« Reply #7 on: May 27, 2022, 03:19:01 pm »
Thanks to the development team. Will this new feature be available in the next release version of fpc?

It will be in the next major release which is not yet scheduled. The next release will be a minor release (namely 3.2.4).

Bi0T1N

  • Jr. Member
  • **
  • Posts: 85
Re: Feature announcement: Function References and Anonymous Functions
« Reply #8 on: May 27, 2022, 09:09:59 pm »
I'm glad to see that it has finally arrived in FPC - good work! This should allow us to use several nice Delphi libraries with FPC. 8-)
I think you can also close the issue now.

Thaddy

  • Hero Member
  • *****
  • Posts: 14197
  • Probably until I exterminate Putin.
Re: Feature announcement: Function References and Anonymous Functions
« Reply #9 on: May 27, 2022, 09:32:11 pm »
I'm glad to see that it has finally arrived in FPC - good work! This should allow us to use several nice Delphi libraries with FPC. 8-)
I think you can also close the issue now.
Not only that: FPC can do more as per PascalDragon's introductary notes...
I have been toying with old D2009 and XE2 examples today and 90% can be made to work in minutes. Much more than I expected.
Specialize a type, not a var.

edwinyzh

  • New Member
  • *
  • Posts: 43
Re: Feature announcement: Function References and Anonymous Functions
« Reply #10 on: May 28, 2022, 05:44:29 am »
Wonderful! Now all must-have syntax I want from FPC is available!
Thank all you guys!

avk

  • Hero Member
  • *****
  • Posts: 752
Re: Feature announcement: Function References and Anonymous Functions
« Reply #11 on: May 28, 2022, 12:34:01 pm »
This is really great news, many thanks to the FPC team!

Thaddy

  • Hero Member
  • *****
  • Posts: 14197
  • Probably until I exterminate Putin.
Re: Feature announcement: Function References and Anonymous Functions
« Reply #12 on: May 28, 2022, 03:12:27 pm »
If you want to experiment, you need the latest trunk.
These defines can help others as well:
Code: Pascal  [Select][+][-]
  1. {$ifdef windows}{$apptype console}{$endif}
  2. {$mode delphi}{$modeswitch functionreferences}{$modeswitch anonymousfunctions}
  3. {$warn 5036 off}// "Warning: (5036) Local variable "$Capturer" does not seem to be initialized"
  4.  
The "warn 5036 off" is just to suppress a warning about the $capturer variable:
I suppose that will be fixed later? It does not really harm, but I sometimes like to compile with -Sew.
Everything else looks OK, except some advanced examples that use extended RTTI, but that is logical and described in the announcement. It just means that some frameworks do not work yet.
I have by now tested some 20+ examples from Delphi that do work and some homebrew.
« Last Edit: May 28, 2022, 03:27:27 pm by Thaddy »
Specialize a type, not a var.

PascalDragon

  • Hero Member
  • *****
  • Posts: 5446
  • Compiler Developer
Re: Feature announcement: Function References and Anonymous Functions
« Reply #13 on: May 28, 2022, 06:24:23 pm »
I think you can also close the issue now.

I wanted to look for that already. Thanks for finding it for me. :)

I'm glad to see that it has finally arrived in FPC - good work! This should allow us to use several nice Delphi libraries with FPC. 8-)
I think you can also close the issue now.
Not only that: FPC can do more as per PascalDragon's introductary notes...
I have been toying with old D2009 and XE2 examples today and 90% can be made to work in minutes. Much more than I expected.

The other 10% would probably be interesting. At least if they don't rely on feature that we currently don't support (extended RTTI, some of the TValue functionality, etc.).

These defines can help others as well:
Code: Pascal  [Select][+][-]
  1. {$ifdef windows}{$apptype console}{$endif}
  2. {$mode delphi}{$modeswitch functionreferences}{$modeswitch anonymousfunctions}
  3. {$warn 5036 off}// "Warning: (5036) Local variable "$Capturer" does not seem to be initialized"
  4.  
The "warn 5036 off" is just to suppress a warning about the $capturer variable:
I suppose that will be fixed later? It does not really harm, but I sometimes like to compile with -Sew.

And why, pray tell, did you not report this? This is the first time I hear of this considering that I wrote a ton of tests... :o Do you have an example where this happens?

Thaddy

  • Hero Member
  • *****
  • Posts: 14197
  • Probably until I exterminate Putin.
Re: Feature announcement: Function References and Anonymous Functions
« Reply #14 on: May 28, 2022, 08:19:38 pm »
Of course, Sarah. See my above Barry Kelly example and comment the warn  8-)
But a great achiviement.

The warnings are on:
testanon.pas(51,1) Warning: (5036) Local variable "$Capturer" does not seem to be initialized
And on line 61 the same.

"$Capturer" is not accessible for the normal user.
There is also a note, but I tend to ignore those.
For completeness: "testanon.pas(60,3) Note: (5027) Local variable "x" is assigned but never used" which is not true...
« Last Edit: May 28, 2022, 08:49:47 pm by Thaddy »
Specialize a type, not a var.

 

TinyPortal © 2005-2018