Help explain this closure example

[Ryan J]

Something which is confusing me about closures.  I would expect the program to print 1 but it prints 2.

I know "x" is a reference to the calling scope but when is it actually copied to the capturer interface so that it survives leaving the scope? X is a pointer but the variables must be copied at some point and it appears to happen at the end of the calling scope (i.e. TestNested), correct?

If that's true I would be curious as to why because in mind it seems like it could happen at either assignment or scope termination and both would be valid approaches.

Code: Pascal  [Select][+][-]

1. {$mode objfpc}
2. {$modeswitch anonymousfunctions}
3. {$modeswitch functionreferences}
4.  
5. program test;
6.  
7. type
8.   TProc = reference to procedure;
9.  
10. function TestNested: TProc;
11. var
12.   x: Integer;
13. begin
14.   x := 1;
15.  
16.   result := procedure
17.   begin
18.     writeln(x);
19.   end;
20.  
21.   inc(x);
22. end;
23.  
24. var
25.   p: TProc;
26. begin
27.   p := TestNested;
28.   p();
29. end.
30.  

[jamie]

not sure what you mean but I think I have an idea.

Return types that fit in registers will not return immediately to the reference when ever you set it within the function, only when the function returns are those registered referenced.

 However, any type that is larger than registers makes a reference back to the return point, so yes, those will effect the return point within the function doing it.

 I did ask for a reference return option for the compiler once function(..) REF Type, but like everything else, mud on the wall 

 Maybe that wasn't it but it was an excuse for me to post that! 

[Fibonacci]

Why would you expect to print 1, if you increase x by 1 (to 2) before you call that anonymous procedure? At the moment the procedure is called the value of x is 2.

First you call TestNested, which does:
- set 1 to x
- set some procedure as result
- increase x by 1

Then you call "p", result of TestNested, anoymous procedure, which:
- prints value of x

Edit: I think I know why you might think that, but even if x becomes invalid (freed?), there will still be 2 in the memory, until something overwrites it

[Ryan J]

I didn't make my question clear enough. I know there is a hidden capturer object but I'm not sure WHEN it copies the variables. I expected the line

Code: Pascal  [Select][+][-]

1. result := procedure
2.   begin
3.     writeln(x);
4.   end;

to basically instantiate the class and copy the variable "x" which would mean that any subsequent changes to "x" would not affect the value inside the closure.

[Fibonacci]

Surely there is no option that x would be copied when creating anonymous function

That keyword "reference", I think because of it the variables will stay valid. If you change the type of TProc to just "procedure" without reference, you dont have access to x, wont compile

https://forum.lazarus.freepascal.org/index.php?topic=59468.0

Did you read it?

Code: Pascal  [Select][+][-]

1. function TestNested: TProc;
2. var
3.   x: Integer;
4. begin    
5.   writeln('1/addr of x = ', IntToHex(DWORD(@x)));
6.   x := 1;
7.  
8.   result := procedure
9.   begin
10.     writeln('2/addr of x = ', IntToHex(DWORD(@x)));
11.     writeln(x);
12.   end;
13.  
14.   inc(x);  
15.   writeln('3/addr of x = ', IntToHex(DWORD(@x)));
16. end;

1/addr of x = 015033E0
3/addr of x = 015033E0
2/addr of x = 015033E0

[Ryan J]

Yes the variables are references but they still need to be copied the capturer so they can escape the scope in which they were declared.

Sorry this is difficult to explain for me I guess I wanted to see like a full code representation of what's actually happening behind the scenes. The feature announcement explains the hidden interface but that's about it. Sven I remember showed some extra details on the mail list but I can't find them now.

[Fibonacci]

I guess Im not the right person to answer this question. I use anonymous functions a lot, but Ive never used them with a "reference to", and after what I just discovered in generated asmlists, I will keep it this way.

[Ryan J]

I guess Im not the right person to answer this question. I use anonymous functions a lot, but Ive never used them with a "reference to", and after what I just discovered in generated asmlists, I will keep it this way.

These are useful mainly for threads but also for callbacks (imagine you can pass a nested function as a callback parameter for a button or something where the function will be invoked after the calling scope terminates).

I know Sven has the answer if he sees this

[PascalDragon]

I know "x" is a reference to the calling scope but when is it actually copied to the capturer interface so that it survives leaving the scope? X is a pointer but the variables must be copied at some point and it appears to happen at the end of the calling scope (i.e. TestNested), correct?

There is nothing copied. The compiler moves the x variable to the capturer object and only uses that to access x for the whole lifetime of the function.

[jamie]

That looks a little stack crazy to me.

I can see how that generates the value 2 because when you return from the TestNested its then when the actual procedure gets executed and of course by that time, X has been incremented. Fetching the reference to the inner procedure only transfers the address to the Registers on return, it does not execute the body.

 But this is what gets me, it appears that X is living on borrowed time, I mean it looks like it's on old stack space?

  I guess this is fine however, what if I were to insert another function/Procedure between P:= TestNested and P()?
  Assuming this inserted code would have use of a stack like some stack variables etc.?

  Just curious actually, but I don't this code is safe! 

[PascalDragon]

Just curious actually, but I don't this code is safe! 

The code is perfectly safe and is one of the main purposes of the function reference functionality. All variables of the surrounding function that are accessed in a function reference are relocated from the stack to an object instance that's instantiated at the start of the function and which is kept alive by all the function references, cause this are in fact interfaces in disguise. So as long as at least one of the function references is still alive (e.g. because it was passed outside the containing function) the variables will be alive.

[Ryan J]

The code is perfectly safe and is one of the main purposes of the function reference functionality. All variables of the surrounding function that are accessed in a function reference are relocated from the stack to an object instance that's instantiated at the start of the function and which is kept alive by all the function references, cause this are in fact interfaces in disguise. So as long as at least one of the function references is still alive (e.g. because it was passed outside the containing function) the variables will be alive.

Ohhhhh the variable is MOVED to the new object so subsequent changes are redirected to a new location. That explains everything. Some further questions:

1) Does the object get allocated for every function in which a function reference is declared or does an assignment need to occur?
2) I know it's been said from the start that there is some interest in making an option to copy the captured the variables instead of reference them. Does this mean that the variable would be copied upon assignment instead of move/swapping the stack variable to the new object?

[Warfley]

This for example allows for some very interesting constructs:

Code: Pascal  [Select][+][-]

1. {$ModeSwitch functionreferences}
2.  
3. type
4.   TMyAdder = record
5.     add: reference to procedure(AValue: Integer);
6.     result: reference to function: Integer;
7.   end;
8.  
9. function Adder: TMyAdder;
10. var
11.   Accumulator: Integer;
12.  
13.   procedure DoAdd(AValue: Integer);
14.   begin
15.     Accumulator += AValue;
16.   end;
17.   function GetResult: Integer;
18.   begin
19.     Result := Accumulator;
20.   end;
21.  
22. begin
23.   Accumulator:=0;
24.   Result.add:=@DoAdd;
25.   Result.result:=@GetResult;
26. end;
27.  
28. var
29.   a: TMyAdder;
30. begin
31.   a:=Adder;
32.   a.add(3);
33.   a.add(5);
34.   WriteLn(a.result());
35.   a:=Adder;
36.   a.add(4);
37.   a.add(10);
38.   WriteLn(a.result());
39. end.

At it's core this is exactly the same as creating a class:

Code: Pascal  [Select][+][-]

1. type
2.   TMyAdder = class
3.   private
4.     Accumulator: Integer;
5.   public
6.     procedure Add(AValue: Integer);
7.     function GetResult: Integer;
8.   end;

Thats why there is this famous joke:

The venerable master Qc Na was walking with his student, Anton. Hoping to prompt the master into a discussion, Anton said "Master, I have heard that objects are a very good thing - is this true?" Qc Na looked pityingly at his student and replied, "Foolish pupil - objects are merely a poor man's closures."

Chastised, Anton took his leave from his master and returned to his cell, intent on studying closures. He carefully read the entire "Lambda: The Ultimate..." series of papers and its cousins, and implemented a small Scheme interpreter with a closure-based object system. He learned much, and looked forward to informing his master of his progress.

On his next walk with Qc Na, Anton attempted to impress his master by saying "Master, I have diligently studied the matter, and now understand that objects are truly a poor man's closures." Qc Na responded by hitting Anton with his stick, saying "When will you learn? Closures are a poor man's object." At that moment, Anton became enlightened.

[Mr.Madguy]

There is nothing copied. The compiler moves the x variable to the capturer object and only uses that to access x for the whole lifetime of the function.

Yeah. Closures - are way to declare classes without actually declaring classes.

This code:

Code: Pascal  [Select][+][-]

1. TProc = reference to procedure;
2.  
3. function TestNested:TProc;
4.   var x:Integer;
5. begin
6.   x := 1;
7.   Result := procedure
8.   begin
9.     WriteLn(x);
10.   end;
11.   Inc(x);
12. end;
13.  

is replaced by

Code: Pascal  [Select][+][-]

1. TProc = procedure of interface;//Same as procedure of object. Actually structure, that stores IClosure and pointer to ClosureProc1.
2.  
3. IClosure = interface
4.   procedure ClosureProc1;
5. end;
6.  
7. TClosure = class(TInterfacedObject, IClosure)
8.   public
9.     x:Integer;
10.     procedure ClosureProc1;
11. end;
12.  
13. procedure TClosure.ClosureProc1;
14. begin
15.   WriteLn(x);
16. end;
17.  
18. function TestNested:TProc;
19.   var Closure:TClosure;
20. begin
21.   Closure := TClosure.Create;
22.   Closure.x := 1;
23.   Result := Closure.ClosureProc1;
24.   Inc(Closure.x)  
25. end;
26.  

As you can see, first variant in much more SHORTER, than second one. That's why closures are so powerful. They allow attaching arbitrary data to callbacks without actually making lots of unnecessary declarations.

[Ryan J]

Thanks @Mr.Madguy for the example, that's very useful. If FPC hasn't already they should add something like this to the official manual so programmers actually know what their code REALLY looks like.