Recent

Author Topic: Help explain this closure example  (Read 1951 times)

Ryan J

  • Full Member
  • ***
  • Posts: 138
Help explain this closure example
« on: September 24, 2023, 02:58:52 pm »
Something which is confusing me about closures.  I would expect the program to print 1 but it prints 2.

I know "x" is a reference to the calling scope but when is it actually copied to the capturer interface so that it survives leaving the scope? X is a pointer but the variables must be copied at some point and it appears to happen at the end of the calling scope (i.e. TestNested), correct?

If that's true I would be curious as to why because in mind it seems like it could happen at either assignment or scope termination and both would be valid approaches.

Code: Pascal  [Select][+][-]
  1. {$mode objfpc}
  2. {$modeswitch anonymousfunctions}
  3. {$modeswitch functionreferences}
  4.  
  5. program test;
  6.  
  7. type
  8.   TProc = reference to procedure;
  9.  
  10. function TestNested: TProc;
  11. var
  12.   x: Integer;
  13. begin
  14.   x := 1;
  15.  
  16.   result := procedure
  17.   begin
  18.     writeln(x);
  19.   end;
  20.  
  21.   inc(x);
  22. end;
  23.  
  24. var
  25.   p: TProc;
  26. begin
  27.   p := TestNested;
  28.   p();
  29. end.
  30.  

jamie

  • Hero Member
  • *****
  • Posts: 6735
Re: Help explain this closure example
« Reply #1 on: September 24, 2023, 03:07:09 pm »
not sure what you mean but I think I have an idea.

Return types that fit in registers will not return immediately to the reference when ever you set it within the function, only when the function returns are those registered referenced.

 However, any type that is larger than registers makes a reference back to the return point, so yes, those will effect the return point within the function doing it.

 I did ask for a reference return option for the compiler once function(..) REF Type, but like everything else, mud on the wall  :D

 Maybe that wasn't it but it was an excuse for me to post that!  :D
The only true wisdom is knowing you know nothing

Fibonacci

  • Hero Member
  • *****
  • Posts: 613
  • Internal Error Hunter
Re: Help explain this closure example
« Reply #2 on: September 24, 2023, 03:15:32 pm »
Why would you expect to print 1, if you increase x by 1 (to 2) before you call that anonymous procedure? At the moment the procedure is called the value of x is 2.

First you call TestNested, which does:
- set 1 to x
- set some procedure as result
- increase x by 1

Then you call "p", result of TestNested, anoymous procedure, which:
- prints value of x

Edit: I think I know why you might think that, but even if x becomes invalid (freed?), there will still be 2 in the memory, until something overwrites it
« Last Edit: September 24, 2023, 03:25:14 pm by Fibonacci »

Ryan J

  • Full Member
  • ***
  • Posts: 138
Re: Help explain this closure example
« Reply #3 on: September 24, 2023, 03:26:40 pm »
I didn't make my question clear enough. I know there is a hidden capturer object but I'm not sure WHEN it copies the variables. I expected the line

Code: Pascal  [Select][+][-]
  1. result := procedure
  2.   begin
  3.     writeln(x);
  4.   end;

to basically instantiate the class and copy the variable "x" which would mean that any subsequent changes to "x" would not affect the value inside the closure.

Fibonacci

  • Hero Member
  • *****
  • Posts: 613
  • Internal Error Hunter
Re: Help explain this closure example
« Reply #4 on: September 24, 2023, 03:33:19 pm »
Surely there is no option that x would be copied when creating anonymous function

That keyword "reference", I think because of it the variables will stay valid. If you change the type of TProc to just "procedure" without reference, you dont have access to x, wont compile

https://forum.lazarus.freepascal.org/index.php?topic=59468.0

Did you read it?

Code: Pascal  [Select][+][-]
  1. function TestNested: TProc;
  2. var
  3.   x: Integer;
  4. begin    
  5.   writeln('1/addr of x = ', IntToHex(DWORD(@x)));
  6.   x := 1;
  7.  
  8.   result := procedure
  9.   begin
  10.     writeln('2/addr of x = ', IntToHex(DWORD(@x)));
  11.     writeln(x);
  12.   end;
  13.  
  14.   inc(x);  
  15.   writeln('3/addr of x = ', IntToHex(DWORD(@x)));
  16. end;

1/addr of x = 015033E0
3/addr of x = 015033E0
2/addr of x = 015033E0
« Last Edit: September 24, 2023, 03:36:30 pm by Fibonacci »

Ryan J

  • Full Member
  • ***
  • Posts: 138
Re: Help explain this closure example
« Reply #5 on: September 24, 2023, 03:42:30 pm »
Yes the variables are references but they still need to be copied the capturer so they can escape the scope in which they were declared.

Sorry this is difficult to explain for me :) I guess I wanted to see like a full code representation of what's actually happening behind the scenes. The feature announcement explains the hidden interface but that's about it. Sven I remember showed some extra details on the mail list but I can't find them now.

Fibonacci

  • Hero Member
  • *****
  • Posts: 613
  • Internal Error Hunter
Re: Help explain this closure example
« Reply #6 on: September 24, 2023, 04:08:51 pm »
I guess Im not the right person to answer this question. I use anonymous functions a lot, but Ive never used them with a "reference to", and after what I just discovered in generated asmlists, I will keep it this way.

Ryan J

  • Full Member
  • ***
  • Posts: 138
Re: Help explain this closure example
« Reply #7 on: September 24, 2023, 04:11:18 pm »
I guess Im not the right person to answer this question. I use anonymous functions a lot, but Ive never used them with a "reference to", and after what I just discovered in generated asmlists, I will keep it this way.

These are useful mainly for threads but also for callbacks (imagine you can pass a nested function as a callback parameter for a button or something where the function will be invoked after the calling scope terminates).

I know Sven has the answer if he sees this :)

PascalDragon

  • Hero Member
  • *****
  • Posts: 5759
  • Compiler Developer
Re: Help explain this closure example
« Reply #8 on: September 24, 2023, 10:16:58 pm »
I know "x" is a reference to the calling scope but when is it actually copied to the capturer interface so that it survives leaving the scope? X is a pointer but the variables must be copied at some point and it appears to happen at the end of the calling scope (i.e. TestNested), correct?

There is nothing copied. The compiler moves the x variable to the capturer object and only uses that to access x for the whole lifetime of the function.

jamie

  • Hero Member
  • *****
  • Posts: 6735
Re: Help explain this closure example
« Reply #9 on: September 24, 2023, 11:00:51 pm »
That looks a little stack crazy to me.

I can see how that generates the value 2 because when you return from the TestNested its then when the actual procedure gets executed and of course by that time, X has been incremented. Fetching the reference to the inner procedure only transfers the address to the Registers on return, it does not execute the body.

 But this is what gets me, it appears that X is living on borrowed time, I mean it looks like it's on old stack space?

  I guess this is fine however, what if I were to insert another function/Procedure between P:= TestNested and P()?
  Assuming this inserted code would have use of a stack like some stack variables etc.?

  Just curious actually, but I don't this code is safe!  :o
The only true wisdom is knowing you know nothing

PascalDragon

  • Hero Member
  • *****
  • Posts: 5759
  • Compiler Developer
Re: Help explain this closure example
« Reply #10 on: September 24, 2023, 11:22:21 pm »
Just curious actually, but I don't this code is safe!  :o

The code is perfectly safe and is one of the main purposes of the function reference functionality. All variables of the surrounding function that are accessed in a function reference are relocated from the stack to an object instance that's instantiated at the start of the function and which is kept alive by all the function references, cause this are in fact interfaces in disguise. So as long as at least one of the function references is still alive (e.g. because it was passed outside the containing function) the variables will be alive.

Ryan J

  • Full Member
  • ***
  • Posts: 138
Re: Help explain this closure example
« Reply #11 on: September 25, 2023, 02:17:05 am »
The code is perfectly safe and is one of the main purposes of the function reference functionality. All variables of the surrounding function that are accessed in a function reference are relocated from the stack to an object instance that's instantiated at the start of the function and which is kept alive by all the function references, cause this are in fact interfaces in disguise. So as long as at least one of the function references is still alive (e.g. because it was passed outside the containing function) the variables will be alive.

Ohhhhh the variable is MOVED to the new object so subsequent changes are redirected to a new location. That explains everything. Some further questions:

1) Does the object get allocated for every function in which a function reference is declared or does an assignment need to occur?
2) I know it's been said from the start that there is some interest in making an option to copy the captured the variables instead of reference them. Does this mean that the variable would be copied upon assignment instead of move/swapping the stack variable to the new object?

Warfley

  • Hero Member
  • *****
  • Posts: 1763
Re: Help explain this closure example
« Reply #12 on: September 25, 2023, 11:30:20 am »
This for example allows for some very interesting constructs:
Code: Pascal  [Select][+][-]
  1. {$ModeSwitch functionreferences}
  2.  
  3. type
  4.   TMyAdder = record
  5.     add: reference to procedure(AValue: Integer);
  6.     result: reference to function: Integer;
  7.   end;
  8.  
  9. function Adder: TMyAdder;
  10. var
  11.   Accumulator: Integer;
  12.  
  13.   procedure DoAdd(AValue: Integer);
  14.   begin
  15.     Accumulator += AValue;
  16.   end;
  17.   function GetResult: Integer;
  18.   begin
  19.     Result := Accumulator;
  20.   end;
  21.  
  22. begin
  23.   Accumulator:=0;
  24.   Result.add:=@DoAdd;
  25.   Result.result:=@GetResult;
  26. end;
  27.  
  28. var
  29.   a: TMyAdder;
  30. begin
  31.   a:=Adder;
  32.   a.add(3);
  33.   a.add(5);
  34.   WriteLn(a.result());
  35.   a:=Adder;
  36.   a.add(4);
  37.   a.add(10);
  38.   WriteLn(a.result());
  39. end.

At it's core this is exactly the same as creating a class:
Code: Pascal  [Select][+][-]
  1. type
  2.   TMyAdder = class
  3.   private
  4.     Accumulator: Integer;
  5.   public
  6.     procedure Add(AValue: Integer);
  7.     function GetResult: Integer;
  8.   end;

Thats why there is this famous joke:
Quote from: Anton van Straaten
The venerable master Qc Na was walking with his student, Anton. Hoping to prompt the master into a discussion, Anton said "Master, I have heard that objects are a very good thing - is this true?" Qc Na looked pityingly at his student and replied, "Foolish pupil - objects are merely a poor man's closures."

Chastised, Anton took his leave from his master and returned to his cell, intent on studying closures. He carefully read the entire "Lambda: The Ultimate..." series of papers and its cousins, and implemented a small Scheme interpreter with a closure-based object system. He learned much, and looked forward to informing his master of his progress.

On his next walk with Qc Na, Anton attempted to impress his master by saying "Master, I have diligently studied the matter, and now understand that objects are truly a poor man's closures." Qc Na responded by hitting Anton with his stick, saying "When will you learn? Closures are a poor man's object." At that moment, Anton became enlightened.
« Last Edit: September 25, 2023, 11:32:11 am by Warfley »

Mr.Madguy

  • Hero Member
  • *****
  • Posts: 859
Re: Help explain this closure example
« Reply #13 on: September 25, 2023, 01:13:05 pm »
There is nothing copied. The compiler moves the x variable to the capturer object and only uses that to access x for the whole lifetime of the function.
Yeah. Closures - are way to declare classes without actually declaring classes.

This code:
Code: Pascal  [Select][+][-]
  1. TProc = reference to procedure;
  2.  
  3. function TestNested:TProc;
  4.   var x:Integer;
  5. begin
  6.   x := 1;
  7.   Result := procedure
  8.   begin
  9.     WriteLn(x);
  10.   end;
  11.   Inc(x);
  12. end;
  13.  
is replaced by
Code: Pascal  [Select][+][-]
  1. TProc = procedure of interface;//Same as procedure of object. Actually structure, that stores IClosure and pointer to ClosureProc1.
  2.  
  3. IClosure = interface
  4.   procedure ClosureProc1;
  5. end;
  6.  
  7. TClosure = class(TInterfacedObject, IClosure)
  8.   public
  9.     x:Integer;
  10.     procedure ClosureProc1;
  11. end;
  12.  
  13. procedure TClosure.ClosureProc1;
  14. begin
  15.   WriteLn(x);
  16. end;
  17.  
  18. function TestNested:TProc;
  19.   var Closure:TClosure;
  20. begin
  21.   Closure := TClosure.Create;
  22.   Closure.x := 1;
  23.   Result := Closure.ClosureProc1;
  24.   Inc(Closure.x)  
  25. end;
  26.  
As you can see, first variant in much more SHORTER, than second one. That's why closures are so powerful. They allow attaching arbitrary data to callbacks without actually making lots of unnecessary declarations.
« Last Edit: September 25, 2023, 01:15:02 pm by Mr.Madguy »
Is it healthy for project not to have regular stable releases?
Just for fun: Code::Blocks, GCC 13 and DOS - is it possible?

Ryan J

  • Full Member
  • ***
  • Posts: 138
Re: Help explain this closure example
« Reply #14 on: September 25, 2023, 01:55:56 pm »
Thanks @Mr.Madguy for the example, that's very useful. If FPC hasn't already they should add something like this to the official manual so programmers actually know what their code REALLY looks like.

 

TinyPortal © 2005-2018