Anonymous Methods

[Mr.Madguy]

Existing contributors just dig in the source until they get the structure in their head and are able to contribute. Though I haven't contributed any such low level code, I've dug the code a bit while debugging for code generator issues. I say it's not that hard to follow, it just employs classic recursive descent parser that generates syntax tree, which is further optimized as requested before given to platform specific code generator.

Yeah, I know. I've already found parser, symbols, nodes, code generators, assemblers, linker and executable generators. What I don't understand now - is overall code flow. I.e. I've found some module and it does some job. But how it's connected to other modules?

P.S. I also use anonymous methods in unit tests. I need to pass bunch of arbitrary data into test cases, so class declaration is needed. With anonymous methods all I need - is list of test case anonymous callbacks.

Also I don't think, that closures are overrated. They're very powerful. They allow you to create object without declaring class - very useful feature, especially in scripts, where amount of code is tried to be made minium (some scripts, like Lua, don't even have class declaration syntax). It saves half of your time for more important tasks.

Example - enumerator. Big advantage, that helped me in Lua a lot - resulting object can be optimized for specific input parameters.

Code: Pascal  [Select][+][-]

1. type
2.     TEnumRef = reference to function(var ADest:TObject):Boolean;
3.  
4.     TObjectList = class
5.         protected
6.             FItems:array of TObject;
7.         public            
8.             function GetEnumerator(AStart:Integer = 0; AEnd:Integer = -1; AStep:Integer = 1):TEnumRef;
9.     end;
10.  
11. function TObjectList.GetEnumerator(AStart, AEnd, AStep:Integer):TEnumRef;
12.     var I:Integer;
13. begin
14.     if AStart < 0 then begin
15.         Inc(AStart, Count);
16.     end;
17.     if AEnd < 0 then begin
18.         Inc(AEnd, Count);
19.     end;
20.     if (AStart < 0) or (AEnd < 0 ) or (AStart >= Count) or (AEnd >= Count) or (AStep <= 0) then begin
21.         raise EEnumException.Create('Wrong parameters');
22.     end;
23.     I := AStart;
24.     if AStart >= AEnd then begin
25.         Result := function(var ADest:TObject):Boolean
26.         begin
27.             Result := I <= AEnd;
28.             if Result then begin
29.                 ADest := FItems[I];
30.                 Inc(I, AStep);
31.             end
32.             else begin
33.                 ADest := nil;
34.             end;
35.         end;
36.     end
37.     else begin
38.         Result := function(var ADest:TObject):Boolean
39.         begin
40.             Result := I >= AEnd;
41.             if Result then begin
42.                 ADest := FItems[I];
43.                 Dec(I, AStep);
44.             end
45.             else begin
46.                 ADest := nil;
47.             end;
48.         end;
49.     end;
50. end;
51.  

But sometimes closures are bad. Can't find example, but sometimes code is so complicated by closures, that you barely can understand it's flow.

[Leledumbo]

Yeah, I know. I've already found parser, symbols, nodes, code generators, assemblers, linker and executable generators. What I don't understand now - is overall code flow. I.e. I've found some module and it does some job. But how it's connected to other modules?

That would be too much to say in words, better dig in yourself. Start from pp.pas (or compiler.pas since the wrapper program does only a little job). A pp.lpi is provided so you can make a good use of Lazarus code tools to navigate between method calls.

[avra]

What I don't understand now - is overall code flow. I.e. I've found some module and it does some job. But how it's connected to other modules?

There are tools like SciTools Understand which could help you a lot. If I remember well free trial worked for 14 days, which should be more then enough for your needs.
https://scitools.com/feature/supported-languages
Other interesting tools are mentioned here:
http://forum.lazarus.freepascal.org/index.php/topic,34910.msg233315.html#msg233315

EDIT1: I have attached some screenshots of SciTools Understand graphs useful for your study. However, they do not do justice since in the tool itself you can click, collapse, expand, and do much, much more.

EDIT2: Unfortunately, forum upload limit made me use file sharing service for one big file. That's DependsOnGraph for compiler.pas in just compiler dir. Here is the link: http://dropcanvas.com/xr0hw

[PascalDragon]

Anonymous functions are a work in progress by a third party developer and are in fact nearly finished. So it shouldn't take too long (TM) until they appear in FPC trunk.

And yes, closures require a heap allocation as internally (in Delphi and FPC) an anonymous function is a class containing the captured variables and an Invoke() method that calls the anonymous function that had been provided by the user and also implementing an interface that allows for automatic reference counting.

[Mr.Madguy]

Anonymous functions are a work in progress by a third party developer and are in fact nearly finished. So it shouldn't take too long (TM) until they appear in FPC trunk.

I understand, but it isn't listed as new feature for 3.1. And you should understand, that this feature was implemented in Delphi back in 2009 (as I know, when Delphi is called 2009 - then it was released at the beginning of 2009 or even at the end of 2008). 8 years passed since that moment. Don't you think, that it's too long for feature, that requires Macro/Meta programming only - i.e. can be implemented at high (source code) level? And as I know, core Open Source philosophy is - "If you want something - do it yourself". That's why I decided to offer some help.

And yes, closures require a heap allocation as internally (in Delphi and FPC) an anonymous function is a class containing the captured variables and an Invoke() method that calls the anonymous function that had been provided by the user and also implementing an interface that allows for automatic reference counting.

Closures are inverted objects. Object - is data with code, attached to it. Closure - is code with data, attached to it. But the difference is in declaration only - at the end both are objects. In scripts closures are implemented naturally, cuz all variables are reference counted. But in compilable languages only interfaces has reference counting, so variables have to be implicitly captured first. And it's hardest part of task, cuz implicit code - is bad thing in compilable languages. It can cause tricky errors. Good thing, if you know asm and know how to use Debug->CPU. But what, if you don't?

[Thaddy]

In Delphi, anonymous methods are implemented as interfaced types because of that. The Invoke method is the actual anonymous method.

[PascalDragon]

Anonymous functions are a work in progress by a third party developer and are in fact nearly finished. So it shouldn't take too long (TM) until they appear in FPC trunk.

I understand, but it isn't listed as new feature for 3.1. And you should understand, that this feature was implemented in Delphi back in 2009 (as I know, when Delphi is called 2009 - then it was released at the beginning of 2009 or even at the end of 2008). 8 years passed since that moment. Don't you think, that it's too long for feature, that requires Macro/Meta programming only - i.e. can be implemented at high (source code) level? And as I know, core Open Source philosophy is - "If you want something - do it yourself". That's why I decided to offer some help.

Of course it isn't mentioned as a new feature, after all it isn't part of trunk yet.

And if we're going by time frame then first dynamic packages would need to be finished as they are the oldest missing feature, then "implements" support would need to be finished and then maybe one could think about anonymous functions. This is an open source project on which people are working in their free time so they are working on topics that they want to see implemented, not necessarily what others want. And anonymous functions are definitely not something "that requires Macro/Meta programming only" cause the compiler needs to do quite some lifting.

The offer for help is appreciated, but not necessary anymore - at least not until the implementation is done and released to the public.

[Mr.Madguy]

Of course it isn't mentioned as a new feature, after all it isn't part of trunk yet.

And if we're going by time frame then first dynamic packages would need to be finished as they are the oldest missing feature, then "implements" support would need to be finished and then maybe one could think about anonymous functions. This is an open source project on which people are working in their free time so they are working on topics that they want to see implemented, not necessarily what others want. And anonymous functions are definitely not something "that requires Macro/Meta programming only" cause the compiler needs to do quite some lifting.

The offer for help is appreciated, but not necessary anymore - at least not until the implementation is done and released to the public.

"Implements" isn't implemented too? Another feature, that requires Macro/Meta programming only? Just cuz it simply redirects method calls to another entity - simple substitution needed. Then I have another problem. Cuz my project uses this feature. Dunno about FPC, but Delphi has some weird behavior, when you try to implement several interfaces, inherited from the same base one, even if you use explicit "procedure/function interface.interfaceMethod = implementingMethod;" syntax. When you make change to module, where this interfaces are implemented or rebuild project - everything works, as intended. But once you make change to any other module - "Missing implementation of interface method" happens. "Implements" - is the only thing, that helps.

Code, that causes problems. As you understand, it would be nice, if all 5 interfaces would be implemented in one single object - not in different ones, cuz it causes waste of memory. I know, that there is another crutch possible - declaring interfaces with different method names, but with memory-compatible VMTs. It's possible, and at some point I used it, but I don't like it - it's potentially unsafe. Yeah, I know, that I love to push compilers to their limits.

Code: Pascal  [Select][+][-]

1. IComparer<T> = interface
2.     function Compare(const AItem1, AItem2:T):Boolean;
3. end;
4.  
5. {Such declaration is REQUIERED, cuz TPairList is inherited from TList, so pair comparers should work for TList too}
6. IPairComparer<TKey, TValue> = interface(IComparer<TPair<TKey, TValue>>)
7.     {Can be implemented via QueryInterface() - GUID is needed to be provided then,
8.     but it's slow - I don't like to use it for internal interfaces}
9.     function GetKeyComparer:IComparer<TKey>;
10.     function GetValueComparer:IComparer<TValue>;
11.     function GetPairKeyComparer:IPairComparer<TKey, TValue>;
12.     function GetPairValueComparer:IPairComparer<TKey, TValue>;
13. end;
14.  
15. {Just aliases for the same interface - should work, but causes weird behavior, I described above.
16. It's not even bug - it's unforeseen situation. Delphi simply assumes, that if they're inherited from the same interface -
17. then implementation of this interface should be the same for all 3. And this is completely wrong.}
18. IPairKeyComparer<TKey, TValue> = interface(IPairComparer<TKey, TValue>);
19. IPairValueComparer<TKey, TValue> = interface(IPairComparer<TKey, TValue>);
20.  
21. TAbstractPairComparer<TKey, TValue> = class(TInterfacedObject, IPairComparer<TKey, TValue>,
22.                                                             IComparer<TKey>, IComparer<TValue>,
23.                                                             IPairKeyComparer<TKey, TValue>, IPairValueComparer<TKey, TValue>)
24.     protected
25.         function Compare(const AItem1, AItem2:TPair<TKey, TValue>):Boolean;virtual;abstract;      
26.         function ComparePairKey(const AItem1, AItem2:TPair<TKey, TValue>):Boolean;virtual;abstract;  
27.         function ComparePairValue(const AItem1, AItem2:TPair<TKey, TValue>):Boolean;virtual;abstract;
28.         {Following lines don't work properly - work, only when current unit is being rebuilt}
29.         function IPairComparer<TKey, TValue>.Compare = Compare;
30.         function IPairKeyComparer<TKey, TValue>.Compare = ComparePairKey;
31.         function IPairValueComparer<TKey, TValue>.Compare = ComparePairValue;
32.     public
33.         function GetPairKeyComparer:IPairComparer<TKey, TValue>;
34.         function GetPairValueComparer:IPairComparer<TKey, TValue>;
35. end;
36.  
37. function TAbstractPairComparer<TKey, TValue>.GetPairKeyComparer:IPairComparer<TKey, TValue>;
38.     var Temp:IPairKeyComparer<TKey, TValue>;
39. begin
40.     {Crutch: "as" doesn't work, when no GUID is specified,
41.     cuz "as" and "is" are just syntax sugar for QueryInterface()
42.     and direct cast isn't handled properly by compiler.
43.     We don't want to use QueryInterface(), cuz it's slow.}
44.     Temp := Self;
45.     Result := IPairComparer<TKey, TValue>(Temp);
46. end;
47.  
48. function TAbstractPairComparer<TKey, TValue>.GetPairValueComparer:IPairComparer<TKey, TValue>;
49.     var Temp:IPairValueComparer<TKey, TValue>;
50. begin
51.     {Crutch: "as" doesn't work, when no GUID is specified,
52.     cuz "as" and "is" are just syntax sugar for QueryInterface()
53.     and direct cast isn't handled properly by compiler.
54.     We don't want to use QueryInterface(), cuz it's slow.}
55.     Temp := Self;
56.     Result := IPairComparer<TKey, TValue>(Temp);
57. end;
58.  

Currently I'm not even sure, that FPC will be able to handle my project. Even Delphi barely can do it - special methods have to be used to avoid "Out of memory" errors. Simply because every time, size of any unit exceeds 2Mb - "Out of memory" errors start to happen, while using Debug config. It's not due to poor structure of my project. It due to way, generics work. I use generics and generics expand into large amount of implicit code. You write just one "MyDictionary:TDictionary<Integer, String>" line, but 4K lines are injected implicitly. But I don't think, that "out of memory" errors are bad - every programmer will face them at some point, if he work with serious projects, so it's better to be prepared.

[PascalDragon]

Of course it isn't mentioned as a new feature, after all it isn't part of trunk yet.

And if we're going by time frame then first dynamic packages would need to be finished as they are the oldest missing feature, then "implements" support would need to be finished and then maybe one could think about anonymous functions. This is an open source project on which people are working in their free time so they are working on topics that they want to see implemented, not necessarily what others want. And anonymous functions are definitely not something "that requires Macro/Meta programming only" cause the compiler needs to do quite some lifting.

The offer for help is appreciated, but not necessary anymore - at least not until the implementation is done and released to the public.

"Implements" isn't implemented too? Another feature, that requires Macro/Meta programming only? Just cuz it simply redirects method calls to another entity - simple substitution needed. Then I have another problem. Cuz my project uses this feature. Dunno about FPC, but Delphi has some weird behavior, when you try to implement several interfaces, inherited from the same base one, even if you use explicit "procedure/function interface.interfaceMethod = implementingMethod;" syntax. When you make change to module, where this interfaces are implemented or rebuild project - everything works, as intended. But once you make change to any other module - "Missing implementation of interface method" happens. "Implements" - is the only thing, that helps.

"implements" itself is implemented, but not completely. It only works with properties having an interface as type, but not with a class. And no, it's also not "macro/meta programming" as the compiler needs to generate stubs for the interface's VMT entries that correctly adjust the self pointer and then call the specified class instance's methods. I've taken a look at it once a few years ago and decided that this is not yet amongst the features I can implement (that might have changed nowadays - I'd need to take a long again, but currently I have other problems to solve ).

Code, that causes problems. As you understand, it would be nice, if all 5 interfaces would be implemented in one single object - not in different ones, cuz it causes waste of memory. I know, that there is another crutch possible - declaring interfaces with different method names, but with memory-compatible VMTs. It's possible, and at some point I used it, but I don't like it - it's potentially unsafe. Yeah, I know, that I love to push compilers to their limits.

Well, only one way to know whether FPC handles the interface method aliases correctly: test it. If you find bugs, please report them. (you should at least use FPC 3.0.0 however, maybe even test FPC 3.1.1 if you find a problem in 3.0.0 before reporting it, especially if the problem is related to generics)

Currently I'm not even sure, that FPC will be able to handle my project. Even Delphi barely can do it - special methods have to be used to avoid "Out of memory" errors. Simply because every time, size of any unit exceeds 2Mb - "Out of memory" errors start to happen, while using Debug config. It's not due to poor structure of my project. It due to way, generics work. I use generics and generics expand into large amount of implicit code. You write just one "MyDictionary:TDictionary<Integer, String>" line, but 4K lines are injected implicitly. But I don't think, that "out of memory" errors are bad - every programmer will face them at some point, if he work with serious projects, so it's better to be prepared.

FPC definitely can handle units that are larger than 2 MB. Take for example the %fpcdir%\packages\odata\src\sharepoint.pas unit which has ~3.7 MB as source and ~15.8 MB .o and ~9.5 MB .ppu on x86_64-win64. Though it might be that the out of memory errors you see might not be directly related to the unit, but extensive use of generics. Might be that FPC handles that more gracefully as well...

[Mr.Madguy]

"implements" itself is implemented, but not completely. It only works with properties having an interface as type, but not with a class. And no, it's also not "macro/meta programming" as the compiler needs to generate stubs for the interface's VMT entries that correctly adjust the self pointer and then call the specified class instance's methods. I've taken a look at it once a few years ago and decided that this is not yet amongst the features I can implement (that might have changed nowadays - I'd need to take a long again, but currently I have other problems to solve ).

I don't think, it's hard. When I say "macro/meta programming" - I mean simple code substitution/injection needed, i.e. this feature can be implemented via some sort of "preprocessor", that takes input source file and turns it into some output via using standard features only. Some sort of programming language inside programming language (that's, why it's called "Meta programming"). Yeah, this stubs should be optimized, but at the end they're something like that:

Code: Pascal  [Select][+][-]

1. procedure TObject1.DoSomething(X:TSomeType);
2. begin
3.      Object2.DoSomething(X);
4. end;
5.  

Simply because Object2 should be created and therefore all we need - to redirect call to it.

FPC definitely can handle units that are larger than 2 MB. Take for example the %fpcdir%\packages\odata\src\sharepoint.pas unit which has ~3.7 MB as source and ~15.8 MB .o and ~9.5 MB .ppu on x86_64-win64. Though it might be that the out of memory errors you see might not be directly related to the unit, but extensive use of generics. Might be that FPC handles that more gracefully as well...

I don't know. Looks like some sort of debug info overflow. May be circular references (one of known reasons for "Out of memory" in Delphi) or too long symbol declarations. Cuz disabling debug info usually helps and it usually has no problems for 64bit targets, that use remote debug info due to IDE, being 32bit. Delphi is well known for having "Out of memory" problems due to 32bit address space limit and sharing this space with relatively heavy IDE.

Problem happens in different situations. Sometimes I add one unit test after another (3-4K lines), then at some point "Linking" becomes really slow and then starts to throw "Out of memory" errors from time to time. After adding another test - program can't be compiled. Sometimes I just add several "TGeneric.Create" (even specialized in different modules) and after 5-6 different generics (every generic adds 200-700K to size of module, implementation itself can be up to 1.5M) - "Out of memory". Sometimes it looks like some sort of bug - that's when program is compiled properly with nested generics, like TPairList<Integer, TPairList<String, TPairList<Boolean, TTargetObject>>>, but fails for TPairList<Integer, TPairList<String, TPairList<Integer, TTargetObject>>>.

Thanks God, there are ways to overcome this problems. May be it's not bad, but I have to separate generic implementations into dlls. Some are big - 5Mb for example (64bit release). 3rd version of my unit tests: helper object - 1Mb, all needed generic implementations - 1.5Mb (32bit release, 2x for 64bit versions).

[guest60499]

FPC definitely can handle units that are larger than 2 MB. Take for example the %fpcdir%\packages\odata\src\sharepoint.pas unit which has ~3.7 MB as source and ~15.8 MB .o and ~9.5 MB .ppu on x86_64-win64. Though it might be that the out of memory errors you see might not be directly related to the unit, but extensive use of generics. Might be that FPC handles that more gracefully as well...

He seems to have been implying that Delphi implements generics by code rewriting. His experience with the errors seems to support this.

Personally it's an extremely odd way to implement generics, but I guess it works. Coincidentally it is also one of the only times I have seen a 32 bit address space become saturated.

[Thaddy]

He seems to have been implying that Delphi implements generics by code rewriting. His experience with the errors seems to support this.

Personally it's an extremely odd way to implement generics, but I guess it works. Coincidentally it is also one of the only times I have seen a 32 bit address space become saturated.

Generics are a form of macro and macro expansion. So yes, that is correct. It is also the ONLY KNOWN way to implement generics.
At least to people who studied compiler engineering.
I would be rather surprised if you came up with something else.... That would be less odd... maybe even even...

So he is not implying, he is correct... Explicitly.

https://en.wikipedia.org/wiki/Generic_programming

In this case wiki is reliable.

[Mr.Madguy]

Generics are a form of macro and macro expansion. So yes, that is correct. It is also the ONLY KNOWN way to implement generics.
I would be rather surprised if you came up with something else.... That would be less odd... maybe even even...

So he is not implying, he is correct...

Yeah, templates/generics - is Macro feature. Every specialization implicitly produces same amount of code, generic declaration has.

Wrapping everything into objects and using one single TGeneric<TBaseObject> implementation - is the only way to solve this problem. I actually have been thinking about it. But at some point I realized, that the whole reason to use compiler would be lost then - script with JIT support would be more reasonable.

I don't know. May be it's my fault, cuz some code can cause "Out of memory" errors. Unfortunately I can't find example, but some array declarations can cause some sort of circular referencing in Delphi. Google "Delphi Fatal F2046 Out of memory", if you want. But such problems are very hard to detect - code is very complex. Of course it would be nice, if Delphi would inform me, what module/class/declaration causes this error, but it doesn't. The only thing, it does - shows extended error code like "F2084 Internal Error: DBG2886", if you try to compile your program again. But this code is useful to Delphi developers only - it contains info about module and source line, where error happens. Bad thing - you don't have access to Delphi sources.

In most cases this problem is caused by the fact, that IDE is still 32bit and it uses internal compiler - not external. When compiling from IDE - dll is used instead of running separate dccxx.exe process. Delphi do it in order to be able to perform dynamic compiling - code is being compiled in background, directly when you write it. And this internal compiler is buggy. It may have memory leaks.

64bit IDE would solve all problems, but due to some reasons (legacy design-time packages?), they can't do it.

[Thaddy]

From a computational point of view there is no reason not to use 31 bits or 33 bits or 65 bits or 63 bits.
It is just that the numeric system chosen for most systems favors 32 bits or 64 bits, multiples of two or dividable by two if you want.  (base 2, 10 or any base. I like base 7: http://www.purplemath.com/modules/numbbase2.htm  https://en.wikipedia.org/wiki/Senary)
But multiples of three and devidability by three has distinct advantages....as can be mathematically proven.
The first! computers were based on that (three! states per "bit") principle.

Don't ever let you find yourself to believe there is only one system: there isn't. Quantum state is even more funny and closer to real life application than you might think.

It is also ridiculous to state an opinion that 32 bit software is necessarily inferior to 64 bit software.
That is simply not true. You are lacking quite a bit of knowledge.

Like that signed integers use a resolution of 31 bits or 63 bits on your home computer ? 
Oh, well. Mathematics. I love it 

[Mr.Madguy]

From a computational point of view there is no reason not to use 31 bits or 33 bits or 65 bits or 63 bits.
It is just that the numeric system chosen for most systems favors 32 bits or 64 bits, multiples of two or dividable by two if you want.  (base 2, 10 or any base. I like base 7: http://www.purplemath.com/modules/numbbase2.htm  https://en.wikipedia.org/wiki/Senary)
But multiples of three and devidability by three has distinct advantages....as can be mathematically proven.
The first! computers were based on that (three! states per "bit") principle.

Don't ever let you find yourself to believe there is only one system: there isn't. Quantum state is even more funny and closer to real life application than you might think.

It is also ridiculous to state an opinion that 32 bit software is necessarily inferior to 64 bit software.
That is simply not true. You are lacking quite a bit of knowledge.

Like that signed integers use a resolution of 31 bits or 63 bits on your home computer ? 
Oh, well. Mathematics. I love it 

Math is great, but computers are binary for reason - it's required by circuitry. No other elements, other than transistors, are possible at such small scale, as 14nm. And any logic, other than binary, requires threshold switches, that are impossible without precise resisters.

Quantum computers are great for bruteforce tasks only.

And, yeah, 4Gbs seemed unreachable back in era of 16bit computers, but now 32bits cause the same bottlenecks, 16bits caused in the past. And I personally have already started to feel it. I don't try to say, that 64bits are better, than 32bits. But I'm not one of these programmers, who refuse to port their programs to 64bit due to "you don't need it" reason.