Compiler Callback API => WAS: Poor optimization of constant folding

[lagprogramming]

During development, the sources can be compiled with an optimization level less than 3 and use level 3 at release. Level 3 has the slow optimizations. Usually the default optimization level used by both gcc and fpc is 2. Because there are different levels, build time should not be an issue.
With poor optimizations, I have doubts developers would chose FPC for low powered devices applications or for CPU intensive applications, which is not good at all. The LLVM backend may be useful up to a point.
The number of core developers for FPC, Lazarus and MSEide/MSEgui is not expected to increase in the near future. Maybe a proposal made in the past should be reconsidered. It was proposed that FPC and Lazarus implement addon capability that could be used by external developers to extend the functionality by using binary files(like library files). The main reason the proposal was ignored by those who read it at that time was because they wanted FPC and Lazarus pure open source, without the possibility to be easily mixed with closed source addons. Lazarus OPM was not released at that time, but it was already under development. FPC core developers are known to be even more conservative than Lazarus ones. As a consequence, interested developers were forced to learn the internals of Lazarus and FPC. Obviously, few of them did it and I have to say that at that time the community was much more vibrant than now. In the meantime, the number of core developers didn't increased and new core developers are less likely to appear.
Maybe now it's the right time to think again about it. Regarding FPC, one stage when the available addons would be called by the compiler is after the assembly code is generated. It would allow addons to do optimizations before the build process finishes. The fact that FPC has an option to keep the assembler files is not enough because after modifying them you can't ask the compiler to resume the building process, retrying to apply internal optimizations. I mean the compiler optimizes code, writes the *.s files, these files are improved using external tools, but the compiler might further improve those externally improved *.s files. It's this cycle that can't be used because FPC doesn't have a usable cross-platform "resume" like parameter to continue a build process from a specific stage, in this example restarting the assembly code optimizations on already optimized code until it remains unmodified by any internal or external(addons) algorithm.

I doubt what I am going to mention can be applied to FPC without significant effort.  The solution to a large number of expression optimization problems, including constant folding, is to evaluate the expression _symbolically_. 

This not only coalesces all constants (such as the above examples) but allows for the recognition of common algebraic patterns that can be substituted for equivalent but simpler (read faster) patterns.

A little bit of Mathematica in the compiler

There is a pascal unit with no 3rd party dependencies that interprets strings and tries to solve equations like Mathematica or Mathlab does, but for now it's source is not public. I estimate at least 70% of what FPC needs is already done in that unit file. However, binary files(like libraries) using this unit may be given to public. Unfortunately, again, FPC doesn't allow an external binary addon to be called with it's scanned pascal source, in order to allow the addon to optimize(simplify) the pascal code. The only way I know to interfere with the pascal source files is by using the ide to automatically call an external program before and after the build process, which is a poor implementation.

[Warfley]

I don't know the reasoning from the fpc developers, but personally I think that allowing for closed source add-ons to the compiler would not be a good idea.

First closed source development environments are historically speaking, not successful, quite the opposite infact. When looking at the most popular languages, opensource development always is more successful than closed source.
And the reason for this is simple, closed source suffers massively more from bitrot than opensource. There is still software that is built with java 6 or 8, two versions which you will not get an official SDK from oracle anymore. But you don't need that because the OpenJDK exists and in the server world (where java is most prolific) the OpenJDK is the default.
C# was closed source early on, then with the success of mono Microsoft decided to also make C# officially an open source language and with .net core even provided an open source runtime.
Python, JavaScript, C, C++, PHP, go, rust, swift, etc, pretty much all popular languages today, are all open source, or those that have closed source compilers (like intel C) are the dominant versions still the open source ones

There is nothing worse then when you have a long lasting project that outlives the technology stack used to build it on. I remember seeing old windows XP machines in hospitals running an old version of internet explorer because this was the only way to get some java applet to work that powers some MRI machine that was built in the late 90s, early 2000s and all of the tools required to run this are EOL but the machine still has to work.

Even if this would give a short term burst to the fpcs capabilities in the long run it will most certainly harm the project, when some projects don't work anymore because they require a plugin that was last updated 10 years ago and doesn't work with any compiler version released in the past decade anymore. Or when you are searching for something and get overwhelmed with a platera of add-ons that all may or may not work together and you only find out after you payd half a fortune to buy a yearly license.

Closed source development environments died 20 years ago and that's a good thing. Don't bring them back

[440bx]

I don't know the reasoning from the fpc developers, but personally I think that allowing for closed source add-ons to the compiler would not be a good idea.

But, the decision would _not_ be about allowing closed or open source add-ons, it would be about providing an interface to the compiler's structures for a plug-in (regardless of it being open or closed) to use for some purpose.

I believe that if a closed source plug-in existed that was genuinely useful, an open source version would eventually come into existence, in part to avoid some of the closed source problems you mentioned.

[lagprogramming]

I don't know the reasoning from the fpc developers, but personally I think that allowing for closed source add-ons to the compiler would not be a good idea.

But, the decision would _not_ be about allowing closed or open source add-ons, it would be about providing an interface to the compiler's structures for a plug-in (regardless of it being open or closed) to use for some purpose.

I believe that if a closed source plug-in existed that was genuinely useful, an open source version would eventually come into existence, in part to avoid some of the closed source problems you mentioned.

Yes, that's the point.
Devices using Android use an open source kernel, which has an integrated addon(plug-in) system that's used by external drivers(including firmware binary files). It's just an example where you can extend the capability using external open or close sourced addons(plug-ins).
Regarding FPC, if FPC had a relatively easy to use addon(plug-in) system I doubt developers would have started projects like MSElang or PicPas the way they did, because those developers had to copy or write many routines already existing in FPC. It's a waste of time, at least at the start of the project. PicPas is a pascal compiler that's built using FPC. To fully integrate it in FPC would take too much effort, but transforming it into an addon should be easier.

[PascalDragon]

Regarding FPC, if FPC had a relatively easy to use addon(plug-in) system I doubt developers would have started projects like MSElang or PicPas the way they did, because those developers had to copy or write many routines already existing in FPC.

A plugin systems means a stable API. We don't want to support something like that. We want to build an open source compiler that compiles (Object) Pascal code and not maintain some API for plugins that would have to change every other day, because we extended/changed/fixed something in the compiler. (Not to mention that plugins - in this case especially for Lazarus - first need dynamic packages fully working with is a big topic in and of itself)

Also MSElang was about creating a compiler for a Pascal variant free from the legacy baggage that FPC must carry with it. This wouldn't be something that you can simply solve by a plugin.
And nothing is stopping anyone from adding additional target processors for the small micro processors that PicPas supports (we do support AVR, Z80 and i8086 already after all), however FPC is not yet good at generating code for such small architectures and that is something that also can't be improved by some plugin, but it's a general code generation problem.

[440bx]

A plugin systems means a stable API. We don't want to support something like that. We want to build an open source compiler that compiles (Object) Pascal code and not maintain some API for plugins that would have to change every other day, because we extended/changed/fixed something in the compiler.

I realize that what follows is a discussion on "hot air" since a plug-in facility is unlikely to be in FPC's future but, creating a stable  plug-in API isn't that difficult.  MS figured out long ago that one way to extend functionality without changing the API is to have structures/records that declare their size and the recipient function returns data corresponding to the record size.  IOW, the API does not need to change to provide additional functionality.

(Not to mention that plugins - in this case especially for Lazarus - first need dynamic packages fully working with is a big topic in and of itself)

The plug-in facility would/should be exclusive to FPC, therefore it should not affect Lazarus one way or the other (presuming that the plug-ins are dlls loaded by FPC.)

I believe the biggest problem is that FPC's architecture isn't easily adapted to support plug-ins without a very significant amount of effort from the developers.  Likely a prohibitive effort.

[marcov]

A plugin systems means a stable API. We don't want to support something like that. We want to build an open source compiler that compiles (Object) Pascal code and not maintain some API for plugins that would have to change every other day, because we extended/changed/fixed something in the compiler.

I realize that what follows is a discussion on "hot air" since a plug-in facility is unlikely to be in FPC's future but, creating a stable  plug-in API isn't that difficult.  MS figured out long ago that one way to extend functionality without changing the API is to have structures/records that declare their size and the recipient function returns data corresponding to the record size.  IOW, the API does not need to change to provide additional functionality.

From 10km in the air everything seems small. Conceptualizing on the back of a beer coaster is not the hurdle. The nitty-gritty of implementing it in an useful way and then maintaining that.

But it is all academic because besides compiler developer resistance, for non GPL client plugins there also would be a licensing hurdle. So even if magically a plugin system got donated and accepted, it still would have to be GPL compatible as re-licensing is difficult.

But the main problem is that it requires the project developers to bend over backwards to support people that want to avoid to contribute for various reasons. (Laziness, not being the cooperative sort or commercial considerations)

[440bx]

From 10km in the air everything seems small. Conceptualizing on the back of a beer coaster is not the hurdle. The nitty-gritty of implementing it in an useful way and then maintaining that.

I stand by what I said.  Whether 10km or 1cm in the air, designing a _stable_ API, while it requires thought and care, is not very difficult. The design of APIs (stable and unstable) is not an occult science. 

That said, I concede and, conceded in the previous post, that the _implementation_ of an API could likely require a prohibitive effort, mostly due to FPC's current architecture.

[marcov]

From 10km in the air everything seems small. Conceptualizing on the back of a beer coaster is not the hurdle. The nitty-gritty of implementing it in an useful way and then maintaining that.

I stand by what I said.  Whether 10km or 1cm in the air, designing a _stable_ API, while it requires thought and care, is not very difficult. The design of APIs (stable and unstable) is not an occult science. 

A stable API is usually the result of some forward thinking (what could they need) combined with feedback found while actually using the API. But usually both require iteration. Even e.g. .NET went through several iterations (1.0, 1.1 and 2.0 were afaik mutually incompatible and only 2.0+ was somewhat stable). And that even with the massive internal use and considerable internal iterations feedback cycles that they had.

That said, I concede and, conceded in the previous post, that the _implementation_ of an API could likely require a prohibitive effort, mostly due to FPC's current architecture.
[/quote]

[lagprogramming]

A rudimentary(basic) plug-in system based on build stages doesn't require high maintenance or complex API because it's simple. For example, the compiler calls a library "processdata(filename{string},stage{enum})" routine for each processed file like:

• processdata('/development/foo.pas',stage1); At stage 1 the addon routine reads the file and if the file content has syntaxes known that can't be understood by the FPC, those are modified(translated). It's like an external compiler mode directive but it might even translate an entire file from a different than Pascal computer language to Pascal.
• processdata('/development/foo.pas',stage2); At stage 2 the compiler already checked the syntax of the source files and arranged it according to it's own standardized code format. For example, all lines are trimmed, all those useless empty lines are removed, macros have been applied, all function calls have a single text line with parameters, a single block for each constant, type and variable declarations(for example: "procedure Foo;var x:integer;var y:integer;begin...end;" would remove the duplicate "var" keyword) and so on. Because the content of the file has been cleaned, arranged and verified for valid syntaxes, it would be easy for the plug-in to work with it. A "repeat" line will always be followed by a line that starts with "until " and ends with ";". A "for" loop line will always start with "for ". An assignment line will always contain ":=" in a greater than 1 position and will end with ";". So, it would be easier to process most of the source code.
• ...
• processdata('/development/foo.s',stageN); At stage N the compiler built the assembly files, files that have the assembly code arranged the same way as the pascal code was arranged in stage 2. Again, the plug-in will analyse the file and make changes to it. For the code presented in the first post, the compiler generates consecutive and identical "neg*" lines, which means that a plug-in can easily remove some of them, it's not rocket science.

So, the plug-in system doesn't necessary have to be something very complex. In the above example, there is a single addon(plug-in) routine that's called. Obviously, file reads and writes should be avoided, a TargetCPU parameter might be usefull, a compiler optimization level parameter might be usefull, and so on. Plug-in capabilities might be presented at compiler's request, capabilities that should appear when calling the compiler with the "-i" series of parameters, but those are details.
To me it looks like it's not a significant effort to implement such a simple(basic, rudimentary) addon(plug-in) system mostly because it's based on compiler's build stages, not on internal routines(functions and procedures) and structures(like records) that are subject to frequent changes. And because it's based on compiler's build stages, similar results should be achievable by using compiler calling parameters. For example, FPC has the "-s" series of parameters to stop the building process before using an assembler, but doesn't have a parameter to stop the process in an earlier stage, when the code is still in Pascal syntax. In addition, FPC doesn't have a "restartstage" or  a "resumebuild" like parameter, forcing you to use external tools directly. So, once FPC has a parameter to stop it's build process in a stage where the code is still in Pascal language and a parameter to resume it's building process, or restart it at a specified stage, most likely 80% of this rudimentary addon infrastructure is done.
Obviously, FPC's current architecture might be so inappropriate to stopping, restarting a stage and resuming it's own building process, that would make implementing even such a stage-based rudimentary addon system a nightmare.

[TRon]

If this thread is going to divert into a discussion on a plugin system/implementation wouldn't it make more sense to split off this topic ?

2 cents

edit: @moderator: Thank you very much !

[marcov]

A rudimentary(basic) plug-in system based on build stages

In Free Pascal, the preprocessor and the tokenizer are pretty much merged and then directly deliver a token stream that is parsed into a tree representation. No other complete-file representation exist in between (not even non-text). So basing a plugin system on text  sounds like you haven't looked at how a/the compiler works.

What you are describing is more something for a source 2 source converter. Most compilers get  rid of textual representation as quickly as possible, simply because it is slowing.

Maybe an overall preprocessor would be theoretically possible. (operating on the whole file before being processed), because that could be inserted before the compiler does anything with the read file. But the rest seems pure fantasy to me.

[Fibonacci]

Maybe an overall preprocessor would be theoretically possible. (operating on the whole file before being processed), because that could be inserted before the compiler does anything with the read file. But the rest seems pure fantasy to me.

Because of lack of such thing I am using my own preprocessing system as "external tools" in the IDE. 2 external tools. First is doing preprocessing, second is reverting back the original source.

Preprocessor and revert-to-original could be executed before build and after build (in project settings), but the after build command is not executed if build fails. So I run this manually before production build.

Essentially, my preprocessing is changing or inserting some things in the source, and while in development mode it uses runtime functions as fallback.

Example, say I need a md5 hash of something that is known at compile time. I could just put the precomputed hash and add a comment what it is, but im calling __md5('something'). Normally, in development, it computes the hash in runtime, its just a normal function. After I run my preprocessor its changed to actual md5 of "something" as string.

Code: Pascal  [Select][+][-]

1. s := __md5('something');

->

Code: Pascal  [Select][+][-]

1. s := #$43#$7B#$93#$0D#$B8#$4B#$80#$79#$C2#$DD#$80#$4A#$71#$93#$6B#$5F{something};

There is few problems with this, files are changed, IDE asks for reload them, then reload again after reverting the original source. Preprocessor parsing the source on the fly without changing files on the disk would solve that. Also for everything I need to write a fallback, so I have a separate unit with all the fallbacks that is not included in production builds.

Plugins could be run on the source code before starting compilation, then on assembly code before linking, then on binary output.

If such plugin system existed someone would probably write a plugin to do the constant folding discussed in the original topic.

[marcov]

If such plugin system existed someone would probably write a plugin to do the constant folding discussed in the original topic.

That seems a rather workaround, rather fix it well and in the compiler. 

[Fibonacci]

That seems a rather workaround, rather fix it well and in the compiler.

Of course, but instead of waiting a decade to do actual fix in the compiler for constant folding a plugin could do it until then