Compiler Callback API => WAS: Poor optimization of constant folding

[Martin_fr]

There is few problems with this, files are changed, IDE asks for reload them, then reload again after reverting the original source. Preprocessor parsing the source on the fly without changing files on the disk would solve that. Also for everything I need to write a fallback, so I have a separate unit with all the fallbacks that is not included in production builds.

off topic to the main thread, but: Put the processed source (all sources) into a temp folder. Substitute the compiler in the IDE by a script of yours, and modify the arguments so it compiles your temp files.

Mind: in 2.2.6  (and 3.0RC1) the debugger may not like it because then the path of pas files in debug info is different. in 3.0. fixes it should be ok (but you may need to remove the temp files after compile, before the debugger starts).

[ccrause]

Plugins could be run on the source code before starting compilation, then on assembly code before linking, then on binary output.

The plugin would then have to duplicate some of the existing compiler functionality, such as parsing text source into some kind of internal representation, operating on this internal representation and then converting back to text source. Perhaps relatively easy to handle the current case of constant folding, but handling complex cases requiring syntax analysis would require more effort to implement (at least in principle, one could reuse one of the existing non-compiler parsers which would not be much extra real effort).  However the above scenarios sounds a bit more like pre- and post processing since it doesn't really interact with the compiler itself (as pointed out already by Marco).

I was thinking that one should rather perform optimizations on the internal node representation generated by the compiler, then perhaps hook into the peephole optimizer stage for processor specific optimizations.  However, this would require an addin to be aware of certain internal structures for information, making this not much different from adding an optimization directly to the source tree.  Although an addin would be outside the compiler source tree, so in theory should be "safer".  The big thing about this (also already pointed out) is that the compiler source is not designed to make this easy, so a lot of effort will be required to enable the compiler to share information with an addin via an API.

[Fibonacci]

There is few problems with this, files are changed, IDE asks for reload them, then reload again after reverting the original source. Preprocessor parsing the source on the fly without changing files on the disk would solve that. Also for everything I need to write a fallback, so I have a separate unit with all the fallbacks that is not included in production builds.

off topic to the main thread, but: Put the processed source (all sources) into a temp folder. Substitute the compiler in the IDE by a script of yours, and modify the arguments so it compiles your temp files.

Mind: in 2.2.6  (and 3.0RC1) the debugger may not like it because then the path of pas files in debug info is different. in 3.0. fixes it should be ok (but you may need to remove the temp files after compile, before the debugger starts).

Actually not bad idea

build.bat as "compiler command"

Code: Pascal  [Select][+][-]

1. @echo off
2.  
3. cd %~dp0
4.  
5. echo build executed >> log.txt
6. echo %* >> log.txt
7.  
8. set COMPPATH=C:\...\fpc.exe
9.  
10. if "%1" equ "-va" ( goto compiler )
11. if "%1" equ "-iWTOTP" ( goto compiler )
12.  
13. parse_and_run_compiler.bat %*
14. exit
15.  
16. :compiler
17. %COMPPATH% %*

parse_and_run_compiler.bat

Code: Pascal  [Select][+][-]

1. @echo off
2.  
3. rem this would be the parser executable
4. rem parse, change paths in command line, execute compiler
5.  
6. cd %~dp0
7.  
8. echo parse and run executed >> log.txt
9. echo %* >> log.txt
10.  
11. set COMPPATH=C:\...\fpc.exe
12.  
13. %COMPPATH% %*

The plugin would then have to duplicate some of the existing compiler functionality, such as parsing text source into some kind of internal representation, operating on this internal representation and then converting back to text source.

It doesnt have to parse to internal representation. Take input as plain source, do whatever, and return plain source.

[440bx]

At this point it's probably worth noting that there are two (2) kinds of plug-ins. 

The first and "simplest" is the type that uses information created by the compiler to derive new information from it, e.g, a source analyzer.

The second and, much more complicated by several orders of magnitude, is a plug-in that either "assists" or possibly even replaces some of the compiler's functions such as some optimizations.  This not only requires accommodating plug-ins but also requires a level of compiler modularization and standardization of internal data representation that isn't easy to accomplish.

As far as source preprocessors go, anyone who has used C is sadly aware of all the problems source preprocessors can, and often do, create.  I believe it was all those problems that led to the addition of "templates"/"generics" in the compiler which is, "preprocessing" (effectively, a macro resolved by the compiler) done by the compiler not by some text manipulation "thingy".

[marcov]

Of course, but instead of waiting a decade to do actual fix in the compiler for constant folding a plugin could do it until then

Not if you first need to do a full re-architecture of the compiler to enable that plugin. That is magnitudes more.

[Warfley]

First, I believe that there is currently work going on for compiletime procedures which would replace such hacky macro replacement

Second, I often work with custom generated code. The solution to most problems is simple, never Mix generated and handwritten code. Always generated your code into newwith things like ide reloading, or Versioncontrolsystems

[lagprogramming]

Of course, but instead of waiting a decade to do actual fix in the compiler for constant folding a plugin could do it until then

Not if you first need to do a full re-architecture of the compiler to enable that plugin. That is magnitudes more.

According to it's current architecture, can FPC be easily modified to resume a process stopped by one of the "-s" series of parameters?
When using the "-sh" parameter FPC writes the assembler files and an additional script that may be used in order to resume the building process. In order to assemble and link those files using the script, it's mandatory to use external tools even though FPC might have the capability to avoid them(depending on the host and target OS-CPU). No matter it's capability, instead of using the generated script, FPC might be called with a "-resumeprocesss" or a "-restartstage_assembly" like parameter. If FPC needs to call external tools to continue building the file that's ok, but FPC should be the one that does that. Makes easier to implement addon-like capability at IDE level.

[marcov]

According to it's current architecture, can FPC be easily modified to resume a process stopped by one of the "-s" series of parameters?

Other than running the script, no. All info the compiler has is stored in that script, there is no context for restarting. Restarting the compiler is a new independent compilation step again.

And keep in mind that in many cases, these points are not points the normal build process will reach, the -s parameters force the compiler to take a different path to generate textual stuff for debug purposes. E.g. Linux and FreeBSD x86 and x86_64 write .o files directly. 

Win32, win64 and wince write .o directly AND link directly to an EXE.