Pascal code instrumentation as a compilation part?

[Rel]

Hello! I'm looking for a best way to integrate FPC compiler with our custom code analysis/modification tools. We are using custom tools to perform different code metrics (like execution time, test coverage and etc) and even do some code obfuscation. So for example for measuring code execution time I need to insert 2 procedure calls into each function/procedure (the first one where the function starts, and the other one for each function exit). The code for these 2 procedures are implemented in a separate translation unit. What is the best way to do these code instrumentations (insert/modify code) with FPC compiler in terms of simplicity and performance? Does it (FPC) supports some kind of code generation plugins? Is it somehow possible to get and modify abstract syntax tree or any other internal compiler representation?

[marcov]

No plugin of any kind, commandline compiler only.

Keep also in mind that the compiler source is GPL.

[hinst]

CodeTools: http://wiki.lazarus.freepascal.org/Codetools
fcl-passrc: http://wiki.freepascal.org/fcl-passrc

CodeTools is perhaps a more suitable library for your task, considering that you want to modify implementation code

What I am thinking about is: in case one uses stack trace information to debug application, stack trace line numbers will have to be adjusted accordingly because after your utility inserts additional procedure calls line numbers will change. One possible solution is to put additional procedure calls on same line where begin & end reserved word is: like
begin StartMeasurement;
StopMeasurement; end;

These two libraries I mentioned are not integrated with compiler

[Martin_fr]

Have you seen:
http://wiki.lazarus.freepascal.org/FPProfiler

[Laksen]

If you want to do AST modifications you can have a look at way the nodetree optimizations are done in psub.pas

[kpeters58]

Maybe these guys have something suitable?

http://www.prodelphi.de/

We have been using their Delphi versions with great success - easy to use, very powerful and top notch support

[Rel]

It is sad that FPC doesn't support plugins of any kind, as a lot of modern compilers do (like Clang, GHC and etc). But lets say that I've wrote my custom preprocessor that insert instrumentation code into pascal source files (lets say it is python script or a custom app). What is the best way to integrate my preprocessor script/app into FPC's toolchain? I need to perform my instrumentations on all code, including runtime library and all the dependencies, and rebuild it all from scratch. When I compile pascal programs compiler figures out all the dependencies itself and compiles/link them in. How to make the compiler perform my custom preprocessing on all the dependencies including runtime library? Is it even possible?

[Leledumbo]

It is sad that FPC doesn't support plugins of any kind, as a lot of modern compilers do (like Clang, GHC and etc). But lets say that I've wrote my custom preprocessor that insert instrumentation code into pascal source files (lets say it is python script or a custom app). What is the best way to integrate my preprocessor script/app into FPC's toolchain? I need to perform my instrumentations on all code, including runtime library and all the dependencies, and rebuild it all from scratch. When I compile pascal programs compiler figures out all the dependencies itself and compiles/link them in. How to make the compiler perform my custom preprocessing on all the dependencies including runtime library? Is it even possible?

marco has given you a BIG hint. If you don't realize it yet: the compiler code is open, you can study and modify it yourself. FPC is no way like Clang (which is supported by Apple) or GHC (which is supported by Glasgow University). The developers are totally volunteers and they code in their spare time.

[marcov]

It is sad that FPC doesn't support plugins of any kind, as a lot of modern compilers do (like Clang, GHC and etc). But lets say that I've wrote my custom preprocessor that insert instrumentation code into pascal source files (lets say it is python script or a custom app). What is the best way to integrate my preprocessor script/app into FPC's toolchain? I need to perform my instrumentations on all code, including runtime library and all the dependencies, and rebuild it all from scratch. When I compile pascal programs compiler figures out all the dependencies itself and compiles/link them in. How to make the compiler perform my custom preprocessing on all the dependencies including runtime library? Is it even possible?

It's always possible. Simply don't do it on the fly, but have original source (-tree) that you run through the preprocessor, and then store in a new tree. Then invoke the compiler.

[Rel]

marco has given you a BIG hint. If you don't realize it yet: the compiler code is open, you can study and modify it yourself. FPC is no way like Clang (which is supported by Apple) or GHC (which is supported by Glasgow University). The developers are totally volunteers and they code in their spare time.

Well, I realized that the first time. But generally speaking I don't think that modifying and recompiling the compiler is a good idea. Firstly I'm not a Pascal expert, I have good experience in other languages, but still. Secondly the compiler's code base is huge and is not documented. So I can easily mess up something, that may break code generation for example. And fighting with some new (introduced by me) bugs inside the compiler is the last thing I'd like to do. You know, compiler's bugs is the worst bugs to find ever.

It's always possible. Simply don't do it on the fly, but have original source (-tree) that you run through the preprocessor, and then store in a new tree. Then invoke the compiler.

I think you misunderstood my problem. Lets say I write a simple program, like "writeln('Hello');". The actual code base (the source tree) is one file, but still compiler includes a lot of code from runtime library inside resulting executable. At compile time there is no way for me to find out what units my program depends on (only compiler knows it), so I can't take and preprocess them separately. I can take the whole runtime library, store it with the project and preprocess it all before calling FPC. But the runtime library is huge, preprocessing all pascal source in it will take a lot of time, so compile time will increase dramatically.

[Laksen]

Can you perhaps try to give some idea about what it is you want to do more specifically? Maybe with an example with how that would be done with Clang or GHC?

[Leledumbo]

Can you perhaps try to give some idea about what it is you want to do more specifically? Maybe with an example with how that would be done with Clang or GHC?

I don't think it would be that easy, since the problem lies in the architecture. Both Clang and GHC (and GCC since 4.something) has pluggable architecture that inserts several callbacks in the compilation stages. People may write plugins for any of those callbacks to alter the intermediate result (or simply reading it and doing something with it), without recompiling the compiler.

[marcov]

It's always possible. Simply don't do it on the fly, but have original source (-tree) that you run through the preprocessor, and then store in a new tree. Then invoke the compiler.

I think you misunderstood my problem. Lets say I write a simple program, like "writeln('Hello');". The actual code base (the source tree) is one file, but still compiler includes a lot of code from runtime library inside resulting executable. At compile time there is no way for me to find out what units my program depends on (only compiler knows it), so I can't take and preprocess them separately. I can take the whole runtime library, store it with the project and preprocess it all before calling FPC. But the runtime library is huge, preprocessing all pascal source in it will take a lot of time, so compile time will increase dramatically.

Yes. That's all correct, and I understood it correctly. Like most Borland versions, Free Pascal does not make allowances for an external preprocessor, and the few times that you really have a need, just keep it entirely external. It's a tradeoff to keep compiling speedy.

[Rel]

Well... Sadly it seems that there is no good way of solving my task with FPC. Anyway thanks a lot for your responses!

[avra]

Well... Sadly it seems that there is no good way of solving my task with FPC. Anyway thanks a lot for your responses!

Copying sources to project's subdir, injecting code as suggested by hinst and compiling is not good enough? That can all be automated...