Parsing compiler directives with fcl-passrc

[simone]

I have already asked this question on another thread on fcl-passrc related to another matter. For this reason, perhaps it has gone unnoticed. I try again by opening a new thread.

How can I parse compiler directive (i.e. {$ DIRECTIVE [value]}) with fcl-passrc? In other words, how can I detect them through the parser? Thanks.

[jamie]

I don't use that code but have you looked at the "DocComment" of the TpasElement ?

Comments are {….} based generally...

[marcov]

Directive processing happens in the preprocessor pass before the actual pascal parser.  The preprocessor in FPC speak is combined with the tokenizer and called scanner.

I assume you have already looked at pscanner:TPascalScanner?

[simone]

I don't use that code but have you looked at the "DocComment" of the TpasElement ?

Comments are {….} based generally…

I tried to use the DocCommet property, but I didn't extract any information from it.

Directive processing happens in the preprocessor pass before the actual pascal parser.  The preprocessor in FPC speak is called scanner.

I assume you have already looked at pscanner:TPascalScanner?

Yes, I know that during the compilation the directives are examined by the preprocessor before the scanner. I need to recognize them in the parsing phase because I need to know the code context where they appear, especially for the local ones.

I am studying the code of fcl-passrc, following your hint, as unfortunately for this library I have found only a minimal wiki.

[440bx]

This post unfortunately won't be of any help to simone, just something for someone to think about (if they are so inclined)...

There are probably some programmers around here who have heard of Roslyn and the basic idea behind it.  It would be very nice if FPC decided to slowly but surely implement that concept, that is, making the information resulting from the scanning and parsing phases accessible to a "client", most likely through a set of interfaces.

With that, things like fcl-passsrc are quite likely to no longer be of any use and, many very useful utilities could be created to analyze code because the information is coming directly from the compiler and is the same information that is used by the compiler.

Just in case, the above isn't a feature request.

[simone]

It would be very nice if FPC decided to slowly but surely implement that concept, that is, making the information resulting from the scanning and parsing phases accessible to a "client", most likely through a set of interfaces.

This would be very useful for me.

[marcov]

This post unfortunately won't be of any help to simone, just something for someone to think about (if they are so inclined)...

There are probably some programmers around here who have heard of Roslyn and the basic idea behind it.  It would be very nice if FPC decided to slowly but surely implement that concept, that is, making the information resulting from the scanning and parsing phases accessible to a "client", most likely through a set of interfaces.

Not going to happen. Both compilation speed, and having to long term maintain internal compiler structures that way.

FPC is set up as a production compiler, not as a research project.

This is the way why fcl-passrc exists.

[simone]

I understand this motivation. The problem is that in the world of Lazarus / FPC there are at least four parsers:

- the internal compiler parser;
- the FCL parser called passrc;
- the parser in Codetools;
- the parser in Jedi Code Formatter.

Keeping all the last three aligned with the first one requires a lot of effort for the development team.

[Thaddy]

The latter two are not really parsers in the sense of resolving to code. The former two are.
Note that JCF has a lot of issues and is not a serious option. It may be for Delphi, but not for FPC.
I am afraid the codetools option also isn't that serious an option, because it is geared towards near real-time and only has to resolve syntax.

[marcov]

I understand this motivation. The problem is that in the world of Lazarus / FPC there are at least four parsers:

- the internal compiler parser;

Production

- the FCL parser called passrc;

Documentation and transpiler.

- the parser in Codetools;

Must deal with potentially incorrect code. Production parsers halt at the first error.

- the parser in Jedi Code Formatter.

I don't know the motivation about this one. Afaik it was mostly preexisting.

Keeping all the last three aligned with the first one requires a lot of effort for the development team.

But it is done by different people at different times. Centralizing this creates a central point of failure.

I don't think you'll ever earn back attempts at integrating them.

[Thaddy]

And we have also the parser in ptop....(better than jcf)

[simone]

Thanks Thaddy. Yes it is, in the strict technical sense the last two of my list are not parsers. I know Ptop but in my code generator I use JCF. When I made this choice I thought it was better as it was integrated into the IDE. Does Ptop have its own parser (broadly speaking) or does it use fcl-passrc?

[marcov]

Thanks Thaddy. Yes it is, in the strict technical sense the last two of my list are not parsers. I know Ptop but in my code generator I use JCF. When I made this choice I thought it was better as it was integrated into the IDE. Does Ptop have its own parser (broadly speaking) or does it use fcl-passrc?

Ptop has its own very simple ones (it basically search for keywords and adjusts indentation level accordingly,
and is only slightly in excess of 1000 lines.

[440bx]

Not going to happen.

I "suspected" that much.

Both compilation speed,

That's not a good argument, while it does require the compiler to keep track of more information, that information is usable in other areas in the compilation process.  The net effect is usually negligible or well worth the advantages obtained. There is a little extra overhead upfront but, the information it produces saves time in subsequent compilation phases.  The increase in the final usability is _substantial_.

and having to long term maintain internal compiler structures that way.

That is true of the internal structures it is currently using.  There is no difference there.

FPC is set up as a production compiler, not as a research project.

We probably have a very different viewpoint on the subject but, to me, a production compiler provides as much information as it can to the programmer using it.  It doesn't just generate code.  Generating code is the bottom of the basement, IMO, a production compiler goes a bit beyond that.  When I want the "minimum" and I want it fast, I go to McDonalds, they produce "food" (code) and they definitely do it fast but, in the long run, it's a bit short of something consistently desirable.

not as a research project.

Is that a way of saying that a compiler that uses something a bit more modern than 1960's compiler construction techniques is a "research project" ?...  I don't believe Delphi is a research project and, one of the obvious improvements they have implemented is making the compiler a DLL.   That way the RAD environment doesn't have to create a new process for every compilation (by the way, that takes time) nor does it have to communicate with the compiler using pipes (which is definitely far from optimal.) 

This is the way why fcl-passrc exists.

And along with it, the overhead and hassle of having to keep it in synch with the actual compiler.  "Research" has already shown, long ago I might add, that keeping multiple copies of the same thing in synch is nothing but, a never ending bag of headaches.   It is things like the existence and overhead of fcl-passrc that is a "clue" that there is room for improvement there.

Again, just of the record, I don't have any illusions of FPC going that direction anytime soon but, it would be good if those involved in its development took notice of how compiler architecture is changing to better serve the user/programmer.

[PascalDragon]

not as a research project.

Is that a way of saying that a compiler that uses something a bit more modern than 1960's compiler construction techniques is a "research project" ?...  I don't believe Delphi is a research project and, one of the obvious improvements they have implemented is making the compiler a DLL.   That way the RAD environment doesn't have to create a new process for every compilation (by the way, that takes time) nor does it have to communicate with the compiler using pipes (which is definitely far from optimal.) 

Hosting the compiler inside the IDE isn't optimal either (doesn't matter if it's dynamically like in Delphi or statically as it is in FPC's textmode IDE), cause memory leaks will accumulate and it needs to be made sure that the compiler's state is correctly reset upon an error so that it doesn't influence the next run. And yes, memory leaks in the compiler are things that occurred in Delphi. It makes the IDE unusable sooner or later.