Language syntax parsers

[Abelisto]

Sorry for question. But. There are 1) FPC parser to compile the code; 2) CodeTools parser to make Lazarus more complicated; 3) JCF (Jedy Code Formatter) to format the Pascal code; 4) SynEdit to highlite the sources; 5) SVN to find out the differences; 6) JS libraries to show the Pascal texts with keywords highlighting (it was really hard word...). There are absolutely different code doing the same tasks using different code base. Can we to unify those?

[Edson]

I don't know how that parsers are implemnted, but I agree. Although each parser have a different objective and the implementation are focused in that objective (speed, appearance, scanning features, ...).
Maybe a unified lexer (no parser) could be possible and would have save a lot of code.

About SynEdit, it has not a Parser. It has a syntax highlighter, and it is more a lexer, than a parser.

[sam707]

http://wiki.freepascal.org/How_To_Use_TFPExpressionParser

[Edson]

About a unified parser, my highlighter: https://github.com/t-edson/SynFacilSyn
is designed to work too, like a lexer and it's the base for this completion code tool: https://github.com/t-edson/SynFacilCompletion

It's used in this compiler: https://github.com/t-edson/PicPas, so this project use SynFacilSyn in differentes places:
* Like a lexer
* In the parser
* In the text highlighter (in the editor)
* In the completion tool.

[sam707]

I wrote millions lines (some I did copy from magazines while young, some I did invent myself years after years)... millions lines, never activated any highlighter or lexer. found them useless, each one his/her opinion.
Parsers are fundamental to compilers, painting is for painters ...

never planed to build a jpeg compilationationater LOL

[Abelisto]

Ok, there package `fcl-passrc` Is it have any relation to the FPC? Let me to be more specific: is this package have any relation to the FPC itself?

[molly]

@Abelisto
Wiki tells:

This package contains some useful units for dealing with Pascal source files: Parsing, creating structures in memory, and output to files.

Used by: fpdoc

edit: and a quick grep shows that the pas2js package/tool also uses it.

edit2: And taking a closer look:
- fpdoc + makeskel
- pas2js (i was unable to find a page for it)
- unitdiff

[marcov]

  Can we to unify those?

No since they have differing requirements. Compiler parsers are typically define before use, while syntaxhighlighting and codetools parsers might skip whole blocks as long as it recognizes the global blockstructure further along.

Some parsers try to process comments and whitespace (e.g. for pretty printing) some not. And since most are used interactively there are also some speed requirements.

[JuhaManninen]

One related issue:
 http://bugs.freepascal.org/view.php?id=19754
The initial idea was to use fcl-passrc but it does not work as a pretty-printer if I have understood right.
Codetools on the other hand is perfect for modifying code in-place. It is more like a toolbox for scanners and parsers.
Using Codetools for a pretty-printer formatter in Lazarus would have an extra benefit. The existing settings for Codetools behavior could be reused. A user would get the same formatting when typing and completing code, and when running an explicit formatter.

The issue summary says "Jedi Code Format (JCF) parser needs to be replaced with codetools parser" but in practice a whole new formatter must be created. The parsers are so different.
Who volunteers to implement it?

[Abelisto]

Probably I am  new-school, but, as I can imagine from point of OOP: Grammar -> Language family (there are a lot of different syntax: Fortran & Basic & Python dependent on character positions, Prolog where text is not the code actually) -> Language syntax (C/Pascal, actually there are only two most popular dialects (Java,JS are clones of C++, Ada, PLSQL are clones of Pascal, C# is clone of C++ and Pascal)) -> Compiler. Somewhere here lying the common part of compiler/syntax highliter/codetools and any others, lets say, interpreter of the programming language.

Finally: IMO FPC compiler and f.i. JCF should to have the common ancestor.

[JuhaManninen]

@Abelisto, you are now completely confused.
Compiler, interpreter, parser and scanner are very different things.
FPC compiler and JCF obviously cannot have a common ancestor because they do so different things.
I recommend you study some source code yourself to understand the complexity involved.

[Leledumbo]

Sorry for question. But. There are 1) FPC parser to compile the code; 2) CodeTools parser to make Lazarus more complicated; 3) JCF (Jedy Code Formatter) to format the Pascal code; 4) SynEdit to highlite the sources; 5) SVN to find out the differences; 6) JS libraries to show the Pascal texts with keywords highlighting (it was really hard word...). There are absolutely different code doing the same tasks using different code base. Can we to unify those?

Probably I am  new-school, but, as I can imagine from point of OOP: Grammar -> Language family (there are a lot of different syntax: Fortran & Basic & Python dependent on character positions, Prolog where text is not the code actually) -> Language syntax (C/Pascal, actually there are only two most popular dialects (Java,JS are clones of C++, Ada, PLSQL are clones of Pascal, C# is clone of C++ and Pascal)) -> Compiler. Somewhere here lying the common part of compiler/syntax highliter/codetools and any others, lets say, interpreter of the programming language.

Finally: IMO FPC compiler and f.i. JCF should to have the common ancestor.

There is indeed possibility for having unified parser, however there are things that must be sacrificed in order to gain the largest lowest common denominator for all use cases and that's often not acceptable. There are still people who complain the current compilation speed even though it's already blazing fast. With unified parser, it will be even slower. There are people complaining with codetools inability to deal with a little (semantically) incorrect code (but syntactically correct). With unified parser, the number of unacceptable construct will be even larger. And the list goes on and on for more specific use case.

Specific for JCF, it's a result of porting the tool from Delphi written project, which shares none of the existing FPC parser codebase, neither the compiler nor fcl-passrc.

[sam707]

I have a question @Abelisto

why would you unify parsers-highlighters-codefolderers-lexers into one only object/block? to reduce memory use on 4Go/4To machines? to build a full IDE on a smartphone with 128Mo only? I am wondering around the origins of your 1st post, because with nowadays 4 cores and 4/8Go ram standard computers/laptops running 4Ghz fast, I do not understand your basical motivation.

And I am a little bit curious

Thanks

[JuhaManninen]

why would you unify parsers-highlighters-codefolderers-lexers into one only object/block? to reduce memory use on 4Go/4To machines?

In general reusing code is a good thing, not only for memory consumption but also for maintenance.
The parsers, lexers and scanners mentioned here however have so different purposes that uniting them is not realistic.
Codetools is used to parse code on the fly when it is edited and modify it in-place. It can handle temporarily broken code.
Fcl-passrc builds a parse tree, loses comments and does not like syntax errors.
They are meant for different use cases.

The JCF parser is a problem because nobody maintains it actively and it does not support the latest language extensions.
Codetools could be used to make a similar formatter and it is already maintained for other purposes.