Lazarus

Announcements => Third party => Topic started by: avra on September 03, 2018, 11:57:34 am

Title: Gold Parser Lazarus Engine problems
Post by: avra on September 03, 2018, 11:57:34 am
I have built a FCL grammar as mentioned in this thread:
http://forum.lazarus.freepascal.org/index.php/topic,33830.msg296041.html#msg296041

There are no warnings nor errors, and I can successfuly test the grammar in Gold Test Window. However when I compile the grammar and try to parse the same code in LazarusGoldEngine (which can also be found in thread mentioned above) I get parsing errors.

This is a minimal grammar to demonstrate the problem (nice to see that BNF color highlighting works in this forum):
Code: bnf  [Select]
  1. "Case Sensitive" = False
  2. "Start Symbol" = <range>
  3.  
  4. Comment Block   @= { Nesting = All, Advance = Character }
  5. Comment Start    = '{'
  6. Comment End      = '}'
  7. Comment Line     = '//'
  8.                
  9. ASSIGN           = ':='
  10. SEMI             = ';'
  11. LEFTPAR          = '('
  12. RIGHTPAR         = ')'
  13. DOUBLEPOINT      = '..'
  14.  
  15. <range>
  16.         ::= RANGE ASSIGN LEFTPAR 0 DOUBLEPOINT 100 RIGHTPAR SEMI

I compile it in Gold to create CGT file, and load CGT file into Lazarus GoldEngine.

1) Now, if I use this code as test input:
Code: Pascal  [Select]
  1. RANGE := (0..100);
I get this error while parsing in Lazarus GoldEngine:
Quote
Line 1: Syntax Error: Expecting the following tokens: LEFTPAR
RANGE and ASSIGN tokens are recognized, but LEFTPAR not although it is there.

2) Also if I use a comment line as test input:
Code: Pascal  [Select]
  1. // some comment line
I get this error while parsing in Lazarus GoldEngine:
Quote
Line 1: Lexical Error: Cannot recognize token: s

Problem 1 is a major one, and currently a deal breaker since I could not figure out how to overcome it. Problem 2 is a minor one and it looks like GoldEngine ignores whole comment group defined in Gold. Not pleasant, and it would be nice if it could be fixed, but I think that i could eventually overcome it with some effort.

Problems are the same with both old non-generics Lazarus GoldEngine version and new Thaddy's version with generics.

I do hope that someone can take a look at this and help since I would really hate to drop GoldEngine  :'(
Title: Re: Gold Parser Lazarus Engine problems
Post by: Thaddy on September 03, 2018, 12:35:42 pm
The compiled grammars from 5.2 do not always work correctly. Compiled grammars from 1.0 do.
I will try to analyze the problem. Can you attach a cgt?
Title: Re: Gold Parser Lazarus Engine problems
Post by: avra on September 03, 2018, 12:57:39 pm
The compiled grammars from 5.2 do not always work correctly. Compiled grammars from 1.0 do.
I didn't know that. :o I do not have a 1.0 but I do have an old 3.0.1. I will try and report back.

I will try to analyze the problem. Can you attach a cgt?
Of course. Thanks a lot for looking into this problem.  :)
Title: Re: Gold Parser Lazarus Engine problems
Post by: avra on September 03, 2018, 02:01:55 pm
The compiled grammars from 5.2 do not always work correctly. Compiled grammars from 1.0 do.
I didn't know that. :o I do not have a 1.0 but I do have an old 3.0.1. I will try and report back.
You were right. Old version 3.0.1 runs better, although with some quircks.

I had to remove comment group from grammar since old version does not support it. Here is a new grammar:
Code: bnf  [Select]
  1. "Case Sensitive" = False
  2. "Start Symbol" = <range>
  3.  
  4. ASSIGN           = ':='
  5. SEMI             = ';'
  6. LEFTPAR          = '('
  7. RIGHTPAR         = ')'
  8. DOUBLEPOINT      = '..'
  9.  
  10. <range>        ::= 'RANGE' ASSIGN LEFTPAR 0 DOUBLEPOINT 100 RIGHTPAR SEMI

It compiles in both old and new Gold without errors or warnings, and testing inside of both versions works without any problem. I have attached CGT files from both Gold versions. When running Lazarus GoldEngine with these CGT files and this input:
Code: Pascal  [Select]
  1. RANGE := (0..100);
then
a) with 3.0.1 parsing works and AST is shown (but only if there is at least one blank or newline after last character ';' - which seams as a 3.0.1 bug)
b) with 5.2.0 parsing does not work with the same error as before:
Quote
Line 1: Syntax Error: Expecting the following tokens: LEFTPAR

Hope this helps a little  ::)
Title: Re: Gold Parser Lazarus Engine problems
Post by: avra on September 03, 2018, 07:03:14 pm
I have tested my FCL grammar with 3.0.1 and GoldEngine and it works. I haven't yet figured out exact rule for appending some whitespace character at the end of the line (or file?) to overcome the mentioned bug but hopefully that will be sorted out. So, now I have a work around and will continue to invest time in Gold Parser, being it 3.0.1 or 5.20. Whatever.

Thank you Thaddy !!!  8) ;) 8)

Btw. I have to correct my self and say that 3.0.1 supports comment groups. Only comment attributes are not supported. Meaning, this is not compatible with 3.01:
Code: Pascal  [Select]
  1. Comment Block   @= { Nesting = All, Advance = Character }
Good enough for me!  ::)
Title: Re: Gold Parser Lazarus Engine problems
Post by: Thaddy on September 03, 2018, 09:16:23 pm
I have tested my FCL grammar with 3.0.1 and GoldEngine and it works. I haven't yet figured out exact rule for appending some whitespace character at the end of the line (or file?) to overcome the mentioned bug but hopefully that will be sorted out. So, now I have a work around and will continue to invest time in Gold Parser, being it 3.0.1 or 5.20. Whatever.

Thank you Thaddy !!!  8) ;) 8)

Btw. I have to correct my self and say that 3.0.1 supports comment groups. Only comment attributes are not supported. Meaning, this is not compatible with 3.01:
Code: Pascal  [Select]
  1. Comment Block   @= { Nesting = All, Advance = Character }
Good enough for me!  ::)
I am also translating Coco /R to be Freepascal/Lazarus compatible. Almost finished except for two assembler routines that I have to change to Pascal. Code already works for Windows32/64
Coco /R is also nice-to-have.
Title: Re: Gold Parser Lazarus Engine problems
Post by: avra on September 04, 2018, 12:37:56 am
I am also translating Coco /R to be Freepascal/Lazarus compatible.
I looked at Coco/R and Antlr for Delphi but they just didn't "click" for me. Gold, plex/pyacc and Lazarus Lex&Yacc are more to my taste. If you ever decide to try improving GoldEngine compatibility with Gold 5.2.0 then give me a note and I will help as much as possible. I have tried to dig into it myself but that didn't go well.
 :'(
Title: Re: Gold Parser Lazarus Engine problems
Post by: avra on September 10, 2018, 08:32:19 pm
I have found a link to the latest Gold Parser 3.x.x version:
http://www.semitwist.com/download/gold-parser-builder-3-4-4.zip

It works good and uses smaller fonts so you can see more on your screen at once. It still uses the old CGT 1.0 file so it works fine with Lazarus Gold Engine 2018 which was refactored by Thaddy.

I have also found another Lazarus adaptation of the same Delphi engine:
https://github.com/cgarciagl/goldparser-Lazarus
It's a little more commented then the Thaddy's version so might be more appealing to the ones who want to understand the code.

I have saved the best for the end... :D 8-) :D   

There is a Lazarus/FPC engine for latest Gold Parser 5.x.x! Yes, you've read it well. Here it is:
https://gitlab.com/teo-tsirpanis-718/gold-parser-lazarus
It can read EGT v5.0 files. I have tested it it with FCL grammar written and compiled in Gold Parser 5.2.0, and it works!
Title: Re: Gold Parser Lazarus Engine problems
Post by: taazz on September 10, 2018, 09:23:32 pm
thank you for the feedback.
Title: Re: Gold Parser Lazarus Engine problems
Post by: avra on September 10, 2018, 09:52:49 pm
thank you for the feedback.
You're most welcome  ;)
Title: Re: Gold Parser Lazarus Engine problems
Post by: Trenatos on September 11, 2018, 12:07:00 am
This is really cool news.

I've never used Gold but I'll give it a shot after I learn some Lex/Yacc for comparison  :)
Title: Re: Gold Parser Lazarus Engine problems
Post by: Thaddy on September 11, 2018, 09:53:33 am
Avra, thanks for the link. Saves me lot's of work, It looks fine code, but there is some room to improve. I will suggest some pulls.
Title: Re: Gold Parser Lazarus Engine problems
Post by: avra on September 12, 2018, 01:07:24 am
Avra, thanks for the link. Saves me lot's of work, It looks fine code, but there is some room to improve. I will suggest some pulls.
You're welcome. I also have some ideas for improvement.

This GOLD thing got so far that I am making a wiki. Stay tuned...
Title: Re: Gold Parser Lazarus Engine problems
Post by: avra on September 14, 2018, 01:26:36 am
Not a single Delphi derived CGT v1.0 Gold engine can handle {All Valid} character set in CGT v1.0 file produced with Gold 3.x.x (according to Gold documentation it should be supported since Gold 2.6.0).

For example, this JSON grammar:
Code: bnf  [Select]
  1. "Name"     = JSON Grammar
  2. "Author"   = Ars ne von Wyss
  3. "Version"  = 1.0
  4. "About"    = 'Grammar for JSON data, following http://www.json.org/'
  5. ! and compliant with http://www.ietf.org/rfc/rfc4627
  6.  
  7. "Start Symbol" = <Json>
  8. "Case Sensitive" = True
  9. "Character Mapping" = 'Unicode'
  10.  
  11. ! ------------------------------------------------- Sets
  12.  
  13. {Unescaped} = {All Valid} - {&1 .. &19} - ["\]
  14. {Hex} = {Digit} + [ABCDEFabcdef]
  15. {Digit9} = {Digit} - [0]
  16.  
  17. ! ------------------------------------------------- Terminals
  18.  
  19. Number = '-'?('0'|{Digit9}{Digit}*)('.'{Digit}+)?([Ee][+-]?{Digit}+)?
  20. String = ["]({Unescaped}|'\'(["\/bfnrt]|'u'{Hex}{Hex}{Hex}{Hex}))*["]
  21.  
  22. ! ------------------------------------------------- Rules
  23.  
  24. <Json> ::= <Object>
  25.          | <Array>
  26.  
  27. <Object> ::= '{' '}'
  28.            | '{' <Members> '}'
  29.  
  30. <Members> ::= <Pair>
  31.             | <Pair> ',' <Members>
  32.  
  33. <Pair> ::= String ':' <Value>
  34.  
  35. <Array> ::= '[' ']'
  36.           | '[' <Elements> ']'
  37.  
  38. <Elements> ::= <Value>
  39.              | <Value> ',' <Elements>
  40.  
  41. <Value> ::= String
  42.           | Number
  43.           | <Object>
  44.           | <Array>
  45.           | true
  46.           | false
  47.           | null
parses this JSON:
Code: Text  [Select]
  1. {"widget": {
  2.     "debug": "on",
  3.     "window": {
  4.         "title": "Sample Konfabulator Widget",
  5.         "name": "main_window",
  6.         "width": 500,
  7.         "height": 500
  8.     },
  9.     "image": {
  10.         "src": "Images/Sun.png",
  11.         "name": "sun1",
  12.         "hOffset": 250,
  13.         "vOffset": 250,
  14.         "alignment": "center"
  15.     },
  16.     "text": {
  17.         "data": "Click Here",
  18.         "size": 36,
  19.         "style": "bold",
  20.         "name": "text1",
  21.         "hOffset": 250,
  22.         "vOffset": 100,
  23.         "alignment": "center",
  24.         "onMouseUp": "sun1.opacity = (sun1.opacity / 100) * 90;"
  25.     }
  26. }}  
without a problem in Gold Parser Builder. Unfortunately none of the CGT v1.0 engines can load such CGT file. Problem is when {All Valid} is used with "Character Mapping" = 'Unicode'. If you simply replace {All Valid} with {Printable}, all engines load CGT file and parsing works.

However, the already mentioned "all mighty" EGT v5.0 engine CAN handle EGT files with {All Valid} character set, and it doesn't care if file is CGT v1.0 or EGT v5.0, written with Gold Parser Builder 3.x.x or 5.x.x. (although I have found a few complex grammars which run fine as EGT but not as CGT).

One more plus for EGT v5.0 engine  8)
Title: Re: Gold Parser Lazarus Engine problems
Post by: Thaddy on September 14, 2018, 08:22:12 am
Yes, indeed. I also recommend 5 (even if I wrote one of the older engines)
Title: Re: Gold Parser Lazarus Engine problems
Post by: avra on August 07, 2019, 12:51:05 pm
Theodore has fixed an important 5.0 engine bug and memory leak with Comment1, Comment2 and Comment3 interpretation:
https://gitlab.com/teo-tsirpanis/gold-parser-lazarus/issues/1

If you need Gold Parser 5.0 engine then you should update ASAP.
Title: Re: Gold Parser Lazarus Engine problems
Post by: Thaddy on August 07, 2019, 01:28:53 pm
Which engine original? Mine? or one of the other ones?
Title: Re: Gold Parser Lazarus Engine problems
Post by: avra on August 07, 2019, 03:43:14 pm
The one that Theodore translated from original Java 5.0 engine.