Recent

Author Topic: Gold Parser Lazarus Engine problems  (Read 3810 times)

avra

  • Hero Member
  • *****
  • Posts: 1738
    • Additional info
Gold Parser Lazarus Engine problems
« on: September 03, 2018, 11:57:34 am »
I have built a FCL grammar as mentioned in this thread:
http://forum.lazarus.freepascal.org/index.php/topic,33830.msg296041.html#msg296041

There are no warnings nor errors, and I can successfuly test the grammar in Gold Test Window. However when I compile the grammar and try to parse the same code in LazarusGoldEngine (which can also be found in thread mentioned above) I get parsing errors.

This is a minimal grammar to demonstrate the problem (nice to see that BNF color highlighting works in this forum):
Code: bnf  [Select]
  1. "Case Sensitive" = False
  2. "Start Symbol" = <range>
  3.  
  4. Comment Block   @= { Nesting = All, Advance = Character }
  5. Comment Start    = '{'
  6. Comment End      = '}'
  7. Comment Line     = '//'
  8.                
  9. ASSIGN           = ':='
  10. SEMI             = ';'
  11. LEFTPAR          = '('
  12. RIGHTPAR         = ')'
  13. DOUBLEPOINT      = '..'
  14.  
  15. <range>
  16.         ::= RANGE ASSIGN LEFTPAR 0 DOUBLEPOINT 100 RIGHTPAR SEMI

I compile it in Gold to create CGT file, and load CGT file into Lazarus GoldEngine.

1) Now, if I use this code as test input:
Code: Pascal  [Select]
  1. RANGE := (0..100);
I get this error while parsing in Lazarus GoldEngine:
Quote
Line 1: Syntax Error: Expecting the following tokens: LEFTPAR
RANGE and ASSIGN tokens are recognized, but LEFTPAR not although it is there.

2) Also if I use a comment line as test input:
Code: Pascal  [Select]
  1. // some comment line
I get this error while parsing in Lazarus GoldEngine:
Quote
Line 1: Lexical Error: Cannot recognize token: s

Problem 1 is a major one, and currently a deal breaker since I could not figure out how to overcome it. Problem 2 is a minor one and it looks like GoldEngine ignores whole comment group defined in Gold. Not pleasant, and it would be nice if it could be fixed, but I think that i could eventually overcome it with some effort.

Problems are the same with both old non-generics Lazarus GoldEngine version and new Thaddy's version with generics.

I do hope that someone can take a look at this and help since I would really hate to drop GoldEngine  :'(
« Last Edit: September 03, 2018, 12:04:07 pm by avra »
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

Thaddy

  • Hero Member
  • *****
  • Posts: 9303
Re: Gold Parser Lazarus Engine problems
« Reply #1 on: September 03, 2018, 12:35:42 pm »
The compiled grammars from 5.2 do not always work correctly. Compiled grammars from 1.0 do.
I will try to analyze the problem. Can you attach a cgt?
also related to equus asinus.

avra

  • Hero Member
  • *****
  • Posts: 1738
    • Additional info
Re: Gold Parser Lazarus Engine problems
« Reply #2 on: September 03, 2018, 12:57:39 pm »
The compiled grammars from 5.2 do not always work correctly. Compiled grammars from 1.0 do.
I didn't know that. :o I do not have a 1.0 but I do have an old 3.0.1. I will try and report back.

I will try to analyze the problem. Can you attach a cgt?
Of course. Thanks a lot for looking into this problem.  :)
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

avra

  • Hero Member
  • *****
  • Posts: 1738
    • Additional info
Re: Gold Parser Lazarus Engine problems
« Reply #3 on: September 03, 2018, 02:01:55 pm »
The compiled grammars from 5.2 do not always work correctly. Compiled grammars from 1.0 do.
I didn't know that. :o I do not have a 1.0 but I do have an old 3.0.1. I will try and report back.
You were right. Old version 3.0.1 runs better, although with some quircks.

I had to remove comment group from grammar since old version does not support it. Here is a new grammar:
Code: bnf  [Select]
  1. "Case Sensitive" = False
  2. "Start Symbol" = <range>
  3.  
  4. ASSIGN           = ':='
  5. SEMI             = ';'
  6. LEFTPAR          = '('
  7. RIGHTPAR         = ')'
  8. DOUBLEPOINT      = '..'
  9.  
  10. <range>        ::= 'RANGE' ASSIGN LEFTPAR 0 DOUBLEPOINT 100 RIGHTPAR SEMI

It compiles in both old and new Gold without errors or warnings, and testing inside of both versions works without any problem. I have attached CGT files from both Gold versions. When running Lazarus GoldEngine with these CGT files and this input:
Code: Pascal  [Select]
  1. RANGE := (0..100);
then
a) with 3.0.1 parsing works and AST is shown (but only if there is at least one blank or newline after last character ';' - which seams as a 3.0.1 bug)
b) with 5.2.0 parsing does not work with the same error as before:
Quote
Line 1: Syntax Error: Expecting the following tokens: LEFTPAR

Hope this helps a little  ::)
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

avra

  • Hero Member
  • *****
  • Posts: 1738
    • Additional info
Re: Gold Parser Lazarus Engine problems
« Reply #4 on: September 03, 2018, 07:03:14 pm »
I have tested my FCL grammar with 3.0.1 and GoldEngine and it works. I haven't yet figured out exact rule for appending some whitespace character at the end of the line (or file?) to overcome the mentioned bug but hopefully that will be sorted out. So, now I have a work around and will continue to invest time in Gold Parser, being it 3.0.1 or 5.20. Whatever.

Thank you Thaddy !!!  8) ;) 8)

Btw. I have to correct my self and say that 3.0.1 supports comment groups. Only comment attributes are not supported. Meaning, this is not compatible with 3.01:
Code: Pascal  [Select]
  1. Comment Block   @= { Nesting = All, Advance = Character }
Good enough for me!  ::)
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

Thaddy

  • Hero Member
  • *****
  • Posts: 9303
Re: Gold Parser Lazarus Engine problems
« Reply #5 on: September 03, 2018, 09:16:23 pm »
I have tested my FCL grammar with 3.0.1 and GoldEngine and it works. I haven't yet figured out exact rule for appending some whitespace character at the end of the line (or file?) to overcome the mentioned bug but hopefully that will be sorted out. So, now I have a work around and will continue to invest time in Gold Parser, being it 3.0.1 or 5.20. Whatever.

Thank you Thaddy !!!  8) ;) 8)

Btw. I have to correct my self and say that 3.0.1 supports comment groups. Only comment attributes are not supported. Meaning, this is not compatible with 3.01:
Code: Pascal  [Select]
  1. Comment Block   @= { Nesting = All, Advance = Character }
Good enough for me!  ::)
I am also translating Coco /R to be Freepascal/Lazarus compatible. Almost finished except for two assembler routines that I have to change to Pascal. Code already works for Windows32/64
Coco /R is also nice-to-have.
also related to equus asinus.

avra

  • Hero Member
  • *****
  • Posts: 1738
    • Additional info
Re: Gold Parser Lazarus Engine problems
« Reply #6 on: September 04, 2018, 12:37:56 am »
I am also translating Coco /R to be Freepascal/Lazarus compatible.
I looked at Coco/R and Antlr for Delphi but they just didn't "click" for me. Gold, plex/pyacc and Lazarus Lex&Yacc are more to my taste. If you ever decide to try improving GoldEngine compatibility with Gold 5.2.0 then give me a note and I will help as much as possible. I have tried to dig into it myself but that didn't go well.
 :'(
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

avra

  • Hero Member
  • *****
  • Posts: 1738
    • Additional info
Re: Gold Parser Lazarus Engine problems
« Reply #7 on: September 10, 2018, 08:32:19 pm »
I have found a link to the latest Gold Parser 3.x.x version:
http://www.semitwist.com/download/gold-parser-builder-3-4-4.zip

It works good and uses smaller fonts so you can see more on your screen at once. It still uses the old CGT 1.0 file so it works fine with Lazarus Gold Engine 2018 which was refactored by Thaddy.

I have also found another Lazarus adaptation of the same Delphi engine:
https://github.com/cgarciagl/goldparser-Lazarus
It's a little more commented then the Thaddy's version so might be more appealing to the ones who want to understand the code.

I have saved the best for the end... :D 8-) :D   

There is a Lazarus/FPC engine for latest Gold Parser 5.x.x! Yes, you've read it well. Here it is:
https://gitlab.com/teo-tsirpanis-718/gold-parser-lazarus
It can read EGT v5.0 files. I have tested it it with FCL grammar written and compiled in Gold Parser 5.2.0, and it works!
« Last Edit: September 14, 2018, 03:20:22 am by avra »
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

taazz

  • Hero Member
  • *****
  • Posts: 5363
Re: Gold Parser Lazarus Engine problems
« Reply #8 on: September 10, 2018, 09:23:32 pm »
thank you for the feedback.
Good judgement is the result of experience … Experience is the result of bad judgement.

OS : Windows 7 64 bit
Laz: Lazarus 1.4.4 FPC 2.6.4 i386-win32-win32/win64

avra

  • Hero Member
  • *****
  • Posts: 1738
    • Additional info
Re: Gold Parser Lazarus Engine problems
« Reply #9 on: September 10, 2018, 09:52:49 pm »
thank you for the feedback.
You're most welcome  ;)
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

Trenatos

  • Hero Member
  • *****
  • Posts: 508
  • Software developer - Open source contributor
    • MarcusFernstrom.com
Re: Gold Parser Lazarus Engine problems
« Reply #10 on: September 11, 2018, 12:07:00 am »
This is really cool news.

I've never used Gold but I'll give it a shot after I learn some Lex/Yacc for comparison  :)

Thaddy

  • Hero Member
  • *****
  • Posts: 9303
Re: Gold Parser Lazarus Engine problems
« Reply #11 on: September 11, 2018, 09:53:33 am »
Avra, thanks for the link. Saves me lot's of work, It looks fine code, but there is some room to improve. I will suggest some pulls.
also related to equus asinus.

avra

  • Hero Member
  • *****
  • Posts: 1738
    • Additional info
Re: Gold Parser Lazarus Engine problems
« Reply #12 on: September 12, 2018, 01:07:24 am »
Avra, thanks for the link. Saves me lot's of work, It looks fine code, but there is some room to improve. I will suggest some pulls.
You're welcome. I also have some ideas for improvement.

This GOLD thing got so far that I am making a wiki. Stay tuned...
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

avra

  • Hero Member
  • *****
  • Posts: 1738
    • Additional info
Re: Gold Parser Lazarus Engine problems
« Reply #13 on: September 14, 2018, 01:26:36 am »
Not a single Delphi derived CGT v1.0 Gold engine can handle {All Valid} character set in CGT v1.0 file produced with Gold 3.x.x (according to Gold documentation it should be supported since Gold 2.6.0).

For example, this JSON grammar:
Code: bnf  [Select]
  1. "Name"     = JSON Grammar
  2. "Author"   = Ars ne von Wyss
  3. "Version"  = 1.0
  4. "About"    = 'Grammar for JSON data, following http://www.json.org/'
  5. ! and compliant with http://www.ietf.org/rfc/rfc4627
  6.  
  7. "Start Symbol" = <Json>
  8. "Case Sensitive" = True
  9. "Character Mapping" = 'Unicode'
  10.  
  11. ! ------------------------------------------------- Sets
  12.  
  13. {Unescaped} = {All Valid} - {&1 .. &19} - ["\]
  14. {Hex} = {Digit} + [ABCDEFabcdef]
  15. {Digit9} = {Digit} - [0]
  16.  
  17. ! ------------------------------------------------- Terminals
  18.  
  19. Number = '-'?('0'|{Digit9}{Digit}*)('.'{Digit}+)?([Ee][+-]?{Digit}+)?
  20. String = ["]({Unescaped}|'\'(["\/bfnrt]|'u'{Hex}{Hex}{Hex}{Hex}))*["]
  21.  
  22. ! ------------------------------------------------- Rules
  23.  
  24. <Json> ::= <Object>
  25.          | <Array>
  26.  
  27. <Object> ::= '{' '}'
  28.            | '{' <Members> '}'
  29.  
  30. <Members> ::= <Pair>
  31.             | <Pair> ',' <Members>
  32.  
  33. <Pair> ::= String ':' <Value>
  34.  
  35. <Array> ::= '[' ']'
  36.           | '[' <Elements> ']'
  37.  
  38. <Elements> ::= <Value>
  39.              | <Value> ',' <Elements>
  40.  
  41. <Value> ::= String
  42.           | Number
  43.           | <Object>
  44.           | <Array>
  45.           | true
  46.           | false
  47.           | null
parses this JSON:
Code: Text  [Select]
  1. {"widget": {
  2.     "debug": "on",
  3.     "window": {
  4.         "title": "Sample Konfabulator Widget",
  5.         "name": "main_window",
  6.         "width": 500,
  7.         "height": 500
  8.     },
  9.     "image": {
  10.         "src": "Images/Sun.png",
  11.         "name": "sun1",
  12.         "hOffset": 250,
  13.         "vOffset": 250,
  14.         "alignment": "center"
  15.     },
  16.     "text": {
  17.         "data": "Click Here",
  18.         "size": 36,
  19.         "style": "bold",
  20.         "name": "text1",
  21.         "hOffset": 250,
  22.         "vOffset": 100,
  23.         "alignment": "center",
  24.         "onMouseUp": "sun1.opacity = (sun1.opacity / 100) * 90;"
  25.     }
  26. }}  
without a problem in Gold Parser Builder. Unfortunately none of the CGT v1.0 engines can load such CGT file. Problem is when {All Valid} is used with "Character Mapping" = 'Unicode'. If you simply replace {All Valid} with {Printable}, all engines load CGT file and parsing works.

However, the already mentioned "all mighty" EGT v5.0 engine CAN handle EGT files with {All Valid} character set, and it doesn't care if file is CGT v1.0 or EGT v5.0, written with Gold Parser Builder 3.x.x or 5.x.x. (although I have found a few complex grammars which run fine as EGT but not as CGT).

One more plus for EGT v5.0 engine  8)
« Last Edit: September 14, 2018, 03:22:21 am by avra »
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

Thaddy

  • Hero Member
  • *****
  • Posts: 9303
Re: Gold Parser Lazarus Engine problems
« Reply #14 on: September 14, 2018, 08:22:12 am »
Yes, indeed. I also recommend 5 (even if I wrote one of the older engines)
also related to equus asinus.