Making sense of pparser, pastree and pscanner - help needed.

[JernejL]

I have tried to use unit parser included in fpc. followed instructions here:
 
https://wiki.freepascal.org/fcl-passrc

I used TSimpleEngine from there, and some other sources to see how resolver,scanner and parser work together.

However, if a class file uses {$include} directive, it cannot find includes:

Code: Pascal  [Select][+][-]

1. game_Actors.pas(10,24) Error: Could not find include file 'compiler.inc'

I cannot figure out why as i did set BaseDirectory and add AddIncludePath to TStreamResolver where the include file is. Does anybody know how to set this up properly? Since fpdoc uses it, i assume there is a way to make this work.
 
for example, just drop a memo, button and opendialog onto form:

Code: Pascal  [Select][+][-]

1.  
2. unit Unit1;
3.  
4. {$mode objfpc}{$H+}
5.  
6. interface
7.  
8. uses
9.         Classes, SysUtils, Forms, Controls, Graphics, Dialogs, Buttons, StdCtrls, pparser, pastree, pscanner, strutils;
10.  
11. type
12.  
13.  // maybe better / different example https://github.com/fpc/FPCSource/blob/fd677978e7661b22a15fb3a9bce3f5c4b2ae8134/utils/fpdoc/unitdiff.pp#L61
14.  
15.         { TForm1 }
16.  
17.  TForm1 = class(TForm)
18.                 BitBtn1: TBitBtn;
19.                 Memo1: TMemo;
20.                 OpenDialog1: TOpenDialog;
21.                 procedure BitBtn1Click(Sender: TObject);
22.  
23.         private
24.  
25.         public
26.  
27.         end;
28.  
29.  TSimpleEngine = class(TPasTreeContainer)
30.  public
31.    function CreateElement(AClass: TPTreeElement; const AName: String;
32.      AParent: TPasElement; AVisibility: TPasMemberVisibility;
33.      const ASourceFilename: String; ASourceLinenumber: Integer): TPasElement;
34.      override;
35.    function FindElement(const AName: String): TPasElement; override;
36.  end;
37.  
38.  
39.  
40. var
41.         Form1: TForm1;
42.  
43.     procedure ParseFile(const FileName: string);
44.  
45. implementation
46.  
47. function TSimpleEngine.CreateElement(AClass: TPTreeElement; const AName: String;
48.  AParent: TPasElement; AVisibility: TPasMemberVisibility;
49.  const ASourceFilename: String; ASourceLinenumber: Integer): TPasElement;
50. begin
51.  Result := AClass.Create(AName, AParent);
52.  Result.Visibility := AVisibility;
53.  Result.SourceFilename := ASourceFilename;
54.  Result.SourceLinenumber := ASourceLinenumber;
55. end;
56.  
57. function TSimpleEngine.FindElement(const AName: String): TPasElement;
58. begin
59.  { dummy implementation, see TFPDocEngine.FindElement for a real example }
60.  Result := nil;
61. end;
62.  
63.  
64. procedure ParseFile(const FileName: string);
65. var
66.   I: Integer;
67.  
68.    Var
69.      R : TStreamResolver;
70.      S : TPascalScanner;
71.      P : TPasParser;
72.      M : TPasModule;
73.      C : TPasTreeContainer;
74.          ms: TMemoryStream;
75.  
76. begin
77.  
78.    R:= TStreamResolver.Create;
79.    try
80.  
81.      ms:= Tmemorystream.create();
82.      ms.LoadFromFile(filename);
83.  
84.      r.BaseDirectory:= ExtractFilePath(filename);
85.      r.AddIncludePath( ExtractFilePath(filename) );
86.  
87.      R.AddStream(ExtractFileName(filename), ms);
88.  
89.      S:=TPascalScanner.Create(R);
90.  
91.  
92.      try
93.        S.OpenFile(ExtractFileName(filename));
94.  
95.        C:= TPasTreeContainer.Create();
96.        try
97.          C.InterfaceOnly:=True;
98.  
99.          P:=TPasParser.Create(S, R, C);
100.          try
101.            P.ParseUnit(M);
102.          finally
103.            P.Free;
104.          end;
105.        finally
106.          C.Free;
107.        end;
108.      finally
109.        S.Free;
110.      end;
111.    finally
112.      R.Free;
113.    end;
114.  
115.  
116. end;
117.  
118. {$R *.lfm}
119.  
120. { TForm1 }
121.  
122. procedure TForm1.BitBtn1Click(Sender: TObject);
123. begin
124.  
125.   if opendialog1.execute = false then exit;
126.  
127.   ParseFile(opendialog1.FileName);
128.  
129. end;
130.  
131. end.
132.  
133.  
134.  

[dsiders]

I have tried to use unit parser included in fpc. followed instructions here:
 
https://wiki.freepascal.org/fcl-passrc

I used TSimpleEngine from there, and some other sources to see how resolver,scanner and parser work together.

However, if a class file uses {$include} directive, it cannot find includes:

Code: Pascal  [Select][+][-]

1. game_Actors.pas(10,24) Error: Could not find include file 'compiler.inc'

I cannot figure out why as i did set BaseDirectory and add AddIncludePath to TStreamResolver where the include file is. Does anybody know how to set this up properly? Since fpdoc uses it, i assume there is a way to make this work.
 
for example, just drop a memo, button and opendialog onto form:

Code: Pascal  [Select][+][-]

1.  
2. unit Unit1;
3.  
4. {$mode objfpc}{$H+}
5.  
6. interface
7.  
8. uses
9.         Classes, SysUtils, Forms, Controls, Graphics, Dialogs, Buttons, StdCtrls, pparser, pastree, pscanner, strutils;
10.  
11. type
12.  
13.  // maybe better / different example https://github.com/fpc/FPCSource/blob/fd677978e7661b22a15fb3a9bce3f5c4b2ae8134/utils/fpdoc/unitdiff.pp#L61
14.  
15.         { TForm1 }
16.  
17.  TForm1 = class(TForm)
18.                 BitBtn1: TBitBtn;
19.                 Memo1: TMemo;
20.                 OpenDialog1: TOpenDialog;
21.                 procedure BitBtn1Click(Sender: TObject);
22.  
23.         private
24.  
25.         public
26.  
27.         end;
28.  
29.  TSimpleEngine = class(TPasTreeContainer)
30.  public
31.    function CreateElement(AClass: TPTreeElement; const AName: String;
32.      AParent: TPasElement; AVisibility: TPasMemberVisibility;
33.      const ASourceFilename: String; ASourceLinenumber: Integer): TPasElement;
34.      override;
35.    function FindElement(const AName: String): TPasElement; override;
36.  end;
37.  
38.  
39.  
40. var
41.         Form1: TForm1;
42.  
43.     procedure ParseFile(const FileName: string);
44.  
45. implementation
46.  
47. function TSimpleEngine.CreateElement(AClass: TPTreeElement; const AName: String;
48.  AParent: TPasElement; AVisibility: TPasMemberVisibility;
49.  const ASourceFilename: String; ASourceLinenumber: Integer): TPasElement;
50. begin
51.  Result := AClass.Create(AName, AParent);
52.  Result.Visibility := AVisibility;
53.  Result.SourceFilename := ASourceFilename;
54.  Result.SourceLinenumber := ASourceLinenumber;
55. end;
56.  
57. function TSimpleEngine.FindElement(const AName: String): TPasElement;
58. begin
59.  { dummy implementation, see TFPDocEngine.FindElement for a real example }
60.  Result := nil;
61. end;
62.  
63.  
64. procedure ParseFile(const FileName: string);
65. var
66.   I: Integer;
67.  
68.    Var
69.      R : TStreamResolver;
70.      S : TPascalScanner;
71.      P : TPasParser;
72.      M : TPasModule;
73.      C : TPasTreeContainer;
74.          ms: TMemoryStream;
75.  
76. begin
77.  
78.    R:= TStreamResolver.Create;
79.    try
80.  
81.      ms:= Tmemorystream.create();
82.      ms.LoadFromFile(filename);
83.  
84.      r.BaseDirectory:= ExtractFilePath(filename);
85.      r.AddIncludePath( ExtractFilePath(filename) );
86.  
87.      R.AddStream(ExtractFileName(filename), ms);
88.  
89.      S:=TPascalScanner.Create(R);
90.  
91.  
92.      try
93.        S.OpenFile(ExtractFileName(filename));
94.  
95.        C:= TPasTreeContainer.Create();
96.        try
97.          C.InterfaceOnly:=True;
98.  
99.          P:=TPasParser.Create(S, R, C);
100.          try
101.            P.ParseUnit(M);
102.          finally
103.            P.Free;
104.          end;
105.        finally
106.          C.Free;
107.        end;
108.      finally
109.        S.Free;
110.      end;
111.    finally
112.      R.Free;
113.    end;
114.  
115.  
116. end;
117.  
118. {$R *.lfm}
119.  
120. { TForm1 }
121.  
122. procedure TForm1.BitBtn1Click(Sender: TObject);
123. begin
124.  
125.   if opendialog1.execute = false then exit;
126.  
127.   ParseFile(opendialog1.FileName);
128.  
129. end;
130.  
131. end.
132.  
133.  
134.  

I'm no expert, but...

 I believe the include path needs to be relative to the directory where the source file is located. And, if it's the same directory the include path needs to be '.'.

[JernejL]

Seems like i need to set both relative and absolute to make any progress, which is odd.

Code: Pascal  [Select][+][-]

1.      r.BaseDirectory:= ExtractFilePath(filename);
2.      r.AddIncludePath( '.' );
3.      r.AddIncludePath( ExtractFilePath(filename) );
4.  

There is no more complaint about include missing, but it does die somewhere else:
 

Code: Pascal  [Select][+][-]

1. Project rttiprocessor raised exception class 'EStringListError' with message:
2. List index (-1) out of bounds

I guess i will need to compile RTL with debug symbols to find out where this problem is.
 
stack trace is only partially helpful, no proper line numbers:
 

Code: Pascal  [Select][+][-]

1. #0 fpc_raiseexception at :0
2. #1 CLASSES$_$TSTRINGS_$__$$_ERROR$ANSISTRING$LONGINT at :0
3. #2 CLASSES$_$TSTRINGLIST_$__$$_CHECKINDEX$LONGINT at :0
4. #3 CLASSES$_$TSTRINGLIST_$__$$_GET$LONGINT$$ANSISTRING at :0
5.  
6. #4 PSCANNER$_$TSTREAMRESOLVER_$__$$_FINDSTREAM$ANSISTRING$BOOLEAN$$TSTREAM at :0
7. #5 PSCANNER$_$TSTREAMRESOLVER_$__$$_FINDSTREAMREADER$ANSISTRING$BOOLEAN$$TLINEREADER at :0
8. #6 PSCANNER$_$TSTREAMRESOLVER_$__$$_FINDINCLUDEFILE$ANSISTRING$$TLINEREADER at :0
9. #7 PSCANNER$_$TPASCALSCANNER_$__$$_HANDLEINCLUDEFILE$ANSISTRING at :0
10. #8 PSCANNER$_$TPASCALSCANNER_$__$$_HANDLEINCLUDE$ANSISTRING$$TTOKEN at :0
11. #9 PSCANNER$_$TPASCALSCANNER_$__$$_HANDLEDIRECTIVE$ANSISTRING$$TTOKEN at :0
12. #10 PSCANNER$_$TPASCALSCANNER_$__$$_DOFETCHTOKEN$$TTOKEN at :0
13. #11 PSCANNER$_$TPASCALSCANNER_$__$$_FETCHTOKEN$$TTOKEN at :0
14.  

[JernejL]

Apparently the parser itself is rather poorly implemented and does not support initialize operator within records.
 

Unknown operator type: Initialize at token "Identifier Initialize" in file game_Actors.pas at line 175 column 24 line:175 column:24 file:game_Actors.pas
Exception at 00455023: EPasWriter:

 
I will report this as a bug but i guess i'll need to go with some other solution in the meantime at least.
 

[JernejL]

apparently i was an idiot with that.
 
Still, in the end i found out that the parser has no support for -Sc / {$COPERATORS ON}

  //EparserError on C++ style
  //X+=Y;      { Same as X := X+Y, needs -Sc command line switch}
  //x-=y;
  //X/=2;      { Same as X := X/2, needs -Sc command line switch}
  //x*=y;

[Thaddy]

Good, I like that, because {$COPERATORS ON} is silly.
A candidate for removal, were it not that it breaks a minority of code, because some people actually use it.

[JernejL]

Good, I like that, because {$COPERATORS ON} is silly.
A candidate for removal, were it not that it breaks a minority of code, because some people actually use it.

 
I figured out the latest version of pscanner from trunk works well and supports COPERATORS, plus i managed to get most of it working based on pasrewrite example.
 
I will now write my own paswrite-based module so i can generate what i needes (i need to generate rpc mock classes).
 
Regarding {$COPERATORS ON}:  I guess you never wrote a lot of math intensive code to need it (like for game).
Repeating tons of multi-property expressions is not very fun, i love this feature as it makes code much more readable.