Recent

Author Topic: How to export Project data to UML?  (Read 2287 times)

garlar27

  • Hero Member
  • *****
  • Posts: 618
How to export Project data to UML?
« on: January 21, 2019, 07:09:50 pm »
Hi!!

I want to make a plugin to import (reverse engineering) FPC/Lazarus project to StarUML 3 (http://staruml.io/). I've been reading their tutorials (in case someone wish to look at them https://docs.staruml.io/developing-extensions/getting-started).

I don't want to write a tokenizer/parser in Java Script when everything is already made in pascal.

My plan is to generate a JSON file with the info needed to at least create a Class Diagram (in a first stage) or at least import the classes and types without a diagram.


So the question are:
   o- Where do I start reading?
   o- Someone has tried to do something like that?


By the way. I've read the thread Wiki stub: Make your own compiler, interpreter, parser, or expression analyzer and Make your own compiler, interpreter, parser, or expression analyzer and it is quite interesting and I can use that info.-

ETA:
   I want to build a Package to export project types, classes, etc. to a JSON file (JSON definition will be decided afterwards)
« Last Edit: January 21, 2019, 10:55:36 pm by garlar27 »

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 7493
Re: How to export roject data to UML?
« Reply #1 on: January 21, 2019, 08:43:57 pm »
Post on the staruml forum? They are probably more knowledgable about how to instrument their parsers than we are?!?

garlar27

  • Hero Member
  • *****
  • Posts: 618
Re: How to export roject data to UML?
« Reply #2 on: January 21, 2019, 09:06:26 pm »
Post on the staruml forum? They are probably more knowledgable about how to instrument their parsers than we are?!?
It seems pretty easy to access diagrams and elements on them. But

I don't want to write a tokenizer/parser in Java Script when everything is already made in pascal.

I want to build a package for lazarus to export a JSON with info of all classes a project has

valdir.marcos

  • Hero Member
  • *****
  • Posts: 830
Re: How to export roject data to UML?
« Reply #3 on: January 21, 2019, 10:15:54 pm »
Hi!!

I want to make a plugin to import (reverse engineering) FPC/Lazarus project to StarUML 3 (http://staruml.io/). I've been reading their tutorials (in case someone wish to look at them https://docs.staruml.io/developing-extensions/getting-started).

I don't want to write a tokenizer/parser in Java Script when everything is already made in pascal.

My plan is to generate a JSON file with the info needed to at least create a Class Diagram (in a first stage) or at least import the classes and types without a diagram.


So the question are:
   o- Where do I start reading?
   o- Someone has tried to do something like that?


By the way. I've read the thread Wiki stub: Make your own compiler, interpreter, parser, or expression analyzer and Make your own compiler, interpreter, parser, or expression analyzer and it is quite interesting and I can use that info.-

ETA:
   I want to build a Package to export project types, classes, etc. to a JSON file (JSON definition will be decided afterwards)
What about studying?
C:\lazarus\components\ideintf\objectinspector.pp
C:\lazarus\components\ideintf\objectinspector.lfm

C:\lazarus\examples\objectinspector\oiexample.lpi

avra

  • Hero Member
  • *****
  • Posts: 1711
    • Additional info
Re: How to export Project data to UML?
« Reply #4 on: January 22, 2019, 12:38:21 am »
I want to make a plugin to import (reverse engineering) FPC/Lazarus project to StarUML 3
Are you aware that StarUML used to be an open source Delphi project? There is a fork of last Delphi version here:
https://sourceforge.net/projects/whitestaruml/

I don't want to write a tokenizer/parser in Java Script when everything is already made in pascal.
Well, maybe you might want to write in pascal but just output javascript? http://wiki.freepascal.org/pas2js

My plan is to generate a JSON file with the info needed to at least create a Class Diagram (in a first stage) or at least import the classes and types without a diagram
You might want to take a look at EssModel (https://sourceforge.net/projects/essmodelforlaza/), or use something like Gold (http://wiki.lazarus.freepascal.org/Gold). Gold and other parsers are fine when zooming into 99% of your code (grammars can not handle every possible case). When you need that 1% left then you need to go manual way.
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

garlar27

  • Hero Member
  • *****
  • Posts: 618
Re: How to export Project data to UML?
« Reply #5 on: January 22, 2019, 01:01:35 am »
AFAIK Object inspector can access published methods and properties and I need private protected and public too.
also I need the parameter list with name and type, if the method is abstract, virtual, overridden, overloaded. And collect as much data as possible.

Lets say I have this code:
Code: Pascal  [Select]
  1. type
  2. // C:\Lazarus\Lazarus.1.8.2-FPC.3.0.4-(32)\fpc\3.0.4\source\packages\fcl-json\src\fpjsonrtti.pp
  3.  
  4.   TJSONFiler = Class(TComponent)
  5.   Protected
  6.     Procedure Error(Const Msg : String);
  7.     Procedure Error(Const FMT : String;  Args : Array of const);
  8.   end;
  9.  
  10.   TJSONStreamer = Class(TJSONFiler)
  11.   private
  12.     FAfterStreamObject: TJSONStreamEvent;
  13.     FBeforeStreamObject: TJSONStreamEvent;
  14.     FChildProperty: String;
  15.     FDateTimeFormat: String;
  16.     FOnStreamProperty: TJSONPropertyEvent;
  17.     FOptions: TJSONStreamOptions;
  18.     function GetChildProperty: String;
  19.     function IsChildStored: boolean;
  20.     function StreamChildren(AComp: TComponent): TJSONArray;
  21.   protected
  22.     function StreamClassProperty(Const AObject: TObject): TJSONData; virtual;
  23.     Function StreamProperty(Const AObject : TObject; Const PropertyName : String) : TJSONData;
  24.     Function StreamProperty(Const AObject : TObject; PropertyInfo : PPropInfo) : TJSONData;
  25.     Function FormatDateProp(const DateTime : TDateTime) : TJSONString;
  26.   Public
  27.     Constructor Create(AOwner : TComponent); override;
  28.     Destructor Destroy;override;
  29.     //
  30.     // Basic functions
  31.     //
  32.     // Use RTTI to stream object.
  33.     // If AObject is of type TStrings or TCollection, special treatment occurs:
  34.     // TStrings results in { Strings: [S,S,S] } or { Strings: { "S1" : O1, "S2" : O2 }} depending on Options.
  35.     // Collection results in { Items: [I,I,I] }
  36.     Function ObjectToJSON(Const AObject : TObject) : TJSONObject;
  37.     // Stream a collection - always returns an array
  38.     function StreamCollection(Const ACollection: TCollection): TJSONArray;
  39.     // Stream an objectlist - always returns an array
  40.     function StreamObjectList(Const AnObjectList: TObjectList): TJSONArray;
  41.     // Stream a List - always returns an array
  42.     function StreamTList(Const AList: TList): TJSONArray;
  43.     // Stream a TStrings instance as an array
  44.     function StreamTStringsArray(Const AStrings: TStrings): TJSONArray;
  45.     // Stream a TStrings instance as an object
  46.     function StreamTStringsObject(Const AStrings: TStrings): TJSONObject;
  47.     // Stream a TStrings instance. Takes into account Options.
  48.     function StreamTStrings(Const AStrings: TStrings): TJSONData;
  49.     // Stream a variant as JSON.
  50.     function StreamVariant(const Data: Variant): TJSONData; virtual;
  51.     //
  52.     // Some utility functions.
  53.     //
  54.     // Call ObjectToJSON and convert result to JSON String.
  55.     Function ObjectToJSONString(AObject : TObject) : TJSONStringType;
  56.     // Convert TSTrings to JSON string with array or Object.
  57.     Function StringsToJSON(Const Strings : TStrings; AsObject : Boolean = False) : TJSONStringType;
  58.     // Convert collection to JSON string
  59.     Function CollectionToJSON(Const ACollection : TCollection) : TJSONStringType;
  60.     // Convert variant to JSON String
  61.     Function VariantToJSON(Const Data : Variant) : TJSONStringType;
  62.   Published
  63.     // Format used when formatting DateTime values. Only used in conjunction with jsoDateTimeToString
  64.     Property DateTimeFormat : String Read FDateTimeFormat Write FDateTimeFormat;
  65.     // Options to use when streaming
  66.     Property Options : TJSONStreamOptions Read FOptions Write FOptions;
  67.     // Called before streaming an object with ObjectToJSON
  68.     Property BeforeStreamObject : TJSONStreamEvent Read FBeforeStreamObject Write FBeforeStreamObject;
  69.     // Called After streaming an object with ObjectToJSON
  70.     Property AfterStreamObject : TJSONStreamEvent Read FAfterStreamObject Write FAfterStreamObject;
  71.     // Called whenever a property was streamed. If Res is nil on return, no property is added.
  72.     Property OnStreamProperty : TJSONPropertyEvent Read FOnStreamProperty Write FOnStreamProperty;
  73.     // Property name to use when streaming child components. Default is "Children"
  74.     Property ChildProperty : String Read GetChildProperty Write FChildProperty Stored IsChildStored;
  75.   end;
  76.  
  77.   TJSONDeStreamer = Class(TJSONFiler)
  78.   private
  79.     FAfterReadObject: TJSONStreamEvent;
  80.     FBeforeReadObject: TJSONStreamEvent;
  81.     FDateTimeFormat: String;
  82.     FOnGetObject: TJSONGetObjectEvent;
  83.     FOnPropError: TJSONpropertyErrorEvent;
  84.     FOnRestoreProp: TJSONRestorePropertyEvent;
  85.     FCaseInsensitive : Boolean;
  86.     FOptions: TJSONDestreamOptions;
  87.     procedure DeStreamClassProperty(AObject: TObject; PropInfo: PPropInfo; PropData: TJSONData);
  88.     function GetCaseInsensitive: Boolean;
  89.     procedure SetCaseInsensitive(AValue: Boolean);
  90.   protected
  91.     // Try to parse a date.
  92.     Function ExtractDateTime(S : String): TDateTime;
  93.     function GetObject(AInstance : TObject; const APropName: TJSONStringType; D: TJSONObject; PropInfo: PPropInfo): TObject;
  94.     procedure DoRestoreProperty(AObject: TObject; PropInfo: PPropInfo;  PropData: TJSONData); virtual;
  95.     Function ObjectFromString(Const JSON : TJSONStringType) : TJSONData; virtual;
  96.     procedure RestoreProperty(AObject: TObject; PropInfo: PPropInfo; PropData: TJSONData);
  97.   Public
  98.     Constructor Create(AOwner : TComponent); override;
  99.     Destructor Destroy; override;
  100.     // Convert JSON object to properties of AObject
  101.     Procedure JSONToObject(Const JSON : TJSONStringType; AObject : TObject);
  102.     Procedure JSONToObject(Const JSON : TJSONObject; AObject : TObject);
  103.     // Convert JSON object/array to collection.
  104.     Procedure JSONToCollection(Const JSON : TJSONStringType; ACollection : TCollection);
  105.     Procedure JSONToCollection(Const JSON : TJSONData; ACollection : TCollection);
  106.     // Convert JSON array/object/string to TStrings
  107.     Procedure JSONToStrings(Const JSON : TJSONStringType; AStrings : TSTrings);
  108.     Procedure JSONToStrings(Const JSON : TJSONData; AStrings : TSTrings);
  109.     // Convert JSON data to a variant. Supports simple data types and arrays.
  110.     Function JSONToVariant(Data: TJSONData): Variant;
  111.     Function JSONToVariant(Data: TJSONStringType): Variant;
  112.     // Triggered at the start of each call to JSONToObject
  113.     Property BeforeReadObject : TJSONStreamEvent Read FBeforeReadObject Write FBeforeReadObject;
  114.     // Triggered at the end of each call to JSONToObject (not if exception happens)
  115.     Property AfterReadObject : TJSONStreamEvent Read FAfterReadObject Write FAfterReadObject;
  116.     // Called when a property will be restored. If 'Handled' is True on return, property is considered restored.
  117.     Property OnRestoreProperty : TJSONRestorePropertyEvent Read FOnRestoreProp Write FOnRestoreProp;
  118.     // Called when an error occurs when restoring a property. If Continue is False on return, exception is re-raised.
  119.     Property OnPropertyError : TJSONpropertyErrorEvent Read FOnPropError Write FOnPropError;
  120.     // Called when a object-typed property must be restored, and the property is Nil. Must return an instance for the property.
  121.     // Published Properties of the instance will be further restored with available data.
  122.     Property OngetObject : TJSONGetObjectEvent Read FOnGetObject Write FOnGetObject;
  123.     // JSON is by definition case sensitive. Should properties be looked up case-insentive ?
  124.     Property CaseInsensitive : Boolean Read GetCaseInsensitive Write SetCaseInsensitive ; deprecated;
  125.     // DateTime format. If not set, RFC3339DateTimeFormat is assumed.
  126.     // If set, it will be used as an argument to ScanDateTime. If that fails, StrToDateTime is used.
  127.     Property DateTimeFormat : String Read FDateTimeFormat Write FDateTimeFormat;
  128.     // Options overning the behaviour
  129.     Property Options : TJSONDestreamOptions Read FOptions Write FOptions;
  130.   end;
  131.  

I would like to buid a JSON like this (more or less):
Code: Text  [Select]
  1. {
  2.    "element" :
  3.       [
  4.          {
  5.             "name" : "TJSONFiler", "type" : "Class", "inheritsFrom" : "TComponent", "visibility" : "public", "isAbstract" : "false", "documentation" : "",
  6.             "operations" :
  7.                [
  8.                   {"name" : "Error", "visibility" : "protected", "isAbstract" : "false", "isVirtual" : "false", "returns" : "", "documentation" : "", "parameters" :
  9.                      [
  10.                         {"name" : "Msg", "type" : "String"}
  11.                      ]
  12.                   },
  13.                   {"name" : "Error", "visibility" : "protected", "isAbstract" : "false", "isVirtual" : "false", "returns" : "", "documentation" : "", "parameters" :
  14.                      [
  15.                         {"name" : "FMT", "type" : "String"},
  16.                         {"name" : "Args", "type" : "Array of const"}
  17.                      ]
  18.                   }
  19.                ]
  20.          },
  21.          {
  22.             "name" : "TJSONStreamer", "type" : "Class", "inheritsFrom" : "TJSONFiler", "visibility" : "public", "isAbstract" : "false", "documentation" : "",
  23.             "attributes" :
  24.                [
  25.                   {"name" : "FAfterStreamObject", "type" : "TJSONStreamEvent", "visibility" : "private", "documentation" : ""},
  26.                   {"name" : "FBeforeStreamObject", "type" : "TJSONStreamEvent", "visibility" : "private", "documentation" : ""},
  27.                   {"name" : "FChildProperty", "type" : "String", "visibility" : "private", "documentation" : ""},
  28.                   {"name" : "FDateTimeFormat", "type" : "String", "visibility" : "private", "documentation" : ""},
  29.                   {"name" : "FOnStreamProperty", "type" : "TJSONPropertyEvent", "visibility" : "private", "documentation" : ""},
  30.                   {"name" : "FOptions", "type" : "TJSONStreamOptions", "visibility" : "private", "documentation" : ""}
  31.                ],
  32.             "operations" :
  33.                [
  34.                   {"name" : "GetChildProperty", "visibility" : "private", "returns" : "String", "documentation" : "", "parameters" : []},
  35.                   {"name" : "IsChildStored", "visibility" : "private", "returns" : "boolean", "documentation" : "", "parameters" : []},
  36.                   {"name" : "StreamChildren", "visibility" : "private", "returns" : "TJSONArray", "documentation" : "", "parameters" :
  37.                      [
  38.                         {"name" : "AComp", "type" : "TComponent"}
  39.                      ]
  40.                   },
  41.                   {"name" : "StreamClassProperty", "visibility" : "protected", "isAbstract" : "false", "isVirtual" : "true", "returns" : "TJSONData", "documentation" : "", "parameters" :
  42.                      [
  43.                         {"name" : "AObject", "type" : "TObject"}
  44.                      ]
  45.                   },
  46.                   {"name" : "StreamProperty", "visibility" : "protected", "isAbstract" : "false", "isVirtual" : "false", "returns" : "TJSONData", "documentation" : "", "parameters" :
  47.                      [
  48.                         {"name" : "AObject", "type" : "TObject"},
  49.                         {"name" : "PropertyName", "type" : "String"}
  50.                      ]
  51.                   },
  52.                   {"name" : "StreamProperty", "visibility" : "protected", "isAbstract" : "false", "isVirtual" : "false", "returns" : "TJSONData", "documentation" : "", "parameters" :
  53.                      [
  54.                         {"name" : "AObject", "type" : "TObject"},
  55.                         {"name" : "PropertyInfo", "type" : "PPropInfo"}
  56.                      ]
  57.                   },
  58.                   {"name" : "FormatDateProp", "visibility" : "protected", "isAbstract" : "false", "isVirtual" : "false", "returns" : "TJSONString", "documentation" : "", "parameters" :
  59.                      [
  60.                         {"name" : "DateTime", "type" : "TDateTime"}
  61.                      ]
  62.                   },
  63.                   {"name" : "ObjectToJSON", "visibility" : "public", "isAbstract" : "false", "isVirtual" : "false", "returns" : "TJSONObject", "documentation" : "Basic functions\n\nUse RTTI to stream object.\nIf AObject is of type TStrings or TCollection, special treatment occurs:\nTStrings results in { Strings: [S,S,S] } or { Strings: { "S1" : O1, "S2" : O2 }} depending on Options.\nCollection results in { Items: [I,I,I] }\n", "parameters" :
  64.                      [
  65.                         {"name" : "AObject", "type" : "TObject"}
  66.                      ]
  67.                   },
  68.                   {"name" : "", "visibility" : "public", "isAbstract" : "false", "isVirtual" : "false", "returns" : "", "documentation" : "", "parameters" :
  69.                      [
  70.                         {"name" : "", "type" : ""},
  71.                         {"name" : "", "type" : ""},
  72.                         {"name" : "", "type" : ""},
  73.                         {"name" : "", "type" : ""}
  74.                      ]
  75.                   },
  76.                   {"name" : "", "visibility" : "public", "isAbstract" : "false", "isVirtual" : "false", "returns" : "", "documentation" : "", "parameters" :
  77.                      [
  78.                         {"name" : "", "type" : ""},
  79.                         {"name" : "", "type" : ""},
  80.                         {"name" : "", "type" : ""},
  81.                         {"name" : "", "type" : ""}
  82.                      ]
  83.                   },
  84.                  
  85.                ]
  86.          }
  87.       ]
  88. }
  89.  

ETA:
I plan to feed the StarUML extension with a file like that.
I have a long way to go.....  %)
« Last Edit: January 22, 2019, 01:06:17 am by garlar27 »

garlar27

  • Hero Member
  • *****
  • Posts: 618
Re: How to export Project data to UML?
« Reply #6 on: January 22, 2019, 01:05:01 am »
I want to make a plugin to import (reverse engineering) FPC/Lazarus project to StarUML 3
Are you aware that StarUML used to be an open source Delphi project? There is a fork of last Delphi version here:
https://sourceforge.net/projects/whitestaruml/
Yeah, I knew it was made with Delphi but didn't know of the fork...

I don't want to write a tokenizer/parser in Java Script when everything is already made in pascal.
Well, maybe you might want to write in pascal but just output javascript? http://wiki.freepascal.org/pas2js
Not ATM

My plan is to generate a JSON file with the info needed to at least create a Class Diagram (in a first stage) or at least import the classes and types without a diagram
You might want to take a look at EssModel (https://sourceforge.net/projects/essmodelforlaza/), or use something like Gold (http://wiki.lazarus.freepascal.org/Gold). Gold and other parsers are fine when zooming into 99% of your code (grammars can not handle every possible case). When you need that 1% left then you need to go manual way.
Tomorrow will take a look at them!!!

af0815

  • Sr. Member
  • ****
  • Posts: 359
Re: How to export Project data to UML?
« Reply #7 on: January 22, 2019, 06:39:05 am »
Some time ago, i have looked into fpdoc, and it is also IMHO building a parsertree for making the docs, maybe one idea/input more.   
regards
Andreas