Recent

Author Topic: Examples of third-party packages in the new Examples Window  (Read 2387 times)

wp

  • Hero Member
  • *****
  • Posts: 11853
Examples of third-party packages in the new Examples Window
« on: March 03, 2023, 12:37:42 am »
What do I have to do to add sample projects coming with third-party packages to the new Examples Window?

The wiki in https://wiki.lazarus.freepascal.org/Lazarus_Examples_Window says that I should write an .ex-meta file. I copied an existing one and modified the contents accordingly. But where should it be stored? I guess, in the folder containing the sample project.

The first line of the json should identify the subdirectory of the sample project. This cannot be an absolute path and thus must be relativ. But relative to what? To the package installation directory? Or to the directory of the lpk file?

Furthermore, the wiki says that the Examples Window application searches for ex-meta files in "the Lazarus Source Tree and the Primary Config Path". But on my system all the packages that I maintain are in other folders. Does this mean that the ex-meta files are not found then? Wouldn't it be better if the Examples Window application would scan the packagefiles.xml in the PCP which contains the path to the lpk files of all available user packages? (However, the packagefile.xml does not contain the root of the package installation - therefore, it will be difficult to find all the meta files since the lpk file can be located in a subdirectory of the installation).

dbannon

  • Hero Member
  • *****
  • Posts: 2786
    • tomboy-ng, a rewrite of the classic Tomboy
Re: Examples of third-party packages in the new Examples Window
« Reply #1 on: March 03, 2023, 05:56:24 am »
What do I have to do to add sample projects coming with third-party packages to the new Examples Window?
By third-party packages, you mean packages of your own, not distributed with Lazarus ?
That would be a bit out of scope for the Examples unit IMHO. But would be quite possible I think....

Furthermore, the wiki says that the Examples Window application searches for ex-meta files in "the Lazarus Source Tree and the Primary Config Path". But on my system all the packages that I maintain are in other folders. Does this mean that the ex-meta files are not found then?
Yep, thats what it means. The Examples window scans the Lazarus SRC tree and then it scans the config directory because thats where packages installed by OPM are found. It could scan absolutely anywhere you care to point it but at the time of its writing, there did not seem to be anywhere else it needed to scan.

Maybe it needs a config option to add a tree to also look into ?  When it was first made, you indicated you wanted to be able to move the Examples Working directory, that is where examples are copied to to allow people to build and play with them. Its a whole tab in Options with only one item so would look better with one extra option there. ;-)


Wouldn't it be better if the Examples Window application would scan the packagefiles.xml in the PCP which contains the path to the lpk files of all available user packages? (However, the packagefile.xml does not contain the root of the package installation - therefore, it will be difficult to find all the meta files since the lpk file can be located in a subdirectory of the installation).

I guess I am unsure of just what packages I'd expect to find listed in packagefile.xml, I might be a bit uncomfortable scanning outside the Lazarus space it self. But if a user adds a directory, then they are allowing that scanning and it sounds better. So, adding an option for a list of extra directories does sound like a better approach. An empty list "out of the box".

Davo

 
Lazarus 3, Linux (and reluctantly Win10/11, OSX Monterey)
My Project - https://github.com/tomboy-notes/tomboy-ng and my github - https://github.com/davidbannon

wp

  • Hero Member
  • *****
  • Posts: 11853
Re: Examples of third-party packages in the new Examples Window
« Reply #2 on: March 03, 2023, 10:43:03 am »
This way it should work:
  • The package parameters must be extended by a value for the Examples directory used by this particular package. It must contain the relative path to the Examples root directory as seen from the lpk file. It must be added as a node to the lpk xml structure, and as a page to the package editor (similar to "i18n")
  • The examples window application must scan the Lazarus source tree for sample projects, as before
  • But rather than blindly scanning the primary config path (OPM installations can be relocated to be at any place, not just in the PCP), it should load the file "packagefiles.xml" from the PCP. Here is an excerpt:
Code: XML  [Select][+][-]
  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <CONFIG>
  3.   <UserPkgLinks Version="3" Count="92">
  4.     <Item1>
  5.       <Name Value="Abbrevia"/>
  6.       <Version Major="5"/>
  7.       <Filename Value="D:\Prog_Lazarus\svn\abbrevia\packages\Lazarus\abbrevia.lpk"/>
  8.       <LastUsed Value="2023/02/26 17:25:08"/>
  9.     </Item1>
  10.     <Item2>
  11.       <Name Value="BGRABitmapPack"/>
  12.       <Version Major="11" Minor="3" Release="1"/>
  13.       <Filename Value="D:\Prog_Lazarus\packages\bgrabitmap-master\bgrabitmap\bgrabitmappack.lpk"/>
  14.       <LastUsed Value="2023/03/03 10:29:57"/>
  15.     </Item2>
  16.     <Item3>
  17.       <Name Value="callight_pkg"/>
  18.       <Version Minor="3" Release="11"/>
  19.       <Filename Value="D:\Prog_Lazarus\svn\lazarus-ccr\components\callite\callight_pkg.lpk"/>
  20.       <LastUsed Value="2023/03/03 10:29:57"/>
  21.     </Item3>
  • Iterate over the child nodes of <UserPkgLinks> which represent all installed non-default packages. Find the name of the lpk file from the node <FileName>.
  • Load the lpk file and find the new node with the Examples directory specification
  • Now that the directory is known in which the current package stores its example projects it can be scanned to find all the meta files.
  • This way all packages, not just those installed by OPM in its default directory, are detected by the Examples Window application.

[EDIT]
Here's a (tested) code snippet which collects the names of all installed third-party packages in a TStrings instance:
Code: Pascal  [Select][+][-]
  1. uses
  2.   laz2_xmlread, laz2_dom;
  3.  
  4. // PkgFilesXML is the full path of the file "packagefiles.xml" which resides in the Lazarus primary config path.
  5. procedure TForm1.CollectThirdPartyPackages(PkgFilesXML: String; AList: TStrings);
  6. var
  7.   doc: TXMLDocument;
  8.   userPkgLinks: TDOMNode;
  9.   pkgNode: TDOMNode;
  10.   filenameNode: TDOMNode;
  11.   filenameAttr: TDOMNode;
  12. begin
  13.   if not FileExists(PkgFilesXML) then
  14.     exit;
  15.  
  16.   ReadXMLFile(doc, PkgFilesXML);
  17.   try
  18.     userPkgLinks := doc.DocumentElement.FindNode('UserPkgLinks');
  19.     if userPkgLinks = nil then
  20.       exit;
  21.     pkgNode := userPkgLinks.FirstChild;
  22.     while pkgNode <> nil do
  23.     begin
  24.       filenameNode := pkgNode.FindNode('Filename');
  25.       if filenameNode <> nil then
  26.       begin
  27.         filenameAttr := filenameNode.Attributes.GetNamedItem('Value');
  28.         if filenameAttr <> nil then
  29.           AList.Add(filenameAttr.Nodevalue);
  30.       end;
  31.       pkgNode := pkgNode.NextSibling;
  32.     end;
  33.   finally
  34.     doc.Free;
  35.   end;
  36. end;
« Last Edit: March 03, 2023, 11:10:14 am by wp »

dbannon

  • Hero Member
  • *****
  • Posts: 2786
    • tomboy-ng, a rewrite of the classic Tomboy
Re: Examples of third-party packages in the new Examples Window
« Reply #3 on: March 03, 2023, 10:56:45 pm »
OK Werner, that makes good sense. Are you going to go ahead with it ?

Davo
Lazarus 3, Linux (and reluctantly Win10/11, OSX Monterey)
My Project - https://github.com/tomboy-notes/tomboy-ng and my github - https://github.com/davidbannon

wp

  • Hero Member
  • *****
  • Posts: 11853
Re: Examples of third-party packages in the new Examples Window
« Reply #4 on: March 03, 2023, 11:06:27 pm »
Are you going to go ahead with it ?
Hmmm... Well, I wanted to avoid diving into the depths of your application. I can help to integrate the new version into the IDE (i.e. provide a page "Example projects" in the package options and to write the examples root path into the lpk file, but this should not be done before the new code has been tested with at least one external package in which the examples root path has been added to the lpk manually.

dbannon

  • Hero Member
  • *****
  • Posts: 2786
    • tomboy-ng, a rewrite of the classic Tomboy
Re: Examples of third-party packages in the new Examples Window
« Reply #5 on: March 04, 2023, 06:16:46 am »
Yep, thats OK, it just seemed to me you were itching to start.  I can do it but not get a lot done immediately due to some other commitments right now, couple of weeks, easy !
Overall, I don't think it should be too hard .....

Thanks for highlighting the issue.

Davo
Lazarus 3, Linux (and reluctantly Win10/11, OSX Monterey)
My Project - https://github.com/tomboy-notes/tomboy-ng and my github - https://github.com/davidbannon

dbannon

  • Hero Member
  • *****
  • Posts: 2786
    • tomboy-ng, a rewrite of the classic Tomboy
Re: Examples of third-party packages in the new Examples Window
« Reply #6 on: March 27, 2023, 06:17:41 am »
OK WP, finally found the time to play with this.
As you note in your initial post, there is a problem in the fact that packagefile.xml records, in the case of a manually added package, the location of the actual lpk file used to install. And its position, wrt any examples is indeterminate. I have it working fine finding OPM packages,  packagefile records the package top level directory then. Easy.

But to get the manually installed packages working is a lot harder. You seem to want to make an addition to the LPK format and add a section in the Lazarus->Project->Options->Miscellaneous (?) for the user to enter where they will store Examples. I suggest a lot milder change may be 'almost' as good ?

I know you (wp) manage LazMapViewer, in its  lazmapviewerpkg.lpk file it has -
Code: [Select]
<CONFIG>
  <Package Version="5">
   ....
    <CompilerOptions>
      <SearchPaths>
        <OtherUnitFiles Value="source"/>

I use KControls, it has -
Code: [Select]
<CONFIG>
  <Package Version="4">
    ....
    <CompilerOptions>
      <SearchPaths>
        <OtherUnitFiles Value="..\..\source"/>

That is becoming a widely used syntax, one or two directories down from package root, there is a 'source' directory. In most cases, the sensible place to put some Examples is parallel to 'source'. (If its not called 'source', we could whatever is there, less happy with that.)

So, this would need no change to LPK format, no change to Project Options screen, will work for most, maybe almost all projects. Quick and easy. What do you think ?

Davo
Lazarus 3, Linux (and reluctantly Win10/11, OSX Monterey)
My Project - https://github.com/tomboy-notes/tomboy-ng and my github - https://github.com/davidbannon

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4459
  • I like bugs.
Re: Examples of third-party packages in the new Examples Window
« Reply #7 on: March 27, 2023, 09:49:17 am »
Code: [Select]
        <OtherUnitFiles Value="..\..\source"/>
That is becoming a widely used syntax, one or two directories down from package root, there is a 'source' directory.
Actually that is two directories up from package root. It looks unusual to me.

I also was thinking that scanning example project metafiles in all package directories listed in UserPkgLinks should be enough, without any extension to .lpk file format.
However it affects performance. Many packages have no examples included. Their examples may be a different download. Thus an extension to .lpk actually sounds good.

Yes, the options should have a list of user defined extra directories to scan.
Mostly Lazarus trunk and FPC 3.2 on Manjaro Linux 64-bit.

dbannon

  • Hero Member
  • *****
  • Posts: 2786
    • tomboy-ng, a rewrite of the classic Tomboy
Re: Examples of third-party packages in the new Examples Window
« Reply #8 on: March 28, 2023, 01:11:00 am »
Actually that is two directories up from package root. It looks unusual to me.
Yes, unusual but not "wrong" in any way. My (current development) code (using "OtherUnitFiles" as a hint where to look) allows any number (0..n) of directories between 'source' and the LPK file mentioned in packagefiles.xml.
Quote from: JuhaManninen
I also was thinking that scanning example project metafiles in all package directories listed in UserPkgLinks should be enough, without any extension to .lpk file format.
However it affects performance. Many packages have no examples included. Their examples may be a different download. Thus an extension to .lpk actually sounds good.
I only scan Installed packages, I'd think most users only have a few such installed packages (OPM and manual). And we only scan when using the Examples window. Do you think thats an important performance hit ?
Quote from: JuhaManninen
Yes, the options should have a list of user defined extra directories to scan.
Thats a substantial change, as well as the lpk format I'll have to add a field in the IDE's Package Options screen for the user to enter either the fact that the project does have examples or to specify where they are. And it won't work for older packages, any package with examples will need to be repackaged with a new version of Lazarus with the extra capabilities.

Given that many third party packages out there have not been touched in several years, just adding a meta data file is easy, repackaging may not be.

Another problem I note, if you rebuild Lazarus (eg after a git pull) and don't flush the PCP, any previously installed packages are still listed in packagefiles.xml but are obviously not installed in Lazarus. I don't see an easy solution to that.

Davo

edit: I said Project Options, I meant Package Options ....
« Last Edit: March 28, 2023, 07:09:12 am by dbannon »
Lazarus 3, Linux (and reluctantly Win10/11, OSX Monterey)
My Project - https://github.com/tomboy-notes/tomboy-ng and my github - https://github.com/davidbannon

dbannon

  • Hero Member
  • *****
  • Posts: 2786
    • tomboy-ng, a rewrite of the classic Tomboy
Re: Examples of third-party packages in the new Examples Window
« Reply #9 on: March 28, 2023, 06:09:07 am »

Assuming you do want the "extra field in LPK file model", I propose it will be called "ExamplesDirectory" and live in the LPK file directly under "Package". It will contain a relative path to the package file where we should start searching. In practice, the IDE will provide just a tick box, if ticked, the field will be added and it will always point to top level directory of package tree.  Obviously, that does not mean the example directories need start at the top level, its just that we will start searching there and recurse down. Again thats to minimize changes to a package's layout if possible.

eg

Code: XML  [Select][+][-]
  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <CONFIG>
  3.   <Package Version="5">
  4.     <Name Value="NewPackage"/>
  5.     <Type Value="RunAndDesignTime"/>
  6.     <ExamplesDirectory Value="../"/>

And given a directory structure like this -
Code: Pascal  [Select][+][-]
  1. PackageRoot
  2.      source
  3.      package   // contains the package's LPK file.
  4.      OneExample
  5.      Examples
  6.           Demo1
  7.           Demo1
We would be looking for Example Metadata files in 'source", "OneExample", "Demo1" and "Demo2". Hopefully, find none in source ....
The most common content for ExamplesDirectory will be blank (?) or './.' from what I have seen.

I have not yet looked at how the package making unit will work out how many directories down the relevant lpk file is.


Davo
Lazarus 3, Linux (and reluctantly Win10/11, OSX Monterey)
My Project - https://github.com/tomboy-notes/tomboy-ng and my github - https://github.com/davidbannon

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4459
  • I like bugs.
Re: Examples of third-party packages in the new Examples Window
« Reply #10 on: March 28, 2023, 06:42:04 pm »
I only scan Installed packages, I'd think most users only have a few such installed packages (OPM and manual). And we only scan when using the Examples window. Do you think thats an important performance hit ?
It scans the actual package directories mentioned in packagefiles.xml? That is good, performance should be OK.

Quote
Quote from: JuhaManninen
Yes, the options should have a list of user defined extra directories to scan.
Thats a substantial change, as well as the lpk format I'll have to add a field in the IDE's Package Options screen for the user to enter either the fact that the project does have examples or to specify where they are. And it won't work for older packages, any package with examples will need to be repackaged with a new version of Lazarus with the extra capabilities.
My idea was not related to packages at all. A user may have a collection of example projects in his own computer and wants to select them through the Examples Window.

Quote
Another problem I note, if you rebuild Lazarus (eg after a git pull) and don't flush the PCP, any previously installed packages are still listed in packagefiles.xml but are obviously not installed in Lazarus. I don't see an easy solution to that.
No solution is needed, that is by design. The list of installed packages will be applied when the IDE is rebuilt. Otherwise a user would have to select the set of packages again every time after running "make clean all" or "make clean bigide".
Mostly Lazarus trunk and FPC 3.2 on Manjaro Linux 64-bit.

dbannon

  • Hero Member
  • *****
  • Posts: 2786
    • tomboy-ng, a rewrite of the classic Tomboy
Re: Examples of third-party packages in the new Examples Window
« Reply #11 on: March 29, 2023, 12:17:57 pm »
....
Quote
Quote from: JuhaManninen
Yes, the options should have a list of user defined extra directories to scan.
OK, sorry, I misunderstood that statement. Thats what I first thought WP was requesting. Not hard but I am not convinced thats needed, examples need to be in context, install a package and its examples come along with it. Without the package, the examples are useless. WP's suggestion that we can find ALL the extra packages, not just OPM installed in default place ones, is, IMHO a better approach.

But there are two ways to get to the Examples in installed packages -
  • Have an additional field in the LPK file that tells us how far down the LPK file is in the package directory tree. That involves changing the LPK format (a bit) and only packages with a new LPK file will work with the Example Window. But is a better, more reliable job by far.
  • Make a 'guess' based on the <OtherUnitFiles Value="source"/> xml element as to where the effective top of the package tree is.  I think it works for 'most' packages and it might be easier to change the few its not valid for than all the packages currently out there. (Still need a valid meta file of course).

I think I will do a wider survey of packages and see just how safe an assumption the second one is. I have a working version of it now, changes to just one file. Easy to test if any one want to.

Quote
Quote
Another problem I note, if you rebuild Lazarus (eg after a git pull) and don't flush the PCP, any previously installed packages are still listed in packagefiles.xml but are obviously not installed in Lazarus. I don't see an easy solution to that.
No solution is needed, that is by design. The list of installed packages will be applied when the IDE is rebuilt. Otherwise a user would have to select the set of packages again every time after running "make clean all" or "make clean bigide".
Interesting. I don't think that works to be honest. I always have to reinstall install eg KControls. Maybe it works for OPM packages ? Further research is indicated ....

Davo

EDIT : Yep, doing   make clean; make bigide<enter>   does not restore previously installed packages. But doing a rebuild of the IDE from within the IDE, eg installing another package, does restore all the packages. I had not realised that.
« Last Edit: March 29, 2023, 12:43:41 pm by dbannon »
Lazarus 3, Linux (and reluctantly Win10/11, OSX Monterey)
My Project - https://github.com/tomboy-notes/tomboy-ng and my github - https://github.com/davidbannon

wp

  • Hero Member
  • *****
  • Posts: 11853
Re: Examples of third-party packages in the new Examples Window
« Reply #12 on: March 29, 2023, 01:49:59 pm »
Make a 'guess' based on the <OtherUnitFiles Value="source"/> xml element as to where the effective top of the package tree is.  I think it works for 'most' packages and it might be easier to change the few its not valid for than all the packages currently out there. (Still need a valid meta file of course).
I agree that this will work for most packages. It will not work for fpspreadsheet, however, which has this structure:

Code: [Select]
  package root
    lpk file - "OtherUnitFiles" points to "source/common"
    [source]
      [common]
    [examples]

Since OtherUnitFiles points to "source/common" and "source" is at the same level as "examples", the examples folder will not be found starting at the OtherUnitFiles. Adding the location of the lpk file to the search, however, would be successful (assuming that the search reaches recursively into deeper folders). However, then the source/common folder will be searched twice...

Having a separate entry in the package Paths certainly will be safer and faster.



JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4459
  • I like bugs.
Re: Examples of third-party packages in the new Examples Window
« Reply #13 on: March 29, 2023, 08:42:00 pm »
EDIT : Yep, doing   make clean; make bigide<enter>   does not restore previously installed packages. But doing a rebuild of the IDE from within the IDE, eg installing another package, does restore all the packages. I had not realised that.
"bigide" has a set of packages used for Lazarus release.
From "make help" output :
Code: [Select]
Main targets
                  Without any target, target 'all' will be invoked.
   all            build all needed, i.e. minimal IDE, lazbuild, startlazarus.
   clean          deletes files that 'bigide' creates. It does not clean up all possible
                  targets. Clean other target: make clean LCL_PLATFORM=qt
   distclean      Clean all targets and common leftovers.
   lazbuild       build lazbuild and lcl with nogui widgetset
   bigide         as all, except that the IDE is built with a lot of extra packages
   useride        calls lazbuild to build an IDE with your active profile, requires lazbuild
   install        installs Lazarus under /usr/local/share/lazarus
                  You can change the directory by appending INSTALL_PREFIX=/some/path
Usually there is no need to run "make". Just rebuild Lazarus from its Tools menu, "Build Lazarus with Profile: ...". It does the same as "useride" make target but compiles dependent packages in parallel.
Only when you "git bisect" an old bug or otherwise want to build a revision from history, you may need "make". Until recently the changes in Lazarus base packages required it but then Mattias improved the logic. Now the GUI offers you a choice to build clean using LazBuild when it notices a change in the set of base packages.
After normal pulling of latest changes from GitLab, rebuilding "clean" is needed very seldom. The compiler is able to compile changed units and the units that depend on them. Always rebuilding everything gives no benefit. It only wastes time.

BTW, I have added the "Build Lazarus with Profile: ..." command to the editor toolbar thus making the rebuild even easier. I have unchecked the "Restart after building IDE" because I want to see compiler messages first.
« Last Edit: March 29, 2023, 08:56:45 pm by JuhaManninen »
Mostly Lazarus trunk and FPC 3.2 on Manjaro Linux 64-bit.

dbannon

  • Hero Member
  • *****
  • Posts: 2786
    • tomboy-ng, a rewrite of the classic Tomboy
Re: Examples of third-party packages in the new Examples Window
« Reply #14 on: March 30, 2023, 03:27:21 am »
I agree that this will work for most packages. It will not work for fpspreadsheet, however, which has this structure:

Code: [Select]
  package root
    lpk file - "OtherUnitFiles" points to "source/common"
    [source]
      [common]
    [examples]

Since OtherUnitFiles points to "source/common" and "source" is at the same level as "examples", the examples folder will not be found starting at the OtherUnitFiles.
No, my code finds that fine. I just tested it.

I follow the value in OtherUnitFiles up, but not down, I rely on recursion down through the whole tree once I find its top. While inefficient, its only the installed packages.

Code looks like this, ADir has the value of OtherUnitFiles, FullPkgFileName starts with path and name of package file, ends up being the top level directory.

Code: Pascal  [Select][+][-]
  1.         while ADir.StartsWith('..') do begin
  2.             ADir := ADir.Remove(0, 3);
  3.             FullPkgFileName := ExtractFileDir(FullPkgFileName);
  4.         end;
Quote from: wp
Having a separate entry in the package Paths certainly will be safer and faster.

Agreed, but maybe not by a significant margin ?  The 'guess' approach will be far easier to get supported in all the external packages. I'll make a metafile for each and send them to authors as I come across them.

Davo

@JuhaManninen - thats cool. I should be user useride rather than bigide ! Thanks.
Lazarus 3, Linux (and reluctantly Win10/11, OSX Monterey)
My Project - https://github.com/tomboy-notes/tomboy-ng and my github - https://github.com/davidbannon

 

TinyPortal © 2005-2018