Lazarus

Using the Lazarus IDE => Designer => Topic started by: JuhaManninen on December 29, 2021, 01:55:57 pm

Title: Example projects window
Post by: JuhaManninen on December 29, 2021, 01:55:57 pm
This is moved from the "Lazarus Object Inspector is inactive" thread.

So, do we want a quick and easy fix or a full coverage -
* redesign the GUI
* provide a category system for all examples, maybe in the (compulsory) readme file ?
Yes, a readme file is already shown in a memo if it exists. A "Category: xxx" syntax can be added. Later somebody could add a proper readme file for all example projects.

Quote
* Copy the deb/rpm installed example to user space when 'chosen' by the user.
That sounds strange. Why should we use Linux distro package system for Lazarus examples?

Quote
Thats quite a significant change, I'm not keen to get stared unless there is some indication its the way we really want to go ??
The GUI can be improved at the same time while wp or others improve the actual example projects.
My idea of the necessary "Example Projects" window improvements:

I attached a mockup GUI following my idea. It is simple.
Title: Re: Example projects window
Post by: dseligo on December 29, 2021, 03:47:07 pm
I like Online Package Manager tool.
It would be nice if similar tool would exist for examples (maybe in Help menu of Lazarus).

It would be fast and easy way to get to the examples and easy to point someone to.

And examples could be 'live' - when someone make new example project  (or update it), it would be available in this tool.
Title: Re: Example projects window
Post by: JuhaManninen on December 29, 2021, 04:04:44 pm
I like Online Package Manager tool.
It would be nice if similar tool would exist for examples (maybe in Help menu of Lazarus).
It would be fast and easy way to get to the examples and easy to point someone to.
And examples could be 'live' - when someone make new example project  (or update it), it would be available in this tool.
Online example projects. Yes, it is a good idea. It could use the same framework with Online Package Manager. Maybe GetMem has an idea.
Sometime in future OPM should use the the fppkg system and allow authors to upload their packages (+ maybe example projects).

Online example projects is a separate and bigger project. The current window should be simplified and improved in any case.

A proper menu for example projects is tricky. Help menu is for actual help data so far. Project menu is now about the current project only.
Opinions are welcome.
Title: Re: Example projects window
Post by: af0815 on December 29, 2021, 05:06:33 pm
Maybe some ideas from the actual Example-Screen and OPM. And help is not Example, and can reference to the example system.

BTW, is the fppkg system for Lazarus too ? OPM is well known in Lazarus, but fppkg more a hidden secret system for IMHO the most user.
Title: Re: Example projects window
Post by: JuhaManninen on December 29, 2021, 05:21:48 pm
Maybe some ideas from the actual Example-Screen and OPM.
Do you mean ideas for an Online Example Project system?

Quote
And help is not Example, and can reference to the example system.
Do you mean the Help menu should have entries for example projects, too?

Quote
BTW, is the fppkg system for Lazarus too ? OPM is well known in Lazarus, but fppkg more a hidden secret system for IMHO the most user.
The design goal was that fppkg supports both FPC and Lazarus code. It does not work well yet. It only has a small set of FPC packages to test with. I don't know more about the technical challenges there.
Title: Re: Example projects window
Post by: af0815 on December 29, 2021, 09:30:26 pm
Maybe some ideas from the actual Example-Screen and OPM.
Do you mean ideas for an Online Example Project system?
Yes, you must be able to search by a keyword. In the ExampleProjects , this is often done by enter few letters and you hope to find a project with this word :-)

OPM is IMHO based on a json, with a lot more features and information. So a keyword based search is more possible. A program can use eventually the information from VersionsInfo to make it similar to a package in OPM.

Quote
And help is not Example, and can reference to the example system.
Do you mean the Help menu should have entries for example projects, too?
Yes or inserted links to the sample projects. Or the keywords can bypassed from lhelp to the sampleproject manager.

Quote
BTW, is the fppkg system for Lazarus too ? OPM is well known in Lazarus, but fppkg more a hidden secret system for IMHO the most user.
The design goal was that fppkg supports both FPC and Lazarus code. It does not work well yet. It only has a small set of FPC packages to test with. I don't know more about the technical challenges there.
But to change a well working system to a 'not well working' system, is not a target. And the actual OPM system is stable. So if you want to seperate the samples from Lazarus, you should use a stable base. Actual i think a fast way is to use the technique of OPM and start. On the other side you have to wait for a realy working fppkg system (for Lazarus too). I think fppkg will make a brake through with dynamics packages in the future.
 
Title: Re: Example projects window
Post by: Martin_fr on December 29, 2021, 10:30:58 pm
I like Online Package Manager tool.
It would be nice if similar tool would exist for examples (maybe in Help menu of Lazarus).

I don't think we need a new tool for that.

If someone want to provide examples, the OPM can be used as it.
- Just make an empty package
- create an example folder in the package
- put your examples into that folder
- get the package added to OPM

The only thing that needs to be done, is for the "example window" in the IDE, to search all the <packagedir>/example/ folders (optional).

If (and that is an "if and only if") this takes off (i.e. if contributor pop up, and create example packages) the OPM window can easily have a button added "open example manager" (and set a filter for the examples of the selected package). But that is an "if and only if". No point adding, unless there will be packages with examples.

Doing it via opm, and the "example window" searching packages, also offers the advance that existing packages (e.g. packages that add new components), can include examples for those components.




That said, it will still be a good idea to have some examples included in the IDE distro itself. So that still needs addressing.

And, no matter if packages come via OPM or are included, if they should be presented, that presentation may need to be polished. And that presentation is the "example window"

Title: Re: Example projects window
Post by: wp on December 29, 2021, 11:31:18 pm
Sorry, Martin. This is not a good idea: It will be an unacceptable burdon to GetMem who handles all the additions to OPM manually. And unless "examples" are separated from "components" in some kind of GUI, components will be buried underneath loads of examples.
Title: Re: Example projects window
Post by: dbannon on December 29, 2021, 11:57:44 pm
Quote
Quote
* Copy the deb/rpm installed example to user space when 'chosen' by the user.
That sounds strange. Why should we use Linux distro package system for Lazarus examples?

Because, from what I understand, one of the issues in the bug report is that end users who install Lazarus using deb/rpm/pakman etc end up with it all in Read Only space.  Its not a packaging thing, its a ReadOnly issue. So, if they try a build an example project, it cannot write. I, personally always build from source but lots and lots of people install distro packages.

This, of course a *nix problem and WP is a Windows user. Hmm....

I like the other suggestions made.

WP, the category model ?  I'd really like to see all the examples have a readme file (and its contents displays as per the existing GUI) and that seems like a great way to categorize them at the same time. And maybe some keywords ?  So, maybe the first two line of the readme has (eg) -

Category : beginner
Keywords : buttons, memo

Categories need to be agreed on, [ beginner, example, test ...]

Davo



Title: Re: Example projects window
Post by: trev on December 30, 2021, 12:11:04 am
My two cents worth:

1) Remove all the examples from the distributions
2) Add an Examples entry to the Lazarus IDE Help menu (separate FPC and Lazarus?)
3) Link to a to-be-created Wiki page which lists example category pages
4) Add examples to appropriate yet-to-be-created category pages
5) Example entry in Wiki links to SourceForge/Github/Gitlab for the source for each example

Benefits:

* Reduces distribution size (I didn't even know there were examples)
* Avoids read-only issues
* Adds missing "readme" info
* Allows ANYONE to add examples, info etc
* Removes any load on the dev team(s)
Title: Re: Example projects window
Post by: dbannon on December 30, 2021, 02:03:07 am
....
3) Link to a to-be-created Wiki page which lists example category pages

yes, interesting idea. I'd sure support it if we can be sure (pretty sure) that there is no security/privacy issues happening. Further on this topic below.

> Reduces distribution size (I didn't even know there were examples)
"mostly" in ./examples - 15M

> Avoids read-only issues
Yep

>Allows ANYONE to add examples, info etc
>Removes any load on the dev team(s)
There is a safety,  quality and relevance issue there I have to say. On the other hand, the quality of what we have now is not great due to old content (rather than poor content).  Personally, I am a big believer (and a contributor to) the wiki but this wiki, and all wikis do tend to drift into chaotic quite quickly.

I see a danger in "anyone" being able to load a project on the wiki, we tell the beginners "go look there" and they download a project that does something nasty. Sorry, I see that as quite a serious danger.

Davo



Title: Re: Example projects window
Post by: speter on December 30, 2021, 05:01:27 am
This sounds like a good idea.

The forum (presumably) has "classes" of users, does the wiki have similar? Maybe restrict edits to this section of the wiki to people who are forum "Global Moderators" / "Hero Members" (or whatever).

cheers
S.
Title: Re: Example projects window
Post by: trev on December 30, 2021, 05:20:06 am
I see a danger in "anyone" being able to load a project on the wiki, we tell the beginners "go look there" and they download a project that does something nasty. Sorry, I see that as quite a serious danger.

You could say the same about any of the existing code (https://wiki.lazarus.freepascal.org/Category:Code) on the Wiki and all the FPC + Lazarus projects (https://wiki.lazarus.freepascal.org/Projects_using_Free_Pascal) linked from the Wiki not to mention all the tutorials, components and libraries also linked from the Wiki.
Title: Re: Example projects window
Post by: dbannon on December 30, 2021, 05:40:29 am
You could say the same about any of the existing code (https://wiki.lazarus.freepascal.org/Category:Code) on the Wiki and all the FPC + Lazarus projects (https://wiki.lazarus.freepascal.org/Projects_using_Free_Pascal) linked from the Wiki not to mention all the tutorials, components and libraries also linked from the Wiki.

No, I disagree, code pasted in a wiki page or appearing in forum are discrete lumps of viewable code. By definition, its reasonably short and lots of people run their eyes over it. That should not be compared to a zip file that could contain many units, thousands of lines of code, frankly, its not going to be checked.

Code, elsewhere, linked from our wiki is not officially endorsed to the same extent. Indeed, I expect there is a disclaimer somewhere, no one reads them anyway....

You must remember that we are talking about, in many cases here, rank beginners. Ones who will follow instructions, line by line.

Having some sort of barrier is, IMHO, necessary and we would be considered as failing in our duty of care we did not have a proper process in place. I am definitely not against the idea, it just needs to be done appropriately.

I am assuming there is a way (and people are happy) to make the wiki allow uploads of zip files ?  Or do we need an alternative location. Hmm, gitlab ?

Davo 
Title: Re: Example projects window
Post by: trev on December 30, 2021, 05:49:40 am
You could say the same about any of the existing code (https://wiki.lazarus.freepascal.org/Category:Code) on the Wiki and all the FPC + Lazarus projects (https://wiki.lazarus.freepascal.org/Projects_using_Free_Pascal) linked from the Wiki not to mention all the tutorials, components and libraries also linked from the Wiki.

No, I disagree, code pasted in a wiki page or appearing in forum are discrete lumps of viewable code. By definition, its reasonably short and lots of people run their eyes over it. That should not be compared to a zip file that could contain many units, thousands of lines of code, frankly, its not going to be checked.

I never advocated a ZIP file - I advocated links to SourceForge et al code repos.

Quote
I am assuming there is a way (and people are happy) to make the wiki allow uploads of zip files ?  Or do we need an alternative location. Hmm, gitlab ?

Again, I never advocated either ZIP files or uploading them to the Wiki.

You seem to have setup a straw man just so you could demolish it :)
Title: Re: Example projects window
Post by: dbannon on December 30, 2021, 06:36:24 am
I never advocated a ZIP file - I advocated links to SourceForge et al code repos.
Example projects need to be bundled into something and zip is both the Lazarus preferred model and is reasonably cross platform. if not zip, what else would we use ? Contents could just sit as files in a directory on, eg Gitlab, one directory per project but maybe a little harder to manage.

Again, I never advocated either ZIP files or uploading them to the Wiki.
No, you did not. And I clearly misread that. But you equally clearly missed my words that "I am definitely not against the idea".

You seem to have setup a straw man just so you could demolish it :)
I think that is unfair and uncalled for. It really does not matter where we keep gz/xz/zip/dir contents, if it has access similar to the wiki, its not safe enough. If it has gatekeepers, that can work as long as we can manage the extra load. Its the usual security v. convenience argument, neither SourceForge nor Github/lab would allow unauthenticated uploads anyway.

This idea is far from a bad one, it just needs some discussion, please encourage it.

Davo

 
Title: Re: Example projects window
Post by: JuhaManninen on December 30, 2021, 12:40:17 pm
Because, from what I understand, one of the issues in the bug report is that end users who install Lazarus using deb/rpm/pakman etc end up with it all in Read Only space.  Its not a packaging thing, its a ReadOnly issue. So, if they try a build an example project, it cannot write. I, personally always build from source but lots and lots of people install distro packages.
A user must be able to compile the code while it is open in the IDE. He may want to modify and debug it, just to learn and experiment.
All those Linux packages would require lots of maintenance work. It is not realistic. They are not even very useful because the ReadOnly issue remains.
An online example project system indeed would solve the ReadOnly issue.

Quote
Category : beginner
Keywords : buttons, memo
Yes, both Category and Keywords sound good.

Returning to the original issue, improving the current Example Projects GUI...
It can and should be done anyway, regardless of any future online systems.
Is somebody willing to do it? What do you think of my mockup GUI?
If nobody else volunteers, I will look at it during this winter.
Title: Re: Example projects window
Post by: af0815 on December 30, 2021, 01:10:39 pm
A example system like OPM - i said like !! The goal are, it can be maintained and you can download only the examples you want. Not needed to load all examples at once. If you use gitlab, how can you saperate all the examples ? And you must make it possible to test the examples for the actual stable (and comming) version of Lazarus and FPC. Nothing more boring than a not working sample.

But this is definitly not a job for the actual OPM. It can use the techniques, but must have a team to handle this.

ANd what is about fppkg, as nmed in the first post - is it useable/working for Lazarus too ? I have never heard in Lazarus (except of problem at installtime).
Title: Re: Example projects window
Post by: Martin_fr on December 30, 2021, 01:28:21 pm
Sorry, Martin. This is not a good idea: It will be an unacceptable burdon to GetMem who handles all the additions to OPM manually. And unless "examples" are separated from "components" in some kind of GUI, components will be buried underneath loads of examples.

Well, yes: But the burden can be shared. With the correct infrastructure to maintain the OPM database, other could help.

If the database is moved to a gitlab repo (or the database is (auto-)generated from a git repo), then all team members (or selected team members) could help.


And, in any case
Doing it via opm, and the "example window" searching packages, also offers the advance that existing packages (e.g. packages that add new components), can include examples for those components.
needs to be taken care off.

Already today "ATSynEdit" has a folder "app" which contains demos.
It would be great, if the "Example Window" could show them, wouldn't it?

But if, that is the case, then we have an entirely working system for "online examples". Should we then really duplicate it?
Or should we think about how to distribute the maintenance work?
Title: Re: Example projects window
Post by: Martin_fr on December 30, 2021, 01:33:30 pm
Because, from what I understand, one of the issues in the bug report is that end users who install Lazarus using deb/rpm/pakman etc end up with it all in Read Only space.  Its not a packaging thing, its a ReadOnly issue. So, if they try a build an example project, it cannot write. I, personally always build from source but lots and lots of people install distro packages.

When rebuilding the IDE, ppu files (and the new exe) are written to a folder inside the primary config path.

A similar concept is needed here.
Plus, if the user wants to change the files, then the project should be saved to a new location at this point. (the IDE could offer that, whenever there is a write permission issue).
Or the "example window" could offer to copy the files before opening the example.

Such functionality could also go along with "files saved for compilation" should not be saved in the project folder, because you may not want to change your files. Compilation should be done by saving to a temp folder (all that as an optional feature). But that part is off topic.
Title: Re: Example projects window
Post by: Martin_fr on December 30, 2021, 01:46:58 pm
1) Remove all the examples from the distributions
Quote
* Reduces distribution size (I didn't even know there were examples)
It's 60 MB, but text only. Compressed maybe 10 MB.
I think that is the least of the issues.

Quote
3) Link to a to-be-created Wiki page which lists example category pages
4) Add examples to appropriate yet-to-be-created category pages
5) Example entry in Wiki links to SourceForge/Github/Gitlab for the source for each example

IMHO, some examples should be included in the main distro.
Maybe that could be more selective.

If the examples are included. And if they actually work, then that removes one (big-ish) source of problem for users.
Examples would probably more often be used by less experienced users. So that actually matters.

If some examples would be removed from the default-distro, then there should be a download via OPM (or if needs must, something similar). IMHO Download links to "SourceForge/Github/Gitlab"  are not a good replacement.

If Examples are stored online (for opm), or even included in the distro, then nothing stops us from having the description in a readme, and displayed in the IDE example window.
An additional wiki page is still ok (some examples already have this)
Title: Re: Example projects window
Post by: wp on December 30, 2021, 03:31:04 pm
One thing to be considered when Examples are removed from the main distribution: Sample code is coupled to the distribution. When a Laz version introduces a new feature the corresponding sample project will adopt to the new feature. As a consequence the sample will not be usable by older Laz versions. This means that OPM (or whatever makes the samples available) will have to keep sample projects at a variety of versions!

Isn't it possible to have the installer copy the Examples folder after installation to user space?
Title: Re: Example projects window
Post by: Martin_fr on December 30, 2021, 03:53:11 pm
Isn't it possible to have the installer copy the Examples folder after installation to user space?

Well this is mainly relevant for installations to the default/global install folder.

Such an install can then be used by any user on this system. Even by users (accounts) that are created at a later time.
So those users do not have a home folder yet. Where should the installer move the files too?

The IDE, could copy the example folder to the primary-conf-path, on its very first run. It does the same with the default "environmentoptions.xml" that is in the main folder of the install. That is, if the pcp does not yet have an "environmentoptions.xml".

But that creates a new problem. The very problem that we already have for the "lazarus.dci" file.

The "example folder" and the "lazarus.dci" can have updates in newer versions. But the copy in the pcp can contain user changes.
So with the "lazarus.dci" currently you do not get new templates when you update your IDE.
And that would happen to examples too.

If we copy, examples on request, we have the same problem. But at least on a smaller scale. Also in that case, because we still need to implement this copy-code, we can add a dialog and ask if a new copy from the install folder should be created.
I.e. we can list the copy, and the original as two entries. We can check, if there is a diff between them (and it does not matter if the user made changes, or the install was updated). If the user made changes, they may want the option to continue with them, revert their copy, or make a new copy in a different folder.
Title: Re: Example projects window
Post by: dbannon on December 31, 2021, 01:56:42 am
OK, so we have two broad models available. Both models require a review of and (indexing, meta data) additions to existing set of examples.

Built In
An improved version of what we now have. It requires -


Online Model
I think overloading the OPM would be a bad stratagic mistake. The OPM is an incredibly valuable resource and one we should not complicate or enlarge without due consideration. Too many eggs in one basket argument applies.

But a new stand alone, online service may well be a good idea. I have been working recently with an app that hits the Github API, its very rich and easy to use. I expect Gitlab has the same approach.  Issues include -

In answer to Juha's question, I am certainly willing to be involved but we both know Juha would do a far better job (but good use of his time ?). But we must agree on an approach first. We seem to be a long way from there now.

OK, some wild and uninformed speculation of how an online model would work -

Examples and Demos are defined as small projects demonstrating only a small number of features. This is not a show case and a projects with more than X lines are not accepted. Given the small size of projects, pull requests may not be the best way to handle additions/updates ??

A github/lab site, maybe under the existing FPC one. Several gatekeepers are given commit access. Top level is category, below that are all the projects in that category, pairs of files, a metadata file and a archive (a zip file).

Gatekeepers are sent a metadata file and a zip file of a standalone project, maybe a lazarus project or just a single code file. Only the gatekeepers must have git installed.

They need to examine code for safety (and correctness ??) If happy,  its pushed.

A script reads all the metadata (or just the new one) and category based pages are generated. Either on the Lazarus/FPC wiki or the gitlab wiki ?? A master metadata file is generated.

Potential end users can download those zip files themselves.

Lazarus has a GUI window that reads and displays the master metadata file, a click will download the selected zip (into ~/.lazarus config area) and open it in the IDE.

Too easy !

EDIT : WP's issue about changes due to version of FPC/Lazarus. Most examples would be FPC/Lazarus independent. Those that demonstrate something that exists in one version but not another would be marked accordingly.

Davo



 
Title: Re: Example projects window
Post by: dbannon on December 31, 2021, 12:45:39 pm
OK, here is a first cut at a gitlab examples repository

https://gitlab.com/dbannon/fpexamples

I have put two categories up, Beginner and NoDesigner, just as eg.  But I have put two versions of each up, Beginner and BeginnerAlt, difference being how files are stored in there. The BeginnerAlt is more traditional, unarchived and is more readable. Thoughts ?

And, yep, dead easy to use their API to read the metadata and to download files from there.

Davo
Title: Re: Example projects window
Post by: Martin_fr on December 31, 2021, 01:49:07 pm
OK, here is a first cut at a gitlab examples repository

https://gitlab.com/dbannon/fpexamples

Not sure why the name is "FP..."?


1) Why zip?
- If we have IDE integration, we can download files/folders as required
- If there is more than one example, GitLab offers "download folder as zix" => so all that is needed is "each example has a folder of its own"
- Without zip, examples can be browsed online.


2) "Hello World"
I try to imagine a user to whom this is of value....

IMHO, this really would need to be a tutorial.
=> That is, an IDE plugin that would open a "guide window" leading the user step-by-step through the process of creating the "hello word" project.
Starting the tutorial, the user would get instructions to
- New Project: Application
- Select TMemo on the component bar
- Click the form, and drag the size, or resize afterwards
- but the buttons on the form
- add the events
- run
(actually, one button to change the form.caption would be a fine start...)

Just seeing a form, with two buttons, even with some text explaining how the got there....
Yes, on the wiki, with images, that would be different. But that would defeat the idea of "like OPM" (download, and explore locally).


3) Examples for components
Like IIRC the AtSynEdit demo. A form with the component, and the buttons, checkboxes, ... to modify different properties of the component.
So the user can choose different "line-wrap" options, and immediately see the effect.

Only problem(s), as pointed out before:
- If you use OPM to get AtSynEdit, then you need a 2nd download for the examples (if indeed the decision is to have examples in a diff repo).
- If you explore the OnlineExamples, and download AtSynEdit-examples, then there must be a dependency to trigger OPM.

As far as I can see, there is a given dependency between OnlineExamples and OPM.
Attempting to divide them, will create extra complexity.

We do already have OPM packages, that include examples (component + examples). I doubt we can prevent this.
Whatever "tool" (example window , online-example-mgr) we will have in the IDE:
=> It must display
- examples from the installer
- examples from OPM packages (at least the "component + examples")
- maybe OPM packages, not yet downloaded, but know to have examples included? (they would need to have a flag)
- and (if indeed created): the "online example only repo"




Title: Re: Example projects window
Post by: JuhaManninen on December 31, 2021, 07:09:32 pm
In answer to Juha's question, I am certainly willing to be involved but we both know Juha would do a far better job (but good use of his time ?). But we must agree on an approach first. We seem to be a long way from there now.
Actually I only promised to work on the window for local example projects, yet I am happy if somebody else does it. Whoever does it, let other people know to prevent duplicate work.

For an online project system it makes no sense to store ZIP packages in a revision control system like GitLab. Revision control is meant for source code.
ZIP files can be stored in the same server as the OPM packages. Remember, the sources of OPM packages are also maintained in some external revision control system.
A competing concept would be to load source code directly from GitLab or GitHub. It was already discussed when OPM was designed. For OPM it didn't fit well because quality control is difficult to enforce. For example projects it would work better as quality control is not as important.
There is such a plugin for Delphi's packages and projects IIRC.
Title: Re: Example projects window
Post by: dbannon on January 01, 2022, 01:56:23 am
Not sure why the name is "FP..."?
It stands for Fine Programing.

- If there is more than one example, GitLab offers "download folder as zix" => so all that is needed is "each example has a folder of its own"

Indeed ! I did not know that (Github, my usual platform does not allow that). I have restructured it. Thats heaps better !

2) "Hello World, I try to imagine a user to whom this is of value...."
Its only an example of an example. But new user of a programming language, build system etc looks for Hello World. Its of no earthly use expect allowing to new user to confirm compiler works as expected and binary runs. It takes up no room but there is no commitment to it being there at the end if you find it a problem.

IMHO, this really would need to be a tutorial.

Sure, this model would support a tutorial series, but it would be silly discussing content before the framework is agreed on.



Only problem(s), as pointed out before:
- If you use OPM to get AtSynEdit, then you need a 2nd download for the examples (if indeed the decision is to have examples in a diff repo).
- If you explore the OnlineExamples, and download AtSynEdit-examples, then there must be a dependency to trigger OPM.
.....

The need to resolve dependencies is in just about any development environment. People who are going to stay with us and develop real apps must learn that process. The example project we are addressing here is just where they start out, we cannot undertake to solve all problems beyond that stage. That said, its sure no worse than where we are now, if AtSynEdit decides to move its examples to this project, the meta data will say, "but first install AtSynEdit" and docs for AtSynEdit will say look for the examples in Lazarus Examples. I guess it will depend on just how "3rd Party" a package is.

Thanks for your input Martin

Davo
Title: Re: Example projects window
Post by: dbannon on January 01, 2022, 02:16:22 am
Actually I only promised to work on the window for local example projects, yet I am happy if somebody else does it. Whoever does it, let other people know to prevent duplicate work.

I already have code that can upload and download from Github, superficial experiments yesterday indicates Gitlab is much the same. So, maybe I get the transport side of things working and you make it look presentable ? I'd just use my temp gitlab repo, later we'd need to redirect it to somewhere more official. And then integrate that into Lazarus.

The task of reviewing existing projects, adding necessary meta data, pushing up to a repo ......

My experience, from a past life, is managing projects like this can be done from a wiki page, does that suit ? 

For an online project system it makes no sense to store ZIP packages ...
Yes, Martin pointed out that Gitlab (unlike Github) allows an interactive user to pull down a directory, converting to zip on the fly, thats excellent.

The question that needs to be resolved is, IMHO, is this the way we want to go ?

(Please note, I am NOT trying to grab ownership of all this, far from it, but I am just trying to push it along ....)

Davo
Title: Re: Example projects window
Post by: Martin_fr on January 01, 2022, 03:27:28 pm
2) "Hello World, I try to imagine a user to whom this is of value...."
Its only an example of an example. But new user of a programming language, build system etc looks for Hello World. Its of no earthly use expect allowing to new user to confirm compiler works as expected and binary runs. It takes up no room but there is no commitment to it being there at the end if you find it a problem.
Sorry, I did not meant to make this about "finding a problem". But rather "pointing out more potential".

I am aware, this is an "example of an example". And for that, it could have been an empty "program Foo; begin end;".
- It shows the infra structure (e.g. meta data, repo structure, ...).
- It gives some ideas of what the "example system" would have to comprise.

So, indeed my introduction about the lack of "value" was wrong. My apologies for that.

I thoughtlessly used this as an opener, to bring up the idea of tutorials (see meta-data below) as something to comprise. Your example just happens to the kind, that IMHO could be made into a tutorial.


Quote
IMHO, this really would need to be a tutorial.
Sure, this model would support a tutorial series, but it would be silly discussing content before the framework is agreed on.

framework and content are somewhat bound together....

The more is known about the content, the better the framework can be planed.
Sure "KISS" is a could principle. And we can add more as we get there.
But
- meta-data files will have to be extended too, and once the first version of the "example mgr" is out, meta-data requires backward compatibility.
  Planing ahead, can help with that. E.g. planing for different example kinds: component-behaviour, tutorial, comparing-related-features, ...
- The form and place of storage (folder vs zip ...) are also long term decisions.
- Compatibility (and/or integration) with OPM, is yet another "hard to change" decision.
  As pointed out: OPM packages already contain examples. So our meta data, and storage requirements should allow such OPM packages to be recognized.

So the below is not about what to implement now.
But about, what to consider when making decisions.



The OPM integration has still a lot of unconsidered points.

Well, I guess - ideally - the "example window" (in its new form) will show everything that is (or has) example(s).
- installed
- online
- and if our online DB is separated from OPM, then also OPM packages with examples

But in reverse, if a user browses OPM packages, and if they (or some of them) have meta-data about their examples:
- Should then the local OPM browser window, not also allow to filter by "has example" ?
- And if the local OPM browser window, can show "only with example", should then the local OPM browser window not also be able to show the entries from the new example repo?
  (Especially, since the new example repo could also contain examples for OPM packages. Just examples that have not been bundled with the OPM package itself.)

Note that:
- "local OPM browser window" and "OPM online database" are separate entities.
   Changes to the "local OPM browser window" do not affect the maintenance work of the "OPM online database".
- Adding "example meta data" to OPM packages may affect that maintenance.
   But this is independent of changing the "local OPM browser window". This would be desirable, even just for the new "example window".



So, unless we want to exclude OPM packages from having examples (within the example window), it seems to me that the two need to be very closely coupled.

We can have separate databases for the two indeed. IMHO that works well with any point I mentioned above.
It does mean however:
- the example system must access both databases.
- Contributors must deal with two teams of maintainers to submit their work.

The latter is, if they submit work in both categories. OPM packages, that just happen to include examples would be dealt with by OPM maintenance, and thus the OPM maintainer would also decide about some additions to the examples list.



Anyway: Just my two pennies.

If it is thought better to limit the extend, then that's that.
Also if the idea to add those later, and the hope is that the code will be easy to change, even if it was designed without taking those possibilities into account, well then hopefully this will work out.

Otherwise, consider taking them into account now.
Title: Re: Example projects window
Post by: Martin_fr on January 01, 2022, 04:09:24 pm
Btw, another point that may be worth considering: translations.

- Could the meta data, contain fields for translations?
- Could "readme" or similar documents, be provided in more than one language?

And, if the example itself does not have a translation, could the design allow for a lookup db to be added at some point, so other contributors can add more translations?
Title: Re: Example projects window
Post by: af0815 on January 01, 2022, 06:00:47 pm
Btw, another point that may be worth considering: translations.
Can the actual FPDoc this ?! Because this is the main tool for (Code-) documentation.
Title: Re: Example projects window
Post by: dbannon on January 02, 2022, 02:01:35 am

Sorry, very long answer warning !
Summary -


Sorry, I did not meant to make this about "finding a problem". But rather "pointing out more potential".

No 'sorry' needed !  This is all about discussion.  And you are discussing, thanks !  I'd much rather be told I am on the wrong track now than after a couple of days work. And a sequential tutorial capability is probably almost there, should be considered as we go.

framework and content are somewhat bound together....

- meta-data files will have to be extended too, and once the first version of the "example mgr" is out, meta-data requires backward compatibility.

Yes, right now there are about 250 example projects just in core Lazarus. There is a lot of work to be done to make them into what we are considering and I agree, as soon as thats done, we'll realise there is more to it. Right now I am playing with a brief "technical feasibility" project, just to see if what I imagine works will (given my familiarity is with Github, not Gitlab). And the answer is yes, technically, its all pretty easy. Will post something soon. I think my best approach is a standalone app, not integrated (yet) into Lazarus that we can use to prove the access and metadata models. It will be rough and ready and any part of can be walked away from without causing me any distress at all !

Using that we can go over the existing examples and demo (I am disregarding Tests) and build up a workable set in that Gitlab repo of mine, so no annoyance to the Laz Devs yet.   

OPM Integration

  As pointed out: OPM packages already contain examples. So our meta data, and storage requirements should allow such OPM packages to be recognized.

What I am playing with is model where the client (ultimately built into Lazarus) dowloads a master.meta file, checked daily (?) and keeps a cached copy. Suppose when doing its daily checks (or a manual refresh) it also scans the area in .lazarus where OPM packages are and if it finds any recognisable example projects, adds them to its local list ?

Obviously, this only happens if a user opens the Examples window.


Well, I guess - ideally - the "example window" (in its new form) will show everything that is (or has) example(s).
- installed
- online
- and if our online DB is separated from OPM, then also OPM packages with examples
The model I mention a couple of line up would show -
* all "lazarus examples"
* all OPM examples it found. That is, ones delivered as part of a OPM install.

It would NOT show OPM examples from packages not installed. And I think that makes sense. 


But in reverse, if a user browses OPM packages, and if they (or some of them) have meta-data about their examples:
 - Should then the local OPM browser window, not also allow to filter by "has example" ?
- And if the local OPM browser window, can show "only with example", should then the local OPM browser window not also be able to show the entries from the new example repo?
While an issue for OPM, I'd say yes, I'd also like a filter for "show ones with good docs" but thats getting a bit subjective :-)



Especially, since the new example repo could also contain examples for OPM packages. Just examples that have not been bundled with the OPM package itself.)

I am sort of thinking, given that idea of the Example Unit scanning locally of OPM Examples, that OPM Examples belong in OPM Packages and 'should' comply with the Examples standard we come up with.


So, unless we want to exclude OPM packages from having examples (within the example window), it seems to me that the two need to be very closely coupled.

No, again, looking at the "Example Unit scanning locally of OPM Examples" model minimizes coupling between the two services. And its one way only. And I think thats quite important. Over time, we can encourage OPM Package maintainers to include the standard looking examples and thats all that needs to be done.

And when I say standard looking examples, its a Lazarus Project with a 6 or 7 line metadata (although, as you noted, that could increase).




translations
Quote
Btw, another point that may be worth considering: translations.

Definitely we must allow for that. Brain Storming -

As the meta files are by definition, quite short, maybe its just having a po directory in the project dir with translated files ? Can the same model support in project po behaviour too ?

My model, at present is that a gatekeeper when added or changing an example project, will run an app that scans over the repo building a master.meta that is uploaded to gitlab along with the content changes. Now, if that scanner finds translated meta files, it could also build translated master.meta files. Then, when the (Lazarus) client tool runs, it would prefer an appropriate translated file if one is available.

An end user who decides to translate an existing project just need send in a translated file, gatekeeper names it appropriatly, adds it to his/her local git repo, runs the scanned, adds, commits and push.




A snipit manager ?

I have also been considering how this could be used in a snipit mode, instead of opening the Example Project, open just eg uini1.pas without closing the project the user is working on. That way, a quick copy and paste is possible without disturbing the user's work flow.  Might be version two.




fpdoc
Quote
Can the actual FPDoc this ?! Because this is the main tool for (Code-) documentation.
My initial though is that fpdoc would be overkill for the faily miminal docs we are planning to use, just a couple of relevant keywords and a sentence or so of what the project demonstrates. But  af0815 if you are familiar with fpdoc, could you elaborate ?

(phew ! longest post I have ever made here. Far too long !)

Davo




Title: Re: Example projects window
Post by: JuhaManninen on January 02, 2022, 11:32:51 am
As dbannon wrote, the example project system can search projects under the OPM install directory. No other integration is needed.
I realized there must be multiple directories to search from.
One is "examples" under Lazarus tree which is the default now.
The "components" directory must be searched, too. The local packages also include example projects.
Third one is the OPM directory which is known by reading the OPM settings.
Then a user can define more directories.
Title: Re: Example projects window
Post by: Martin_fr on January 02, 2022, 01:02:58 pm
OE => "Online Examples", depending on context "Online Example Registry/Database"

It would NOT show OPM examples from packages not installed. And I think that makes sense. 
Makes sense. In fact, showing examples for packages "not yet even downloaded" would be confusing. (There OPM could be extended, to allow filtering by "has example", but that is off-topic).

However:
- What about "examples" (pure examples, stored in the OE) that require a package?
-- 1) If that package is OPM ? (And not yet downloaded / or needs an update downloaded)
-- 2) If that package is avail on local storage (but maybe not installed in the IDE)
-- 3) A package, unknown to the IDE (maybe removed, but example DB not yet updated) -> This would be an error, but needs handling.

In case -2- examples could still be listed:
- "installation" (for design time packages) can be offered when opening the example.
- "installation" does not have to be mandatory. It is (usually) only needed to open the form in the designer. Installation is (usually) not needed to compile and run the example. (So once the user has been informed, it is their choice)

In case -1-:
I am sort of thinking, given that idea of the Example Unit scanning locally of OPM Examples, that OPM Examples belong in OPM Packages and 'should' comply with the Examples standard we come up with.
Well, ideally yes.

But, what if a person outside the "OPM package"-team provides examples? (I.e.: If I were to write examples for OPM-foo)
Ideally, I would submit them to the maintainer of that OPM package.
But if they do not want to add it? Maybe because their code is Laz&Delphi, but the example is Laz-only?
Then it could still be a good example...
-> Yes, this is a very special case.... Ok to worry later, if we don't have a good idea now.

As dbannon wrote, the example project system can search projects under the OPM install directory. No other integration is needed.
Depends, on how the above is handled.

IMHO we need at least the ability to specify "required packages" for the example.
And probably should at least prepare, to store some info on the packages
- min version (maybe max version too)
- Only required for design.
--  Some packages have separate *Dsgn packages, then the example can run without this (i.e. the package does not need to exist at all, example can be compiled/run without it)
     But the user must be told what happens if he does not install it.
--  Other packages have IDE integration and runtime in one package. So even if the user skips installation, the package must exist for compilation.
- Origin of package: Laz-Installer, OPM, unknown

We may not need all of those right now, and hence not add them now.
But the design of the meta file should allow to easily add them, including backward compatibilty.




Translations:
As the meta files are by definition, quite short, maybe its just having a po directory in the project dir with translated files ? Can the same model support in project po behaviour too ?
Good start. Translated files, must not contain "version info", or any info at all, except translated strings. (Other elements could be allowed, but have to be ignored).

That depends a bit, on how well compatibility info "lazarus version" is represented.





Right now I am playing with a brief "technical feasibility" project, just to see if what I imagine works will (given my familiarity is with Github, not Gitlab).

What I am playing with is model where the client (ultimately built into Lazarus) dowloads a master.meta file, checked daily (?) and keeps a cached copy. Suppose when doing its daily checks (or a manual refresh) it also scans the area in .lazarus where OPM packages are and if it finds any recognisable example projects, adds them to its local list ?

A few notes:

1)
I would expect that contributors maintain their own GIT repo. (I guess we start with "git repo" as storage, probably keep it at this, but extension would be an issue for later...)

We would then have a central "OE" repo. Containing
* A list of all examples
* A copy of all meta files (and maybe translation)
* A copy of the current stable version of the example (actually not a must, but desirable in case the contributor's repo becomes unavailable.)

Ideally the meta files can keep info on:
- older versions
- Which example version to use with which IDE version

However, we do not need to have copies of older version. (Yet we probably will have).

==> IMHO we may find, that downloading data from our OE repo (and/or externals) will be very similar to what OPM already does.
   So, it could be quite likely we can re-use some of the code from the OPM package. (just pass in our OE Url / some config/opts)

Like OPM, we could also allow the user, to get the latest example from the contributor. This would of course not have been reviewed by us.
(If the meta has a version map, this could be the "latest matching the IDE version" or the "latest on main branch")


2)
The meta file.

As long as we allow git only as storage for contributors, we can relay on references.
I.e. if the meta file would have:
Code: Text  [Select][+][-]
  1. "Compatibilty": {
  2.   "Laz-2-2": "1.3",
  3.   "Laz-2-0": "1.2"
  4. };
  5. "Versions": {
  6.    "1.3": "<sha1>123",
  7.    "1.2": "<sha1>234"
  8. }

The "sha1" is a signature (and the meta with that sha1 is stored in our OE, under our control).
If we do not have the version for the sha1 in our OE, we can load it from the external git. If the contributor changes something, the sha1 no longer exists. But if it exists, it is the version that we know.

However, the copies in our OE repo have different sha1. So using the above, we need a "meta-sha1-map" file. This would contain a mapping from the external sha1 to our sha1.


Another options here is, to have a single global "meta-ide-version" file.

Lets say, our OE was updated for the 2.0 release.
The commit 987 has all the correct entries, for Laz-2.0.
Then "meta-ide-version" would contain
Code: Text  [Select][+][-]
  1. "VersionMap": {
  2.   "2.0": "987"
  3. }

We can add new meta data, for the ongoing development versions of the IDE.

"987" can be on a branch. So if we need to add fix-updates, we can.
We would get commit "98A" on the branch.
And in the "main" branch we would change the info to: "2.0": "98A"

If we release Laz 2.2  we add:
Code: Text  [Select][+][-]
  1. "VersionMap": {
  2.   "2.0": "987",
  3.   "2.2": "ABC"
  4. }

The example system can always download "meta-ide-version" from the "main" branch, and lookup the correct revision for the IDE version the user has installed.


Translations and versions:
Once the revision for the IDE-version has been found, all other data, including translation, is loaded from that revision.

Title: Re: Example projects window
Post by: dbannon on January 02, 2022, 01:03:20 pm
As dbannon wrote, the example project system can search projects under the OPM install directory. No other integration is needed.
I realized there must be multiple directories to search from.
One is "examples" under Lazarus tree which is the default now.
The "components" directory must be searched, too. The local packages also include example projects.
Third one is the OPM directory which is known by reading the OPM settings.
Then a user can define more directories.

Hmm, maybe not quite the way I am leaning Juha.  I am thinking that the existing Lazarus Examples will move to a Gitlab repo, ideally under freepascal.org.

I suggested local scanning for Examples only for things like the OPM ones. Benefits, solves the Read Only problem, makes for a smaller Lazarus Distribution, people using an older Lazarus can get new Examples - I see several more less easily and quickly described.

Anyway, please consider !

I have posted a very, very early draft of the unit doing the work and a rough and ready gui to exercise it. Rather appropriately in https://gitlab.com/dbannon/fpexamples/-/tree/main/Utility/ExScanner

It can load its internal List of Examples data direct from the above repository, from a locally cached file or from directly reading a Examples Gatekeeper's local Examples Git repo.  It makes a master.meta, displays a list of Examples (yes, in a ListView) and shows the description when an item is clicked.  Its far more about proof of concept than a proposal at this stage.

Davo

Title: Re: Example projects window
Post by: dbannon on January 02, 2022, 01:09:17 pm
Sorry Martin, our posts crossed.

I will read yours in the morning, its late here and hopefully just getting cool enough to sleep. Been hot for a few days, yawn !

Davo
Title: Re: Example projects window
Post by: Martin_fr on January 02, 2022, 01:32:54 pm
I suggested local scanning for Examples only for things like the OPM ones. Benefits, solves the Read Only problem, makes for a smaller Lazarus Distribution, people using an older Lazarus can get new Examples - I see several more less easily and quickly described.

I would definitely keep some examples in the distro.

Not everyone wants their IDE to go online. Some are concerned about privacy, others are still exposed to cost-per-byte (and yes, of course the installer has those bytes, though maybe they downloaded this while on a cheaper connection...)
Title: Re: Example projects window
Post by: avra on January 02, 2022, 03:04:04 pm
I would definitely keep some examples in the distro.

Not everyone wants their IDE to go online.
+1
Title: Re: Example projects window
Post by: wp on January 02, 2022, 05:27:46 pm
I would definitely keep some examples in the distro.

Not everyone wants their IDE to go online.
+1
My opinion, too.

And I get the impression from the discussion that the solution will somehow depend on features of GitLab. I think this is not a good idea. What if we once were forced to move away from GitLab? Then the example infrastructure would break away...
Title: Re: Example projects window
Post by: Martin_fr on January 02, 2022, 06:15:37 pm
And I get the impression from the discussion that the solution will somehow depend on features of GitLab. I think this is not a good idea. What if we once were forced to move away from GitLab? Then the example infrastructure would break away...

The only "gitlab" specific thing I recall is the "download folder as zip". But that is not required for the IDE integration. This is nice, for users who want to download a single folder.

It seems OPM just hosts a bunch of zip files.
IMHO allowing online-browsing, is a nice idea too.

So, well we have to see how the OE could do the downloads....
Actually, if it follows OPM, then all it needs is a link to a zip file.
- if hosted on gitlab, link to the zip that gitlab generates.
- otherwise link to a zip that was uploaded somewhere.



There are a few "git" related proposals. Such as using the sha1. That works with any git-server.

However, if we use git as storage for the OE database, then the client will need some config:

Path to main file: https://gitlab/some/path/main-meta.json
Path template for other files: https://gitlab/some/path/{%filename%}
Path template for other files an given sha1: https://gitlab/some/path/{%sha1%}/files/{%filename%}

Something like that should be adjustable to any git repo with https access.
Title: Re: Example projects window
Post by: JuhaManninen on January 02, 2022, 09:26:19 pm
IMHO we need at least the ability to specify "required packages" for the example.
And probably should at least prepare, to store some info on the packages
- min version (maybe max version too)
- Only required for design.
--  Some packages have separate *Dsgn packages, then the example can run without this (i.e. the package does not need to exist at all, example can be compiled/run without it)
     But the user must be told what happens if he does not install it.
--  Other packages have IDE integration and runtime in one package. So even if the user skips installation, the package must exist for compilation.
- Origin of package: Laz-Installer, OPM, unknown
That is over-engineering and duplicate logic. All the information is already in project/package dependencies.
An improved algorithm to find a dependent package would benefit all projects and packages, not only the example projects discussed here.
For example a converted Delphi project can have its dependent package already ported but with a different name. Adding more meta-data for an online package would then help to locate it.

... hopefully just getting cool enough to sleep. Been hot for a few days, yawn !
Oh boy! We had -15°C last night here. It is cool enough to sleep. :)
Title: Re: Example projects window
Post by: Martin_fr on January 02, 2022, 10:17:02 pm
IMHO we need at least the ability to specify "required packages" for the example.
And probably should at least prepare, to store some info on the packages
That is over-engineering and duplicate logic. All the information is already in project/package dependencies.

Yes, and no.

The meta data, is downloaded for all examples.
lpi/lpk are only downloaded, once a user wants to try the example.

A user searching for an example to draw some 3d cube, might not expect to have to download the entire castle-game engine. (just because an example for this matches his search term).
Might sound far fetched, but then again I don't think it classifies for "so far the universe is winning".

But ok, to leave out (for now).
Title: Re: Example projects window
Post by: dbannon on January 02, 2022, 11:30:41 pm
...
I would definitely keep some examples in the distro.
Not everyone wants their IDE to go online. Some are concerned about privacy, ....

Then we have to solve the *nix readonly problem.
Personally, I share the privacy concerns but neither gitlab nor github track non-logging in downloads. Maybe they will in the future...

Davo
Title: Re: Example projects window
Post by: Martin_fr on January 02, 2022, 11:35:44 pm
...
I would definitely keep some examples in the distro.

Then we have to solve the *nix readonly problem.

Well, treat the installed files, like files on a server. "Download" them, and save them to a writeable directory.
Title: Re: Example projects window
Post by: dbannon on January 03, 2022, 12:13:48 am
... a number of issues
> What about "examples" (pure examples, stored in the OE) that require a package?
I envisage only the Lazarus Examples stored in OE, examples belonging to external packages belong in external packages.

> -- 3) A package, unknown to the IDE (maybe removed, but example DB not yet updated) -> This would be an error, but needs handling.
The local Examples dbase could be updated daily (when user opens Example Window) or manually. If they remove a package and later that day expect to find the associated Example there, well, and error message is all we can do. But the point is, the Example has gone when the OPM has gone.

> IMHO we need at least the ability to specify "required packages" for the example.
While agree, I think it can be done with just FP defines in the project code and a mention in the Project Description, "This demo requires Lazarus 2.0.12 or later". We are not trying to make an unattended Build System, a new user using an old Lazarus will learn something from the experience.
 

> But the design of the meta file should allow to easily add them, including backward compatibilty.
Right now, due to its simplicity, the meta data is just plain text. If we intend to make it, the meta data, more complicated, then probably need to move to xml or json etc. My demo uses json for the master.meta file.

> I would expect that contributors maintain their own GIT repo. (I guess we start with "git repo" as storage, probably keep it at this, but extension would be an issue for later...)
See earlier discussion with Trev about safety. Suppose I have an example in my own github repo, there is a link in the Lazarus Repo to it. It demonstrates how to read all files on disk into a TStringList.  Been there for years. I have too much to drink one night and add a two lines -

Code: Pascal  [Select][+][-]
  1. for St in STL do
  2.     deletefile(St);
  3.  

Or an example that encrypts the user's disk until they find some bitcoin ....


> So, it could be quite likely we can re-use some of the code from the OPM package. (just pass in our OE Url / some config/opts)
That side of it is all working now in the demo I mentioned yesterday. Its quite easy....

> Like OPM, we could also allow the user, to get the latest example from the contributor. This would of course not have been reviewed by us.
With an appropriate warning / disclaimer.

(disregarding the next section because -
* I do not believe we should be automatically pointing to repositories out of our control
* required versions of FPC/Lazarus can be handled in project code

Martin, I suspect you see this facility as supporting large code base projects. I don't, its about small, self contained demonstrations of how to do one particular thing each. Nothing here should be an "application", just enough code to compile and demonstrate one useful trick. Most of the existing Examples are small, all thats lacking is a review and meta data.)


Davo
Title: Re: Example projects window
Post by: dbannon on January 03, 2022, 12:19:30 am
And I get the impression from the discussion that the solution will somehow depend on features of GitLab.

Not intrinsically. I choose gitlab because FPC/Lazarus did. If gitlab closes down, several methods in https://gitlab.com/dbannon/fpexamples/-/blob/main/Utility/ExScanner/exampledata.pas would need to be changed to use github. Or some other service. Or a machine hosted in my back shed (just joking).

If gitlab no longer meets our needs, we have a lot more to worry about then OE (as Martin has named it).

Davo
Title: Re: Example projects window
Post by: dbannon on January 03, 2022, 12:38:53 am
Just to labour the point a bit about keeping examples in the distro.

I believe if we keep some examples in the distro, we should keep all examples there. To make it work with just a handful of perhaps very basic ones is no harder than making it work for a whole lot.

Right now, we have lots of examples in the distro, they are generally speaking unusable for many users because -

1. Undocumented - all you get is a project name for most.
2. Unwriteable - (Linux package installs)
3. Unfindable - un-categorized as to target audience, applicability etc.
4. Inflexible - cannot, for example find installed OPM Package Examples.

So, something needs be done, please, lets not have two parallel models, its either in disto or online. I'm happy with either but I do consider an on-line model will provide a more manageable and flexible service. For much the same reason that so many other things are moving to on-line models.

Its all new and shiny !   :D

Davo

Title: Re: Example projects window
Post by: Martin_fr on January 03, 2022, 01:02:21 am
> I would expect that contributors maintain their own GIT repo. (I guess we start with "git repo" as storage, probably keep it at this, but extension would be an issue for later...)
See earlier discussion with Trev about safety. Suppose I have an example in my own github repo, there is a link in the Lazarus Repo to it. It demonstrates how to read all files on disk into a TStringList.  Been there for years. I have too much to drink one night and add a two lines -

Code: Pascal  [Select][+][-]
  1. for St in STL do
  2.     deletefile(St);
  3.  

Or an example that encrypts the user's disk until they find some bitcoin ....

I do thing we will have a copy of their code. Just though things don't break if they have some downtime, or delete/move their repo.

But, a link is save too. As long as it is to a specific commit by sha1.
https://gitlab.com/freepascal.org/lazarus/lazarus/-/commit/c8feb2d0e8d72e8145c52b9a0b6055adbdc8344c

Unless you can deliberately generate content (different from the current), that will get the same sha. If you can do that, a lot of stuff on the net suddenly becomes a lot less secure.
Title: Re: Example projects window
Post by: dbannon on January 03, 2022, 06:42:45 am
But, a link is safe too. As long as it is to a specific commit by sha1.
https://gitlab.com/freepascal.org/lazarus/lazarus/-/commit/c8feb2d0e8d72e8145c52b9a0b6055adbdc8344c

I think you mean (your link was to the commit) -
https://gitlab.com/freepascal.org/lazarus/lazarus/-/tree/c8feb2d0e8d72e8145c52b9a0b6055adbdc8344c

Yes, I agree, that's probably pretty safe !  But I come back to my question of we really here in the business of reviewing projects of the complexity that someone feels they even belong under revision control ? These are supposed the be small, simple examples of how to do some specific thing, not a full blown application.  Hmm, someone using a package may well have a directory containing Examples, based on their experiments perhaps, so its possible.  But its probably going to be rare ?  So we design our meta data model to support it "at some time in the future" ?

Now, talking of Delphi (someone was) has Delphi addressed this issue ? Do Delphi Examples have meta data ? (I have a legal copy of Delhi 2 somewhere).

Hmm, jumping around a bit. We'll have Examples that are OS specific, is it enough to say so in the Description, its free form text -

Quote
This application demonstrates reading the Windows Registery, its Windows Specific .....

Or rely on keywords saying "Registery Windows ..." ?

Or, more positive, we define some suggested keywords with specific meaning - Windows-Only, Unix-Only, BSD-on-PPC-64-Little-Endian-Tuesdays-Only (maybe not the last one). But having a keywords field does let us extend it easily.

I'd really like to try and keep it as simple as possible. I work with Debian Packaging and its so (and needs to be so) specific about everything, people just won't bother to contribute if we let it get too regulated.

Davo
Title: Re: Example projects window
Post by: dbannon on January 04, 2022, 05:29:57 am
OK, lest this topic go cold, I have established a (hopefully short lived) wiki page at

https://wiki.freepascal.org/Laz_Examples_Window

EDIT : https://wiki.freepascal.org/Lazarus_Examples_Window

I suggest anyone could add or remove proposals, content, make arguments either way there. But 'votes' for one model or another are perhaps better registered here ?

Davo
Title: Re: Example projects window
Post by: howardpc on January 04, 2022, 06:05:27 am
For the metadata example you give
Keywords : Buttons Forms Hello

I would suggest rather

Keywords: button TButton form TForm memo TMemo Hello

although I thought most "Hello World" programs used a label rather than a memo?
Title: Re: Example projects window
Post by: dbannon on January 04, 2022, 07:00:54 am
Thanks Howard, I guess the example there is more about the format rather than content of the metadata right now.  But if we go ahead, some "guidelines" would be a good idea.  I'd also like to see multi word words too -

'Hello World', 'Windows Only' etc, single quotes to stay readable in json ....

I think we MUST keep the metadata format simple but its also important it do the job, sigh ...

Davo
Title: Re: Example projects window
Post by: Martin_fr on January 04, 2022, 01:09:13 pm
About the meta data. I would strongly suggest to use either json or xml.
There is pro and contra for both.

Pro "json": Even if we keep Examples separate from OPM, there is a good chance that some OPM code may be re-used. After all there are similar tasks to be achieved.

Pro "xml": It is more or less the IDE's default format for data.
Title: Re: Example projects window
Post by: dbannon on January 05, 2022, 04:10:42 am
About the meta data. I would strongly suggest to use either json or xml.
There is pro and contra for both.

OK Martin, if thats the call, done. Personally I would prefer json because its easier to read using human eyes. That important if we go for an on-line solution, being able to browse through the repo content sensibly would be useful. OTOH, if its an in-distro solution, really does not matter, maybe conforming to an existing Lazarus precedent is worthwhile so xml.

So, we really need a decision, on-line or in-distro ?  My vote is on-line repo and json meta data.

Davo

Browsing ? Please see - https://gitlab.com/dbannon/fpexamples

Title: Re: Example projects window
Post by: dbannon on January 09, 2022, 09:37:36 am
I now have a reasonable demonstration of the On-Line version at -

https://gitlab.com/dbannon/fpexamples/-/tree/main/Utility/ExScanner

Its a stand alone application, has one window that might eventually evolve into a replacement for the Lazarus Examples Window. In the mean time, it serves to demonstrate the second, headless  unit that transports files from gitlab, builds a master meta file, searches for local Example meta data and, finally manages a (fplist) list of metadata allowing it to be poked into a listview and optionally filtered by category.

The App windows has a "drop down" extension showing some setup information, obviously not needed in the real thing. Similarly, there are some "Source" radiobuttons used, here, to exercise the unit.

I have tested on Linux and Windows (Mac will need some directory issues sorted out) and apart from some scalling issues, works OK.

I note to my surprise that GetEnvironmentVariable('HOMEPATH') on windows does not seem to prepend the "c:" thingo to users home path, should it ?   And TListView, in windows,you have to click in the leftmost column to get its attention, why is that ?

Summary
I have demonstrated that the on-line system is feasible and it does have some advantages over the in-distro model. Not much more I can do now unless I get some feedback as to which way we are proceeding.

See also

https://wiki.freepascal.org/Lazarus_Examples_Window - note this is a new address, someone renamed it (Laz -> Lazarus)

https://gitlab.com/dbannon/fpexamples - my badly named demo repository.

Davo

[Edit: Fixed Wiki link + explained]
Title: Re: Example projects window
Post by: dbannon on January 16, 2022, 08:43:45 am
OK, a week after my last post here, apparently no interest so I abandone the proposal.

I have removed the gitlab demo page, marked the wiki page as abandoned (Trev tells me it cannot be deleted).

I think its a wasted opportunity, we need to make it easier for (especially new) users and easy access to examples would, IMHO, make a big difference.

Thanks folks to those who contributed early on !

Davo
Title: Re: Example projects window
Post by: JuhaManninen on January 16, 2022, 09:02:23 am
OK, a week after my last post here, apparently no interest so I abandone the proposal.
I have removed the gitlab demo page, marked the wiki page as abandoned (Trev tells me it cannot be deleted).
I think its a wasted opportunity, we need to make it easier for (especially new) users and easy access to examples would, IMHO, make a big difference.
I planned to study it but then other things came up. Have you really deleted the whole thing now? No way to test?

A branch in GitLab Lazarus sources would be a good fit for such experiment. You could replace the examples window in Tools menu right away there and then fine tune without hurry.
Title: Re: Example projects window
Post by: dbannon on January 16, 2022, 10:15:50 am
I planned to study it but then other things came up. Have you really deleted the whole thing now? No way to test?
Sorry, I was too impatient !

I saved the demo app because I plan to use the gitlab code in my own app, it already works with github.  And the few demo projects I built are still in my trash bin here. So I will recreate it.

Quote
A branch in GitLab Lazarus sources would be a good fit for such experiment. You could replace the examples window in Tools menu right away there and then fine tune without hurry.

If you think the idea has some merit, then I will continue (I do have time on my hands, heat wave, C19 lockdown, watching cricket on TV) to develop it, initially as a standalone thing so we can get the existing example projects under control. From what I have seen, there is a fair bit of work to be done there. My standalone app can easily be turned into a replacement for the existing window but no point in doing that until it has a reasonable number of projects ready.

I'd rather manage the updating of existing projects in a standalone app and a standalone repository for now, the devs don't want me continuously sending pull requests in. I'll call for volunteers to help with the projects, its a big task but one that needs to be done no matter which path we take.

Davo

 

Title: Re: Example projects window
Post by: dbannon on January 16, 2022, 10:53:18 am
OK, its all back.
The temp Gitlab repo has a more reasonable name now, its -
https://gitlab.com/dbannon/laz_examples

I will proceed on the basis that -


If anyone is unhappy with that, please speak up now !

Davo
Title: Re: Example projects window
Post by: wp on January 16, 2022, 11:17:58 am
li]I'll copy a user selected project into a dir tree under 'examples' in the Lazarus config dir.
Make this directory configurable by the user, like it is in OPM. I have numerous Lazarus installations on my HD, and I'd like to re-use the examples of one installation by another installation for example to check whether sample code is compatible between versions.
Title: Re: Example projects window
Post by: dbannon on January 16, 2022, 01:23:09 pm
Make this directory configurable by the user,....

Noted, makes a lot of sense.

Davo
Title: Re: Example projects window
Post by: dseligo on January 16, 2022, 09:04:46 pm
https://gitlab.com/dbannon/laz_examples

It says 'Page not found'.
Title: Re: Example projects window
Post by: dbannon on January 17, 2022, 12:09:58 am
It says 'Page not found'.

Sorry, I had left it 'private'. Fixed now.

Davo
Title: Re: Example projects window
Post by: dseligo on January 17, 2022, 01:20:16 am
I tried ExScanner application, you did a great work.

You are missing closing single quote in line 154 of unit1.pas (EditLocalExamples.Text := Home + '\AppData\Local\lazarus\;) and in line 156 I changed the number to 32866275 (32480729 was probably old repo number).

Stand alone application is OK, but I think it would be best if ExScanner is turned into a package and put in OPM. This package would add menu entry in Lazarus to start ExScanner window. Then anybody could install package from OPM, and then browse and use online examples. This way it would be easy to add this functionality to existing Lazarus (since it would probably take a lot of time for this to be built-in Lazarus).

Oh, and is it possible to delete project1 file from gitlab, it's 25MB binary?
Title: Re: Example projects window
Post by: dbannon on January 17, 2022, 04:01:28 am
You are missing closing single quote in line 154 of unit1.pas
Thats pretty careless, thanks.

Quote
(32480729 was probably old repo number).
Indeed, just about to do another commit, trying to break apart the user from Admin content.

Quote
Stand alone application is OK,

Standalone is only a development stage. Its intended that a version of it will replace the Examples Window in Lazarus but I need something to work with now that allows easy viewing.

Quote

Oh, and is it possible to delete project1 file from gitlab, it's 25MB binary?
OK, thats happened when I pushed it all up again. Quite silly !  I will sure fix that.

I will shortly push up a version addressing the above but also restructured so its easy the see the (proto) Examples Window as an end user will see it and, separately, provide the tools someone administering the project can use.

Thanks for your feedback.

Davo
Title: Re: Example projects window
Post by: dbannon on January 22, 2022, 01:47:51 am
Some Progress -

I now have the app that demo's the prototype Lazarus Examples Window also provide a number of tools to manage the process of attaching a metadata file to each Example. Works OK but only superficially tested. So we are now at the stage where -

* Testers and their comments very welcome.
* Contributors willing to review Examples and write the very minimal metadata file needed.

Fine the app in https://gitlab.com/dbannon/laz_examples/-/tree/main/Utility/ExScanner

Its self documenting (yeah, and pigs might fly) ....

Davo
Title: Re: Example projects window
Post by: dbannon on February 12, 2022, 04:14:00 am
OK, time for a Stage Report.

--- Examples Done ---
So far, I have reviewed, occasionally edited, made a metadata file for and added to my gitlab repo just over a 100 examples, all sourced from <laz-dir>/examples . Thats just about all viable examples in that dir. I have excluded a number, most because they are "old school", built without ObjectInspector. Yes, I know its viable and even, some times a good thing to do it all in code but for someone looking for a specific example, its a distraction. I suggest one or two examples of that approach is all thats needed.

Some projects made it to gitlab with the keyword "Need Work", ones that need better docs or some other fix. I have taken the liberty of fixing some obvious ones myself.

If you want to look through the list, in random order, see https://gitlab.com/dbannon/laz_examples/-/blob/main/projects.md

--- Tools ---
I have a viable tool to help with this processing that also launches a potential "Examples Window", what the end user will see if it makes it into Lazarus. The Examples Window will download a master metadata file (if required) and displays the contents, allows selecting by category and allows searching by keyword. And will download an indicated project if required.

Should build cleanly for most people but most will need to alter the default directories I use. Easy for anyone on this list.

https://gitlab.com/dbannon/laz_examples/-/tree/main/Utility/ExScanner

--- Other Examples ---
The Lazarus bundle has a large number (maybe 200 or so) more projects scattered through the tree, mostly associated with particular components. Here I need some discussion of what should be included.

Tests - lots of what is currently considered Examples are in fact Tests, sometimes for a specif bug and sometimes to exercise a particular component or set of components. Some are useful as examples and some are not, maybe because they focus too much on testing and have no educational intention.

I believe Tests should stay where they are and ones that are useful as Examples be also copied to the Examples system.

Non Tests - ones intended as examples, some are good, some broken, some irrelevant. Quite a lot depend on optional units but without metadata user has not idea. Some appear to depend on units that are now part of other, default installed, units. In other words, its going to be time consuming and ideally done by someone with some familiarity with the specialized component, DBase, LazReport, Chart etc.

--- Questions ---

* Am I on the right track ?

* Am I correct in excluding all but a few example projects that don't use the Object Inspector ?

* Am I being too ambitious ? Should we leave the Example code where it is and just graft on the metadata somehow ?

Davo


 

Title: Re: Example projects window
Post by: howardpc on February 12, 2022, 06:59:37 am
* Am I on the right track ?
Yes and no. I don't see how you can be motivated to do much more work on this without closer collaboration with the developers. Why not start from the existing Tools > Example Projects... dialog and improve it?
Remove the tests and bugs "examples" (they should be placed somewhere else in the tree by the developers, if they are still needed).
Provide metadata for each example considered worthy to be kept, and provide a tool for local editing of that metadata and the example itself, and prune the examples to remove the outdated and unhelpful ones, and a tool for uploading the improved example with its metadata to your temporary repository.

Quote
* Am I correct in excluding all but a few example projects that don't use the Object Inspector ?
Certainly. There could be a category in which you place a couple of good examples of non-RAD, code-only utilities, useful for people who are no longer beginners.


Quote
* Am I being too ambitious ? Should we leave the Example code where it is and just graft on the metadata somehow ?
I think you need to work more closely with the developers, or you risk all your work just being rejected or ignored.
Title: Re: Example projects window
Post by: dbannon on February 13, 2022, 08:02:34 am
Yeah, thanks Howard.  I understand your message.

I felt that this was/is the best approach but that I (or someone) needed to get some partially working model happening to demonstrate its practicality.

Where it is up to now would be relatively easy to scale back from, to make a version that is not on-line for example, it would be a case of replacing the existing examples dir with my one, it has a number of edits and a meta file for each project. And it has projects moved into their own unique directory. But would all work with the existing Lazarus Examples Window (as it works now). When all such Examples are to treated, work could start on the Examples Window.

Grafting my prototype Examples Window onto Lazarus would also be relatively easy (I think) but I think it all would be a huge diff file so would need reassurance there is some interest and, sensibly, some parallel testing.

Making incremental changes, firstly to the existing examples, then to the Lazarus Examples Window is a possibility and perhaps, a lot less risky.

Davo
Title: Re: Example projects window
Post by: JuhaManninen on February 15, 2022, 02:59:37 pm
Sorry for the delay, I had some other activities. (*)

Yes, we need online example projects. IMO you are on the right track. There are 2 separate things now:
1. The GUI to download and run example projects.
2. Improving the actual example projects.
These two can be done in parallel.
Sometimes a big revamp is needed.
It turned out many example projects are old and don't use .lfm files which are quite essential in RAD. Most of them should be updated. Of course there can be an example of how to do it in Pascal code only, but that is more like an advanced topic.
Do we need the "examples" directory under Lazarus sources at all? Should all projects be moved to an online repository? I think they should.
It is good to think out of the box so to speak. Projects under "examples" directory should be copied to a writable directory in standard *nix system installs which is clumsy. We do things in clumsy way? From an online repo they will always go to a writable directory.
Some components will have their own examples included. Should the GUI support them somehow? I don't know. Those examples can always be opened manually.
These are my thoughts, other developers may see things differently.
BTW, currently the Lazarus Code and Component Repository (CCR) also has example projects.

@dbannon, you should fork Lazarus sources in GitLab and create an IDE integration package. The settings should be registered as a new page under the global options window. OPM should do the same thing BTW.
Your GUI must replace the existing Example Projects ... window. It can be merged to trunk (main branch) when everybody agrees. The trunk can be under construction sometimes, no problem.
If you want to maintain a standalone app, then you need 2 packages. One for the example manager functionality and one for registering the IDE integration. Both the standalone app and the IDE integration package would use the first package.
A standalone app may help with debugging, otherwise it has very little usage.
Later you will need more rights for Lazarus sources. You clearly have energy and motivation now for this project. It should not be wasted.

Juha

(*) I bought a 18" bar (X-Force) and chain (X-Cut) for my Jonsered CS2152 saw and did some lumber jack stuff as the picture proves:
 https://drive.google.com/file/d/1sNfKzB-1zmwUl9FWEWv2m3BUts9AXgpG/view
I also bought some forest. It is now or never. I am getting old. When I get very old, I will not be able to work in a forest any more.
Title: Re: Example projects window
Post by: Thaddy on February 15, 2022, 05:40:03 pm
Sorry for the delay, I had some other activities. (*)

(*) I bought a 18" bar (X-Force) and chain (X-Cut) for my Jonsered CS2152 saw and did some lumber jack stuff as the picture proves:
 https://drive.google.com/file/d/1sNfKzB-1zmwUl9FWEWv2m3BUts9AXgpG/view
I also bought some forest. It is now or never. I am getting old. When I get very old, I will not be able to work in a forest any more.
Serious power tool! Can I rent it? We have a piece of forest in Lithuania (inherited, 7 Ha) Should be right next door across the Baltic  :D
( Now playing "when I'am 64" have to wait till Monday....)
Title: Re: Example projects window
Post by: JuhaManninen on February 15, 2022, 06:44:18 pm
Serious power tool! Can I rent it? We have a piece of forest in Lithuania (inherited, 7 Ha) Should be right next door across the Baltic  :D
( Now playing "when I'am 64" have to wait till Monday....)
Wow!
I added a follow-up thread with details here :
 https://forum.lazarus.freepascal.org/index.php/topic,58345.0.html
Title: Re: Example projects window
Post by: ASBzone on February 15, 2022, 07:01:36 pm
Yeah, thanks Howard.  I understand your message.

I felt that this was/is the best approach but that I (or someone) needed to get some partially working model happening to demonstrate its practicality.

Davo

Awesome work, Davo

I agree with the need for dev support to move forward, but I also agree that what you did to get is jumpstarted was likely needed as well.
Title: Re: Example projects window
Post by: balazsszekely on February 15, 2022, 08:24:25 pm
@Davo

To save a few hours of work I attach a package. After compile/install/rebuild, you can access the main, for now empty, form via LazMenu->Tools->Online Example Projects(Ctr + Alt + E). What is now laz_examples(ExScanner) should go into main form and main_examples(ExScanner) should be an options dialog accessible from laz_examples aka. package main form. It sounds confusing, but you know what I meant to say.
Please consider adding a tree structure, replacing the listview, like in the attached picture, also master.ex-meta does not pass an online json validator. For translation purposes, each and every caption/text should be a resourcestring, please see uconst.pas for more details.


Other then this it's an excellent start. Good luck with the example project!


Title: Re: Example projects window
Post by: dbannon on February 16, 2022, 04:20:19 am
OK, thanks folks for some valuable (and encouraging) feedback.

Juha,
Sorry, I come from a background where a lot of emphasis was placed on planning, I need to see and "socialize" (an IBM term) my plan before proceeding. So -
1. Move my examples gitlab tree to somewhere under FPC/Lazarus. Might save everyone a lot of time if I am granted access to (just) that new tree. This is easily reversed if we decide against this model.
2. Replace Lazarus Examples Window in Lazarus with the new one - it will see only ~/examples/* (about 100). As you suggest, I'll bring that up in a fork. Or Getmem's package idea ?
3. Remove  examples from ~/examples. Leave five there, they will provide the initial "recent projects" list in a new install.
4. Start processing remaining examples under eg ~/components/*/examples|demos but leave 'tests' in place. Some tests are also usable as examples as well and these will be duplicated, many tests are not good examples and tests belong with their components.

Step 2 and 3 can happen after a lot of testing and will require developer merging of course. People who know their way around the src tree will still find the ~/components/*/examples but no so easily. But, as you say, its trunk !

Step 2 could be made a lot easier with GetMem's package suggestion, maybe thats a good approach, then, the transition would be considerably softened ?  Your thoughts please ?

Yep, agree 100% that most non-ObjectInspector examples  should go. In most cases, they are really only demonstrating how to to declare and create visual components anyway so rewriting is not useful.

CCR - my current scanner looks for the OPM metafiles in Lazarus Config. If a metafile is added to a CCR project and a user installs that project using OPM, then the scanner will find and list it. In all cases, IMHO, examples found in installed OPM projects should be copied to the defined space (also under Lazconfig dir) so the original code remains untouched.

Standalone App - no, it has no future beyond the transition. Good testbed, code will be reused and maybe a small tool to write good JSON metadata will emerge but whats there now is just for now.

GetMem
Wow, thats a big help and a very sensible approach, removes quite a lot of the transition problems. I will have a play.

ASBzone - Thanks !

Davo
Title: Re: Example projects window
Post by: dbannon on February 16, 2022, 08:53:02 am
...also master.ex-meta does not pass an online json validator.

Thats very, very messy. JSON does not allow newline characters in a JSON string, but, as luck would have it, our JSON parser (and JSONTools and, apparently many others) is quite happy with it. I'd be quite unhappy if this had gone on much longer before being spotted, thanks !

I'll have to convert to \\n and do some cross platform testing. Sadly, it negates my reason for choosing JSON, readable content. Tempting as it is, I cannot rely on a non-compliant feature of our parser. Sigh....

Quote
each and every caption/text should be a resourcestring
I have a few string literals in there that will need converting, most of the GUI content will be done automatically.  In the event that someone wants to translate the metadata, my plan is to have a separate files in each directory with a po file like 'es', 'ru' etc. The actual translation would be quite a big job and my guess is it will be quite selective, so the master metadata for each language will probably be mostly English unless someone gets really keen. Code changes to support it is minimal.

Quote
TreeView v. ListView.
Now, I am not sure about that. A treeview approach will require a user to click to expand each item to view it, the listview requires just a scroll. But I will definitly consider and maybe code up an example ....

Again, thanks for the feedback, especially picking up that bad json - its not a good situation at all.

I have tried out your "Package Template", that is really neat, makes it all look so easy ! Very grateful.

 Davo
Title: Re: Example projects window
Post by: balazsszekely on February 16, 2022, 10:17:10 am
@Davo

It's not yet clear what will happen with the example folder. Personally I would move it to an external server, but I'm not the one who decides such things. As always things are not black and white,  there are pros and cons no matter what solution is chosen. Anyways if the example folder remains part of Lazarus, you should reuse the existing Tools->Example projects dialog. If the folder is moved to a server, then the package approach is a far better one. Code moved from IDE to a package, will make Lazarus faster and slimmer, because a large part of the packages are only loaded in the bigide.

PS: Other implementation details can be discussed later.


Title: Re: Example projects window
Post by: wp on February 16, 2022, 10:37:36 am
Personally I would move it to an external server, but I'm not the one who decides such things. As always things are not black and white,  there are pros and cons no matter what solution is chosen.
One con of the external server is that examples will change with the Lazarus version (new examples added, deprecated ones removed, some edited due to changes in properties or FPC calling parameters etc). Keeping the examples within the Lazarus distribution makes sure that they always are opened by the correct version.
Title: Re: Example projects window
Post by: trev on February 16, 2022, 10:45:39 am
One con of the external server is that examples will change with the Lazarus version (new examples added, deprecated ones removed, some edited due to changes in properties or FPC calling parameters etc). Keeping the examples within the Lazarus distribution makes sure that they always are opened by the correct version.

Why not version the external server examples tree eg using version branches or tags or whatever. Lazarus should know its own version.
Title: Re: Example projects window
Post by: JuhaManninen on February 16, 2022, 11:57:18 am
It's not yet clear what will happen with the example folder. Personally I would move it to an external server, but I'm not the one who decides such things. As always things are not black and white,  there are pros and cons no matter what solution is chosen. Anyways if the example folder remains part of Lazarus, you should reuse the existing Tools->Example projects dialog. If the folder is moved to a server, then the package approach is a far better one. Code moved from IDE to a package, will make Lazarus faster and slimmer, because a large part of the packages are only loaded in the bigide.
Don't reuse the existing Tools->Example Projects dialog. It needed a big revamp anyways. Now the revamp will be thorough. Please remove the old GUI when adding the new one.
Yes, the new example projects GUI should be a package which registers itself under Tools menu and to global options window. We should make Lazarus more modular anyways. For example the Delphi converter should be a package, too.

Why not version the external server examples tree eg using version branches or tags or whatever. Lazarus should know its own version.
The same discussion now happens in Lazarus dev list and here, but no problem.
The examples must be tested with few recent versions. Typically examples are simple and libraries are more or less backwards compatible. I believe the examples can be made to work with older versions quite easily. No big problems there.
Yes, the examples should be under revision control where dbannon and current Lazarus devs have write access. I am not sure what is the right place.
Title: Re: Example projects window
Post by: dbannon on February 16, 2022, 01:13:06 pm
Personally, I think just noting, in the metadata, any constraint is more user friendly. If we enforce it, hiding or refusing to download for example, the user feels dis-empowered. Better to tell them about the issue and let them experiment with it if they like.

We would need to document just what the issue is anyway. And some constraints can be flexible.

"This example requires Lazarus 2.0.12 or later because it uses BlarBlar"

"This example works with Lazarus 2.0.8 and before if you uncomment the {.$DEFINE OLDLAZARUS}"

Davo
Title: Re: Example projects window
Post by: trev on February 16, 2022, 10:36:45 pm
"This example requires Lazarus 2.0.12 or later because it uses BlarBlar"
"This example works with Lazarus 2.0.8 and before if you uncomment the {.$DEFINE OLDLAZARUS}"

Why not simply omit the example from the versioned tree? It may be "fixed" later and re-included or it may simply disappear into the bit bucket of history.

Remember, this all began with a discussion about beginners and the included examples. You may already have restraints based on OS and in some cases OS version. No need to make it more complicated than necessary or add to the amount of labour required for continued maintenance which is something I often find to be overlooked (in a previous life in loose-leaf publishing, but the same issue applies here).
Title: Re: Example projects window
Post by: JuhaManninen on February 20, 2022, 10:18:03 am
It seems the example directory must stay there in Lazarus sources. It is OK, part of the projects can be moved from there gradually.
@dbannon, can you please create a GitLab fork with a GUI replacing the current Example Projects dialog, and options page registered to the global Options.
More people will pay attention then and maybe propose further improvements.
The final location for online examples can be discussed later. I have no opinion or preference for it.
Title: Re: Example projects window
Post by: dbannon on February 20, 2022, 10:48:10 am
OK, I guess a decision made elsewhere ?

Thats OK, no problem. I have a working package already but honestly, thats all down to the package template Getmem sent me, made it trivial. And we could have that working in parallel to existing one, so easy and safe transition. But no problems Juha, I'll fix it.

A more serious problem might be the fact that I cannot, totally dependably, get my https downloader working on MacOS. We cannot depend on the user having installed openssl and the Mac provided one is dodgy (IMHO) in earlier releases of MacOS.

Unless this can be sorted out, Examples on line is looking risky too I am afraid. Sigh...

Davo
Title: Re: Example projects window
Post by: JuhaManninen on February 20, 2022, 12:02:54 pm
OK, I guess a decision made elsewhere ?
No decision really. The discussion stalled after few comments. The best way to get things forward is to provide code to test. Let's try to get it into Lazarus trunk which is tested by many people.

Quote
Thats OK, no problem. I have a working package already but honestly, thats all down to the package template Getmem sent me, made it trivial. And we could have that working in parallel to existing one, so easy and safe transition. But no problems Juha, I'll fix it.
Package is good. I didn't study GetMem's package but the IDE integration at least should use a package. The existing Example Projects dialog is not a package. It can stay there in parallel if you want, but can also be removed.
Remember, this is not a critical feature. It can be under construction in trunk for a while. Remember also that the open source development model is basically meritocracy. Voting power goes up with accomplishments. You don't only need to take commands from others.

Quote
A more serious problem might be the fact that I cannot, totally dependably, get my https downloader working on MacOS. We cannot depend on the user having installed openssl and the Mac provided one is dodgy (IMHO) in earlier releases of MacOS.
Unless this can be sorted out, Examples on line is looking risky too I am afraid. Sigh...
I believe it can be sorted out. OPM works in MacOS, right? It can be temporarily broken for MacOS, no problem.
Title: Re: Example projects window
Post by: dbannon on February 20, 2022, 01:21:14 pm
OK, now I am afraid I don't understand what you are saying Juha.

Maybe if I read your message a bit more carefully.   :)

> It seems the example directory must stay there in Lazarus sources. It is OK, part of the projects can be moved from there gradually.

OK, so you mean the dir (and its contents) must stay in the Laz SRC tree. But not necessarily be used by the new Examples Window ? Fine by me. Maybe a good idea to update it with the work I have done on the Examples but no rush.

> @dbannon, can you please create a GitLab fork with a GUI replacing the current Example Projects dialog, and options page registered to the global Options.
> Package is good.

I really don't understand that one. I do not need to fork Lazarus to make a package, Lazarus already has all the support I need built in, nothing need change in Lazarus at this stage. Except, perhaps, to intercept a user click on the Project Wizard -> View Examples Project.

In https://gitlab.com/dbannon/laz_examples/-/tree/main/Utility/pkg you will find a working Package that adds a menu item to the Tools menu, it opens an Examples Window that does most of what we need. Copy the files somewhere and open the lpk file.

Unless, of course, if you are using a Mac.

I have no idea how to connect that Package to the Project Wizard but as a try out now, its up and running via the Tools Menu.

> The existing Example Projects dialog is not a package.

No, it was a test bed for some code and some ideas. Please make out I never mentioned it.

Davo

Title: Re: Example projects window
Post by: JuhaManninen on February 20, 2022, 04:07:32 pm
OK, so you mean the dir (and its contents) must stay in the Laz SRC tree. But not necessarily be used by the new Examples Window ?
As long as the directory stays there, it should be used by the new Examples Window.

Quote
I really don't understand that one. I do not need to fork Lazarus to make a package, Lazarus already has all the support I need built in, nothing need change in Lazarus at this stage. Except, perhaps, to intercept a user click on the Project Wizard -> View Examples Project.

In https://gitlab.com/dbannon/laz_examples/-/tree/main/Utility/pkg you will find a working Package that adds a menu item to the Tools menu, it opens an Examples Window that does most of what we need. Copy the files somewhere and open the lpk file.
They must be copied under the Components directory at some point. It makes things easier if you already have them in the correct place in your GitLab fork. There will be Makefiles which need help from others etc.
Then merging to Lazarus trunk can be done easily.
I can look at your latest package version next week and add it to trunk. Do you have the options page registration there? Once it is in trunk, you must continue with a GitLab Lazarus fork anyways.
(Sorry, I am not at my development computer now.)

Quote
I have no idea how to connect that Package to the Project Wizard but as a try out now, its up and running via the Tools Menu.
Which Project Wizard you mean? The Project -> New Project dialog?
Title: Re: Example projects window
Post by: dbannon on February 21, 2022, 03:19:58 am
Juha, I'll make some more readable notes on https://wiki.freepascal.org/Lazarus_Examples_Window but for now, this is how I see where we are.

As long as the directory stays there, it should be used by the new Examples Window.
OK, some issues. My version of the Examples Dir has a number of changes, if they are going to be used, we need to decide which of these changes need be pushed up to the fork -
1. The metadata files - thats a must.
2. Directory Structure, I have built a dir structure based on Category just to make it more manageable. Its not necessary, just tidier.
3. Changes made to examples. Strongly Recommended. See note A.

So, easiest approach for me and probably best outcome for Lazarus is for me to replace all the Examples in ~/examples with my set but leave out my new directory structure. So, implement 1 and 3 above. I'll reverse the directory restructure (because it will confuse the existing Examples Window). This will be first merge request. It should not affect anything except removing misleading examples  :-\

Then I will need to make some changes to new Examples Window so that it looks to the examples in the local directory rather than on-line. Easy, I'll use some {$ifdef so we don't loose (yet) the online capability. This solves my MacOS problem too. 

If, at some later stage, we do decide to go online, the underlying structure will still be there.

I'll then push my new Examples Window package up, it will appear directly under ~/components and, if installed, will only change the tools menu but will provide improved access to ~/examples and will solve the read only issue (because when a user chooses to first open an example, we'll make a working copy in user space). This will be the second merge.

Then, all the other examples (eg under ~/components) will need to be tested and have metadata added (they will remain where they are in the Laz src tree but will also be copied to user space when needed). This will be quite long series of merge requests.
 
Quote
They must be copied under the Components directory at some point...... options page registration ?

OK, right now, its a Package, its gets everything it needs from LazarusIDE variable, it does not need any options set at present (see note B). So, no, no "options page registration". I have no idea what that means by the way.

Quote
Which Project Wizard you mean? The Project -> New Project dialog?
Yep, thats what I mean.  its caption is "Project Wizard" and it currently has a Button that triggers the current Examples Window. Maybe that can be done with the LazarusIDE variable, it seems to be amazingly flexible.

Note A - A number of Examples have been edited, updated or skipped because they are no longer useful. For example some Examples needed extra units "used" as things have been moved. Many Examples have been skipped because they pre-dated the Object Inspector. Some Examples are still marked as needing some more work. That was substantial amount of effort and was needed for any of the outcomes. While it does improve the overall quality, more needs to be done. So far, the Examples have only been tested on Linux, they will require testing on Windows and MacOS too.

Note B - The new Example Window reads things like LazConfigDir from the LazarusIDE variable. Will need to read LazSrcDir. At present it has no options of its own but at least one person has requested the ability to direct downloaded (or copied) Examples to a directory of their choice rather than automatically under LazConfigDir. I assumed this would a small config screen associated with the new Examples Window, to be done later.  Is this what you are referring to as "options page registration" ?

Title: Re: Example projects window
Post by: JuhaManninen on February 21, 2022, 08:49:04 am
I'll reverse the directory restructure (because it will confuse the existing Examples Window).
Just remove the existing one as I initially suggested. I don't know why you want to keep it there.

Quote
Then I will need to make some changes to new Examples Window so that it looks to the examples in the local directory rather than on-line. Easy, I'll use some {$ifdef so we don't loose (yet) the online capability. This solves my MacOS problem too. 
If, at some later stage, we do decide to go online, the underlying structure will still be there.
I think online is the way to go. Supporting both local and online examples would be best.

Quote
Note B - The new Example Window reads things like LazConfigDir from the LazarusIDE variable. Will need to read LazSrcDir. At present it has no options of its own but at least one person has requested the ability to direct downloaded (or copied) Examples to a directory of their choice rather than automatically under LazConfigDir. I assumed this would a small config screen associated with the new Examples Window, to be done later.  Is this what you are referring to as "options page registration" ?
Yes, a package that integrates in the IDE and needs settings should add a configuration page (or pane) to the global options window, Tools->Options.
IdeIntf has an API for it. I can find an example package that already does it later. ToDoList maybe...
Title: Re: Example projects window
Post by: dbannon on February 21, 2022, 10:48:20 am
Just remove the existing one as I initially suggested. I don't know why you want to keep it there.
OK, I'll do a fork, remove existing examples, replace it with my version of that directory and merge. A pull request or a diff file ?

Quote
I think online is the way to go. Supporting both local and online examples would be best.
Yep, thats what I'll do.

Quote
.... settings .... Tools->Options.
IdeIntf has an API for it. I can find an example package that already does it later. ToDoList maybe...
When you get a chance, yes please. But its not holding me up now.

What is holding me up right now is I cannot find a way to read the LazarusDirectory from a package. Must be possible but right now, I am considering reading LazConfDir/EnvironmentOptions.xml - and I'll watch to see who laughs !

Davo
Title: Re: Example projects window
Post by: balazsszekely on February 21, 2022, 12:01:19 pm
Quote
What is holding me up right now is I cannot find a way to read the LazarusDirectory from a package. Must be possible but right now, I am considering reading LazConfDir/EnvironmentOptions.xml - and I'll watch to see who laughs !
Nobody gonna laugh. You can read LazarusDirectory form EnvironmentOptions.xml like this:
Code: Pascal  [Select][+][-]
  1. uses LazIDEIntf, LazFileUtils, Laz2_XMLCfg;
  2.  
  3. function GetLazarusDirectory: String;
  4. var
  5.   XML: TXMLConfig;
  6.   ConfigPath: String;
  7. begin
  8.   Result := '';
  9.   ConfigPath := AppendPathDelim(LazarusIDE.GetPrimaryConfigPath) + 'environmentoptions.xml';
  10.   XML := TXMLConfig.Create(ConfigPath);
  11.   try
  12.     Result := XML.GetValue('EnvironmentOptions/LazarusDirectory/Value', '');
  13.     if Trim(Result) <> '' then
  14.       Result := AppendPathDelim(Result);
  15.   finally
  16.     XML.Free;
  17.   end;
  18. end;

also since the package is running inside the IDE:
Code: Pascal  [Select][+][-]
  1. LazarusDirectory := ExtractFilePath(ParamStr(0));
should be enough.
Title: Re: Example projects window
Post by: dbannon on February 21, 2022, 01:11:28 pm
OK thanks Getmem, I have just done something similar but, as I'd expect, yours is shorter and neater. I'll need to sit and have a good read of that, has stuff I am unaware of. TXMLConfig.  Sigh...

I'll skip the ParamStr(0) approach because its frowned upon in the Unix world. 

Thanks, I feel a lot better now (that I know no one is laughing) !

Davo  :D

Title: Re: Example projects window
Post by: PascalDragon on February 21, 2022, 01:22:38 pm
also since the package is running inside the IDE:
Code: Pascal  [Select][+][-]
  1. LazarusDirectory := ExtractFilePath(ParamStr(0));
should be enough.

If the Lazarus executable was rebuild to a different location, because the original location is readonly, this will not be true.
Title: Re: Example projects window
Post by: balazsszekely on February 21, 2022, 01:40:03 pm
@PascalDragon
Quote
If the Lazarus executable was rebuild to a different location, because the original location is readonly, this will not be true.
The option dialog does the same(IDE/frames/file_options.pas):
Code: Pascal  [Select][+][-]
  1. LazarusDirComboBox.Items.Add(ProgramDirectoryWithBundle);  
where ProgramDirectoryWithBundle is a function in fileutil(lazutils), which uses function ProgramDirectory, which begins with:
Code: Pascal  [Select][+][-]
  1. Result:=ParamStrUTF8(0)

Title: Re: Example projects window
Post by: PascalDragon on February 21, 2022, 01:44:59 pm
But that dialog uses that only to fill in the suggestions.
Title: Re: Example projects window
Post by: JuhaManninen on February 22, 2022, 09:59:13 am
The API should have a way to get LazarusDirectory directly. I think it is a missing feature.
Or does it impose some potential problem?
Title: Re: Example projects window
Post by: balazsszekely on February 22, 2022, 10:49:21 am
@JuhaManninen
Quote
The API should have a way to get LazarusDirectory directly. I think it is a missing feature.
Yes. LazarusIDE from LazIDEIntf already expose GetPrimaryConfigPath, shouldn't be too difficult to add LazarusDirectory.
Although in my opinion the best place to put LazarusDirectory is TIDEEnvironmentOptionsfrom form IDEOptionsIntf, which is the base class for TEnvironmentOptions

Quote
Or does it impose some potential problem?
No potential problem AFAICS. It's just a getter.
Title: Re: Example projects window
Post by: dbannon on February 22, 2022, 11:39:08 am
The API should have a way to get LazarusDirectory directly. I think it is a missing feature.
Or does it impose some potential problem?

Yes, I agree. If nothing else, I spent quite some time going through that API because I was sure it should show LazarusDirectory and must have been overlooking it. Perhaps, I thought, it has some other name ....

Davo
Title: Re: Example projects window
Post by: JuhaManninen on February 22, 2022, 03:12:47 pm
Quote
.... settings .... Tools->Options.
IdeIntf has an API for it. I can find an example package that already does it later. ToDoList maybe...
When you get a chance, yes please. But its not holding me up now.
Ok, ToDoList does not have it. All its settings are on the main GUI form.
Jedi Code Format package has it. The initialization section of unit frFiles has:
  RegisterIDEOptionsEditor(JCFOptionsGroup, TfFiles, JCFOptionFormatFile);

Please search for "RegisterIDEOptionsEditor" under Components directory. They are plenty.

Quote
What is holding me up right now is I cannot find a way to read the LazarusDirectory from a package. Must be possible but right now, I am considering reading LazConfDir/EnvironmentOptions.xml - and I'll watch to see who laughs !
Can you please add the LazarusDirectory getter. True, it should be in class TIDEEnvironmentOptions which is part of BuildIntf. This setting has no dependency for GUI and may be needed by cmd line tools as well.
Maybe GetPrimaryConfigPath and GetSecondaryConfigPath should be there, too, but that is another issue.
Title: Re: Example projects window
Post by: dbannon on February 23, 2022, 05:43:13 am

Can you please add the LazarusDirectory getter. True, it should be in class TIDEEnvironmentOptions which is part of BuildIntf. This setting has no dependency for GUI and may be needed by cmd line tools as well.
Maybe GetPrimaryConfigPath and GetSecondaryConfigPath should be there, too, but that is another issue.

OK, TIDEEnvironmentOptions is defined in IDEOptionsIntf.pas And a var IDEEnvironmentOptions of that type is declared.

In ide/EnvironmentOpts.pp we  define
 TEnvironmentOptions = class(TIDEEnvironmentOptions)
and add the property LazarusDirectory, along with quite a lot of other useful stuff.

We then declare a var EnvironmentOptions of that type.

But in ide/BuildManager.pas we go backwards, we create the (EnvironmentOpts.pp)EnvironmentOptions var and assign it to the (IDEOptionsIntf.pas) IDEEnvironmentOptions var so losing access to LazarusDirectory.

Now, we cannot (easily) see either  ide/BuildManager.pas nor  ide/EnvironmentOpts.pp so need to get by with the quite uninformed  TIDEEnvironmentOptions.

From where I sit, it would make more sense to move "LazarusDirectory" to the parent, TIDEEnvironmentOptions, but that raises questions about the Get and Set functions.

Easier to add a "TheLazarusDirectory" (as a string ??, no such thing as an abstract property ? ) to TIDEEnvironmentOptions and do
    IDEEnvironmentOptions.TheLazarusDirectory  := EnvironmentOptions.LazarusDirectory;
in BuildManager, ugly IMHO.

So, must be a better approach, what ?

Davo
Title: Re: Example projects window
Post by: balazsszekely on February 23, 2022, 07:24:55 am
@davo

It turned out that is already a function named GetParsedLazarusDirectory in EnvironmentOpts. I made it available for packages in trunk/main. Please use it like this:
Code: Pascal  [Select][+][-]
  1. uses IDEOptionsIntf;
  2.  
  3. var
  4.   LazarusDirectory: String;
  5. begin
  6.   LazarusDirectory := IDEEnvironmentOptions.GetParsedLazarusDirectory;        
  7.   //...
  8. end;

PS: Sorry for the confusion!
Title: Re: Example projects window
Post by: dbannon on February 24, 2022, 01:16:45 pm
@davo

It turned out that is already a function named GetParsedLazarusDirectory in EnvironmentOpts. I made it available for packages in trunk/main. Please use it like this:
Code: Pascal  [Select][+][-]
  1. uses IDEOptionsIntf;
  2.  
  3. var
  4.   LazarusDirectory: String;
  5. begin
  6.   LazarusDirectory := IDEEnvironmentOptions.GetParsedLazarusDirectory;        
  7.   //...
  8. end;

PS: Sorry for the confusion!

OK, that seems to work. Thanks Getmem, thats a lot tidier solution. I looked for a solution there but clearly did not have a good enough look. Will have to examine your handicraft.

Sorry I have been so slow to answer, discovering the joys of CamelCase in a case sensitive OS. Going to have to revisit my Examples list .....

Davos
Title: Re: Example projects window
Post by: PascalDragon on February 25, 2022, 03:46:58 pm
Sorry I have been so slow to answer, discovering the joys of CamelCase in a case sensitive OS. Going to have to revisit my Examples list .....

Yes, that's very much a joy. That's why Lazarus stores units by default as lowercase and FPC searches for as-is, lowercase and uppercase. ;)
Title: Re: Example projects window
Post by: dbannon on February 27, 2022, 01:02:36 am
Yes, its a cute issue.

Many examples have camelcase names, makes it heaps easier to understand so I wanted to preserve that. But that name also needs to be reflected in the project directory. So, I now require the spelling but not the case to match, and have a custom DirectoryExits function that is NOT case sensitive for the last term. Works ...

Juha, I have a forked Lazarus gitlab repo, it now contains my worked over ~/examples directory.  How do you suggest I submit that 'stage one' to Lazarus, as a pull request or an old school diff file ? 

I have the 'package' that provides the new Lazarus Examples Window all working nicely with local examples (instead of online as you suggested). All it needs now is to pick up its one necessary config item (alt place to store working examples) from the lazarus config system, you have sent me a pointer to do that so its just a matter of finding time.

But maybe a policy matter ?  Its currently called Online Examples (in, eg, the menu item).  As its not exclusively OnLine and may never be, is a different name appropriate ?  I don't want to call it "New Examples" and at some stage it will be old, any suggestions ?

Davo
Title: Re: Example projects window
Post by: JuhaManninen on February 27, 2022, 12:22:07 pm
Juha, I have a forked Lazarus gitlab repo, it now contains my worked over ~/examples directory.  How do you suggest I submit that 'stage one' to Lazarus, as a pull request or an old school diff file ? 
I will look at it tomorrow. Today I have other things going on...

Quote
But maybe a policy matter ?  Its currently called Online Examples (in, eg, the menu item).  As its not exclusively OnLine and may never be, is a different name appropriate ?  I don't want to call it "New Examples" and at some stage it will be old, any suggestions ?
"Example projects" is still very descriptive.
Title: Re: Example projects window
Post by: dbannon on February 27, 2022, 11:38:17 pm
No rush Juha, we are traveling at present so my time is limited too.

"Example projects" is still very descriptive.

OK, from that I assume that "Stage Two" will have to include removal of the existing Example Window, it already owns "Example projects".  I was planning on being a bit more incremental than that, but quite happy to go boots and all.

My fork with the worked over examples is at https://gitlab.com/dbannon/lazarus/-/tree/newexamples, its a branch of my main fork that can track the official one.

I'll bring up another branch to accept the Example Window changes.

Davo 
Title: Re: Example projects window
Post by: JuhaManninen on March 02, 2022, 04:02:50 pm
OK, from that I assume that "Stage Two" will have to include removal of the existing Example Window, it already owns "Example projects".  I was planning on being a bit more incremental than that, but quite happy to go boots and all.
IMO the old Example Window can be removed at the same time the new one is added. Two competing Example Windows will only confuse users.
The idea is that people test the new one in trunk. The old one can be found in revision history any time.

Quote
My fork with the worked over examples is at https://gitlab.com/dbannon/lazarus/-/tree/newexamples, its a branch of my main fork that can track the official one.
Sorry but https://gitlab.com/dbannon/lazarus/ does not exist. Under https://gitlab.com/dbannon/ I only see your Laz_Examples project.
Maybe I cannot use Gitlab properly yet. How to get to the Lazarus fork?
We could also give you rights to the main Lazarus repo and then create a Git branch for your work. It would work as any Git branch, Gitlab actions would not be needed.
Title: Re: Example projects window
Post by: dbannon on March 03, 2022, 12:22:59 am
Oh Dear, I suspect I ave been caught again by by GitLab's model of making new repos PRIVATE by default. In Github its always public initially.  I'll check it shortly.

Sorry ....

Davo 

EDIT : yes, it was 'Private', seems that is the GitLab default. Its marked public now.

> IMO the old Example Window can be removed

Yep, thats how I am approaching it now. Just a bit preoccupied at present. Will get there ....

Davo
Title: Re: Example projects window
Post by: dbannon on March 04, 2022, 11:54:28 am
OK, the Package now works with the Global Options Window. Not an easy task understanding that I must admit.

I will push content up into https://gitlab.com/dbannon/lazarus when we have made some progress with the Examples them selves.  I don't want to make one huge merge, would be a bit scary !

It will then be easier to strip out existing Examples Window.

Davo


Title: Re: Example projects window
Post by: JuhaManninen on March 04, 2022, 03:33:56 pm
OK, the Package now works with the Global Options Window. Not an easy task understanding that I must admit.
I will push content up into https://gitlab.com/dbannon/lazarus when we have made some progress with the Examples them selves.
How am I supposed to test your code if you don't push it?
I cloned your Lazarus fork repo and found the newexamples thread with 2 commits for the actual example projects. No other commits from you anywhere.

Quote
I don't want to make one huge merge, would be a bit scary !
Merge into where? The only purpose of your Lazarus fork in Gitlab is to let other people test it. You can do local experiments without forking by just cloning the official Lazarus Gitlab repo. I don't fully understand what is going on.

Quote
It will then be easier to strip out existing Examples Window.
It can be stripped out any time. I can do it for you if you want. I don't know why you are so attached to it.
Title: Re: Example projects window
Post by: dbannon on March 05, 2022, 12:20:31 am
Sigh, we really don't seem to be communicating very well here.  I thought I had discussed the model and you were comfortable with it. But obviously not !

The 'dbannon/lazarus/newexamples' branch has all the examples projects in ~/examples cleaned up and metadata added. While two commits it does represent quite a lot of change (and work).

It now sounds to me like you want to see everything, the complete transform, all in that branch ?  Sure, I can do that, almost immediately.  That will be everything EXCEPT removal of existing Examples Window, I'll start on that next.

Quote
It can be stripped out any time. I can do it for you if you want. I don't know why you are so attached to it.

OK, OK, I'll do that next.

I was trying to get as much as possible done that did not alter core LazarusIDE code. Thats because its better to base a branch on as recent as possible main to avoid merge conflicts. My plan was to merge the Examples, merge the Examples Package and then branch off for the parts that require changes to core LazarusIDE.

As it is, newexamples is already ten days old. Its likely, people will have made some changes to the files I will need to change since then. And I will need several more days to remove the old Examples Window. Lets hope its still a reasonably easy merge.

Davo

EDIT : OK, dbannon/lazarus/newexamples now contains the code to display the new Examples Window. Its gets its one solitary bit of config from the Lazarus Options Window.  But I have, at this stage -

1) Not removed the old Examples Window, it will appear if you click the button on the Project Wizard or the upper "Example Projects" menu item under the Tools Menu.

2) Not added this Package to the Install/Uninstall Packages Window, I don't yet know how to.  To install this package, you should use Package -> Open Package File (.lpk) and browse to <lazarusDir>/components/examplesWindow/exampleprojects.lpk

3) Not extensively tested on anything except a Lazarus installed from SRC on Linux. There are bound to be dragons....

4) Not tested or provided metadata for the many examples that appear under directories other than ~/examples

I will address those four points as my free time permits.

Davo
Title: Re: Example projects window
Post by: JuhaManninen on March 05, 2022, 02:12:40 pm
Sigh, we really don't seem to be communicating very well here.  I thought I had discussed the model and you were comfortable with it. But obviously not !
The 'dbannon/lazarus/newexamples' branch has all the examples projects in ~/examples cleaned up and metadata added. While two commits it does represent quite a lot of change (and work).
Yes, I believe there was some work involved. I was eager to see the user experience. Now I pulled you fork and yes, I can see a commit "New Examples Window, old one still exists". I will test it soon ...
Thanks for your efforts so far.
I don't think anybody else modified the example projects recently meaning your changes can be merged easily. You can always check it by rebasing your branch to the latest "main" branch HEAD.
Title: Re: Example projects window
Post by: JuhaManninen on March 05, 2022, 04:36:29 pm
OK, I built Lazarus with the new package. It shows a GUI but no example projects are listed yet. No problem, it is under construction.
True, the old GUI can stay there until the new one shows the local example projects.

I went through the actual examples. Looks good. I created patches from your Git branch and applied the 2 first ones to my development repo. No conflicts, good.
I created a new patch from my repo and uploaded it here :
 https://gitlab.com/freepascal.org/lazarus/lazarus/-/issues/35446
to make it easy for anybody to test. The patch includes your original "metadata tweaks" commit.
Shall we apply it to Lazarus trunk (main branch) soon? I will wait for comments from others a while.
Title: Re: Example projects window
Post by: avra on March 06, 2022, 12:56:09 am
Does it make sense to have screenshot of some example projects? If it does, then you could check for screenshot.png in project directory and show it.
Title: Re: Example projects window
Post by: JuhaManninen on March 06, 2022, 07:57:34 am
Does it make sense to have screenshot of some example projects? If it does, then you could check for screenshot.png in project directory and show it.
Yes, a good idea!
Title: Re: Example projects window
Post by: dbannon on March 06, 2022, 10:23:44 am
It shows a GUI but no example projects are listed yet.

OK, thats bad. You definitely should see 104 example projects (and fully functional examples window, view metadata, copy a project to a working area, open that project in lazarus).

I see no obvious reason why you would not, so have to assume I have overlooked 'something', but cannot replicate it here. There should be a debug statement that will drop three lines (to stdio, I am guessing you are on Linux Juha ??) when you open the Examples Window.  In my case, like this -

Code: Pascal  [Select][+][-]
  1. TExampleData.LoadExData - found 0 examples
  2. Lazarus Dir (ie source tree) = /home/dbannon/bin/Lazarus/lazarus-drb/
  3. Lazarus Config Dir = /home/dbannon/bin/LazConfig/lazarus-drb/
  4. Examples Home Dir  = /home/dbannon/bin/LazConfig/lazarus-drb/examples_work_dir/

Make sure those first two directories are valid, they may give me an idea whats happening. The first one should point to the top of your Lazarus source tree, there should be the ~/examples dir there and all the examples below that.

(Just to be sure, I have just pushed a small change so that the debug statements appear always, not just when there are no projects found. You could 'pull' and re-add the package to the IDE.)

Shall we apply it to Lazarus trunk (main branch) soon? I will wait for comments from others a while.

Well, as I indicated previously, I believe the examples can and should be committed now. They still work with the old Examples Window but have a number of problem ones removed and a few problems fixed.  Right now, the new Examples Window code is all there as an uninstalled package so does no harm and its easily removed if the overall project is rejected. But, as you noted, it has at least one bug !  So, lets get to bottom of that first.

Davo
 
Title: Re: Example projects window
Post by: dbannon on March 06, 2022, 10:35:25 am
Does it make sense to have screenshot of some example projects? If it does, then you could check for screenshot.png in project directory and show it.

Yes, good idea but maybe not now while the examples are being distributed as part of the Source ?  I am thinking of the extra size we would be adding to the source tree, we could double it.  But would sure help a user determine if its the project that answers their particular question.

Davo
Title: Re: Example projects window
Post by: JuhaManninen on March 06, 2022, 01:03:18 pm
Code: [Select]
TExampleData.LoadExData - found 0 examples
Lazarus Dir (ie source tree) = /home/dbannon/bin/Lazarus/lazarus-drb/
Lazarus Config Dir = /home/dbannon/bin/LazConfig/lazarus-drb/
Examples Home Dir  = /home/dbannon/bin/LazConfig/lazarus-drb/examples_work_dir/

My version:
Code: [Select]
TExampleData.LoadExData - found examples = 1
Lazarus Dir (ie source tree) = ../SW/lazarus/
Lazarus Config Dir = /home/juha/.lazarus/
Examples Home Dir  = /home/juha/.lazarus/examples_work_dir/
My Lazarus sources are in ~/SW/lazarus. I start the executable directly from there as "./lazarus &".
"../SW/lazarus/" is wrong. It should be "../lazarus/" or just "."

Quote
(Just to be sure, I have just pushed a small change so that the debug statements appear always, not just when there are no projects found. You could 'pull' and re-add the package to the IDE.)
It was useful. I test now with my main development repo (instead of your fork), and the GUI finds one example, Beginner/listview. Maube it came from testing your standalone app earlier.

Is anybody else testing this? Please do.

Yes, good idea but maybe not now while the examples are being distributed as part of the Source ?  I am thinking of the extra size we would be adding to the source tree, we could double it.  But would sure help a user determine if its the project that answers their particular question.
True, we must not bloat the distribution more. Instead we should move more examples to an online place.
For an online project a screenshot would be useful. It is a good use case for a thread. The GUI would download a screenshot in background and show it when ready.
Title: Re: Example projects window
Post by: dbannon on March 07, 2022, 12:51:34 am

EDIT:  I normally put a lazarus.cfg file in my lazarus installs, has a --PCP= setting to make sure I don't cross connect.

If I remove that and let it use the default ~/.lazarus config dir, the files in there all have relative paths !  Gives me somewhere to start experimenting from !





Lazarus Dir (ie source tree) = /home/dbannon/bin/Lazarus/lazarus-drb/

My version:
Code: [Select]
Lazarus Dir (ie source tree) = ../SW/lazarus/My Lazarus sources are in ~/SW/lazarus. I start the executable directly from there as "./lazarus &".
"../SW/lazarus/" is wrong. It should be "../lazarus/" or just "."

IMHO, it should be (eg) /home/juha/SW/lazarus/  I prefer full paths to avoid problems with Lazarus internally changing default directory. But you are getting a relative directory, puzzling. I read the Lazarus Dir directly from environmentoptions.xml and I see this (from the LazConfig dir) on my system -

Code: Bash  [Select][+][-]
  1. dbannon@dell:~/bin/LazConfig/lazarus-drb$ cat environmentoptions.xml | grep LazarusDirectory
  2.     <LazarusDirectory Value="/home/dbannon/bin/Lazarus/lazarus-drb">
  3.     </LazarusDirectory>

I am guessing you have a relative path there. While its easy to convert that to absolute, if Lazarus has already changed its default dir (why would it ?) it will still get the wrong answer.  So, I have two rhetorical questions -
1. Why do you have a relative path in there and I have absolute ? I have five different installs of Lazarus and they all have absolute paths to LazarusDirectory.
2. Why is Lazarus changing its working dir from what it was when that relative path was written  ?  Thats going to break other things too IMHO. 

I added ResolveDots() to the code that reads environmentoptions.xml and I manually edited environmentoptions.xml to have a relative path ../. and then started Lazarus. The initial Configure Lazarus screen popped up and put it back the way it was !  I do not understand why your system is behaving so differently.

(I can now push up a version that does not have the older Examples Window, starts mine from ProjectWizard etc. But would rather wait until the above issue is clarified)

Davo





Title: Re: Example projects window
Post by: dbannon on March 07, 2022, 05:43:19 am
OK, this is pretty strange, my Lazarus Src is in ~/bin/Lazarus/lazarus-drb
Code: Bash  [Select][+][-]
  1. dbannon@dell:~/bin/Lazarus/lazarus-drb$ ls -l ~/bin/Lazarus/lazarus-drb
  2. total 306696
  3. drwxrwxr-x 87 dbannon dbannon      4096 Mar  5 10:23 components
  4. drwxrwxr-x  2 dbannon dbannon      4096 Feb 24 09:24 converter
  5. -rw-rw-r--  1 dbannon dbannon     18088 Feb 24 09:24 COPYING.GPL.txt
  6. .....
  7. dbannon@dell:~/bin/Lazarus/lazarus-drb$ pwd
  8. /home/dbannon/bin/Lazarus/lazarus-drb
  9.  
  10. dbannon@dell:~/bin/Lazarus/lazarus-drb$ rm -Rf ~/.lazarus
  11.  

And thats where I am and I removed the default lazarus config files. So, I start lazarus, let it create new config, exit and have a look at LazrusDir in ~/.lazarus/environmentoptions.xml 
Code: Bash  [Select][+][-]
  1. dbannon@dell:~/bin/Lazarus/lazarus-drb$ ./lazarus
  2. Hint: (lazarus) [TMainIDE.ParseCmdLineOptions] PrimaryConfigPath="/home/dbannon/.lazarus"
  3. Hint: (lazarus) [TMainIDE.ParseCmdLineOptions] SecondaryConfigPath="/etc/lazarus"
  4. ....
  5.  
  6. dbannon@dell:~/bin/Lazarus/lazarus-drb$ grep LazarusDir ~/.lazarus/environmentoptions.xml
  7.     <LazarusDirectory Value="../bin/Lazarus/lazarus-drb">
  8.     </LazarusDirectory>
  9. dbannon@dell:~/bin/Lazarus/lazarus-drb$ ls ../bin/Lazarus/lazarus-drb
  10. ls: cannot access '../bin/Lazarus/lazarus-drb': No such file or directory
  11.  
  12. dbannon@dell:~/bin/Lazarus/lazarus-drb$ ls -l ../lazarus-drb
  13. total 306696
  14. drwxrwxr-x 87 dbannon dbannon      4096 Mar  5 10:23 components
  15. drwxrwxr-x  2 dbannon dbannon      4096 Feb 24 09:24 converter
  16.  

Its actually, in this case, two directories out !

But if we force a more complicated LazConfig dir and Lazarus decides not to try to save a relative path, all is well -
Code: Bash  [Select][+][-]
  1. dbannon@dell:~/bin/Lazarus/lazarus-drb$ ./lazarus --pcp=~/bin/LazConfig/lazarus-test
  2. ....
  3. dbannon@dell:~/bin/Lazarus/lazarus-drb$ grep LazarusDir ~/bin/LazConfig/lazarus-test/environmentoptions.xml
  4.     <LazarusDirectory Value="/home/dbannon/bin/Lazarus/lazarus-drb">
  5.     </LazarusDirectory>

So, the problem definitely relates to the conversion to a relative path, a conversion that is, IMHO, quite unnecessary. I am seeing the same thing with Lazarus 2.2.0

Anyone have another view ?

Davo







Title: Re: Example projects window
Post by: JuhaManninen on March 07, 2022, 07:14:32 am
Yes, it was a bug in Lazarus code. I fix it in 9f142eef14.
I had added the relative path myself but the IDE accepted only a false one because of the bug. Now it works.
I guess everybody has used absolute paths so far.

Some notions about the examples GUI:
- Projects also from my writable Lazarus source directory are copied to the config dir. They could be compiled in place.
- The Key Words column for some projects is so wide that the next Path column shifts far away. You cannot even adjust the width manually.
- The Category CheckBoxes could all be ON initially. Now they are OFF but all items in the list show anyways.
- You could consider using TListViewFilterEdit as a filter control.
- The source files don't have any license headers. (L)GPL requires such a header.

Is there a way to download online examples? I guess not yet.
Title: Re: Example projects window
Post by: dbannon on March 07, 2022, 11:18:13 am
Yes, it was a bug in Lazarus code. I fix it in 9f142eef14.
OK, so I will definitely need to do some rebasing now  :-\

- Projects also from my writable Lazarus source directory are copied to the config dir. They could be compiled in place.
Thats intentional. It means that you can have a play with the code, maybe mess it up completely and refresh from the original and start again. Is one of my KPIs.

- The Key Words column for some projects is so wide that the next Path column shifts far away. You cannot even adjust the width manually.
Yes, I am not that happy about that, should be able to make it adjustable but not much more can be done as far as I can see. Truncating the keywords seems worse than hiding the path, in practice, no one really needs to see the path anyway. Maybe better to not display it at all ?

- The Category CheckBoxes could all be ON initially. Now they are OFF but all items in the list show anyways.
Ah, thats an historical accident, one I should have fixed long ago.

- You could consider using TListViewFilterEdit as a filter control.
Hmm, never heard of it before now. Almost no docs. The filtering happens in two separate ways (Category and Keyword) and is performed by the list I maintain in the data unit. I am not sure it can be done in the UI space.

- The source files don't have any license headers. (L)GPL requires such a header.
Easily fixed.

Is there a way to download online examples? I guess not yet.
Not easily. When you suggested sticking to keeping the examples in the src tree, I concentrated on that approach. The on-line code is in there hidden with some $ifdefs and can be brought back and updated in light of what I know now.  But we cannot use httpclient on MacOS to download https files so until that situation improves, I don't see it as a good approach. https://forum.lazarus.freepascal.org/index.php/topic,58392.0.html

I will address the things you mention, rebase and see what it all looks like. I have a src tree here now with all the old Examples Window and its infrastructure stripped out. But will need to do that rebase I suspect.

Davo


Title: Re: Example projects window
Post by: JuhaManninen on March 07, 2022, 12:07:48 pm
OK, so I will definitely need to do some rebasing now  :-\
Yes, run "git rebase main". Then force push to GitLab.
Anybody who has cloned your fork repo must then delete their local "newexamples" branch and create it again. No big deal. I think I am the only one who has done that.
Title: Re: Example projects window
Post by: dbannon on March 07, 2022, 12:17:11 pm
OK, rebase worked like a charm. I have used it in the past and regretted it (indeed, I tared up my working dir before I started). The changes to remove the old Examples Window is not really extensive but it does touch a good number of files. I will rebase frequently from now on ....

So, its usable now.  I'll address at least some of the matters you mention if I have time tomorrow.

Davo
Title: Re: Example projects window
Post by: dbannon on March 08, 2022, 08:33:19 am
OK, I have simplified the ListView, it now shows only Example name and keywords. If user clicks it, the full path to the example is printed in the memo along with metadata.

The category checkboxes work more sensibly now and I have added a license statement, the usual Lazarus one I hope, to each source file.

Davo

Title: Re: Example projects window
Post by: JuhaManninen on March 08, 2022, 01:36:32 pm
OK, rebase worked like a charm. I have used it in the past and regretted it (indeed, I tared up my working dir before I started).
Regretted because of conflicting commits? Now there is only a small chance for them.
In some situations when rebasing over many commits causing conflicts, you must solve too many of them. "git rebase" iterates over the commit history one by one. If many commits change the same file causing a conflict, every such conflict must be solved separately.
In that case "git merge" is better. It just joins the branches together leaving the intermediate steps out.

Quote
The changes to remove the old Examples Window is not really extensive but it does touch a good number of files. I will rebase frequently from now on ....
Your change to lazarus.lpi was not right. You added more lines instead of removing ExampleManager.
Otherwise your changes look good.
I confess I don't always understand the changes that "gitk --all" shows. I attach here a screenshot of how it shows your latest "newexamples" branch. Why are the same commits shown twice? It is not a problem in your commits, I just don't fully understand Git's logic yet.

Other notions: You should remove code straight away instead of commenting it out. The removed code can always be found in revision control history.
For example:
Code: Pascal  [Select][+][-]
  1. -  ecManageExamples          = ecFirstLazarus + 826;
  2. +  // ecManageExamples          = ecFirstLazarus + 826;
The first line removing ecManageExamples is enough.
Title: Re: Example projects window
Post by: dbannon on March 09, 2022, 01:21:04 am
Your change to lazarus.lpi was not right. You added more lines instead of removing ExampleManager.
Interesting. While I have removed ExampleManger, running a diff against a month old lazarus.lpi indicates that I somehow added ProjectWizardDialog to the lpi file, I did not do so consciously  (the local lazarus.lpi is the now current one in my repo, the ~/Downloads/lazarus.lpi  is a month old).

Code: Bash  [Select][+][-]
  1. dbannon@dell:~/bin/Lazarus/lazarus-drb/ide$ sdiff -bs lazarus.lpi ~/Downloads/lazarus.lpi
  2.         <ComponentName Value="ProjectWizardDialog"/>          <
  3.         <ResourceBaseClass Value="Form"/>                     <
  4.         <UnitName Value="ProjectWizardDlg"/>                  <
  5.         <UnitName Value="KeyMapping"/>                        <
  6.                                                               >       </Unit>
  7.                                                               >       <Unit>
  8.                                                               >         <Filename Value="examplemanager.pas"/>
  9.                                                               >         <IsPartOfProject Value="True"/>
  10.                                                               >         <HasResources Value="True"/>
  11. dbannon@dell:~/bin/Lazarus/lazarus-drb/ide$ grep ProjectWizardDialog lazarus.lpi
  12.         <ComponentName Value="ProjectWizardDialog"/>
  13. dbannon@dell:~/bin/Lazarus/lazarus-drb/ide$ grep ProjectWizardDialog ~/Downloads/lazarus.lpi

I believe that ProjectWizardDialog should be mentioned in the lpi file, while I can remove it I think it would be a bad mistake. I suspect it appeared there in a manner related to the rebasing and is someone else's commit (in official Lazarus repo) as discussed below. I think ....

Quote
I confess I don't always understand the changes that "gitk --all" shows. I attach here a screenshot of how it shows your latest "newexamples" branch. Why are the same commits shown twice? It is not a problem in your commits, I just don't fully understand Git's logic yet.
No, neither do I. I think it relates to the rebasing.  Looking at some affected files, I see content changes that do not relate to what I did in the first of the each pair of commits. That has to be someone else's commit off in the official branch that appears here during the refresh of the code base. 

( a comment : Gihub has a lot better GUI than Gitlab when it comes to inspecting past commits. Using gitlab, such "what changed when ?" questions need to be answered using git back here at home, on Github, I can browse to the commit and view it directly.)

Quote
Other notions: You should remove code straight away instead of commenting it out.
Yes, noted. My usual model with a fairly complicated edit is that I comment out code and stamp it with a 'DRB', then I can compile, repeat. Leaving the lines there helps me find related things that might be otherwise left dangling.  Then, when I am happy, I grep for the "DRB"s and remove them. In the case you mention, appears I forgot to stamp those lines.  :(

Davo
Title: Re: Example projects window
Post by: dbannon on March 09, 2022, 05:37:51 am
Hmm, bad new. Seems I cannot rebase any more.

This is, I must say, not particularly surprising to me, I have seen it before. I am getting a lot of conflicts like this -

Code: [Select]
CONFLICT (rename/delete): examples/testall.pp deleted in Examples tested, categorised, metadata added, restructured and renamed to components/codetools/codetools.inc in HEAD. Version HEAD of components/codetools/codetools.inc left in tree.

CONFLICT (rename/delete): examples/helloform.pp deleted in Examples tested, categorised, metadata added, restructured and renamed to doceditor/frmabout.pp in HEAD. Version HEAD of doceditor/frmabout.pp left in tree.

("Examples tested, categorised, metadata added, restructured" is the name of the commit. Git has spotted a file being deleted (or moved) and has searched around for its "replacement", in this case, there is none).

Most are non plain text files, harder to auto merge I expect. But harder to manually edit out the merge messages too, try it with an icon or .res file !

The issue is that there are simply too many edits in there, its too much for one rebase on top of another to handle. I am not sure if its a stack type problem, seems that git is guessing and realizes its guessing wrongly. Notice how it seems to have searched the whole source tree for what might be a replacement for the file its deleting ?  While I admire its persistence, its all wrong !

 There is absolutely no way we can risk sending official lazarus a merge request based on this current code base.

I believe that we must revert to my first suggestion, break it up into several, smaller merges -

1. Contents of ~/examples.   This by far the biggest change and the one that is triggering the rebase problem. And, its totally non controversial. What I have done there is still usable with existing system and if my plans are rejected, all that needs happen is removal of a hundred or so metadata files. The restructure, fix ups and removal of irrelevant examples is a good thing in any case.

2. Add the new package into ~/components - again, does no harm, easily removed if someone decides its a bad thing. It does not stop the existing one working, it allows someone to choose to install the package and see it all work along side the existing Examples Manager.

3. Finally, the potentially destructive bit. Changes to a number of core files in ~/ide, mostly the removal of the existing Examples Manager. 

4. Adding metadata files to all of the examples in other parts of the source tree. I have not yet started on this stage.

I can very easily recover the files needed to do step one and two.  It may be possible to recover already edited files from step three (or manually reedit may be safer).  No point in starting that if stage one and stage two have not been accepted into the official main. Too much chance of this happening again and my time and effort does need to be assigned some value.

I will have to take it out of the git environment, create a lazarus fork, make a new tree, move my stage one and two file in.

Davo

David
Title: Re: Example projects window
Post by: JuhaManninen on March 09, 2022, 10:46:50 am
Hmm, bad new. Seems I cannot rebase any more.
No problem. Moving and modifying files may be a challenge for rebase indeed. I will merge your changes to the official repo through patches.
Merge requests are not needed. They are clumsy anyways. No modification from my side is possible with them.

Quote
While I have removed ExampleManger, running a diff against a month old lazarus.lpi indicates that I somehow added ProjectWizardDialog to the lpi file, I did not do so consciously  (the local lazarus.lpi is the now current one in my repo, the ~/Downloads/lazarus.lpi  is a month old).
ProjectWizardDialog was there already. The IDE sometimes adds tags like <ComponentName Value="ProjectWizardDialog"/> etc. for some reason. You can see your own changes easily with gitk.

Quote
Gihub has a lot better GUI than Gitlab when it comes to inspecting past commits. Using gitlab, such "what changed when ?" questions need to be answered using git back here at home, on Github, I can browse to the commit and view it directly.)
You don't need Gitlab or Gihub GUI for inspecting past commits! There are much better tools for that, especially gitk. It shows a single branch by default and all branches with "gitk --all".
It has 3 panes with all the relevant information:
- List of commits with their messages, author and date.
- List of files in a selected commit.
- Contents of a selected commit as a diff.

Moving between commits is easy also without mouse clicks. IMO it is a superior tool also compared to TortoiseGit which many people praise.
Gitlab web GUI is needed mostly for bug reports after the initial cloning. All development and pulling / pushing happens with local Git tools.

The earlier issue of identical commits in the history was a result of your rebase / merge exercises somehow. I don't know how. :)  I was not able to flatten the history there.
No problem, I moved all your changes to my development repo through patches, then joined them to 3 commits and pushed.
Congratulations! Your revamped example projects and your new GUI are there now.
You should delete your existing newexamples branch and create a new one from "main" HEAD.

One more notion about the GUI: Double-click downloads the selected project but does not open it which is counter-intuitive. A user expects it to open.
The whole concept of "downloading" a local example project is weird. You should hide the copying process then from a user.
With online examples the word "download" makes sense.
Title: Re: Example projects window
Post by: dbannon on March 09, 2022, 11:26:52 am
No problem. Moving and modifying files may be a challenge for rebase indeed. I will merge your changes to the official repo through patches.
Merge requests are not needed. They are clumsy anyways. No modification from my side is possible with them.
Right, I will remember that. Patches don't take any history with them, much better for this way of working !

Quote
....There are much better tools for that, especially gitk.
I saw you mention gitk, thought it was a typo !  TCL/TK ? really ?
OK, installed, yes it works. But still looks like a TCL/TK app.

Quote
Congratulations! Your revamped example projects and your new GUI are there now.
And thanks for your patience, I have learnt quite a lot. Will now start, time permitting, on the remainder of the examples in the tree. All they will need is an appropriate metadata file added, that should not worry git at all.

Quote
One more notion about the GUI: Double-click downloads the selected project but does not open it which is counter-intuitive. A user expects it to open.

The whole concept of "downloading" a local example project is weird. You should hide the copying process then from a user.
With online examples the word "download" makes sense.
Agree. I am sure we'll find more problems before this project is finished.
I'll batch them up as I work through the 'other' examples, will get you another big patch at some stage.

Thanks for your help and patience here Juha !

Davo

EDIT- Also still need to have the new package auto installed. I guess thats a makefile thing ?
EDIT - worse than that, my code to grey out the Examples button on the ProjectWizard is not working, so, fresh install, no Examples until you manually install the package and no errro msg. Sigh ....
Title: Re: Example projects window
Post by: JuhaManninen on March 09, 2022, 01:01:59 pm
I saw you mention gitk, thought it was a typo !  TCL/TK ? really ?
OK, installed, yes it works. But still looks like a TCL/TK app.
:)
There are many new GUI apps with bells and whistles and fancy graphics but their usability sucks.
Gitk is the opposite. Its GUI looks outdated but the usability is super. It is even quick in its actions despite TCL.

TCL is not a nice language, I know. I had to use it in a objet DB system where it was a macro language.

I guess you already know "git gui". I use it always to commit code. It allows you to select files and even lines inside a file to include in the commit. Also allows to easily join your new changes with the last commit.
Super good usability again.
Title: Re: Example projects window
Post by: dbannon on March 11, 2022, 10:47:38 am
I tend to work from the command line where I can and have not looked for a (desktop) git GUI.

Github is pretty good for those situation where a GUI is indicated, examining commits and so on, being on line must be a slower but I have not found that a problem.  I'll try gitk for gitlab related things as it seem my {best|only} choice. Thats all good.

OK, now that I know its preferred to generate a patch rather than git merge, I think I will create two branches of my existing gitlab fork, one will show changes to the Package (as mentioned above) and the other will slowly accrue the metadata files in all the examples that exist in directories other than ~/examples

As neither will involve mass renaming of files, rebasing should be safe.

I do not, at present, understand how I would make the new package part of a standard install. I suspect its Makefile and fpcmake stuff ?  Not my favorite place to play.

Davo



 
Title: Re: Example projects window
Post by: wp on March 11, 2022, 12:44:22 pm
I tend to work from the command line where I can and have not looked for a (desktop) git GUI.

Github is pretty good for those situation where a GUI is indicated, examining commits and so on, being on line must be a slower but I have not found that a problem.  I'll try gitk for gitlab related things as it seem my {best|only} choice. Thats all good.

OK, now that I know its preferred to generate a patch rather than git merge, I think I will create two branches of my existing gitlab fork, one will show changes to the Package (as mentioned above) and the other will slowly accrue the metadata files in all the examples that exist in directories other than ~/examples

As neither will involve mass renaming of files, rebasing should be safe.

I do not, at present, understand how I would make the new package part of a standard install. I suspect its Makefile and fpcmake stuff ?  Not my favorite place to play.

Davo
I don't know whether I like the new structure of the examples folder in the IDE. There is a "top-level" folder "Beginner" which contains all the single mini sample projects that where all in the examples folder of the older version, but now each in its own folder - that's perfect, I like it. But there is also a folder "Components", it contains subfolders for "cell_overflow", "columneditors", "embedded_images", "merged_cells" - all these were contained in a "grid_examples" folder in the old version, and it was clear that all these sample project somehow dealt with the Lazarus grid. This information is lost now. I know that the meta-data of these projects contain the information that they are about grids, but this forces me to use the new example projects windows. So far, I found everything needed from the file system alone. Giving up this feature clearly would be a weakness of the new example system.

Can't you move the mentioned project folders into a subfolder "grid_examples" of "Components"? (BTW: I would prefer all folder names to be in lower-case)

I did not read everything about this topic, and I am not sure if the next topic is planned anyway: There are a lot of examples in the (Lazarus)/Components folder. They should be accessible from the new examples window, too. But without moving them to the examples folder: When I work with TAChart I expect the examples to be in the TAChart installation folder, not somewhere else.  The meta-data structure should be flexible enough to accept a project at any location of the Lazarus source tree.
Title: Re: Example projects window
Post by: JuhaManninen on March 11, 2022, 03:21:32 pm
Github is pretty good for those situation where a GUI is indicated, examining commits and so on, being on line must be a slower but I have not found that a problem.  I'll try gitk for gitlab related things as it seem my {best|only} choice. Thats all good.
Github and Gitlab are servers. You don't need their web GUI for your development process. I personally use only the bug tracker.
A big benefit of Git is its local commits and revision history. Why would you examine commits with a clumsy online browser interface when you can do it locally?
"gitk" + "git gui" are not your only choice. There is for example SmartGit, a Java app, free for FOSS development. It has a modern GUI and nice graphics etc., you may like it. I don't see any usability benefit over "gitk" and "git gui" though.

Quote
OK, now that I know its preferred to generate a patch rather than git merge, I think I will create two branches of my existing gitlab fork, one will show changes to the Package (as mentioned above) and the other will slowly accrue the metadata files in all the examples that exist in directories other than ~/examples
You may not need your fork at all soon. I have asked commit rights for you into the official repository. I have duties during the spring and summer and don't want to work as a middle man.
You can have many local branches and experiment with them. It is possible to push a branch to the Gitlab server for others to test, too, although most people just test the "main" branch.

Quote
I do not, at present, understand how I would make the new package part of a standard install. I suspect its Makefile and fpcmake stuff ?  Not my favorite place to play.
We need help with that. Mattias knows the Make systems best.

BTW, there is a reported problem with MacOS :
 https://forum.lazarus.freepascal.org/index.php/topic,58631
On Linux things work.
Title: Re: Example projects window
Post by: dbannon on March 18, 2022, 11:28:22 am
OK Juha, I have been adding meta data to examples in ~/components and have decided I need to back track a bit.

All along I have been saying that examples belong in a dir with same name as the Example, made sense and was easy when they were moving on-line but is hopeless while they live in the src tree. Better to leave them as they are now but when copying to work area they get the more informative dir name.

So, I have made changes to only the files in ~/components/exampleswindow and they now live in https://gitlab.com/dbannon/lazarus/-/tree/fixpackage

This branch also has something you suggested, a double click in the list now "downloads" AND opens a Example. I cannot think of a better word than "download" I am afraid, "copy" , "stage", "copy to work area" ?

Now checks for duplicate Example Names and I have removed some unnecessary debug statements.  I can rebase it if you like, should be trivial.

Please do not take anything from the other branch there, "otherexamples", its all wrong !   I will remove it shortly, will start a new branch based on the approach I mention above. 

All the stuff in ~/examples is fine, not affected.

Davo
Title: Re: Example projects window
Post by: dbannon on March 21, 2022, 12:35:41 pm
OK, some progress.

https://gitlab.com/dbannon/lazarus/-/tree/fixpackage contains some fixes to the package itself. Now tested on Windows, a couple of tweaks compared to whats in main now. Changes are limited to code in ~/components/exampleswindow. I have tried to rev the version number but its beyond me.

https://gitlab.com/dbannon/lazarus/-/tree/fixpackage contains about 50 Examples, processed, tested and fixed where necessary.  All in ~/components.   Its a lot slower than I expected because many Examples require specific packages be installed and its far from easy to determine what package in some cases.

A lot more to do ....

In both branches, minimal deletions and moves means rebasing should be viable. But be good to get whats there now merged before too long, so to avoid issues.

Davo

 
Title: Re: Example projects window
Post by: JuhaManninen on March 23, 2022, 10:14:10 am
Sorry for the delay. I am a little busy with other things.
@dbannon, could you please create merge requests after all. Other developers prefer to see the changes there and then are able to merge them. I may be busy during the spring and summer.
The initial commits were an exception because so many things changed.
Actually I should learn more about Git usage. I could have linked your forked repo and then fetched it.
I will study the latest changes soon ...
Title: Re: Example projects window
Post by: dbannon on March 24, 2022, 12:16:03 am
No Worries Juha. I hope you have been working on your forestry project !

I'll do some tidying up and generate two pull requests. I can remove some code from the package now I have a better idea of where we are heading. So, Package first, after that is done another 50 or so examples. All very straightforward merges. I'll log a bug report with each pull request.

As we are not doing the on-line model right now, I am building in a way for the user to open a example in their browser. This lets them look at the example code, maybe copy a block, without closing the project they are working on.

Thanks,  Davo



Title: Re: Example projects window
Post by: dbannon on March 30, 2022, 11:56:53 am
Hmm, I have put that merge request in, should be an easy one but I have not heard back from anyone in five days.  So, I am wondering if I also should have put a Bug Report pointing to that Merge Request ??

Not complaining, just wondering if I have followed protocol    :)

I do have a number of 'processed' examples also waiting to be merged too but need to get this first merge done.  And worry if I slip too far behind then git will have problems doing a simple merge. Rebase has its limits ....

Davo
Title: Re: Example projects window
Post by: dbannon on May 13, 2022, 09:33:49 am
Quite a lot of progress has been made with the project to tidy up the Lazarus Examples.

If you are using a reasonably recent "main" you now have over 200 Examples, tested, categorized and with a little text explaining what each does, what extra packages are needed and many fixes.

The new window works for Linux and Unix users who's examples are in read only diskspace and all users will find it easy to refresh an really messed up example or clean up afterwards.

But there are still a number of Example Projects that still have problems, 26 last time I looked and I am sure not the best person to deal with them all. So, please, can you have a look and see if any of them sound a bit familiar ?

If you are using main, open the Examples Window and search for "Needs Work" or, non main users can browse the list of problem examples at https://gitlab.com/dbannon/lazarus/-/issues

Maybe even a decision that a particular example no longer belongs in the repo would help. And some new Examples would be good too !

Probably best I collate the fixes/additions to minimize the total number of commits that have to happen.

Davo

Title: Re: Example projects window
Post by: wp on May 13, 2022, 11:27:39 am
I would appreciate if you'd restore the original folder structure in the (lazarus)/examples folder. I am accustomed to open projects from the file system, not from the new examples window, and the new folder structure makes it hard to find a sample project once you knew where it used to be. Of course, I could train myself to use the examples window. But why should I? I never used it, and why should I "copy the project to workarea" (I am on windows, and the installation is writable)?

In relation with the "need work" examples I am using the new examples window for the first time and I would like to propose a couple of improvements:
Title: Re: Example projects window
Post by: dbannon on May 13, 2022, 01:49:31 pm

A lot of it is historical WP.

Why are some examples under directory names such as XXX ?
There were a large number of examples 'loose' under ~/examples, they needed to be moved into a directory of their own (why ? see below) and, initially I saw category as a reasonable model. Later I realized having fixed, pre-determined categories as too inflexible. But by then I had moved some 100 examples.

The examples under ~/examples represent only about half the total by the way.

Why should I "copy the project to workarea" ?

There are several reasons.
1. For Unix users, the current system is utterly unworkable if Lazarus has been installed under /usr
2. If an inexperienced user edits an Example, a copy is easy to refresh if a fatal mistake is made.
3. By having the Example compiled out of the source tree, its easy to remove unwanted bulky binary files. 
4. Initially, and still IMHO a possibility, the plan was to move the Examples online, downloading as required (or en mass for those who so require).

Why have each example in its own directory ?
1. It makes the above possible.
2. Its just plain tidier and allows the beginner to see just what files belong to a given Example.
3. See item 4 above.

Why is there a "DBase" folder ...
Sheer ignorance I am afraid. I don't do database stuff and did not realise that the term dbase meant a specific database model. Last time I did any database work was Delphi 2. Before that it was dbase III, an IBM product I grew to hate.
I have been using the term dbase from a very early stage of this project, a bit sad it was not noticed any earlier.

The prime purpose of these changes was to make it easier for new users. I guess users like yourself were not really considered as needing Examples to be honest. Sorry !

I could return some Examples to their original location but definitely not all and about half have not moved anyway. The Examples Window requires only a double click, to both copy and open.

IMHO copying a Example before letting the user loose is necessary.

I can correct the use of "DBase" instead of database if its necessary but getting merge requests acted on is a slow business !  Fortunately the Examples Window adapts to whatever category names it finds so that would not require any change.

Davo
Title: Re: Example projects window
Post by: wp on May 13, 2022, 02:40:30 pm
One more issue in the examples window: Looking at the meta text of the TAChart_html demo I see that you added "\n" for line breaks in the description. But these symbols are not resolved as such in the form's memo showing the description, the text after the '\n' just continues in the same line. I did not look at your sources, but every '\n' should be replaced by a LineEnding.
Title: Re: Example projects window
Post by: dbannon on May 13, 2022, 03:04:23 pm

Works fine on Linux. Hmm, a Windows specific bug, I did check for that sort of issue but may have introduced a regression since then ?

The \n are required to be valid json, I initially used LineEndings but that was wrong, I guess I am expecting the json parser to do the necessary conversion.  I'll boot into Windows and see what I can find.

Thanks for picking that up !

Davo
Title: Re: Example projects window
Post by: wp on May 13, 2022, 03:07:32 pm
1. For Unix users, the current system is utterly unworkable if Lazarus has been installed under /usr
I am aware of this, but the example subsystem should not force users of other systems in which Lazarus has been installed in a writeable folder (e.g. fpcupdeluxe) to use it. Note that all this is based on an installable package. What if a user does not install the "ExampleProjects" package? He is rather lost regarding examples.

Returning to the installation under /usr. This affects only the examples coming with Lazarus. When a third-party package, e.g. fpspreadsheet having lots of examples, is installed it goes into writeable space. Is the implication that your examples window supports only the examples distributed with Lazarus?

Why have each example in its own directory ?
1. It makes the above possible.
2. Its just plain tidier and allows the beginner to see just what files belong to a given Example.
Of coarse, each sample project should be in its own folder. But I don't like all these folders to be at the same level. Before your changes, all the grid examples were inside the equally named subfolder of "examples". This way it was easy to find "all grid" examples at one glance. When this is not possible any more your new example window has a severe problem with scalability.

The prime purpose of these changes was to make it easier for new users. I guess users like yourself were not really considered as needing Examples to be honest.
Sure, every effort to make life easier for new users is great. But not every user is a new user. I would not say that I don't need the examples - I am very happy to be able to look how something is solved which I am not familiar with. And on the other hand, I do maintain the examples which are interesting to me and which belong to the components that I maintain. Here the "copy in user space" is useless to me since my fixes are not in version control, and I have extra work to copy it back to git.

Sorry for this criticism. It does not mean that I don't like your work. In contrast, I think it is great that you take your time and do this, and you do this well, and this entire subject can become a great thing such as the Online Package Manager. But I just want to point out that focussing on "new users" alone should not introduce too many issues for old users.
Title: Re: Example projects window
Post by: wp on May 13, 2022, 03:45:49 pm
At the moment I checking the TAChart_html demo. I modified the ex-meta file of this demo and uploaded it to gitlab. Afterwards I noted that I had not edited the description correctly, fixed it and - of course - introduced another error: I quoted a word incorrectly (it is not yet online). Trying to open the examples window now fails with some json error:

'ERROR EJSONParser- invalid JSON Error at line 5, Pos 96: Expected comma (,) or square bracket (]), got token "unusual".'

First wish: Would it be possible to display the name of the erroneous file? Now I know what causes the issue, but I could imagine that in many cases it would be helpful to know the filename.

Next, more severe problem: I changed the text, removed the quoted word. Saved. I still cannot open the examples window any more, always the same error. Did a clean rebuild of the IDE - no luck... Are there some cached files somewhere? How can I clear that cache?

When something like this happens, I think the parser should continue reading, skipt this project, and add the other projects. I am not too familiar with json, is this possible?

[EDIT]
Looking at the debug logs I found the offending file: (lazarus)/components/tachart/demo/html/backup/TAChart_html.ex-meta. Deleted it, and it works again.

Conclusion: Do not seek examples in the backup folders which Lazarus creates by default.

Title: Re: Example projects window
Post by: dbannon on May 15, 2022, 11:12:42 am
I am aware of this (Unix read only issues), but the example subsystem should not force users of other systems
But the Unix Read Only problem is only one of the reasons that I mentioned. For the "average" user, its a good idea, maybe for you, not so good. But harmful ?

Quote
Note that all this is based on an installable package. What if a user does not install the "ExampleProjects" package?
Yes, I agree. But I was advised it was 'policy' to use packages where possible. Making Lazarus more modular does make sense.

Is the implication that your examples window supports only the examples distributed with Lazarus?
No, the Examples Window searches first the Lazarus source tree then the Lazarus Config dir, that way, picking up any compliant examples that have come in from, eg, OPM. It is my intention to make some metafiles for viable examples I find in, at least the OPM packages I use. Hopefully more.
Certain, pre determined directories in config are skipped, eg, we don't pickup working copies or the working copies backup directories there.

Quote
Before your changes, all the grid examples were inside the equally named subfolder of "examples"...
Indeed, some were logically located. Most were not. You obviously feel very strongly about this, I don't so I will restore the ones in ~/examples that I can.

Quote
... I do maintain the examples which are interesting to me and which belong to the components that I maintain. Here the "copy in user space" is useless to me since my fixes are not in version control, and I have extra work to copy it back to git.
I agree thats a backward step for you specifically. Maybe for others who do maintain some examples. My overall impression was that the examples where, on average, quite unloved.  But I wonder if you don't use the example window to find your Example, and I restore the ones important to you, it should be business as usual ?

Quote
Sorry for this criticism.....
Not at all, I need constructive criticism because I want my contribution to be useful. Yes, my focus is on new users and unashamedly so but we cannot, in that process, make things harder to maintain. So, important that I be told if I am getting off track in any way.

// Other message
> error after editing the JSON, display the name of the erroneous file?
Yes, should be easy, I'll look at it.  As an aside, I have a tool I made to edit the meta files, checks for correctness before saving. But most casual editing will be done with vi or equivalent.

> Picking up broken metafile in backup dir.
Sounds like I should exclude files in backup, too easy.

> Proceeding after encountering an error in a metafile.
Yes, I agree, a more graceful recovery is a must. Should be easy, somewhat surprised about your experience. Will deal.

// Related
The issue with newlines in JSON under Windows has left me somewhat surprised. I now understand that "\n" in C is not the same as "\n" in Java and therefore JS. (and my respect for Java has dropped even further). I'll deal with that issue too.

(Sincerely) thanks for feedback !

Davo
Title: Re: Example projects window
Post by: dbannon on May 16, 2022, 01:57:22 pm
I have lodged two merge requests dealing with the above.

 https://gitlab.com/freepascal.org/lazarus/lazarus/-/merge_requests/94
Will restore the ~/examples directory structure much like it was before I started this "project", not completely because -

* All example projects are now in a stand alone subdirectory, many existed as loose files all in the one dir previously.
* Some have been omitted as discussed.

and https://gitlab.com/freepascal.org/lazarus/lazarus/-/merge_requests/93

* Solves the  problem Windows was experiencing with newlines in Example Description.
* When the ExamplesWindow encounters a JSON error in a metafile, now handled gracefully and an attempt is made to display the actual file name. The file name is also reported to console as previously.
* The above error does not abort the scanning process any more.
Title: Re: Example projects window
Post by: CharlyTango on January 25, 2024, 11:26:36 am
@dbannon:

I like the idea of modularising Lazarus, and the approach of preparing the examples for a beginner is great.
But the topic of "examples" should go much further.

Of course, a structured description of examples and demos is desirable. The idea that only examples that can be compiled at the moment are displayed can also help a beginner. However, only some of the components are under the direct control of the lazarus team.

In the course of time, I have accumulated many components which I have stored for testing and also as an archive for possible components in programmes.
I therefore regularly search my archive for examples and demos to get an impression of the functionality of the respective component.

Calling up and compiling such examples directly in Lazarus is very efficient.

However, many such examples have hardly any description, are poorly maintained, possibly still based on old standards, use *.lpi files and many other shortcomings. Nevertheless, I would like to see them in a list and checking installed components would not be an issue.

I would like to have a really stunning window that can play all the pieces you could wish for as a beginner and experienced programmer. Possibly in a normal and advanced function set, switchable and only visible to the advanced programmer. Then perhaps the runtimes would no longer be a real problem and you could also cache search results.

It would be like a fusion of the old and new functions on drugs ;-)

I would be happy to discuss this.

Where should I report bugs?

- Listview - anchor right, +10 missing
- Listview, sorting does not work
- Listview, examples from OPM are not found (Win10, Laz3.0)

Best Regards
Title: Re: Example projects window
Post by: dbannon on January 25, 2024, 11:56:47 pm
Wow Charly, this is an old thread !

The work refered to is now in Lazarus 3.0 and Lazarus 3.0 is released. So, first of all, are you looking at Lazarus 3.0 ?

....
In the course of time, I have accumulated many components which I have stored for testing and also as an archive for possible components in programmes.
I therefore regularly search my archive for examples and demos to get an impression of the functionality of the respective component.

The Examples system is intended to be extensible. In your case, you could add entries referring to your private examples at the end of the ~/lazarus_3_0/examples/examples.txt file and add a simple meta file in each to achieve what I suspect you want. Probably best scripted.

Now that lazarus_3_0 and the new examples code has been released, I should update https://wiki.freepascal.org/Lazarus_Examples_Window - there is some good cricket on this afternoon/evening, I will update while I watch that !

Its also possible for packages (that can be added to the IDE) to list their own examples. So, if you add your own code as a set of packages, you could do that too.

... examples have hardly any description, are poorly maintained, possibly still based on old standards...
All the example presented in lazarus_3_0 where checked to be functional and have, as a minium, a one liner description. But thats all I am afraid. Very many do have bugs (all software has bugs !) sometimes noted in the description. There is, as you mention, massive room for improvement!

I would be happy to discuss this.
Excellent !

Where should I report bugs?

- Listview - anchor right, +10 missing
- Listview, sorting does not work
- Listview, examples from OPM are not found (Win10, Laz3.0)

In the normal Lazarus bug report system. Its been my experience that a bug report that includes a patch is very well received. Focus on one issue per patch, test extensively, explain why its necessary and your contribution will be very welcome !

Davo
TinyPortal © 2005-2018