Recent

Author Topic: Example projects window  (Read 54661 times)

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9857
  • Debugger - SynEdit - and more
    • wiki
Re: Example projects window
« Reply #30 on: January 01, 2022, 04:09:24 pm »
Btw, another point that may be worth considering: translations.

- Could the meta data, contain fields for translations?
- Could "readme" or similar documents, be provided in more than one language?

And, if the example itself does not have a translation, could the design allow for a lookup db to be added at some point, so other contributors can add more translations?

af0815

  • Hero Member
  • *****
  • Posts: 1291
Re: Example projects window
« Reply #31 on: January 01, 2022, 06:00:47 pm »
Btw, another point that may be worth considering: translations.
Can the actual FPDoc this ?! Because this is the main tool for (Code-) documentation.
« Last Edit: January 01, 2022, 06:03:10 pm by af0815 »
regards
Andreas

dbannon

  • Hero Member
  • *****
  • Posts: 2791
    • tomboy-ng, a rewrite of the classic Tomboy
Re: Example projects window
« Reply #32 on: January 02, 2022, 02:01:35 am »

Sorry, very long answer warning !
Summary -

  • Proposed solution for weak, one way coupling to OPM
  • Discussion about Translations
  • Short term dev plans
  • Long term - a snipit manager ??
  • fpdoc ?

Sorry, I did not meant to make this about "finding a problem". But rather "pointing out more potential".

No 'sorry' needed !  This is all about discussion.  And you are discussing, thanks !  I'd much rather be told I am on the wrong track now than after a couple of days work. And a sequential tutorial capability is probably almost there, should be considered as we go.

framework and content are somewhat bound together....

- meta-data files will have to be extended too, and once the first version of the "example mgr" is out, meta-data requires backward compatibility.

Yes, right now there are about 250 example projects just in core Lazarus. There is a lot of work to be done to make them into what we are considering and I agree, as soon as thats done, we'll realise there is more to it. Right now I am playing with a brief "technical feasibility" project, just to see if what I imagine works will (given my familiarity is with Github, not Gitlab). And the answer is yes, technically, its all pretty easy. Will post something soon. I think my best approach is a standalone app, not integrated (yet) into Lazarus that we can use to prove the access and metadata models. It will be rough and ready and any part of can be walked away from without causing me any distress at all !

Using that we can go over the existing examples and demo (I am disregarding Tests) and build up a workable set in that Gitlab repo of mine, so no annoyance to the Laz Devs yet.   

OPM Integration

  As pointed out: OPM packages already contain examples. So our meta data, and storage requirements should allow such OPM packages to be recognized.

What I am playing with is model where the client (ultimately built into Lazarus) dowloads a master.meta file, checked daily (?) and keeps a cached copy. Suppose when doing its daily checks (or a manual refresh) it also scans the area in .lazarus where OPM packages are and if it finds any recognisable example projects, adds them to its local list ?

Obviously, this only happens if a user opens the Examples window.


Well, I guess - ideally - the "example window" (in its new form) will show everything that is (or has) example(s).
- installed
- online
- and if our online DB is separated from OPM, then also OPM packages with examples
The model I mention a couple of line up would show -
* all "lazarus examples"
* all OPM examples it found. That is, ones delivered as part of a OPM install.

It would NOT show OPM examples from packages not installed. And I think that makes sense. 


But in reverse, if a user browses OPM packages, and if they (or some of them) have meta-data about their examples:
 - Should then the local OPM browser window, not also allow to filter by "has example" ?
- And if the local OPM browser window, can show "only with example", should then the local OPM browser window not also be able to show the entries from the new example repo?
While an issue for OPM, I'd say yes, I'd also like a filter for "show ones with good docs" but thats getting a bit subjective :-)



Especially, since the new example repo could also contain examples for OPM packages. Just examples that have not been bundled with the OPM package itself.)

I am sort of thinking, given that idea of the Example Unit scanning locally of OPM Examples, that OPM Examples belong in OPM Packages and 'should' comply with the Examples standard we come up with.


So, unless we want to exclude OPM packages from having examples (within the example window), it seems to me that the two need to be very closely coupled.

No, again, looking at the "Example Unit scanning locally of OPM Examples" model minimizes coupling between the two services. And its one way only. And I think thats quite important. Over time, we can encourage OPM Package maintainers to include the standard looking examples and thats all that needs to be done.

And when I say standard looking examples, its a Lazarus Project with a 6 or 7 line metadata (although, as you noted, that could increase).




translations
Quote
Btw, another point that may be worth considering: translations.

Definitely we must allow for that. Brain Storming -

As the meta files are by definition, quite short, maybe its just having a po directory in the project dir with translated files ? Can the same model support in project po behaviour too ?

My model, at present is that a gatekeeper when added or changing an example project, will run an app that scans over the repo building a master.meta that is uploaded to gitlab along with the content changes. Now, if that scanner finds translated meta files, it could also build translated master.meta files. Then, when the (Lazarus) client tool runs, it would prefer an appropriate translated file if one is available.

An end user who decides to translate an existing project just need send in a translated file, gatekeeper names it appropriatly, adds it to his/her local git repo, runs the scanned, adds, commits and push.




A snipit manager ?

I have also been considering how this could be used in a snipit mode, instead of opening the Example Project, open just eg uini1.pas without closing the project the user is working on. That way, a quick copy and paste is possible without disturbing the user's work flow.  Might be version two.




fpdoc
Quote
Can the actual FPDoc this ?! Because this is the main tool for (Code-) documentation.
My initial though is that fpdoc would be overkill for the faily miminal docs we are planning to use, just a couple of relevant keywords and a sentence or so of what the project demonstrates. But  af0815 if you are familiar with fpdoc, could you elaborate ?

(phew ! longest post I have ever made here. Far too long !)

Davo




Lazarus 3, Linux (and reluctantly Win10/11, OSX Monterey)
My Project - https://github.com/tomboy-notes/tomboy-ng and my github - https://github.com/davidbannon

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4467
  • I like bugs.
Re: Example projects window
« Reply #33 on: January 02, 2022, 11:32:51 am »
As dbannon wrote, the example project system can search projects under the OPM install directory. No other integration is needed.
I realized there must be multiple directories to search from.
One is "examples" under Lazarus tree which is the default now.
The "components" directory must be searched, too. The local packages also include example projects.
Third one is the OPM directory which is known by reading the OPM settings.
Then a user can define more directories.
Mostly Lazarus trunk and FPC 3.2 on Manjaro Linux 64-bit.

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9857
  • Debugger - SynEdit - and more
    • wiki
Re: Example projects window
« Reply #34 on: January 02, 2022, 01:02:58 pm »
OE => "Online Examples", depending on context "Online Example Registry/Database"

It would NOT show OPM examples from packages not installed. And I think that makes sense. 
Makes sense. In fact, showing examples for packages "not yet even downloaded" would be confusing. (There OPM could be extended, to allow filtering by "has example", but that is off-topic).

However:
- What about "examples" (pure examples, stored in the OE) that require a package?
-- 1) If that package is OPM ? (And not yet downloaded / or needs an update downloaded)
-- 2) If that package is avail on local storage (but maybe not installed in the IDE)
-- 3) A package, unknown to the IDE (maybe removed, but example DB not yet updated) -> This would be an error, but needs handling.

In case -2- examples could still be listed:
- "installation" (for design time packages) can be offered when opening the example.
- "installation" does not have to be mandatory. It is (usually) only needed to open the form in the designer. Installation is (usually) not needed to compile and run the example. (So once the user has been informed, it is their choice)

In case -1-:
I am sort of thinking, given that idea of the Example Unit scanning locally of OPM Examples, that OPM Examples belong in OPM Packages and 'should' comply with the Examples standard we come up with.
Well, ideally yes.

But, what if a person outside the "OPM package"-team provides examples? (I.e.: If I were to write examples for OPM-foo)
Ideally, I would submit them to the maintainer of that OPM package.
But if they do not want to add it? Maybe because their code is Laz&Delphi, but the example is Laz-only?
Then it could still be a good example...
-> Yes, this is a very special case.... Ok to worry later, if we don't have a good idea now.

As dbannon wrote, the example project system can search projects under the OPM install directory. No other integration is needed.
Depends, on how the above is handled.

IMHO we need at least the ability to specify "required packages" for the example.
And probably should at least prepare, to store some info on the packages
- min version (maybe max version too)
- Only required for design.
--  Some packages have separate *Dsgn packages, then the example can run without this (i.e. the package does not need to exist at all, example can be compiled/run without it)
     But the user must be told what happens if he does not install it.
--  Other packages have IDE integration and runtime in one package. So even if the user skips installation, the package must exist for compilation.
- Origin of package: Laz-Installer, OPM, unknown

We may not need all of those right now, and hence not add them now.
But the design of the meta file should allow to easily add them, including backward compatibilty.




Translations:
As the meta files are by definition, quite short, maybe its just having a po directory in the project dir with translated files ? Can the same model support in project po behaviour too ?
Good start. Translated files, must not contain "version info", or any info at all, except translated strings. (Other elements could be allowed, but have to be ignored).

That depends a bit, on how well compatibility info "lazarus version" is represented.





Right now I am playing with a brief "technical feasibility" project, just to see if what I imagine works will (given my familiarity is with Github, not Gitlab).

What I am playing with is model where the client (ultimately built into Lazarus) dowloads a master.meta file, checked daily (?) and keeps a cached copy. Suppose when doing its daily checks (or a manual refresh) it also scans the area in .lazarus where OPM packages are and if it finds any recognisable example projects, adds them to its local list ?

A few notes:

1)
I would expect that contributors maintain their own GIT repo. (I guess we start with "git repo" as storage, probably keep it at this, but extension would be an issue for later...)

We would then have a central "OE" repo. Containing
* A list of all examples
* A copy of all meta files (and maybe translation)
* A copy of the current stable version of the example (actually not a must, but desirable in case the contributor's repo becomes unavailable.)

Ideally the meta files can keep info on:
- older versions
- Which example version to use with which IDE version

However, we do not need to have copies of older version. (Yet we probably will have).

==> IMHO we may find, that downloading data from our OE repo (and/or externals) will be very similar to what OPM already does.
   So, it could be quite likely we can re-use some of the code from the OPM package. (just pass in our OE Url / some config/opts)

Like OPM, we could also allow the user, to get the latest example from the contributor. This would of course not have been reviewed by us.
(If the meta has a version map, this could be the "latest matching the IDE version" or the "latest on main branch")


2)
The meta file.

As long as we allow git only as storage for contributors, we can relay on references.
I.e. if the meta file would have:
Code: Text  [Select][+][-]
  1. "Compatibilty": {
  2.   "Laz-2-2": "1.3",
  3.   "Laz-2-0": "1.2"
  4. };
  5. "Versions": {
  6.    "1.3": "<sha1>123",
  7.    "1.2": "<sha1>234"
  8. }

The "sha1" is a signature (and the meta with that sha1 is stored in our OE, under our control).
If we do not have the version for the sha1 in our OE, we can load it from the external git. If the contributor changes something, the sha1 no longer exists. But if it exists, it is the version that we know.

However, the copies in our OE repo have different sha1. So using the above, we need a "meta-sha1-map" file. This would contain a mapping from the external sha1 to our sha1.


Another options here is, to have a single global "meta-ide-version" file.

Lets say, our OE was updated for the 2.0 release.
The commit 987 has all the correct entries, for Laz-2.0.
Then "meta-ide-version" would contain
Code: Text  [Select][+][-]
  1. "VersionMap": {
  2.   "2.0": "987"
  3. }

We can add new meta data, for the ongoing development versions of the IDE.

"987" can be on a branch. So if we need to add fix-updates, we can.
We would get commit "98A" on the branch.
And in the "main" branch we would change the info to: "2.0": "98A"

If we release Laz 2.2  we add:
Code: Text  [Select][+][-]
  1. "VersionMap": {
  2.   "2.0": "987",
  3.   "2.2": "ABC"
  4. }

The example system can always download "meta-ide-version" from the "main" branch, and lookup the correct revision for the IDE version the user has installed.


Translations and versions:
Once the revision for the IDE-version has been found, all other data, including translation, is loaded from that revision.


dbannon

  • Hero Member
  • *****
  • Posts: 2791
    • tomboy-ng, a rewrite of the classic Tomboy
Re: Example projects window
« Reply #35 on: January 02, 2022, 01:03:20 pm »
As dbannon wrote, the example project system can search projects under the OPM install directory. No other integration is needed.
I realized there must be multiple directories to search from.
One is "examples" under Lazarus tree which is the default now.
The "components" directory must be searched, too. The local packages also include example projects.
Third one is the OPM directory which is known by reading the OPM settings.
Then a user can define more directories.

Hmm, maybe not quite the way I am leaning Juha.  I am thinking that the existing Lazarus Examples will move to a Gitlab repo, ideally under freepascal.org.

I suggested local scanning for Examples only for things like the OPM ones. Benefits, solves the Read Only problem, makes for a smaller Lazarus Distribution, people using an older Lazarus can get new Examples - I see several more less easily and quickly described.

Anyway, please consider !

I have posted a very, very early draft of the unit doing the work and a rough and ready gui to exercise it. Rather appropriately in https://gitlab.com/dbannon/fpexamples/-/tree/main/Utility/ExScanner

It can load its internal List of Examples data direct from the above repository, from a locally cached file or from directly reading a Examples Gatekeeper's local Examples Git repo.  It makes a master.meta, displays a list of Examples (yes, in a ListView) and shows the description when an item is clicked.  Its far more about proof of concept than a proposal at this stage.

Davo

Lazarus 3, Linux (and reluctantly Win10/11, OSX Monterey)
My Project - https://github.com/tomboy-notes/tomboy-ng and my github - https://github.com/davidbannon

dbannon

  • Hero Member
  • *****
  • Posts: 2791
    • tomboy-ng, a rewrite of the classic Tomboy
Re: Example projects window
« Reply #36 on: January 02, 2022, 01:09:17 pm »
Sorry Martin, our posts crossed.

I will read yours in the morning, its late here and hopefully just getting cool enough to sleep. Been hot for a few days, yawn !

Davo
Lazarus 3, Linux (and reluctantly Win10/11, OSX Monterey)
My Project - https://github.com/tomboy-notes/tomboy-ng and my github - https://github.com/davidbannon

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9857
  • Debugger - SynEdit - and more
    • wiki
Re: Example projects window
« Reply #37 on: January 02, 2022, 01:32:54 pm »
I suggested local scanning for Examples only for things like the OPM ones. Benefits, solves the Read Only problem, makes for a smaller Lazarus Distribution, people using an older Lazarus can get new Examples - I see several more less easily and quickly described.

I would definitely keep some examples in the distro.

Not everyone wants their IDE to go online. Some are concerned about privacy, others are still exposed to cost-per-byte (and yes, of course the installer has those bytes, though maybe they downloaded this while on a cheaper connection...)

avra

  • Hero Member
  • *****
  • Posts: 2514
    • Additional info
Re: Example projects window
« Reply #38 on: January 02, 2022, 03:04:04 pm »
I would definitely keep some examples in the distro.

Not everyone wants their IDE to go online.
+1
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

wp

  • Hero Member
  • *****
  • Posts: 11910
Re: Example projects window
« Reply #39 on: January 02, 2022, 05:27:46 pm »
I would definitely keep some examples in the distro.

Not everyone wants their IDE to go online.
+1
My opinion, too.

And I get the impression from the discussion that the solution will somehow depend on features of GitLab. I think this is not a good idea. What if we once were forced to move away from GitLab? Then the example infrastructure would break away...

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9857
  • Debugger - SynEdit - and more
    • wiki
Re: Example projects window
« Reply #40 on: January 02, 2022, 06:15:37 pm »
And I get the impression from the discussion that the solution will somehow depend on features of GitLab. I think this is not a good idea. What if we once were forced to move away from GitLab? Then the example infrastructure would break away...

The only "gitlab" specific thing I recall is the "download folder as zip". But that is not required for the IDE integration. This is nice, for users who want to download a single folder.

It seems OPM just hosts a bunch of zip files.
IMHO allowing online-browsing, is a nice idea too.

So, well we have to see how the OE could do the downloads....
Actually, if it follows OPM, then all it needs is a link to a zip file.
- if hosted on gitlab, link to the zip that gitlab generates.
- otherwise link to a zip that was uploaded somewhere.



There are a few "git" related proposals. Such as using the sha1. That works with any git-server.

However, if we use git as storage for the OE database, then the client will need some config:

Path to main file: https://gitlab/some/path/main-meta.json
Path template for other files: https://gitlab/some/path/{%filename%}
Path template for other files an given sha1: https://gitlab/some/path/{%sha1%}/files/{%filename%}

Something like that should be adjustable to any git repo with https access.

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4467
  • I like bugs.
Re: Example projects window
« Reply #41 on: January 02, 2022, 09:26:19 pm »
IMHO we need at least the ability to specify "required packages" for the example.
And probably should at least prepare, to store some info on the packages
- min version (maybe max version too)
- Only required for design.
--  Some packages have separate *Dsgn packages, then the example can run without this (i.e. the package does not need to exist at all, example can be compiled/run without it)
     But the user must be told what happens if he does not install it.
--  Other packages have IDE integration and runtime in one package. So even if the user skips installation, the package must exist for compilation.
- Origin of package: Laz-Installer, OPM, unknown
That is over-engineering and duplicate logic. All the information is already in project/package dependencies.
An improved algorithm to find a dependent package would benefit all projects and packages, not only the example projects discussed here.
For example a converted Delphi project can have its dependent package already ported but with a different name. Adding more meta-data for an online package would then help to locate it.

... hopefully just getting cool enough to sleep. Been hot for a few days, yawn !
Oh boy! We had -15°C last night here. It is cool enough to sleep. :)
Mostly Lazarus trunk and FPC 3.2 on Manjaro Linux 64-bit.

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9857
  • Debugger - SynEdit - and more
    • wiki
Re: Example projects window
« Reply #42 on: January 02, 2022, 10:17:02 pm »
IMHO we need at least the ability to specify "required packages" for the example.
And probably should at least prepare, to store some info on the packages
That is over-engineering and duplicate logic. All the information is already in project/package dependencies.

Yes, and no.

The meta data, is downloaded for all examples.
lpi/lpk are only downloaded, once a user wants to try the example.

A user searching for an example to draw some 3d cube, might not expect to have to download the entire castle-game engine. (just because an example for this matches his search term).
Might sound far fetched, but then again I don't think it classifies for "so far the universe is winning".

But ok, to leave out (for now).

dbannon

  • Hero Member
  • *****
  • Posts: 2791
    • tomboy-ng, a rewrite of the classic Tomboy
Re: Example projects window
« Reply #43 on: January 02, 2022, 11:30:41 pm »
...
I would definitely keep some examples in the distro.
Not everyone wants their IDE to go online. Some are concerned about privacy, ....

Then we have to solve the *nix readonly problem.
Personally, I share the privacy concerns but neither gitlab nor github track non-logging in downloads. Maybe they will in the future...

Davo
Lazarus 3, Linux (and reluctantly Win10/11, OSX Monterey)
My Project - https://github.com/tomboy-notes/tomboy-ng and my github - https://github.com/davidbannon

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9857
  • Debugger - SynEdit - and more
    • wiki
Re: Example projects window
« Reply #44 on: January 02, 2022, 11:35:44 pm »
...
I would definitely keep some examples in the distro.

Then we have to solve the *nix readonly problem.

Well, treat the installed files, like files on a server. "Download" them, and save them to a writeable directory.

 

TinyPortal © 2005-2018