Example projects window

[dbannon]

OK Juha, I have been adding meta data to examples in ~/components and have decided I need to back track a bit.

All along I have been saying that examples belong in a dir with same name as the Example, made sense and was easy when they were moving on-line but is hopeless while they live in the src tree. Better to leave them as they are now but when copying to work area they get the more informative dir name.

So, I have made changes to only the files in ~/components/exampleswindow and they now live in https://gitlab.com/dbannon/lazarus/-/tree/fixpackage

This branch also has something you suggested, a double click in the list now "downloads" AND opens a Example. I cannot think of a better word than "download" I am afraid, "copy" , "stage", "copy to work area" ?

Now checks for duplicate Example Names and I have removed some unnecessary debug statements.  I can rebase it if you like, should be trivial.

Please do not take anything from the other branch there, "otherexamples", its all wrong !   I will remove it shortly, will start a new branch based on the approach I mention above. 

All the stuff in ~/examples is fine, not affected.

Davo

[dbannon]

OK, some progress.

https://gitlab.com/dbannon/lazarus/-/tree/fixpackage contains some fixes to the package itself. Now tested on Windows, a couple of tweaks compared to whats in main now. Changes are limited to code in ~/components/exampleswindow. I have tried to rev the version number but its beyond me.

https://gitlab.com/dbannon/lazarus/-/tree/fixpackage contains about 50 Examples, processed, tested and fixed where necessary.  All in ~/components.   Its a lot slower than I expected because many Examples require specific packages be installed and its far from easy to determine what package in some cases.

A lot more to do ....

In both branches, minimal deletions and moves means rebasing should be viable. But be good to get whats there now merged before too long, so to avoid issues.

Davo

 

[JuhaManninen]

Sorry for the delay. I am a little busy with other things.
@dbannon, could you please create merge requests after all. Other developers prefer to see the changes there and then are able to merge them. I may be busy during the spring and summer.
The initial commits were an exception because so many things changed.
Actually I should learn more about Git usage. I could have linked your forked repo and then fetched it.
I will study the latest changes soon ...

[dbannon]

No Worries Juha. I hope you have been working on your forestry project !

I'll do some tidying up and generate two pull requests. I can remove some code from the package now I have a better idea of where we are heading. So, Package first, after that is done another 50 or so examples. All very straightforward merges. I'll log a bug report with each pull request.

As we are not doing the on-line model right now, I am building in a way for the user to open a example in their browser. This lets them look at the example code, maybe copy a block, without closing the project they are working on.

Thanks,  Davo

[dbannon]

Hmm, I have put that merge request in, should be an easy one but I have not heard back from anyone in five days.  So, I am wondering if I also should have put a Bug Report pointing to that Merge Request ??

Not complaining, just wondering if I have followed protocol   

I do have a number of 'processed' examples also waiting to be merged too but need to get this first merge done.  And worry if I slip too far behind then git will have problems doing a simple merge. Rebase has its limits ....

Davo

[dbannon]

Quite a lot of progress has been made with the project to tidy up the Lazarus Examples.

If you are using a reasonably recent "main" you now have over 200 Examples, tested, categorized and with a little text explaining what each does, what extra packages are needed and many fixes.

The new window works for Linux and Unix users who's examples are in read only diskspace and all users will find it easy to refresh an really messed up example or clean up afterwards.

But there are still a number of Example Projects that still have problems, 26 last time I looked and I am sure not the best person to deal with them all. So, please, can you have a look and see if any of them sound a bit familiar ?

If you are using main, open the Examples Window and search for "Needs Work" or, non main users can browse the list of problem examples at https://gitlab.com/dbannon/lazarus/-/issues

Maybe even a decision that a particular example no longer belongs in the repo would help. And some new Examples would be good too !

Probably best I collate the fixes/additions to minimize the total number of commits that have to happen.

Davo

[wp]

I would appreciate if you'd restore the original folder structure in the (lazarus)/examples folder. I am accustomed to open projects from the file system, not from the new examples window, and the new folder structure makes it hard to find a sample project once you knew where it used to be. Of course, I could train myself to use the examples window. But why should I? I never used it, and why should I "copy the project to workarea" (I am on windows, and the installation is writable)?

In relation with the "need work" examples I am using the new examples window for the first time and I would like to propose a couple of improvements:

• The search line at the top displays the text "Search here", defined as value of the Text property of the TEdit. This has the problem that sometimes I may have to delete this text (when the edit fails to select the entire text) before I can enter my search criteria. A better option would be to erase the Text property and put the "Search here" into the TextHint property where it automatically disappears when the user enters something
• Why is there a "DBase" folder and a "DBase" category? Why not the same for SQLite3, Firebird etc? Or is this an abbreviation for "Database"? If yes, please name them so.
• The form closes when I "open" a copied project. This is bad because when this is not the project that I was seeking I have to start all over again, i.e find the correct menu item, enter the search text again. It would be helpful if the form would stay open when "Open" is clicked.
• The proiect list is not sorted, but there is a sort indicator in the header of the first column. Please remove it, or make the list sortable by clicking in the header.
• When project is to be copied over an existing project in the work a messagebox appears asking for "Refresh existing example?". It has a "Yes" and a "No" button. It should also have a "Cancel" button to return to the example window for the case that the wrong example has been selected and the user notices that before the project is loaded. Without the Cancel button the Examples windows has the be reloaded, and the search phrase has to be typed again - see above.

[dbannon]

A lot of it is historical WP.

Why are some examples under directory names such as XXX ?
There were a large number of examples 'loose' under ~/examples, they needed to be moved into a directory of their own (why ? see below) and, initially I saw category as a reasonable model. Later I realized having fixed, pre-determined categories as too inflexible. But by then I had moved some 100 examples.

The examples under ~/examples represent only about half the total by the way.

Why should I "copy the project to workarea" ?

There are several reasons.
1. For Unix users, the current system is utterly unworkable if Lazarus has been installed under /usr
2. If an inexperienced user edits an Example, a copy is easy to refresh if a fatal mistake is made.
3. By having the Example compiled out of the source tree, its easy to remove unwanted bulky binary files. 
4. Initially, and still IMHO a possibility, the plan was to move the Examples online, downloading as required (or en mass for those who so require).

Why have each example in its own directory ?
1. It makes the above possible.
2. Its just plain tidier and allows the beginner to see just what files belong to a given Example.
3. See item 4 above.

Why is there a "DBase" folder ...
Sheer ignorance I am afraid. I don't do database stuff and did not realise that the term dbase meant a specific database model. Last time I did any database work was Delphi 2. Before that it was dbase III, an IBM product I grew to hate.
I have been using the term dbase from a very early stage of this project, a bit sad it was not noticed any earlier.

The prime purpose of these changes was to make it easier for new users. I guess users like yourself were not really considered as needing Examples to be honest. Sorry !

I could return some Examples to their original location but definitely not all and about half have not moved anyway. The Examples Window requires only a double click, to both copy and open.

IMHO copying a Example before letting the user loose is necessary.

I can correct the use of "DBase" instead of database if its necessary but getting merge requests acted on is a slow business !  Fortunately the Examples Window adapts to whatever category names it finds so that would not require any change.

Davo

[wp]

One more issue in the examples window: Looking at the meta text of the TAChart_html demo I see that you added "\n" for line breaks in the description. But these symbols are not resolved as such in the form's memo showing the description, the text after the '\n' just continues in the same line. I did not look at your sources, but every '\n' should be replaced by a LineEnding.

[dbannon]

Works fine on Linux. Hmm, a Windows specific bug, I did check for that sort of issue but may have introduced a regression since then ?

The \n are required to be valid json, I initially used LineEndings but that was wrong, I guess I am expecting the json parser to do the necessary conversion.  I'll boot into Windows and see what I can find.

Thanks for picking that up !

Davo

[wp]

1. For Unix users, the current system is utterly unworkable if Lazarus has been installed under /usr

I am aware of this, but the example subsystem should not force users of other systems in which Lazarus has been installed in a writeable folder (e.g. fpcupdeluxe) to use it. Note that all this is based on an installable package. What if a user does not install the "ExampleProjects" package? He is rather lost regarding examples.

Returning to the installation under /usr. This affects only the examples coming with Lazarus. When a third-party package, e.g. fpspreadsheet having lots of examples, is installed it goes into writeable space. Is the implication that your examples window supports only the examples distributed with Lazarus?

Why have each example in its own directory ?
1. It makes the above possible.
2. Its just plain tidier and allows the beginner to see just what files belong to a given Example.

Of coarse, each sample project should be in its own folder. But I don't like all these folders to be at the same level. Before your changes, all the grid examples were inside the equally named subfolder of "examples". This way it was easy to find "all grid" examples at one glance. When this is not possible any more your new example window has a severe problem with scalability.

The prime purpose of these changes was to make it easier for new users. I guess users like yourself were not really considered as needing Examples to be honest.

Sure, every effort to make life easier for new users is great. But not every user is a new user. I would not say that I don't need the examples - I am very happy to be able to look how something is solved which I am not familiar with. And on the other hand, I do maintain the examples which are interesting to me and which belong to the components that I maintain. Here the "copy in user space" is useless to me since my fixes are not in version control, and I have extra work to copy it back to git.

Sorry for this criticism. It does not mean that I don't like your work. In contrast, I think it is great that you take your time and do this, and you do this well, and this entire subject can become a great thing such as the Online Package Manager. But I just want to point out that focussing on "new users" alone should not introduce too many issues for old users.

[wp]

At the moment I checking the TAChart_html demo. I modified the ex-meta file of this demo and uploaded it to gitlab. Afterwards I noted that I had not edited the description correctly, fixed it and - of course - introduced another error: I quoted a word incorrectly (it is not yet online). Trying to open the examples window now fails with some json error:

'ERROR EJSONParser- invalid JSON Error at line 5, Pos 96: Expected comma (,) or square bracket (]), got token "unusual".'

First wish: Would it be possible to display the name of the erroneous file? Now I know what causes the issue, but I could imagine that in many cases it would be helpful to know the filename.

Next, more severe problem: I changed the text, removed the quoted word. Saved. I still cannot open the examples window any more, always the same error. Did a clean rebuild of the IDE - no luck... Are there some cached files somewhere? How can I clear that cache?

When something like this happens, I think the parser should continue reading, skipt this project, and add the other projects. I am not too familiar with json, is this possible?

[EDIT]
Looking at the debug logs I found the offending file: (lazarus)/components/tachart/demo/html/backup/TAChart_html.ex-meta. Deleted it, and it works again.

Conclusion: Do not seek examples in the backup folders which Lazarus creates by default.

[dbannon]

I am aware of this (Unix read only issues), but the example subsystem should not force users of other systems

But the Unix Read Only problem is only one of the reasons that I mentioned. For the "average" user, its a good idea, maybe for you, not so good. But harmful ?

Note that all this is based on an installable package. What if a user does not install the "ExampleProjects" package?

Yes, I agree. But I was advised it was 'policy' to use packages where possible. Making Lazarus more modular does make sense.

Is the implication that your examples window supports only the examples distributed with Lazarus?

No, the Examples Window searches first the Lazarus source tree then the Lazarus Config dir, that way, picking up any compliant examples that have come in from, eg, OPM. It is my intention to make some metafiles for viable examples I find in, at least the OPM packages I use. Hopefully more.
Certain, pre determined directories in config are skipped, eg, we don't pickup working copies or the working copies backup directories there.

Before your changes, all the grid examples were inside the equally named subfolder of "examples"...

Indeed, some were logically located. Most were not. You obviously feel very strongly about this, I don't so I will restore the ones in ~/examples that I can.

... I do maintain the examples which are interesting to me and which belong to the components that I maintain. Here the "copy in user space" is useless to me since my fixes are not in version control, and I have extra work to copy it back to git.

I agree thats a backward step for you specifically. Maybe for others who do maintain some examples. My overall impression was that the examples where, on average, quite unloved.  But I wonder if you don't use the example window to find your Example, and I restore the ones important to you, it should be business as usual ?

Sorry for this criticism.....

Not at all, I need constructive criticism because I want my contribution to be useful. Yes, my focus is on new users and unashamedly so but we cannot, in that process, make things harder to maintain. So, important that I be told if I am getting off track in any way.

// Other message
> error after editing the JSON, display the name of the erroneous file?
Yes, should be easy, I'll look at it.  As an aside, I have a tool I made to edit the meta files, checks for correctness before saving. But most casual editing will be done with vi or equivalent.

> Picking up broken metafile in backup dir.
Sounds like I should exclude files in backup, too easy.

> Proceeding after encountering an error in a metafile.
Yes, I agree, a more graceful recovery is a must. Should be easy, somewhat surprised about your experience. Will deal.

// Related
The issue with newlines in JSON under Windows has left me somewhat surprised. I now understand that "\n" in C is not the same as "\n" in Java and therefore JS. (and my respect for Java has dropped even further). I'll deal with that issue too.

(Sincerely) thanks for feedback !

Davo

[dbannon]

I have lodged two merge requests dealing with the above.

 https://gitlab.com/freepascal.org/lazarus/lazarus/-/merge_requests/94
Will restore the ~/examples directory structure much like it was before I started this "project", not completely because -

* All example projects are now in a stand alone subdirectory, many existed as loose files all in the one dir previously.
* Some have been omitted as discussed.

and https://gitlab.com/freepascal.org/lazarus/lazarus/-/merge_requests/93

* Solves the  problem Windows was experiencing with newlines in Example Description.
* When the ExamplesWindow encounters a JSON error in a metafile, now handled gracefully and an attempt is made to display the actual file name. The file name is also reported to console as previously.
* The above error does not abort the scanning process any more.

[CharlyTango]

@dbannon:

I like the idea of modularising Lazarus, and the approach of preparing the examples for a beginner is great.
But the topic of "examples" should go much further.

Of course, a structured description of examples and demos is desirable. The idea that only examples that can be compiled at the moment are displayed can also help a beginner. However, only some of the components are under the direct control of the lazarus team.

In the course of time, I have accumulated many components which I have stored for testing and also as an archive for possible components in programmes.
I therefore regularly search my archive for examples and demos to get an impression of the functionality of the respective component.

Calling up and compiling such examples directly in Lazarus is very efficient.

However, many such examples have hardly any description, are poorly maintained, possibly still based on old standards, use *.lpi files and many other shortcomings. Nevertheless, I would like to see them in a list and checking installed components would not be an issue.

I would like to have a really stunning window that can play all the pieces you could wish for as a beginner and experienced programmer. Possibly in a normal and advanced function set, switchable and only visible to the advanced programmer. Then perhaps the runtimes would no longer be a real problem and you could also cache search results.

It would be like a fusion of the old and new functions on drugs ;-)

I would be happy to discuss this.

Where should I report bugs?

- Listview - anchor right, +10 missing
- Listview, sorting does not work
- Listview, examples from OPM are not found (Win10, Laz3.0)

Best Regards