Online Package Manager

[Groffy]

Hello GetMem

This could only happen if you connect to the old repository with outdated JSON. Open options, then change Remote repository to: http://packages.lazarus-ide.org/

Thank you for your quick response, I forgot about that. Again thank you for your great work!

[balazsszekely]

@Groffy
Thank you for your quick response, I forgot about that. Again thank you for your great work!

You're more then welcome! Feel free to post any bugs you find or suggestions/feature requests.

[Groffy]

Feel free to post any bugs you find or suggestions/feature requests.

Well, its fun to follow the progress, you put a lot of work in it to make it work withing 6weeks (first posting on October 05.). Just a few things/ideas :

1) Filter functionality seems not fully functional. I just tested to filter by category. With any of the defined categories, the result is an empty listbox. (would be nice to store the last assigned filter)
Furthermore there is a filtering by Author, but it seems that the repositiory items do not contain the authors name (at least not visible)

2) Makes it sense to manage more than one repository address?

3) Makes it sense to enable an automatic message when for one of the installed packages an update is available?



My best regards

[balazsszekely]

@Groffy

Filter functionality seems not fully functional. I just tested to filter by category. With any of the defined categories, the result is an empty listbox.

This is a bug,  fixed in r53396.  Please test.

would be nice to store the last assigned filter

This can be easily implemented, but then the list gets filtered by default. It may lead to confusion. What do you think?

Furthermore there is a filtering by Author, but it seems that the repositiory items do not contain the authors name (at least not visible)

Each repository package can contain multiple lpk files. The Author field is located in the lpk node(see attachment).

Makes it sense to manage more than one repository address?

Yes. This is in the todo list.

Makes it sense to enable an automatic message when for one of the installed packages an update is available?

I can implement this feature for the main repository, however checking updates from external webpage(see update feature) can be dangerous(malware, license issues, etc...). I don't want to automate the process.

[JuhaManninen]

I could not install SpkToolbar package because character case in a unit name did not match its file name. They must match exactly when a file name is not all lowercase.

I fixed it in r5371 in Lazarus-CCR.

Index: components/spktoolbar/SpkToolbar/SpkToolbar.pas
===================================================================
--- components/spktoolbar/SpkToolbar/SpkToolbar.pas     (revision 5370)
+++ components/spktoolbar/SpkToolbar/SpkToolbar.pas     (working copy)
@@ -1,4 +1,4 @@
-unit spktoolbar;
+unit SpkToolbar;

It has changed at some point. I remember installing SpkToolbar on Linux a long time ago.

Otherwise the package manager looks good!

For future changes in Options format and default values, a "Version" attribute can be added.
Other config files of the IDE have it.

About the filtering: saving and using the previous filter is not a good idea. Filter could be selected from a history list maybe.
I think the whole filter GUI must be revamped. Now it is clumsy and counter-intuitive.
There should be a separate window where many filter conditions can be set and they are 'AND'ed together.

I also think many packages now included in Lazarus components/ directory should be moved to CCR and provided through the online package manager.

[lainz]

The problem with the CCR is that doesn't support the 'update' option, to get the latest sources.

Why you (Lazarus mantainers?) don't create a GitHub organization and create a separate repository for each package, so all benefit from it: getting the latest with the Online Package Manager is a download of the master.zip

Edit: I mean only for those in active development, else it has no sense at all.

[Groffy]

@GetMem

This can be easily implemented, but then the list gets filtered by default. It may lead to confusion. What do you think?

Right, I was just thinking about filtering into categories, which would be my prefered filter. I agree thats not really helpful for other filter settings. Could lead to confusions.

I like JuhaManninnen's idea (later posting) about saving filter settings into a history

Each repository package can contain multiple lpk files. The Author field is located in the lpk node(see attachment).

Right, did not see/understand it on the first view.

I can implement this feature for the main repository, however checking updates from external webpage(see update feature) can be dangerous(malware, license issues, etc...). I don't want to automate the process.

Hm, good point. However I was just thinking of an automatic information, not to start an update process for a package.


Best regards

[minesadorada]

The problem with the CCR is that doesn't support the 'update' option, to get the latest sources.

Why you (Lazarus mantainers?) don't create a GitHub organization and create a separate repository for each package, so all benefit from it: getting the latest with the Online Package Manager is a download of the master.zip

Edit: I mean only for those in active development, else it has no sense at all.

All my packages are updateable and downloadable via ccr SVN (https://svn.code.sf.net/p/lazarus-ccr/svn) and also zips maintained at SourceForge that OPM currently uses.  I also try to keep the associated Wiki pages up to date.  I like Sourceforge and SVN.

I regularly svn-update to my local disk. - the ccr repository is currently at Revision 5371 with lots of new stuff.

[JuhaManninen]

The problem with the CCR is that doesn't support the 'update' option, to get the latest sources.

'svn' command can be used to update latest sources of individual packages there.

Why you (Lazarus mantainers?) don't create a GitHub organization and create a separate repository for each package, so all benefit from it: getting the latest with the Online Package Manager is a download of the master.zip

I have a gap in my knowledge. Who makes a master.zip file?

I understood earlier that GetMem planned for a direct revision control access in the package manager. I am not sure if I undertstood right.
It would allow loading latest development versions of selected packages by a svn, git or hg etc. command. Such command takes longer than loading a ready zipped package of course.

Maintained packages that have releases, the latest release version must be added to the JSON config file manually now.
I am not sure how GetMem planned to automate the process. There must be a way for the package maintainer to pass the version info without changing download URL (and file name).
This is quite important in future. The online package manager maintainer should have as little admin work as possible. Otherwise we will have an obsolete and outdated package repository in future, years from now.
[Edit] ... or maybe it is not too much maintenance to copy a new package and update the config. Don't know.
In any case it requires action from 2 people: the maintainer of a package and the maintainer of the online package manager.

[wp]

The problem with the CCR is that doesn't support the 'update' option, to get the latest sources.

I do it all the time. In Windows, using TortoiseSVN: Right-click on local ccr repository, "SVN Update" - that's it.

[lainz]

The problem with the CCR is that doesn't support the 'update' option, to get the latest sources.

I do it all the time. In Windows, using TortoiseSVN: Right-click on local ccr repository, "SVN Update" - that's it.

I know wp, I know, I was talking about the update option of this Online package manager that expects a zip file to get the latest sources with it. Else I don't will use it, for example I installed the spktoolbar, then you added some new themes, if I want them with the online package manager I can't, I must use svn (that of course I have and know to update with it) but that's not the point of using the package manager?

[minesadorada]

The problem with the CCR is that doesn't support the 'update' option, to get the latest sources.

I do it all the time. In Windows, using TortoiseSVN: Right-click on local ccr repository, "SVN Update" - that's it.

I know wp, I know, I was talking about the update option of this Online package manager that expects a zip file to get the latest sources with it. Else I don't will use it, for example I installed the spktoolbar, then you added some new themes, if I want them with the online package manager I can't, I must use svn (that of course I have and know to update with it) but that's not the point of using the package manager?

Ideally OPM will eventually use SVN preferentially for downloads and updates.  There is a case to be made for only providing packages with an active svn (or git) repository,  however I like the fallback option of using a hosted zip for developers that don't use svn or git - but only for selected packages i.e. ones where the developer is actively maintaining it.

For easier auto-maintainance, there need to be instructions for developers to make their components 'OPM-friendly'.  Some kind of standard versions.txt file that OPM can check (on user request) as well as including/updating the json file when committing/uploading newer versions.  The instructions should be in the wiki (with a link from OPM)

my 2€

[lainz]

'svn' command can be used to update latest sources of individual packages there.

Good point.

I have a gap in my knowledge. Who makes a master.zip file?

Is automatically done, like this:
https://github.com/bgrabitmap/bgrabitmap/archive/master.zip

So if you have any GitHub repository the zip is already available. It contains the latest sources.

Maintained packages that have releases, the latest release version must be added to the JSON config file manually now.
I am not sure how GetMem planned to automate the process. There must be a way for the package maintainer to pass the version info without changing download URL (and file name).
This is quite important in future. The online package manager maintainer should have as little admin work as possible. Otherwise we will have an obsolete and outdated package repository in future, years from now.
[Edit] ... or maybe it is not too much maintenance to copy a new package and update the config. Don't know.
In any case it requires action from 2 people: the maintainer of a package and the maintainer of the online package manager.

That's the main problem of having a single JSON made by hand. If for example we have a website to upload the packages, then a database will do the trick, you do a query and get the list of packages in a generated JSON, that no one needs to mantain.

For easier auto-maintainance, there need to be instructions for developers to make their components 'OPM-friendly'.  Some kind of standard versions.txt file that OPM can check (on user request) as well as including/updating the json file when committing/uploading newer versions.  The instructions should be in the wiki (with a link from OPM)

That's true. a versions.txt pointing:
- The latest download source of release in zip (an URL)
- The latest version number
- All the other neccessary stuff

So the package manager database will be updated automatically, say one time per month? Of course there's no one testing the quaility of the packages with this way. Is always a responsability of the user what to trust.

So in fact you need only a list of URL's to get the data from (list of versions.txt only), process them and update the database, it will take time to process all, but not too much, since these are a few kb files.

So the submit new app will be something like:

Name of your package: Something
URL of updates: version.txt url containing all the data for the json (generated with the online package manager of course)

[JuhaManninen]

Ideally OPM will eventually use SVN preferentially for downloads and updates.  There is a case to be made for only providing packages with an active svn (or git) repository,  however I like the fallback option of using a hosted zip for developers that don't use svn or git - but only for selected packages i.e. ones where the developer is actively maintaining it.

Oops, now the idea went upside down.
The main purpose of the package manager must be to provide stable versions of packages provided by their authors/maintainers.
The same way that any production quality SW is delivered.

In a perfect world all packages have an active maintainer but in real world they don't.
A fallback option must be to provide direct access to a development SVN / Git / etc. server of  the unmaintained packages. Another option is that the maintainer of our new online package manager picks and tests good revisions and "releases" them. He essentially then partly maintains those packages which is not realistic in long term.

People who participate in testing or developing a certain package will continue to use its revision control access directly.
It is not the main scope of our new package manager.

So if you have any GitHub repository the zip is already available. It contains the latest sources.

Ok, the same thing as getting directly using git command.

That's true. a versions.txt pointing:
- The latest download source of release in zip (an URL)
- The latest version number
- All the other neccessary stuff

So the package manager database will be updated automatically, say one time per month? Of course there's no one testing the quaility of the packages with this way. Is always a responsability of the user what to trust.

Updating automatically once per month sounds clumsy. Why not weekly or daily then?
Actually this works best with external URLs. The maintenance and hosting is moved to a package maintainer then.
I think the process of acknowledging new versions and copying to package repo should be made smoother but maybe not fully automatic.

[minesadorada]

@GetMem - I have a new package for you.  It's a runtime-only non-installable one (maybe a good test for OPM?)
OPM package with JSON:
https://sourceforge.net/projects/lazautoupdate/files/otherpackages/cryptini_opm.zip

I hope I configured it OK.
Updates in: https://sourceforge.net/projects/lazautoupdate/files/otherpackages/cryptini.zip
Wiki Page http://wiki.lazarus.freepascal.org/CryptINI
Testing app/demo:
Source: https://sourceforge.net/projects/lazautoupdate/files/otherpackages/cryptinitest_source.zip
Ready-compiled (Win32/64 + Linux 32/64): https://sourceforge.net/projects/lazautoupdate/files/otherpackages/testcryptini_compiled.zip

Note: When using  the OPM 'Create Repository Package' I noticed that it still puts unnecessary stuff in the zipfile.  I manually removed them.