Lazarus

Announcements => Third party => Topic started by: GetMem on October 05, 2016, 09:33:55 am

Title: Online Package Manager
Post by: GetMem on October 05, 2016, 09:33:55 am
Hi,

 I decided to implement the online package manager(opkman). The first alpha version is ready and it works like this: the packages are zipped and stored in a webserver(repository) along with a json file. I chose zip to minimize server load. The json file contains all the necessary info about the packages. Once installed into the IDE, the opkman downloads, then serialize the json file into a package list. The list is  displayed in a tree. On request the packages can be downloaded or/and installed. For now the repository is read-only, it contains only eight, well known(I guess?) packages. The nature/type of the eight package is irrelevant for now, since we are only testing the functionality of the opkman. Few notes:
 
 1. Supported versions: Laz 1.6+(FPC 3.0.0+). I did the implementation/testing with Lazarus trunk(FPC 3.0.0)
 2. Tested under the following widgetsets: Win32/64, GTK2, Carbon
 3. After install, you can find opkman in the "Package" menu
 4. Link: http://wiki.freepascal.org/Online_Package_Manager
 
 Please test! Suggestions are welcome.
 
 PS: Big thanks to @Leledumbo for hosting the repository.
 
regards,
GetMem


Edit: stable != alpha(thanks Graeme) + screenshot added.
Title: Re: Online Package Manager
Post by: Graeme on October 05, 2016, 11:40:36 am
stable(alpha) version

stable and alpha are two opposites of the scale. I had to laugh when I read that. :)

Thanks for sharing, and good luck with the package manager though.
Title: Re: Online Package Manager
Post by: Bostjan on October 05, 2016, 11:57:50 am
Is this to be something like "NuGet" for MS visual studio ? (https://en.wikipedia.org/wiki/NuGet, https://www.nuget.org/)

If so, I think it is really good idea. Worst part of Delphi/Lazarus for me is that you must hunt various packages across the internet and central repository would be so much better.
Title: Re: Online Package Manager
Post by: GetMem on October 05, 2016, 03:25:48 pm
Quote
@Graeme
stable and alpha are two opposites of the scale. I had to laugh when I read that.
:D What I meant to say is the first alpha that's stable enough for test, but you're right, it's kinda confusing.  Point taken, I edited my post.

Quote
@Bostjan
Is this to be something like "NuGet" for MS visual studio ? (https://en.wikipedia.org/wiki/NuGet, https://www.nuget.org/)
Yes, something similar.

Quote
Worst part of Delphi/Lazarus for me is that you must hunt various packages across the internet and central repository would be so much better.
I agree 100%.
Title: Re: Online Package Manager
Post by: rvk on October 05, 2016, 04:15:22 pm
Nice work :)

Error in trunk:
Code: [Select]
opkman_mainfrm.pas(129,48) Error: Incompatible types: got "<procedure variable type of procedure(TObject;AnsiString;TErrorType;const AnsiString) of object;Register>" expected "<procedure variable type of procedure(TObject;UTF8String;TErrorType;const AnsiString) of object;Register>"
on
Code: Pascal  [Select]
  1.   PackageDownloader.OnJSONDownloadCompleted := @DoOnJSONDownloadCompleted;
Making it this works:
Code: Pascal  [Select]
  1.     procedure DoOnJSONDownloadCompleted(Sender: TObject; AJSON: UTF8String; AErrTyp: TErrorType; const AErrMsg: String = '');
TJSONStringType is an UTF8String in trunk.

Suggestion #1: Making the packages-list more compact. Only expanding when choosing the +

Suggestion #2: Putting the "Online Package Manager" above the "Install/Uninstall packages". I'm very used to be "Install" being the last in that menu :) Although, if this becomes the standard the position at the bottom might be preferred.

Note: The GetIt Package Manager of Delphi resides in the Tools menu. But Delphi doesn't have a dedicated "Packages" menu, only a Component-menu where "Install packages" sits.

Question: What packages will be provided in the end. Will the lazarus-ccr packages/components be added? Now you have the standard packages in the Lazarus distribution, the lazarus-ccr and everything else with separate download via Wiki-links. Maybe all these should be combined.

If these are not combined maybe it should be possible to add multiple "Remote repositories". (That would also allow you to make your own repository in addition to the default ones provided)
Title: Re: Online Package Manager
Post by: Phil on October 05, 2016, 04:38:55 pm
(That would also allow you to make your own repository in addition to the default ones provided)

I would assume this would be the case. That way, a package author only needs to publicize the URL of the package zip. The user types/pastes this URL into the package manager, which does the rest.

I would use a separate .zip for each package. And each .zip would contain its own JSON file for the package. Put a master list of packages elsewhere.

Take a look at how this is implemented in other IDE/apps. For example, QGIS has supported Python plugins for years and has hundreds of plugins to manage. Look at the QGIS External (official) and Experimental plugins dialogs here:

http://www.qgistutorials.com/en/docs/using_plugins.html

Note that official plugins need to be approved. Here's their list of items to pay attention to for fast approval:

http://plugins.qgis.org

Title: Re: Online Package Manager
Post by: lainz on October 05, 2016, 05:08:38 pm
I agree with Phil, so for example I can upload a release on GitHub that in fact is a package, and then anyone can just copy and paste the URL to this program and get the package installed.
Title: Re: Online Package Manager
Post by: Phil on October 05, 2016, 05:19:44 pm
Exactly. The only question in my mind is whether there should be any sort of "official" stamp, or perhaps just a requirement that the package have an appropriate license (modified LGPL, for example), documentation, etc. as indicated by the contents of the JSON file.



Title: Re: Online Package Manager
Post by: howardpc on October 05, 2016, 06:06:01 pm
Apart from questions of improving the functionality, reliability, security etc. of the package manager, careful thought needs to be given to non-software issues such as long term funding of webserver costs, who (or what grouping) has control of the 'official' repositories, what are the criteria for inclusion/rejection of submitted packages, whether some PPA-type system should be in place for addition of private repositories, who will police it in terms of enforcing adherence to the stated policies, and so on.
This is a bold start, but there could be a minefield ahead in terms of future maintenance and oversight, not so much of the software itself, but of the data it manages and the ongoing quality of that data.
Title: Re: Online Package Manager
Post by: lainz on October 05, 2016, 06:49:13 pm
If money is an issue I recommend to host it on GitHub/Sourceforge that has free website that is managed with a repository/ftp for project or organization.

Use standard releases for each package and only mantain a list of json that points to another url where the real zip is available (the zip can be in any host service).

If you choose GitHub/Sourceforge you can add as many mantainers you want, simple give them access to put files in the website.
Title: Re: Online Package Manager
Post by: rvk on October 05, 2016, 07:03:53 pm
Use standard releases for each package and only mantain a list of json that points to another url where the real zip is available (the zip can be in any host service).
If you choose GitHub/Sourceforge you can add as many mantainers you want, simple give them access to put files in the website.
Yes, I would also advise using existing download locations. And because GitHub and Sourceforge are the most commonly used it would be best to implement downloading directly from there. It would also have the benefit of having the latest version and always be up to date.

The json-collection could even be hosted on Github or Sourceforge and downloaded from there on start of the manager. The json-collection would point to the actual download of given project. (Although there should be a solution found for the individual component/packages from lazarus-ccr because they can't be individually downloaded as zip)

Or it should be from here (https://sourceforge.net/projects/lazarus-ccr/files/) but I'm not sure how old those are. The RichMemo in the current manager is more up to date.
Title: Re: Online Package Manager
Post by: Phil on October 05, 2016, 07:16:43 pm
who (or what grouping) has control of the 'official' repositories, what are the criteria for inclusion/rejection of submitted packages, whether some PPA-type system should be in place for addition of private repositories, who will police it in terms of enforcing adherence to the stated policies, and so on.

I'm not necessarily suggesting an "official" repository. That might not work for Lazarus. For example, the QGIS repository has rules that are probably far too rigid for Lazarus (http://plugins.qgis.org). It needs to fit the "culture" of Lazarus. For example, QGIS is a very Euro-centric effort, with many core members being employees of Euro government agencies or their contractors, so a rigid submission policy might naturally fit their bureaucratic modus operandi.

However, that doesn't mean the project manager couldn't have technical requirements. For example (and only as examples):

(1) Package must have a proper license. If it's the same as LCL or FPC RTL, just state so or point to the FPC modified LGPL doc files. If it's something else, anything, just have a link to it (Eclipse, BSD, etc.). The only thing unacceptable might be if nothing is specified in the JSON file. By clearly stating what license is used, the user of the package manager can decide beforehand whether that's even a package they're interested in.

(2) If the package depends on LCL (ie, has LCL under RequiredPgks in .lpk file), then it must specify what LCL interfaces it has been tested against. I typically test against Carbon, Win32, GTK2 at a minimum so that I can legitimately say that the package is cross-platform.

That sort of thing.
Title: Re: Online Package Manager
Post by: molly on October 05, 2016, 07:27:44 pm
Just wanted to mention Delphinus (https://github.com/Memnarch/Delphinus) in case it wasn't already (even though i seem to remember getting the name from this board).

Being able to at least store packages inside your own github account would help tremendously as right now things are scattered all over the internets. At least it would also allow for individuals to create their own packages so these things can be shared.

Decisions about a official or even semi-official package server would perhaps be something for the foundation as well ?
Title: Re: Online Package Manager
Post by: Phil on October 05, 2016, 07:57:51 pm
Decisions about a official or even semi-official package server would perhaps be something for the foundation as well ?

Lazarus already has official packages, which are the ones deemed worthy or essential enough to include with the IDE.

What a package manager can do is establish an unofficial list somewhere between the official IDE packages and the CCR slushpile. At the very least it indicates that a package is still alive and that the author is available to put in the minimal effort of "regularizing" it to meet the package manager's requirements.


Title: Re: Online Package Manager
Post by: Graeme on October 05, 2016, 08:02:30 pm
I agree with Phil, so for example I can upload a release on GitHub that in fact is a package,...
As I already mentioned in the mailing list, and somebody else mentioned it here... The best approach seems to be whan  Delphinus does on Github. Everything is hosted there.  Delphinus use the Github API to detect new packages and automatically present those in the IDE. Anybody can add there own packages via Github plus the required JSON file. No central repository or hosting location is needed. This is the best of all worlds. Now simply port it to Lazarus IDE and enjoy. :)
Title: Re: Online Package Manager
Post by: GetMem on October 05, 2016, 09:19:56 pm
@rvk
You're the first one who actually tested the pacakge, thank you for this!
Quote
TJSONStringType is an UTF8String in trunk.
Fixed! I should have typed "AJSON: TJSONStringType" instead of "AJSON: Ansistring"
Quote
Suggestion #1: Making the packages-list more compact. Only expanding when choosing the +
Done. I updated the link.
Quote
Suggestion #2: Putting the "Online Package Manager" above the "Install/Uninstall packages". I'm very used to be "Install" being the last in that menu :) Although, if this becomes the standard the position at the bottom might be preferred.
I will leave it where it is for now, since we are only testing. It can be changed later if necessary.
Quote
Question: What packages will be provided in the end.
Every package will be accepted in the official repository with one condition, it must be checked first. I will do the work if necessary.
Quote
Will the lazarus-ccr packages/components be added?
Yes.


@Phil, @lainz, @molly, @Graeme
Quote
I would assume this would be the case. That way, a package author only needs to publicize the URL of the package zip. The user types/pastes this URL into the package manager, which does the rest.I would use a separate .zip for each package. And each .zip would contain its own JSON file for the package. Put a master list of packages elsewhere.
I plan to add support for external zip files/repositories, but this is a dangerous game(my only problem with delphinus + the fact, that it force you to use github). The package manager will automatically download/extract the zip, then install it into the IDE. It's a great way to inject a malware into someone's computer. This is why I insist to have an official repository, where the packages can be checked(malware wise + legally as @Phil mentioned). On the other hand, forcing the developers to use only the packages from the official repository it's also wrong, but they must use it at their own risk.


@howardpc
I was hopping that the official repository can be hosted in one of the freepascal.org directory(see fppkg). For now @Leledumbo was kind enough to host the repository on his personal server.

PS: Thank you all for the feedback.
Title: Re: Online Package Manager
Post by: Phil on October 06, 2016, 03:32:09 am
Every package will be accepted in the official repository with one condition, it must be checked first. I will do the work if necessary.

That sounds reasonable.

Quote
I was hopping that the official repository can be hosted in one of the freepascal.org directory(see fppkg). For now @Leledumbo was kind enough to host the repository on his personal server.

It might actually be healthy to be independent of freepascal.org.

In any case, since you guys are doing the work, it's up to you to decide where to host it. Leledumbo's server sounds fine to me.
Title: Re: Online Package Manager
Post by: JuhaManninen on October 06, 2016, 12:52:52 pm
Sorry I haven't tested the package manager yet. I try to do it soon, today or tomorrow.
Earlier discussions have recommended fppkg.
 http://free-pascal-lazarus.989080.n3.nabble.com/Lazarus-An-online-package-manager-tt4043469.html#none
I discussed with GetMem earlier and he says fppkg is not very well suited here. I must take his word on this issue because he is the only person so far to make a functional online package manager.
"Lazarus packagemanager" from Darius Blaszyk uses fppkg but does not work.
The "Aarre" initiative from Mattias and myself has no functional code either.
Several people (including myself) had plans to write such a package manager ... but nobody did until now.

My goal is to include this GetMem's work in Lazarus sources, either as a package or directly in the packages/ source directory.
It also means commit access for him.
It does not need to be perfect initially. As always a piece of code improves by iterations. People can then provide patches to improve it.
Using / not using fppkg is a big design decision but even that can be changed if need be. But again, it needs code from somebody. Simply expressing one's opinion on the issue is not enough.

The management of online packages and the packages included with Lazarus must be integrated seamlessly. Dependencies between all them must be handled automatically.
However this is not required in the initial version either.

The best approach seems to be whan  Delphinus does on Github. Everything is hosted there.  Delphinus use the Github API to detect new packages and automatically present those in the IDE.

Ok, I was planning to ask how it scans all the projects to find Delphi packages. I guess it searches a specific file name using the Github API.
Yes, that is cool and tempting but it must not be the only interface for our package manager. We must support other sources, too. Some packages (Lazarus CCR) are in SourceForge and there are various other options.
So, support for Github API and other APIs should be added later but not initially.

I will answer the Lazarus mailing list post after I have tested the package manager. Please follow it, too.
Talking about mailing list, does it work now for everybody? The known problems were solved by Marc.
Title: Re: Online Package Manager
Post by: JuhaManninen on October 06, 2016, 01:22:13 pm
I plan to add support for external zip files/repositories, but this is a dangerous game(my only problem with delphinus + the fact, that it force you to use github). The package manager will automatically download/extract the zip, then install it into the IDE. It's a great way to inject a malware into someone's computer. This is why I insist to have an official repository, where the packages can be checked(malware wise + legally as @Phil mentioned). On the other hand, forcing the developers to use only the packages from the official repository it's also wrong, but they must use it at their own risk.

A user must not need to learn other alternative URLs for packages, nor copy/paste them. The user must be able to simply search for packages by name + other criteria and get a list of all worthy packages.
It means your JSON config must support an external URL. It should be a trivial change (ok, I write this without testing your code yet).
It means a package author can make the required ZIP file in his repository and ask you to add it to the list. Files don't need to be copied anywhere.
Do SourceForge etc. allow direct download links for files? I think they do. I remember you claimed the opposite. I must study the issue.
Testing your code is now nro 2. in my ToDo list. Wait ...
Title: Re: Online Package Manager
Post by: JuhaManninen on October 06, 2016, 01:36:01 pm
(1) Package must have a proper license. If it's the same as LCL or FPC RTL, just state so or point to the FPC modified LGPL doc files. If it's something else, anything, just have a link to it (Eclipse, BSD, etc.). The only thing unacceptable might be if nothing is specified in the JSON file. By clearly stating what license is used, the user of the package manager can decide beforehand whether that's even a package they're interested in.

An "official" server with a managed configuration file solves the issue. Only legal packages are accepted.
Otherwise the license does not matter, any free package is OK. The license can be commercial, for example if a company wants to deliver a demo of their product, fine. It is not a package manager's task to decide.
The license should be shown in the GUI, yes.

Quote
(2) If the package depends on LCL (ie, has LCL under RequiredPgks in .lpk file), then it must specify what LCL interfaces it has been tested against. I typically test against Carbon, Win32, GTK2 at a minimum so that I can legitimately say that the package is cross-platform.

Yes, that information must be shown, too. There are packages which cannot be cross-platform by definition, for example by using MS Office tools by OLE automation.
Title: Re: Online Package Manager
Post by: GetMem on October 06, 2016, 01:46:17 pm
Quote
@Juha
It means your JSON config must support an external URL. It should be a trivial change (ok, I write this without testing your code yet).
Yes, it is a trivial change.

Quote
It means a package author can make the required ZIP file in his repository and ask you to add it to the list. Files don't need to be copied anywhere.
Ok, and what if another user decide to download and install the the package form the list? The package manager will do it automatically,  without any sanity check. This might lead to problems.
Title: Re: Online Package Manager
Post by: minesadorada on October 06, 2016, 01:46:42 pm
...
Do SourceForge etc. allow direct download links for files? I think they do. I remember you claimed the opposite. I must study the issue.
Yes, LazAutoUpdate uses direct downloads from Sourceforge.
Title: Re: Online Package Manager
Post by: GetMem on October 06, 2016, 01:55:17 pm
Quote
@minesadorada
Yes, LazAutoUpdate uses direct downloads from Sourceforge.
I'm almost certain it's not a direct download. You have to follow a few redirects. Personally I don't like that, though it's easily doable with TFPHTTPClient.
Title: Re: Online Package Manager
Post by: GetMem on October 06, 2016, 01:59:21 pm
@Juha

As an initial step we should make an official repository. I can include a few hundred packages from different location(lazarus ccr, etc...). I think it's more then enough for a start. After that we can add other dependencies/install method and whatnot. We should keep it simple.
Title: Re: Online Package Manager
Post by: JuhaManninen on October 06, 2016, 02:38:58 pm
Ok, and what if another user decide to download and install the the package form the list? The package manager will do it automatically,  without any sanity check. This might lead to problems.
A sanity check is done by you (or other admin) when adding a URL to the JSON configuration file. You must add trusted URLs only.
A centralized and managed configuration has an advantage compared to getting all available packages from GitHub and other services automatically.

As an initial step we should make an official repository. I can include a few hundred packages from different location(lazarus ccr, etc...). I think it's more then enough for a start. After that we can add other dependencies/install method and whatnot. We should keep it simple.
Yes, that is fine initially.
Title: Re: Online Package Manager
Post by: lainz on October 06, 2016, 04:00:26 pm
Hi GetMem, I'm testing it and works nicely.

You can add BGRA-Controls package?

Name: BGRA Controls
Version: 4.1.0.0
Description: BGRA Controls is a set of graphical UI elements that you can use with Lazarus LCL applications.
Author: Dibo, Circular, Lainz and others.
Compatible with: Lazarus: 1.6
Package type and Dependencies: You get them automatically right?
License: LGPL Modified LGPL

You can download it here:
https://github.com/bgrabitmap/bgracontrols/releases


And a suggestion, it should be downloaded himself to get updates of this package manager, not sure if possible but it's something that will be nice.

And show the version number of the package manager somewhere in the window.
Title: Re: Online Package Manager
Post by: Phil on October 06, 2016, 04:08:43 pm
License: LGPL

I read that to mean that it does not include the linking exception that LCL and FPC RTL have. If it does, it needs to include that here.

Title: Re: Online Package Manager
Post by: lainz on October 06, 2016, 04:09:52 pm
License: LGPL

I read that to mean that it does not include the linking exception that LCL and FPC RTL have. If it does, it needs to include that here.

So

License: LGPL with linking exception Modified LGPL

that's the right one.
Title: Re: Online Package Manager
Post by: Groffy on October 06, 2016, 04:34:24 pm
And a suggestion, it should be downloaded himself to get updates of this package manager, not sure if possible but it's something that will be nice.

And show the version number of the package manager somewhere in the window.

Nice idea.

Just installed the package manager, and its working great!

As soon there will be more registered packages, it might be helpfull to introduce categories(?)


Best regards
Title: Re: Online Package Manager
Post by: GetMem on October 06, 2016, 06:14:01 pm
Quote
Hi GetMem, I'm testing it and works nicely.
Thanks for testing, I'm glad it's working.

Quote
You can add BGRA-Controls package?
Yes, but you have to wait a few days, because:
1. The package manager is only completed about 30-40%
2. We didn't decide yet all the details
3. I don't want to bug @Leledumbo each day with a new package.

Quote
Package type and Dependencies: You get them automatically right?
Not yet, I added manually to the JSON. I hope an automated tool will be ready in week or so. However once the dependencies are loaded from the json it will automatically search and resolve each package dependency. For example VirtualStringTree depends on LCLextension. If you try to check VirtualTreeView and LCLExtension is not checked or installed the package manager will warn you. Please try it!

Quote
And a suggestion, it should be downloaded himself to get updates of this package manager, not sure if possible but it's something that will be nice
This is an excellent idea. Since package manager is a package,  will work 100%.

Quote
@Groffy
As soon there will be more registered packages, it might be helpfull to introduce categories(?)
Yes I will add it, but it's not very high on the priority list. For example I just realized that a zip file can contain multiple .lpk files. The package manager can handle it for now. I have to redesign it.
Title: Re: Online Package Manager
Post by: BBasile on October 06, 2016, 06:40:23 pm
Hi,

 I decided to implement the online package manager(opkman). The first alpha version is ready and it works like this: the packages are zipped and stored in a webserver(repository) along with a json file. I chose zip to minimize server load. The json file contains all the necessary info about the packages. Once installed into the IDE, the opkman downloads, then serialize the json file into a package list. The list is  displayed in a tree. On request the packages can be downloaded or/and installed. For now the repository is read-only, it contains only eight, well known(I guess?) packages. The nature/type of the eight package is irrelevant for now, since we are only testing the functionality of the opkman. Few notes

It looks nice. Why don't you use a GitHub or BitBucket repository ?
Title: Re: Online Package Manager
Post by: molly on October 06, 2016, 07:26:18 pm
@GetMem:
You are already getting enough feedback from experienced programmers so i'll stop adding (more) confusion :-)

However, Juha made some good valid points. You can always opt for keeping (manual) control on the package description files yourself.

It is always possible (further down the road) to have different signed packages, official, semi-official, community, and user packages (just to name a few categories). That way it would be possible for individual users to let their own packages be managed by the packagemanager, yet do not bear (any) impact on the official packages. If users  wish to do so they can share their packages amongst each other by providing the descriptions or add personal url repositories to their own copy of the package manager.

It would perhaps be interesting to implement such approach as it allows for users to experiment creating their packages before asking permission to have them added in an 'upgraded' fashion, e.g. from user, to community, from community to semi-official to finally end up in official state (if that would ever happen).

Initially it would be more interesting to get things started, and along the way slowly decide what would be better to implement and what not. There is many many many food for thoughts there.
Title: Re: Online Package Manager
Post by: JuhaManninen on October 06, 2016, 08:27:55 pm
Quote
from user, to community, from community to semi-official to finally end up in official state

Such classification will not work in practice and is not even useful.
There will be good and popular packages, then there will be less good and less popular ones.
Or, maybe an automatic integration to GitHub and SourceForge (if possible) would give "user" packages while the managed packages from server are the "community" packages.
The package manager must differentiate between those 2.

A useful classification would be a user feedback and voting system. It will need server side SW and maybe user authentication.
Let's not think about it yet.

Lazarus distribution has some "official" packages. Some of the current ones can be moved elsewhere when the online system works. One example is the Industrial package.
Ideally Lazarus distribution should include only packages the IDE itself needs. It keeps the distribution slim.
Title: Re: Online Package Manager
Post by: shobits1 on October 06, 2016, 08:30:17 pm
I think it should show the status of the current Installed package more visually, maybe using color codes:
Green: for the already installed packages.
Yellow: for the installed and need update packages.
Neutral: for the other packages.

It may also be helpful(when the list grows) to have checkbox filter for those status.
Title: Re: Online Package Manager
Post by: shobits1 on October 06, 2016, 08:33:02 pm
It should be helpful to provide (if available) URL for Wiki, documentation, Tutorial, Demos... in the package info.
Title: Re: Online Package Manager
Post by: JanRoza on October 06, 2016, 09:29:01 pm
It works great but I see that packages installed via opkman are placed in a subfolder 'Packages' under the opkman folder whereas all other packages installed in Lazarus reside in the folder 'components' under the Lazarus folder.
I think opkman should save the packages in the same location as Lazarus does to keep things uniform and simple.
Other than that: opkman is a great Lazarus add-on.
Title: Re: Online Package Manager
Post by: lainz on October 06, 2016, 09:43:05 pm
I think opkman should save the packages in the same location as Lazarus does to keep things uniform and simple.

I think that is not a good idea.

What if I have multiple lazarus (I have) and I want all the components installed in all of them. I can use this package manager to achieve that but only if the folder is as now: no duplicates, no lost packages.
Title: Re: Online Package Manager
Post by: BBasile on October 06, 2016, 09:43:15 pm
Quote
opkman should save the packages in the same location as Lazarus does to keep things uniform and simple.
No, under Linux it would suck because authentification would be required (Lazarus Components folder is not writable without sudo)
Title: Re: Online Package Manager
Post by: GetMem on October 06, 2016, 09:50:50 pm
Quote
@BBasile
It looks nice. Why don't you use a GitHub or BitBucket repository ?
I will do my best to support GitHub, BitBucket, SourceForge, but at a later stage.

Quote
@molly
Initially it would be more interesting to get things started, and along the way slowly decide what would be better to implement and what not. There is many many many food for thoughts there.
My thoughts exactly.

@Juha
Did you get the chance to run a few tests?

Quote
@shobits1 I think it should show the status of the current Installed package more visually, maybe using color codes:
Green: for the already installed packages.
Yellow: for the installed and need update packages.
Neutral: for the other packages.
Actually this was implemented, but the package manager looked like a cockatoo bird.  :D Too many colors, I gave up.

Quote
@shobits1
It may also be helpful(when the list grows) to have checkbox filter for those status.
Please see attachment.

Quote
@JanRoza
whereas all other packages installed in Lazarus reside in the folder 'components' under the Lazarus folder
Under linux and osx the component folder sometimes is read only. Even under windows(especially win 10) you can run into privilege issues.
Title: Re: Online Package Manager
Post by: Phil on October 06, 2016, 09:56:14 pm
Visually it looks pretty good. One minor typo:

Dependecies --> Dependencies.

If a zip has more than one .lpk, will each package have its own checkbox? Eg, lazxproj_src.zip has two .lpk, one runtime, one installable:

https://macpgmr.github.io/MacXPlatform/LazXProj.html
Title: Re: Online Package Manager
Post by: lainz on October 06, 2016, 10:03:29 pm
Visually it looks pretty good. One minor typo:

Dependecies --> Dependencies.

If a zip has more than one .lpk, will each package have its own checkbox? Eg, lazxproj_src.zip has two .lpk, one runtime, one installable:

https://macpgmr.github.io/MacXPlatform/LazXProj.html

Good question. BGRA Bitmap has 4 packages on it, what is installed? the packages are one for lcl, fpgui, nogui and open gl.
Title: Re: Online Package Manager
Post by: GetMem on October 06, 2016, 10:36:58 pm
Quote
@Phil
isually it looks pretty good. One minor typo:
Dependecies --> Dependencies.
Thanks! Corrected, link updated.

Quote
If a zip has more than one .lpk, will each package have its own checkbox? Eg, lazxproj_src.zip has two .lpk, one runtime, one installable:
Yes, each package will have its own checkbox. I have to redesign the serialization.

Quote
@lainz
Good question. BGRA Bitmap has 4 packages on it, what is installed? the packages are one for lcl, fpgui, nogui and open gl.
One by default, the others is up to user? Except when one depends on the other. This is getting very complicated fast.  :)
Title: Re: Online Package Manager
Post by: shobits1 on October 06, 2016, 11:05:21 pm
Quote
@shobits1
It may also be helpful(when the list grows) to have checkbox filter for those status.
Please see attachment.
It needs a filter for the packages with available new updates.
Title: Re: Online Package Manager
Post by: JuhaManninen on October 06, 2016, 11:56:57 pm
It works great but I see that packages installed via opkman are placed in a subfolder 'Packages' under the opkman folder whereas all other packages installed in Lazarus reside in the folder 'components' under the Lazarus folder.

Not true.
Only the packages included in Lazarus distribution reside under 'components' folder. Other packages can be placed anywhere.
Title: Re: Online Package Manager
Post by: lainz on October 07, 2016, 12:03:39 am
GetMem you can host your package manager on GitHub, so it's easy to get a newer version when you're developing it.
Title: Re: Online Package Manager
Post by: JuhaManninen on October 07, 2016, 12:44:55 am
GetMem you can host your package manager on GitHub ...

+1
I don't know how quickly we can get you commit access to Lazarus repo. I don't even know yet what other Lazarus developers say about this package manager.

I finally tested it. Works well! VirtualTreeview package was installed together with its dependency.
When this gets integrated with the current package management code, dependencies will be handled by it (I think).

You have implemented your own code for TZipper, TUnZipper etc. You should use the TZipper provided by FCL :
  http://wiki.freepascal.org/paszlib#TZipper
BTW, your zipper code has a memory leak somewhere.

You have also added a copy of the VirtualTreeview code into your package. Can't it use the VirtualTreeview package as a dependency? Code duplication is always bad for maintenance.

You used the UTF8 specific functions like DirectoryExistsUTF8(). They are not needed any more with Lazarus + FPC 3.
I think FPC 3+ can be made the minimum requirement for this new code.
Ok, maybe not yet. Don't change them.
Title: Re: Online Package Manager
Post by: Leledumbo on October 07, 2016, 06:24:11 am
GetMem you can host your package manager on GitHub, so it's easy to get a newer version when you're developing it.
I kinda agree with this. Adding packages is a matter of approving merge request then, while maintaining code reviews and other things, with history. Otherwise, I can make my server a git server as well, but it will lack the convenience of github UI.
Title: Re: Online Package Manager
Post by: GetMem on October 07, 2016, 07:29:38 am
@Juha
Quote
When this gets integrated with the current package management code, dependencies will be handled by it (I think).
Yes.

Quote
You have implemented your own code for TZipper, TUnZipper etc. You should use the TZipper provided by FCL :
 http://wiki.freepascal.org/paszlib#TZipper
Not quite! I used the same unit provided by FCL, but with a few modifications, this is also true for httpclient. Neither one provides a timeout function. For example when you try to connect to an offline server with httpclient, sometimes it waits 25 - 30 sec before returns an error. The same is true for zip/unzip from Zipper.pas, once started you have to wait until the entire zip process is over, no possibility to abort the process. More over httpclient doesn't even show a download progress. I had to create a thread timer and a modified streamer class to be able to provide a feedback(see opkman_downloader.pas unit). The possibility to show a progress with TZipper is also limited, (although is there) so I had to enhance the function.
I know this is code duplication. Ideally I should create a patch for those units, but when will be accepted(if ever)? Since zipper.pas and httpclient.pas functionality is more then enough in 99% of the cases,  fpc core members would not accept a patch just to satisfy the online package manger needs(at least this is what I think).

Quote
BTW, your zipper code has a memory leak somewhere.
I always develop with "Debug IDE" profile to avoid leaks. I fixed many leaks in my package, obviously I missed something. Thanks.

Quote
You have also added a copy of the VirtualTreeview code into your package. Can't it use the VirtualTreeview package as a dependency?
Initially I wanted to avoid third party components. Unfortunately TTreeView is so limited in so many ways I decided to switch to VST. The only version which works well in every major widgetset is VST4.8.7R2. The newer one doesn't even compile under osx for example. What if somebody already has a newer version installed? It will lead to conflicts, multiple resource errors, etc. It's not a good practice to force the developers to use a specific version of VST, just to be able to install opkman. So I added VST internally to the package manager, even replaced the resource names with a resource editor to avoid conflicts. This way it can happily coexists with any version of VirtualStringTree. Code duplication again? Perhaps but the benefits outweigh the drawbacks.

Quote
@shobits1
It needs a filter for the packages with available new updates.
Ok. I can add this later.

@lainz, @Leledumbo
I will add the source to github soon.
Title: Re: Online Package Manager
Post by: rvk on October 07, 2016, 07:58:04 am
Since zipper.pas and httpclient.pas functionality is more then enough in 99% of the cases,  fpc core members would not accept a patch just to satisfy the online package manger needs (at least this is what I think).
If that is the case they should at least change the source so that you can inherit the code and enhance it. A lot of the time this is not possible due to unnecessary private variables.

I also hacked one of those http classes once because there was only a download progress option but none for upload (like for uploading a large file via a post). Luckily that could be done via inheritance but often this is not possible.

If a class is designed, so you would need to copy the whole source to add such functionality, I think it is designed poorly.

B.T.W. I thought httpclient did have a download progress option but it lacked an upload progress. (Can't look right now but I think it's the OnDataReceived in dodataread. But it lacks a OnDataSend)
Title: Re: Online Package Manager
Post by: GetMem on October 07, 2016, 08:50:32 am
Quote
@rvk
Can't look right now but I think it's the OnDataReceived in dodataread. But it lacks a OnDataSend
True! With my streamer class can do both download/upload progress threaded. @rvk if you find a way to abort a download/upload process via class inheritence let me know. The same is true for a timeout function.
Title: Re: Online Package Manager
Post by: JuhaManninen on October 07, 2016, 11:02:03 am
I know this is code duplication. Ideally I should create a patch for those units, but when will be accepted(if ever)? Since zipper.pas and httpclient.pas functionality is more then enough in 99% of the cases,  fpc core members would not accept a patch just to satisfy the online package manger needs(at least this is what I think).

Sounds like those features are valid for many use cases, not only for the online package manager.
SW always improves by iterations. This is true for FCL code, too.
I believe a patch for both zipper and httpclient would be accepted. Even more so because an essential Lazarus package must copy/paste FCL code otherwise.
[Edit] A patch may be included in FPC 3.2. It means your copied version is a temporary solution until that compiler version becomes the minimum requirement. We must always think long-term.

VirtualTreeview is more difficult. The porting process continues by Luiz Américo if I understand right.
The long term goal must however be to reuse code. It would mean to move VirtualTreeview to Lazarus codebase together with online package manager once everything works.
I personally like the VirtualTreeview component. I made some innovative SW with it and Delphi years ago.

A thought: if the management of local packages and online packages is integrated in Lazarus code, the non-GUI parts of your code must be moved to Lazarus code base. It means GUI must be separated from logic even more than it is now.
Another choice is to add hooks to the Lazarus package management interface so that available online packages can be registered by external code.
Title: Re: Online Package Manager
Post by: Graeme on October 07, 2016, 11:26:07 am
Good question. BGRA Bitmap has 4 packages on it, what is installed? the packages are one for lcl, fpgui, nogui and open gl.
And fpGUI has one for each GUI backend (GDI, X11, Carbon, OS/2 etc)
Title: Re: Online Package Manager
Post by: rvk on October 07, 2016, 11:30:34 am
Quote
@rvk
Can't look right now but I think it's the OnDataReceived in dodataread. But it lacks a OnDataSend
True! With my streamer class can do both download/upload progress threaded. @rvk if you find a way to abort a download/upload process via class inheritence let me know. The same is true for a timeout function.
I made a helper for TFPHTTPClient (for a WebDav client) in which I override the complete SendRequest() to call a special CopyFromStreamToStream() which has a Callback() function for upload-progress. Maybe you implemented it the same way for download (haven't looked yet). It can be easily changed that the Callback() function can receive an Abort-boolean in which case the CopyFromStreamToStream() can abort it's operation.

The only problem I had was that "socket" wasn't exposed in TFPHTTPClient. And that's a serious problem. So I had to change the original fphttpclient.pp. I just added the following line to my copied fphttpclient.pp and I could use socket in my helper and stream.
Code: Pascal  [Select]
  1.     Property Socket: TInetSocket read FSocket;

But I do agree with JuhaManninen. An upload-callback as well as Abort-option should be added to TFPHTTPClient. It would even be a very small patch. The Download-progress is already there in the DoDataRead (which calls OnDataReceived). So the FOnDataSent would simple to implement. Only 3 lines in TFPHTTPClient but with a complete CopyFromStreamToStream() because TStream.CopyFrom() doesn't have a progress-option. Or should that one be added too :)

I was also very surprised to see the download-progress option in TFPHTTPClient was added but upload-progress option was left out.
Title: Re: Online Package Manager
Post by: JuhaManninen on October 07, 2016, 11:56:45 am
BGRA Bitmap has 4 packages on it, what is installed? the packages are one for lcl, fpgui, nogui and open gl.

I think there should be only one package in a ZIP file.
Having many packages adds a useless extra complication.
Title: Re: Online Package Manager
Post by: GetMem on October 07, 2016, 12:15:43 pm
@Juha, @rvk
Great! Then we need a patch for httpclient/zipper. Right now I'm busy with opkman, but I will do it later. If somebody wants to help, I would be grateful.

Quote
@Juha
I think there should be only one package in a ZIP file.
Having many packages adds a useless extra complication.
That's true, however many big packages(zeos, ibobjects, bgrabitmap) just to mention a few, contains multiple lpk files. Which one should be installed by default? It would be a great help for me if only one package/zip is installed.
Title: Re: Online Package Manager
Post by: rvk on October 07, 2016, 12:34:03 pm
Ok, did some more testing. Here is a list. Don't mind some of the points (they might seem like nitpicking) but they are just some second observations.
(SPM = Standard Package Manager)
Code: [Select]
[~]$ fppkg list
Name                 Installed        Available
embweb               -                0.9.1-968
gecko                -                2.9.0-3
jquery               -                0.9.0-842
lazmkunit            -                0.9.3-1
lnet                 -                0.6.6-2606
webdesign            -                0.9.0-906
Title: Re: Online Package Manager
Post by: JuhaManninen on October 07, 2016, 01:56:50 pm
Maybe a button to go directly to "Install/Uninstall Packages". Not sure if this is possible (opkman is ShowModal and SPM too). But it can be handy to take a quick look in SPM and escape right back into opkman.

That would be clumsy. The goal is to integrate management of online package and local packages. The SPM would then show also online packages and this new VirtualTreeview based GUI would show local packages. In essence the new GUI will be an alternative to the old GUI. Competition is good.

Quote
Regarding the multiple packages in one zip/download. Maybe there can be checkboxes per .lpk inside the treeview which to install. Otherwise there should be some dialog after choosing a package as to which package you really want installed. Moving each .lpk into it's own zip/source is not advised because all those .lpk use the same source and multiple "same sources" will eventually get you in trouble when compiling.

Currently people must select from individual .lpk files anyway, don't they?
I don't know of all the complexities involved but maybe the package manager should add a new meta-package type which includes many actual packages. In the GUI those packages would show as sub-nodes under the meta-package parent node.
Just an idea ...
Title: Re: Online Package Manager
Post by: lainz on October 07, 2016, 02:53:07 pm
BGRA Bitmap has 4 packages on it, what is installed? the packages are one for lcl, fpgui, nogui and open gl.

I think there should be only one package in a ZIP file.
Having many packages adds a useless extra complication.

Yes but why to duplicate the same source code and just for getting a different zip? Maybe for lpk only a file is changed to fit the widgetset, all other files remains the same.
Title: Re: Online Package Manager
Post by: GetMem on October 07, 2016, 06:00:57 pm
@rvk
Quote
When I install a package and choose NOT to rebuild and close opkman and open it again... the opkman shows the package as "Already installed" while it is not yet.
That's not true, it is installed. Close Opkman(without rebuild), then open "Install/Uninstall packages" and check the installed package list. More over, if you close Lazarus and delete the just installed package directory from the filesystem, on next startup Lazarus will complain about a missing package. You cannot use it until an IDE rebuild but it is installed.
Quote
Would the "Downloaded" status be really necessary? When not installing a package and again pressing install, wouldn't re-downloading it and installing it be more logical (you would also have the updated version if any). Or is "Downloaded" also for non-installable packages (like synapse)?
Search for ForceDownload and ForceExtract booleans. The user will choose the appropriate action. It is already implemented(false by default),  not yet visible in the option dialog though.
Quote
"Already installed(Version: 1.0.0.0)". There is no space between installed and (Version). That said... maybe 2 columns with Current version and Online/updated version would be useful. Like the fppkg does with "Installed" and "Available" columns. (See at the bottom of this post)
Escape doesn't work to close the opkman. There should probably a Cancel button for that just like in the "Install/Uninstall Packages" to easily exit the dialog.
Maybe a button to go directly to "Install/Uninstall Packages". Not sure if this is possible (opkman is ShowModal and SPM too). But it can be handy to take a quick look in SPM and escape right back into opkman.
An extra column with the (short)description of the package in the folded-view. There is plenty room to move Status (and current and available version #) in a separate column.
Abbreviate "Status" to something simpler. In that case that extra column doesn't need to be that wide. It's hard to see what's available/installed or downloaded at the moment. Just Available, Downloaded and Installed will be sufficient. And it will be easier on the eyes.
After the Downloaded and Installed could be an update status (like " - Updated").
Only make the " - Updated" status bold. All others should be plain. The bold status also makes it hard to read. (Especially when my eyes are tired :) )
Those are all valid and useful suggestions. I will implement them.
Quote
When those hundreds of packages are added (good luck with that  :P ) a category-pulldown will be useful (I think it was already suggested).
:D Ok fair enough, I won't find that many packages but adding one package to the repository will be a matter of seconds, once the automated tool is ready.
Quote
Regarding the multiple packages in one zip/download. Maybe there can be checkboxes per .lpk inside the treeview which to install. Otherwise there should be some dialog after choosing a package as to which package you really want installed. Moving each .lpk into it's own zip/source is not advised because all those .lpk use the same source and multiple "same sources" will eventually get you in trouble when compiling.
Each lpk will have a checkbox.
Quote
Moving the development of the opkman to SourceForge or GitHub is highly appreciated. We (I) can keep track of progress and automatically SVN a new version into my new trunk-compile scripts :) (unless you have Lazarus repo access real soon)
Done: https://github.com/getmem/LazOpkMan

@Juha, @lainz
I will implement the multiple lpk version. Each lpk will have it's own checkbox, description etc.

   
Title: Re: Online Package Manager
Post by: valdir.marcos on October 07, 2016, 06:56:51 pm
What about big complex packages that fits simultaneously many versions of FPC (for instance 2.6.x, 3.0.x, 3.2.x, etc) or many versions of Lazarus (for example 1.0.x, 1.2.x, 1.4.x, 1.6.x, 1.8.x/2.0.x, etc)? Some of them might have different .lpk files for different versions of FPC or Lazarus.

BGRA Bitmap has 4 packages on it, what is installed? the packages are one for lcl, fpgui, nogui and open gl.

I think there should be only one package in a ZIP file.
Having many packages adds a useless extra complication.

Yes but why to duplicate the same source code and just for getting a different zip? Maybe for lpk only a file is changed to fit the widgetset, all other files remains the same.
Title: Re: Online Package Manager
Post by: valdir.marcos on October 07, 2016, 07:03:02 pm
Just curious.
This Lazarus repository will be similar to Debian repositories that have stable, testing and unstable versions of the packages?
Title: Re: Online Package Manager
Post by: JuhaManninen on October 07, 2016, 07:27:30 pm
BTW, your zipper code has a memory leak somewhere.

It is not in the zipper code.
There is a leak after installing a package and building / restarting the IDE. I could not figure out where exactly.
Title: Re: Online Package Manager
Post by: JuhaManninen on October 07, 2016, 07:28:41 pm
This Lazarus repository will be similar to Debian repositories that have stable, testing and unstable versions of the packages?

No.
Title: Re: Online Package Manager
Post by: GetMem on October 07, 2016, 10:22:03 pm
Quote
@Juha
It is not in the zipper code.
There is a leak after installing a package and building / restarting the IDE. I could not figure out where exactly.
I was able to reproduce the memory leak. It's caused by "DoOpenPackageFile" function from inside pkgmanager.pas called through packageintf.pas. If I remove the function call the memory leak is gone. Something is not freed inside "DoOpenPackageFile".
Title: Re: Online Package Manager
Post by: lainz on October 12, 2016, 05:48:49 pm
How it is going, you created the git repository?

Edit: see the first post, the repository was created 5 days ago.  ::)
Title: Re: Online Package Manager
Post by: GetMem on October 12, 2016, 09:00:46 pm
Quote
@lainz
How it is going?
I'm working on it. To support multiple packages, I had to redesign a large part of the code.(see attachment). When it will be ready? I cannot tell you exactly. Hopefully soon.
Please suggest categories, for now I have:  Networking, Graphics, Multimedia, Reporting, Miscellaneous. What else? BGRABitmap is graphics, mediaplayer multimedia  :D,
no-brainer so far, but in which category can I include VirtualStringTree for example?

Quote
you created the git repository
Yes, I updated my first post a few days ago.
Title: Re: Online Package Manager
Post by: lainz on October 12, 2016, 09:05:23 pm
As I can see in your screenshot:

BGLControls is not BGRAControls! BGLControls is an OpenGL control to use with BGRABitmap for OpenGL.

BGRAControls is located here: https://github.com/bgrabitmap/bgracontrols , and is a separate package/download.
Title: Re: Online Package Manager
Post by: GetMem on October 12, 2016, 09:31:12 pm
Quote
@lainz
BGLControls is not BGRAControls! BGLControls is an OpenGL control to use with BGRABitmap for OpenGL.
Thanks, noted! The eight packages is for testing purposes only, the whole repository will be recreated.
Title: Re: Online Package Manager
Post by: lainz on October 12, 2016, 09:40:47 pm
Ok, is a really nice feature, hope some day get bundled and distributed with lazarus.
Title: Re: Online Package Manager
Post by: howardpc on October 12, 2016, 09:48:37 pm
Please suggest categories, for now I have:  Networking, Graphics, Multimedia, Reporting, Miscellaneous. What else? BGRABitmap is graphics, mediaplayer multimedia  :D ,
no-brainer so far, but in which category can I include VirtualStringTree for example?

DataControls or GUIContainers perhaps.
Could I suggest avoiding Miscellaneous altogether? It's as bad as the name 'Anonymous'  ;D . Or, if you must, how about Other (it's shorter and no less informative).
Title: Re: Online Package Manager
Post by: JuhaManninen on October 12, 2016, 10:08:59 pm
DataControls or GUIContainers perhaps.

GUIContainers or VisualContainers is good.
DataControls brings an association to DB data aware controls.
Title: Re: Online Package Manager
Post by: GetMem on October 12, 2016, 10:10:00 pm
Quote
@howardpc
DataControls or GUIContainers perhaps.
VirtualStringTree is a  GUIContainer. I like that! Thanks.

Quote
Could I suggest avoiding Miscellaneous altogether? It's as bad as the name 'Anonymous'  ;D . Or, if you must, how about Other (it's shorter and no less informative).
Done.

Edit: I need more  :). @Juha?
Title: Re: Online Package Manager
Post by: avra on October 16, 2016, 05:48:12 pm
Please suggest categories, for now I have:  Networking, Graphics, Multimedia, Reporting, Miscellaneous. What else?
Take a look at left frame on http://torry.net/pages.php?id=20. You decide if you want such fragmentation, but anyway you will get an idea about possible categories.
Title: Re: Online Package Manager
Post by: GetMem on October 16, 2016, 06:37:05 pm
Quote
@avra
Take a look at left frame on http://torry.net/pages.php?id=20. You decide if you want such fragmentation, but anyway you will get an idea about possible categories.
Thanks avra! It's useful.
Title: Re: Online Package Manager
Post by: Xor-el on October 16, 2016, 07:23:35 pm
Awesome Job you are doing @GetMem, I have been wanting something like this for a while for FreePascal and it looks like my prayers have been answered.
Thanks a lot.
How about we have a category called Cryptography?
Title: Re: Online Package Manager
Post by: circular on October 16, 2016, 08:22:12 pm
That's a great idea.

I am wondering about updates: how to update the repository with latest version? Let's say I publish a new version of BGRABitmap, how can I update the version number in the package manager?
Title: Re: Online Package Manager
Post by: shobits1 on October 16, 2016, 08:32:56 pm
How about category for the IDE related things, like anchor docking, debuggers, ... maybe call it "IDE Enhancement" or something else (I'm not great with naming).
Title: Re: Online Package Manager
Post by: howardpc on October 16, 2016, 08:42:07 pm
How about category for the IDE related things, like anchor docking, debuggers, ... maybe call it "IDE Enhancement" or something else (I'm not great with naming).
Perhaps LazIDEPlugins ?
Title: Re: Online Package Manager
Post by: GetMem on October 16, 2016, 10:02:53 pm
Thanks guys! I will add a few more categories soon. I just realised we need an entry for supported widget set.
Quote
@circular
I am wondering about updates: how to update the repository with latest version? Let's say I publish a new version of BGRABitmap, how can I update the version number in the package manager?
Each package has a "Home page" and a "Download URL" section(see screenshot from previous page). On request the package manager can download and install the updated package.
Title: Re: Online Package Manager
Post by: Memnarch on October 17, 2016, 03:14:00 pm
Hi,
I am the author of Delphinus, so i thought i'd give my 2 cents:

First of all: Cool to see such a project for the Lazarus-Community. Several Month ago i was asked to port Delphinus to Lazarus, but i declined. I am way to busy with the Delphi-Version. And on top of that, my general knowledge of Lazarus and Freepascal is quite sparse.

About the security/legal-concerns of Delphinus and the reason to control everything on this Package-manager: This is going to be a really heavy task. Even systems like NuGet do not have a QA-Process. The "Report"-Button is there for a reason (the Delphinus Equivalent is just to report a Repository on Github if it violates law or has malicious code).

Your approach has a (quite dangerous) problem: By doing some kind of validation (and not opting out!) you give a false sense of safety and legal security. If something goes wrong, it might all fall back to you. Doing nothing and having an option to report, might be a better one.

Let the community hang the bad people for you ;)

Greetings
Memnarch
Title: Re: Online Package Manager
Post by: JuhaManninen on October 17, 2016, 04:42:32 pm
Hi,
I am the author of Delphinus, so i thought i'd give my 2 cents:
Hi, and congratulations for your Delphinus. I plan to study it and get inspiration from it.

You are right, managing the system will require some work. There will be hundreds of packages at some point.
On the other hand Linux and BSD distributions manage their repositories which contain much more packages, sometimes compiled for many alternative architechtures.
If I understood GetMem right, our package manager will be able to download latest package versions from their trusted URLs. It means managing a package's version moves to its author.

Using the Delphinus approach as a main package manager has also other issues than QA. We would support only one of the commercial source hosting services. People would ask: "Why are you tied to GitHub? Do they pay you to ignore their competition?"
Well, they don't pay us. It would be fair to support Sourceforge, Bitbucket etc. Not all people like Git, why should Lazarus project force it on them?
I hope other hosting services also have APIs like GitHub has. In ideal situation we would support all the major ones, in addition to the managed main repository.

Regards,
Juha
Title: Re: Online Package Manager
Post by: Memnarch on October 17, 2016, 07:28:27 pm
I don't want to hijack this thread, but i'd like to explain why i used github:
None of the others had a suitable searchapi. I checked Bitbucket and Gitlab. Githubs is excellent to do what i wanted: Create a Packageindex without much work. It allows me to get started without managing a remote myself and give the Community control of the packages. That was essential for me for the beginning. If you read some of my older posts on my blog, you'll notice that i talk about adding support for multiple remotes for the future. And that's what is going to be the next project. However, this will not include Bitbucket or one of the other Big hosters. It is more intended to be an interface for defining/adding new types of remotes. I'll provide a Directory-Based one (good for offline environments).

If any of the other hosters (and i still hope for Bitbucket but their API is way beyond GitHubs) at some point extend their search api, i'll support them aswell. After all everything is abstracted by interfaces.

EDIT: But maybe at some point, i'll just abuse the Issue-Trackersystem of Github to allow other sources.
Title: Re: Online Package Manager
Post by: jwhitten on October 19, 2016, 08:52:03 am
Is this to be something like "NuGet" for MS visual studio ? (https://en.wikipedia.org/wiki/NuGet, https://www.nuget.org/)

If so, I think it is really good idea. Worst part of Delphi/Lazarus for me is that you must hunt various packages across the internet and central repository would be so much better.

I completely agree!!

+1,000,000
Title: Re: Online Package Manager
Post by: JuhaManninen on October 19, 2016, 11:31:27 am
Talking about NuGet ... can somebody please tell me how it manages the packages?
I understood it has a centralized place for them but who is allowed to add / remove / update packages?

It has server side SW which enables interactive feedback from users.
It would be useful in our system, too, but will not be implemented initially for sure.
Title: Re: Online Package Manager
Post by: Memnarch on October 19, 2016, 11:46:27 am
Users do:
https://www.nuget.org/users/account/LogOn?ReturnUrl=%2Faccount

Register and you get access to it.
Title: Re: Online Package Manager
Post by: GetMem on October 19, 2016, 05:41:17 pm
@Memnarch

Finally I got the chance to study Delphinus. Very nicely done! Porting to Lazarus though it would require a tremendous effort, it contains a lot of microsoft specific stuff. I'm not even sure it's possible,  at least not without rewriting a large part of the code.
Title: Re: Online Package Manager
Post by: circular on October 19, 2016, 05:46:14 pm
Each package has a "Home page" and a "Download URL" section(see screenshot from previous page). On request the package manager can download and install the updated package.
Oh I see.

Update is down by pressing the green cyclic arrows?
Title: Re: Online Package Manager
Post by: GetMem on October 19, 2016, 05:47:46 pm
Quote
@circular
Update is down by pressing the green cyclic arrows?
That's the refresh button. Update is not yet implemented.
Title: Re: Online Package Manager
Post by: GetMem on October 19, 2016, 05:50:16 pm
The ETA for the next release is monday (24.10.2016). After the release, everyone can create packages for the main repository. Until then a quick preview(win32/64, gtk2, carbon).
Title: Re: Online Package Manager
Post by: Memnarch on October 19, 2016, 05:58:28 pm
@GetMem a lot of interfaces need to be reimplemented. The critical stuff is all encapsulated. But you're right. Behind the scenes i use a lot MS-Specific stuff, which is problematic in that case.
Title: Re: Online Package Manager
Post by: JD on October 22, 2016, 03:58:23 pm
This is wonderful! I'm looking forward to its release. Kudos to GetMem and all who are involved with this project.

JD
Title: Re: Online Package Manager
Post by: lainz on October 22, 2016, 03:59:47 pm
The ETA for the next release is monday (24.10.2016). After the release, everyone can create packages for the main repository. Until then a quick preview(win32/64, gtk2, carbon).

Is looking well, two days left  :D
Title: Re: Online Package Manager
Post by: Xor-el on October 24, 2016, 09:07:24 am
D Day!!!  :-*
Title: Re: Online Package Manager
Post by: lainz on October 24, 2016, 08:54:25 pm
D Day!!!  :-*

Yeah, hope it get's published soon.
Title: Re: Online Package Manager
Post by: GetMem on October 24, 2016, 09:29:04 pm
I'm ready, we have to wait until @Leledumbo updates the repository. So please be patient...
Title: Re: Online Package Manager
Post by: lainz on October 24, 2016, 09:33:28 pm
OK. I've downloaded the package from GitHub. You will create a wiki or something with info on how to create an own repository (website) with packages?
Title: Re: Online Package Manager
Post by: GetMem on October 24, 2016, 09:48:22 pm
Quote
@lainz
OK. I've downloaded the package from GitHub. You will create a wiki or something with info on how to create an own repository (website) with packages?
Unfortunately that feature is not available yet, it will be in the next release though(see attachment). We should populate the central repository with packages first,  this is the main purpose of this release + bug fix. However if you really want a personal repository we can make one ;)

Title: Re: Online Package Manager
Post by: lainz on October 24, 2016, 10:12:53 pm
I mean for example if it's possible to create a personal repository using GitHub pages, so I can put all the packages I want and then download and install them in every PC I want.

If possible if you can create one as example.

Edit: Of course if the packages are all in the central repository it has no sense  8)

If you will create that I will wait for next release.
Title: Re: Online Package Manager
Post by: shobits1 on October 24, 2016, 10:46:13 pm
I keep getting 'Invalid JSON file'??
Title: Re: Online Package Manager
Post by: GetMem on October 24, 2016, 10:48:15 pm
I'm ready, we have to wait until @Leledumbo updates the repository. So please be patient...
Title: Re: Online Package Manager
Post by: shobits1 on October 24, 2016, 10:55:32 pm
I'm ready, we have to wait until @Leledumbo updates the repository. So please be patient...
Ah,, thank you...
sorry I didn't pay much attention -- I started suspecting my crappy internet connection right away.
Title: Re: Online Package Manager
Post by: GetMem on October 24, 2016, 11:00:39 pm
Quote
Ah,, thank you...
sorry I didn't pay much attention -- I started suspecting my crappy internet connection right away.
No problem! I will post a message when everything is up and running.
Title: Re: Online Package Manager
Post by: lainz on October 25, 2016, 03:44:25 pm
Well it's running.

There's a way to send you a new package?
Title: Re: Online Package Manager
Post by: GetMem on October 25, 2016, 04:58:51 pm
The new repository is up and running(thank you @Leledumbo!). Please go to https://github.com/getmem/LazOpkMan download then install lazopkman(I commited a few things recently).
What's new:
- support for multiple packages in a single zip
- from now on everyone can create packages for the main repository
- improved GUI
- a lot of bugfix
- I did try to implement all reasonable suggestion from this thread, if I missed something please let me know

To do:
- update from external site(this one will be ready in a few days)
- create a personal repository

The purpose of this release is to populate the main repository with packages. To create a repository package please go to Create-->Create repository package toolbutton, then:
- select the package main directory(lazopkman will recursively search for all package files inside the folder)
- select the packages you wish to include
- choose a category
- adjust package info if necessary(lazarus/fpc compatibility, suppported widgetset, etc)
- press the create button
The whole process should take no longer then 1-2 min/ package. If all goes well you should get a zip and a json file. Upload somewhere the files and send me the link, alternatively you can send it to getmem1@gmail.com . I will wait a week or so then update the main repository agin. A realistic target is 50+ packages.

PS: What about Lazarus Code and Component Repository? Most of the packages are very old. We should include all of them? Who's familiar with the packages inside LCCR?
 
Title: Re: Online Package Manager
Post by: wp on October 25, 2016, 05:22:24 pm
PS: What about Lazarus Code and Component Repository? Most of the packages are very old. We should include all of them? Who's familiar with the packages inside LCCR?
Yes, many of them are in a bad shape. These are the ones that I am using and which I know are fine:
There are probably more good ones, I just did not try them, so I can't tell. Look at the date of the last update, it's a good indicator of how well the package is maintained.

Title: Re: Online Package Manager
Post by: lainz on October 25, 2016, 06:18:00 pm
Hi I sent you 2 new packages by PM.
Title: Re: Online Package Manager
Post by: rvk on October 25, 2016, 06:43:11 pm
Maybe it's a good idea to add a submission form to the manager (or one somewhere online) from where you get a uniform request for adding a package or updating one. The required json could even be generated through that so that you would only need to review the request before adding it.

Otherwise you'll get a lot of request in a lot of different ways.
Title: Re: Online Package Manager
Post by: lainz on October 25, 2016, 06:54:44 pm
It's a good idea. The form can send you an email with the details. I think is better to add it to the package manager to ensure at least is a Lazarus programmer that sends the request. Spammers will be less, not zero but less.

Edit: The package BGRA Controls I sent you is OK
https://github.com/bgrabitmap/bgracontrols/releases/download/v4.3.1/bgra_controls_431.zip

but the BGRA Controls FX I sent you is now outdated, you need to get it from here:
https://github.com/bgrabitmap/bgracontrolsfx/releases/download/v0.1.4/bgra_controls_fx_014.zip

Edit2: The package bgrabitmap that's in the repository has the wrong category, it must be Graphics.
Title: Re: Online Package Manager
Post by: GetMem on October 25, 2016, 08:05:11 pm
@wp
Thanks! Tomorrow I will add packages to the main repository. I have a question though: when you modify something in one of the packages you replace the zip file? I would like to know if the download link stays the same(this is true on github).

@rvk
Good idea! I' m skeptical though about the large number of request.  :D  I will go with an FTP server because in order to send mails the user must install OpenSSL library.

@lainz
Congratulation! You're the first one who submits two valid packages. You forget the download links though:
https://github.com/bgrabitmap/bgracontrolsfx/archive/master.zip
https://github.com/bgrabitmap/bgracontrols/archive/master.zip

Is the package creation form intuitive? How can we improve it?

Quote
The package bgrabitmap that's in the repository has the wrong category, it must be Graphics.
Thanks! The category is not displayed correctly in the three. I will fix it soon.
Title: Re: Online Package Manager
Post by: molly on October 25, 2016, 08:10:53 pm
Congratulation! You're the first one who submits two valid packages. You forget the download links though:
https://github.com/bgrabitmap/bgracontrolsfx/archive/master.zip
https://github.com/bgrabitmap/bgracontrols/archive/master.zip
Not wanting to play a party pooper there but, how do you know master is in fact the (stable) release of those packages ?

Different people uses different git-flows, e.g. one uses master as development branch and use release tags, others let master be the latest (stable) release.
Title: Re: Online Package Manager
Post by: lainz on October 25, 2016, 08:21:43 pm
Yes I use the GitHub releases to mark a stable release.

Master by default is the work in progress for these two packages.

A link to the releases page will be OK, not directly to any file.

Bgrabitmap is the same. But uses source forge. The releases in GitHub are for lazpaint only.

The creation of the package was easy. I need to copy my GitHub repository to an other folder to avoid the .git folder being included in the zip file.
Title: Re: Online Package Manager
Post by: GetMem on October 25, 2016, 08:24:28 pm
Quote
@molly
Not wanting to play a party pooper there but, how do you know master is in fact the release of those packages ?
I don't, but the package developer does know I suppose  :D. The user will decide if he/she updates the package from the external link or use the default one from the main repository. By the way multiple version of the same package is also allowed.

Did you test it @molly? Feedback is welcome.
Title: Re: Online Package Manager
Post by: molly on October 25, 2016, 09:15:13 pm
Did you test it @molly? Feedback is welcome.
I have now  :D

At first sight things are looking good and polished (it is a form of art so, worth to mention).

The two things that annoyed me (sorry, there is no way for me to tell it more gently).
1) if i press right mouse button to view the json file, i seem unable to 'return' to the initial packager view. All things i tried (click close, escape etc) closes the window (which is not what i wanted).

2) i chose to automatically extract the package, which automatically seems to popup a windows explorer window at the location where the package was extracted. I absolutely most definitely despise those windows popping up as i do not use Windows explorer _ever_ (which is more my personal problem ofc.) :-)

For 1, it needs a solution in the end (unless i missed something obvious) for 2 i would be in your debt in case there is a possibility to prevent that from happening.


In case not already having considered, i would have expected the download location to be part of the options inside the online package manager. At least don't let me start selecting a directory from my computer top-level but in user dir or lazarus dir (i usually install my packages alongside lazarus install dir for lazarus 1.7 series).

After i managed to convince my firewall that lazarus indeed requires some internet access (all requesters before that worked splendid as far as i am able to tell) the list was populated with about 8 packages. The treeview is easy to view and works as expected.

Downloading and extracting seems to have worked like a breeze, at least i did not encounter any issues.

Question: I assume (?) that you also add support for installing the packages, in which case the question pops up: what if i already installed packages manually ? would the online package manager be able to detect that and not try to install the same package twice ?

There are many of such questions that comes to mind but, i promised to keep myself out of things until things become more stable for you. There is no point in speculating until things become more clear for you and you have set a more definitive goal for yourself. Especially because "i want it all, and i want it now" (think electrical guitars and high pitched voices ;-) )

I have to congratulate you on your progress, as things that work seem to work as expected for me. In that regards never mind my encountered annoyances. Most probably you are aware of them already anyways.

Thank you for your hard work on this and actually realizing something that should have been done/implemented ages ago.

edit:
Oh, i almost forgot. If no package selected, download button is active. Dunno how hard it would be for virtualTV to determine number of selected items, otherwise can be solved adding a counter at onclick event/on check event and only activate download button when items are actually selected ? Would imho be better than the nice message box you now show :)
Title: Re: Online Package Manager
Post by: shobits1 on October 25, 2016, 09:28:18 pm
What does `available since` means?? (confused)
does it mean when the package has first seen on the server OR it is the update date for updated packages.
Title: Re: Online Package Manager
Post by: frakno on October 25, 2016, 09:43:47 pm

Quote
@molly
1) if i press right mouse button to view the json file, i seem unable to 'return' to the initial packager view. All things i tried (click close, escape etc) closes the window (which is not what i wanted).

Have you tried   right click  -> hide json ????

that works great
Title: Re: Online Package Manager
Post by: GetMem on October 25, 2016, 10:01:06 pm
@molly
Thank you for testing!

Quote
The two things that annoyed me (sorry, there is no way for me to tell it more gently).
1) if i press right mouse button to view the json file, i seem unable to 'return' to the initial packager view. All things i tried (click close, escape etc) closes the window (which is not what i wanted).
Right click again on the JSON memo, a menu should pop up with "Hide JSON" item. This feature was introduced for debugging purposes, I will probably remove it at the end.

Quote
2. 2) i chose to automatically extract the package, which automatically seems to popup a windows explorer window at the location where the package was extracted. I absolutely most definitely despise those windows popping up as i do not use Windows explorer _ever_ (which is more my personal problem ofc.) :-)
Interestingly enough other people like this feature, but I understand your point. You will be able to set the default behaviour in the options. It's not yet implemented.
Quote
In case not already having considered, i would have expected the download location to be part of the options inside the online package manager. At least don't let me start selecting a directory from my computer top-level but in user dir or lazarus dir (i usually install my packages alongside lazarus install dir for lazarus 1.7 series).
Under linux, osx the lazarus install dir sometimes is read only.

Quote
Question: I assume (?) that you also add support for installing the packages, in which case the question pops up: what if i already installed packages manually ? would the online package manager be able to detect that and not try to install the same package twice ?
I will answer with a screenshot.

Quote
I have to congratulate you on your progress, as things that work seem to work as expected for me. In that regards never mind my encountered annoyances. Most probably you are aware of them already anyways.
Thank you for your hard work on this and actually realizing something that should have been done/implemented ages ago.
Thank you!

Quote
Oh, i almost forgot. If no package selected, download button is active. Dunno how hard it would be for virtualTV to determine number of selected items, otherwise can be solved adding a counter at onclick event/on check event and only activate download button when items are actually selected ? Would imho be better than the nice message box you now show :)
It would be trivial to implement, not very high on the priority list, but I will do it.

Quote
@shobits1
What does `available since` means?? (confused)
does it mean when the package has first seen on the server OR it is the update date for updated packages.
When the package has first seen on the server.
Title: Re: Online Package Manager
Post by: wp on October 25, 2016, 10:03:43 pm
@wp
Thanks! Tomorrow I will add packages to the main repository. I have a question though: when you modify something in one of the packages you replace the zip file? I would like to know if the download link stays the same(this is true on github).
I must confess that I did not follow this thread in detail. Your remark indicates that you need a zip file for the Package Manager. This may be a problem. Much of the work that I put into ccr is work in progress, and I only rarely update the released zip files. When I do I will always add a version number to keep the history. The download link, therefore, will change. But of course I could provide a copy of the current release under a standard name, i.e. the current release of fpspreadsheet (fpspreadsheet-1.6.2.zip (https://sourceforge.net/projects/lazarus-ccr/files/FPSpreadsheet/fpspreadsheet-1.6.2.zip/download)) could be provided also as fpspreadsheet.zip, or fpspreadsheet-release.zip.

As far as I know SourceForge svn does not create snapshots of the development versions in the individual source folders. So, there is not explicit zip snapshot of the current trunk version of, e.g., fpspreadsheet. There is one big snapshot but it contains the entire ccr repository.
Title: Re: Online Package Manager
Post by: GetMem on October 25, 2016, 10:21:02 pm
Quote
As far as I know SourceForge svn does not create snapshots of the development versions in the individual source folders. So, there is not explicit zip snapshot of the current trunk version of, e.g., fpspreadsheet. There is one big snapshot but it contains the entire ccr repository.
This is exactly what I'm looking for(explicit snapshot of the current trunk).
I believe I found the link(ex: color palette): https://sourceforge.net/code-snapshots/svn/l/la/lazarus-ccr/svn/lazarus-ccr-svn-5294.zip

ps: No! You're right, it's the entire ccr repository.
ps1:
Quote
But of course I could provide a copy of the current release under a standard name, i.e. the current release of fpspreadsheet (fpspreadsheet-1.6.2.zip) could be provided also as fpspreadsheet.zip, or fpspreadsheet-release.zip.
That would be helpful. Thanks! On github things are much easier.
Title: Re: Online Package Manager
Post by: molly on October 25, 2016, 10:34:42 pm
Have you tried   right click  -> hide json ????

that works great

Right click again on the JSON memo, a menu should pop up with "Hide JSON" item. This feature was introduced for debugging purposes, I will probably remove it at the end.

Oops. I completely overlooked that one  :-[

Thank you both for showing me my own stupidity  :D

Edit:
@GetMem:
Thank you for your answers, i can live with those given :) Extremely thank you for the screenshot, as that answers some other questions i had as well  :)
Title: Re: Online Package Manager
Post by: lainz on October 25, 2016, 10:57:36 pm
The server containing the repository needs something special? Edit (Seems that not).

Because if not (for example is just a json and zip files) you can use GitHub: create a new git repository and use it as website with github pages, then you can upload new packages using tortoisegit or your favourite tool and the changes are automatic.

You can add trusted users that will contribute uploading packages too for that repository. In fact will be better to create a github organization and manage user groups from that place.

In fact you can use your existing github account and repository, just add a new branch that will contain the packages.

If you need help just ask.-
Title: Re: Online Package Manager
Post by: JuhaManninen on October 26, 2016, 01:07:32 am
Quote
@molly
Not wanting to play a party pooper there but, how do you know master is in fact the release of those packages ?
I don't, but the package developer does know I suppose  :D. The user will decide if he/she updates the package from the external link or use the default one from the main repository. By the way multiple version of the same package is also allowed.
In both cases some admin work is required. The JSON file must be updated with the package's version info etc.
I don't think this can be automated as there must be control over the trusted package sources.

But of course I could provide a copy of the current release under a standard name, i.e. the current release of fpspreadsheet (fpspreadsheet-1.6.2.zip (https://sourceforge.net/projects/lazarus-ccr/files/FPSpreadsheet/fpspreadsheet-1.6.2.zip/download)) could be provided also as fpspreadsheet.zip, or fpspreadsheet-release.zip.
It does not make much difference as the JSON file must be updated anyway.
BTW, you just provided a ZIP-file link to SourceForge server. Clearly it is possible although GetMem doubted it earlier.

I don't think the package manager should in general provide latest development (trunk) versions. The latest "stable" release should be the default.
There can be exceptions like the CCR packages without a maintainer. They have never been stable. Any bug fix they get makes them better. Yet those bug fixes happen so seldom that the package manager maintenance is not a burden either.

Title: Re: Online Package Manager
Post by: JuhaManninen on October 26, 2016, 01:18:14 am
The server containing the repository needs something special? Edit (Seems that not).

Because if not (for example is just a json and zip files) you can use GitHub: create a new git repository and use it as website with github pages, then you can upload new packages using tortoisegit or your favourite tool and the changes are automatic.

It is better to have a proper server. It allows more options in future, including a server side SW to handle user feedback and grading / voting system.
GitHub can be later supported in the same way that Delphinus does now, as an alternative to the contralized server.

Quote
If you need help just ask.-

Somebody could start to think about the server side SW. Something similar with NuGet maybe ...
Title: Re: Online Package Manager
Post by: lainz on October 26, 2016, 01:58:52 am
You are right. Also both GitHub and source forge has a limit in bandwidth, having an own server it can be managed as it grows.

About the server side software I have a poor experience in PHP only. So I'm not the indicated. But maybe is a ready to use solution out there that can be adapted to the repository needs.
Title: Re: Online Package Manager
Post by: GetMem on October 26, 2016, 06:44:54 am
Quote
@Juha
It does not make much difference as the JSON file must be updated anyway.
BTW, you just provided a ZIP-file link to SourceForge server. Clearly it is possible although GetMem doubted it earlier.
Then I fail to see the purpose of the external download link. If the json must be updated each time, it's better to update the package itself, with the automated tool it takes only 1 minute or so. The external link should point to the trunk perhaps? Pointing to a stable version makes no sense at all.
Title: Re: Online Package Manager
Post by: JuhaManninen on October 26, 2016, 08:46:35 am
Then I fail to see the purpose of the external download link. If the json must be updated each time, it's better to update the package itself, with the automated tool it takes only 1 minute or so. The external link should point to the trunk perhaps? Pointing to a stable version makes no sense at all.
Ok, I was thinking of the updated package's version number. I thought it should be visible in the list of available packages. Maybe it is not important, the versions of installed packages are visible anyway.

Having an external link to a stable version makes a lot of sense if the package author can update the package by himself without bothering the maintainer.
Now the changed version info cannot be transferred but we can figure out a way to do that, too, if needed.

In my opinion revision control tools (SVN, Git etc.) are still the best way to get latest cutting edge development versions, at least for packages that are actively developed.
A package author can of course provide development snapshots for package manager, in addition to stable releases, if he has time and energy.
Title: Re: Online Package Manager
Post by: Groffy on October 26, 2016, 12:21:23 pm
Hi,

is it somehow possible to configure a connection with a proxy in between?

Thanks for the great work!
Title: Re: Online Package Manager
Post by: minesadorada on October 26, 2016, 12:27:45 pm
...
It is better to have a proper server. It allows more options in future, including a server side SW to handle user feedback and grading / voting system.
GitHub can be later supported in the same way that Delphinus does now, as an alternative to the contralized server.
..

+1
A repository of packages will only be as good as its maintenance.  This will surely include user feedback - and someone to take notice of it.
If there were only a simple voting system to start with, it would be very useful to potential users.
Title: Re: Online Package Manager
Post by: JuhaManninen on October 26, 2016, 02:26:47 pm
If there were only a simple voting system to start with, it would be very useful to potential users.
No, initially there will not be even a simple voting system. It requires server side SW.
Title: Re: Online Package Manager
Post by: JuhaManninen on October 26, 2016, 03:24:32 pm
I understood some people used the
 Repository  ->  "Create repository package"
feature of the package manager. Is it so?
I tested it with some existing Lazarus packages under Components/ directory, for example with aggpas.
After selecting an output directory the GUI says:
 "Compressing package. Please wait ..."
and it continues forever. Only the Cancel button stops it. A package was created anyway but my ZIP tool (Ark) says it is corrupt, although it shows the contents which look OK to me.

Does it work for others? I can try to debug it.
My system is a 64-bit Manjaro Linux. Could it be a 64-bit issue?
Title: Re: Online Package Manager
Post by: minesadorada on October 26, 2016, 03:34:37 pm

Does it work for others? I can try to debug it.
My system is a 64-bit Manjaro Linux. Could it be a 64-bit issue?
Environment: Win 10 64-bit
I downloaded and installed the package manager today and did the same test as you.  It zipped the component folder quickly (though the manufactured zip contained the /backup and /lib folders which doesn't seem good).
I have WinZip on my system as default archiver.

Maybe it's a linux thing?
Title: Re: Online Package Manager
Post by: molly on October 26, 2016, 03:49:45 pm
I understood some people used the
 Repository  ->  "Create repository package"
feature of the package manager. Is it so?
No i did not but, i have now  :)

Same aggpass example on win32 with lazarus trunk from end of sept.

When the archiving is done (e.g. after "Compressing package. Please wait ..."), an explorer window pops up *grmbl*  ;) pointing to the directory where the .zip file was created (my selection during creation) and a information message dialog pops-up saying "Repository Package successfully created".

PS: most likely a 64 bit thing as the zipper classes had a fair amount of patching for that (i am not sure if there are still bug-reports open for that and how new GetMem's version is).
Title: Re: Online Package Manager
Post by: lainz on October 26, 2016, 04:13:28 pm
It works fine when I created the bgracontrols and bgracontrolsfx packages under Windows 10 x64, but with lazarus 32 bit so I can't say if it's a problem of x64.
Title: Re: Online Package Manager
Post by: GetMem on October 26, 2016, 04:32:27 pm
@Groffy
Quote
is it somehow possible to configure a connection with a proxy in between?
Not yet! I will add that option later.

@Juha
Once I saw that error under linux. It comes from zippper.pp unit(fpc). Please download the latest version, it will display a more meaningful message.
Title: Re: Online Package Manager
Post by: JuhaManninen on October 26, 2016, 09:08:37 pm
Once I saw that error under linux. It comes from zippper.pp unit(fpc). Please download the latest version, it will display a more meaningful message.
It is caused by wrong type of file attributes in the zipper. I always test code with compiler checks and debug flags set, thus an overflow interrupted it.
I reported the issue with a patch:
 http://bugs.freepascal.org/view.php?id=30800
The same change can be made to your copy.
However I recommend that you create a patch of your new zipper events against the latest FPC trunk which now has many changes related to Unicode and other parts, too.
When it is applied, we can copy the latest version from FPC trunk without worries.
The same goes with your http client fork.

[Edit] Now that the zipper works, I got a valid ZIP file and also a JSON file.
If the idea is to download an extra JSON file from the remote URL then it solves the version number problem.
My understanding is that now only the central server has a JSON file.

With aggpas I got:
Code: [Select]
{
  "PackageData0" : {
    "Name" : "aggpas",
    "Category" : 2,
    "RepositoryFileName" : "aggpas.zip",
    "RepositoryFileSize" : 2943239,
    "RepositoryFileHash" : "aa2d6ca1dd9b54f6d1c607cc260aa588",
    "RepositoryDate" : 4.2669000000000000E+004,
    "PackageBaseDir" : "aggpas\\\/"
  },
  "PackageFiles0" : [
    {
      "Name" : "aggpaslcl.lpk",
      "Description" : "AggPas canvas for LCL",
      "Author" : "Mattias Gaertner,  Milan Marusinec",
      "License" : "Permission to copy, use, modify, sell and distribute this software is granted provided this copyright notice appears in all copies.\nThis software is provided \"as is\" without express or implied warranty, and with no claim as to its suitability for any purpose.",
      "RelativeFilePath" : "lazarus\\\/",
      "VersionAsString" : "1.0.0.0",
      "LazCompatibility" : "1.6, Trunk",
      "FPCCompatibility" : "2.6.4, 3.0.0",
      "SupportedWidgetSet" : "win32\/64, gtk2, carbon",
      "PackageType" : 2,
      "DependenciesAsString" : "LCL, FCL(1.0.0.0)",
      "HomePageLink" : "",
      "DownloadLink" : ""
    }
  ]
}
Title: Re: Online Package Manager
Post by: GetMem on October 26, 2016, 10:51:04 pm
@Juha
Quote
I always test code with compiler checks and debug flags set, thus an overflow interrupted it.
Me too, however under Kali Linux(32 bit) the error does not occur.
Quote
The same change can be made to your copy.
However I recommend that you create a patch of your new zipper events against the latest FPC trunk which now has many changes related to Unicode and other parts, too.
When it is applied, we can copy the latest version from FPC trunk without worries.
The same goes with your http client fork.
Thanks for the patch, I will apply it soon. I did try the httpclient from FPC trunk once, but there was a reference to another unit.  I did not want to copy more then one file.

Quote
Now that the zipper works, I got a valid ZIP file and also a JSON file.
If the idea is to download an extra JSON file from the remote URL then it solves the version number problem.
My understanding is that now only the central server has a JSON file.
If somebody submits a repository package, the json file will be integrated into the main JSON. I'm not sure yet what would be the optimal solution for remote URL's.
Anyway things are looking good, we already have 20 packages in the main repository.
Title: Re: Online Package Manager
Post by: rvk on October 26, 2016, 11:20:54 pm
You might want to add onlinepackagemanager to the repository so we can update the manager itself to the latest version easily :D

(I don't see that one yet in your localhost repository)

Or will there be some self-auto-updater?
Title: Re: Online Package Manager
Post by: GetMem on October 27, 2016, 06:10:08 am
Quote
@rvk
You might want to add onlinepackagemanager to the repository so we can update the manager itself to the latest version easily :D
+1. Added.
Title: Re: Online Package Manager
Post by: avra on October 31, 2016, 11:30:15 am
Quote
In case not already having considered, i would have expected the download location to be part of the options inside the online package manager. At least don't let me start selecting a directory from my computer top-level but in user dir or lazarus dir (i usually install my packages alongside lazarus install dir for lazarus 1.7 series).
Under linux, osx the lazarus install dir sometimes is read only.
How about remembering last download location?
Title: Re: Online Package Manager
Post by: Groffy on October 31, 2016, 12:04:03 pm
Quote
+1. Added.

Can't see it in the repository. Is the adress http://104.199.177.61/ still correct?

Thanks
Title: Re: Online Package Manager
Post by: GetMem on October 31, 2016, 12:11:37 pm
Quote
@avra
How about remembering last download location?
Consider it done.

Quote
@Groffy
Can't see it in the repository. Is the adress http://104.199.177.61/ still correct?
Yes, but it's not updated yet. I'm still waiting for packages.
Title: Re: Online Package Manager
Post by: macmike on October 31, 2016, 12:27:26 pm
Great work, this is brilliant
Title: Re: Online Package Manager
Post by: JuhaManninen on October 31, 2016, 02:15:47 pm
Yes, but it's not updated yet. I'm still waiting for packages.
I copy here GetMem's earlier instructions in this same thread. The thread is long and they may get buried and forgotten.
---
To create a repository package please go to Create-->Create repository package toolbutton, then:
- select the package main directory(lazopkman will recursively search for all package files inside the folder)
- select the packages you wish to include
- choose a category
- adjust package info if necessary(lazarus/fpc compatibility, suppported widgetset, etc)
- press the create button
The whole process should take no longer then 1-2 min/ package. If all goes well you should get a zip and a json file. Upload somewhere the files and send me the link, alternatively you can send it to getmem1@gmail.com . I will wait a week or so then update the main repository agin. A realistic target is 50+ packages.
---

One thing: the Create --> "Create repository package" feature should have a filter for file types. Now everything is included. Maybe something similar to the Publish Project filter in Lazarus.
Title: Re: Online Package Manager
Post by: rvk on October 31, 2016, 02:23:46 pm
Yes, but it's not updated yet. I'm still waiting for packages.
I copy here GetMem's earlier instructions in this same thread. The thread is long and they may get buried and forgotten.
Should this only be done by the maintainer of those packages or can anybody submit random packages?

Also... what to do with packages which don't have maintainers (or maintainers who are not part of the Lazarus community like for Ararat Synapse)?
Title: Re: Online Package Manager
Post by: GetMem on October 31, 2016, 02:53:33 pm
Quote
@rvk
Should this only be done by the maintainer of those packages or can anybody submit random packages?
Anyone can submit any package. In a worse case scenario we will have a few duplicates, I will filter them out. Different versions of the same package is also allowed.

Quote
Also... what to do with packages which don't have maintainers (or maintainers who are not part of the Lazarus community like for Ararat Synapse)?
Even more important to submit it. I already added Ararat Synapse because it's a well known package, but my knowledge is limited.

So far we have 32 packages:
Quote
bgra_controls
bgra_controls_fx
bgrabitmap9.2.1
cmdline
colorpalette
csvdocument
dcpcrypt-2.0.4.1
epiktimer
eyecandycontrols_0-9-6
FileMenuHandler
fortes4lazarus 3.24
fortesreport-ce4
fpspreadsheet
HistoryFiles
indy10.6.2.0
jujiboutils
lclextensions-0.6
mplayer0.1.2
OnlinePackageManager
playsoundpackage
plotpanel-lazarus-0.97.1
powerpdf_0.9.14
richmemo1.0
ringwatch
SMNetGradient2.0.6
spktoolbar
synapse40
tvplanit
uniqueinstance
virtualtreeview-4.8.7-R4
virtualtreeview-5.5.3-R1
zeosdbo-7.1.4

PS: We can always extend the package list. There is no time limit. It would be nice though to have 50+ packages to test the functionality of lazopkman.
Title: Re: Online Package Manager
Post by: Groffy on October 31, 2016, 03:23:24 pm
Quote
PS: We can always extend the package list. There is no time limit. It would be nice though to have 50+ packages to test the functionality of lazopkman.

What about the KControls component suite package? They are really great, coming with a powerful grid component.
Title: Re: Online Package Manager
Post by: rvk on October 31, 2016, 03:25:24 pm
I see for Synapse you took the "stable" version 40.
It might also be a good idea to include (or replace it with) the trunk version.
https://sourceforge.net/p/synalist/code/HEAD/tree/trunk/

The stable version 40 is very very old and the trunk version is considered stable enough.
(There are some minor issues with v40 and it's always advised, also here on the forum, to take the trunk version of Synapse, which hasn't changed since februari 2015)
Title: Re: Online Package Manager
Post by: wp on October 31, 2016, 03:46:08 pm
What about the KControls component suite package? They are really great, coming with a powerful grid component.
I would second that. But there is the usual problem of finding the correct sources. The version in ccr is doing fine, but I think it is out-dated. The version on the original site (http://www.tkweb.eu/en/delphicomp/) seems to be last modified in April this year, much more recent. The message is signed by a user "tk". The same name exists also in this forum.

@tk: Are you the author of KControls? If yes: Which is the best version to be included in the Package Manager?
Title: Re: Online Package Manager
Post by: GetMem on October 31, 2016, 09:13:57 pm
Ok, I added:
- Synapse trunk(lpk version 40.1.0.0)
- KControls_1.7(in order to compile, I had to modify the source)
Title: Re: Online Package Manager
Post by: minesadorada on October 31, 2016, 09:27:21 pm
When I used OnlinePackageManager to create a package, it zips up both the /backup and /lib folders if they exist.  This is obviously not a good idea.

Is that current behaviour?
Title: Re: Online Package Manager
Post by: GetMem on October 31, 2016, 09:53:21 pm
Quote
When I used OnlinePackageManager to create a package, it zips up both the /backup and /lib folders if they exist. 
Don't worry about the /backup and /lib folder, I will remove them manually.
Quote
This is obviously not a good idea.
I agree. I can add a feature that skips the /backup and /lib folder, however this won't prevent someone with malicious purpose. A manual check is mandatory.

PS: If you have a package, please send it to me.
Title: Re: Online Package Manager
Post by: JuhaManninen on November 01, 2016, 01:00:15 am
Yes, the Zip file should not contain anything extra. Download traffic should be kept low.
Marc asked me for the needed size and traffic. I could only estimate it is low, even with the DB which will contain only textual data.

If the package manager causes an explosion of Lazarus' popularity, as it may happen, then the traffic could be a problem...
This will be as popular as Visual Studio or NetBeans. Millions of people install packages at the same time. The server overheats and glows red trying to serve them all. :)
Title: Re: Online Package Manager
Post by: lainz on November 01, 2016, 02:11:40 am
Hope it does as you said  :D
Title: Re: Online Package Manager
Post by: GetMem on November 01, 2016, 06:46:57 am
Quote
@Juha
Yes, the Zip file should not contain anything extra. Download traffic should be kept low.
What about documentation, demos(source only)? The total size of the 34 packages is 26.9 MB, it's less then 1MB/package. With today standard 26.9 MB it's a small footprint. Even 100 MB would be small. 

Quote
Marc asked me for the needed size and traffic. I could only estimate it is low, even with the DB which will contain only textual data.
Perhaps @Leledumbo can give us some info about the traffic. He's running the repository for almost a month.

PS: The Lazarus popularity can be increased by supporting mobile/web development and a more straightforward cross-compiling.
Title: Re: Online Package Manager
Post by: JuhaManninen on November 01, 2016, 12:53:24 pm
What about documentation, demos(source only)? The total size of the 34 packages is 26.9 MB, it's less then 1MB/package. With today standard 26.9 MB it's a small footprint. Even 100 MB would be small. 
Sure they can be included, as well as any resource files needed by a package. I was only worried about compiled binary files included accidentally.
Anything below 1GB is small in today's standards. I estimated a few GB will be needed even with the server SW + DB in future. Not much really.

---
Earlier there was discussion about including snapshots of development versions of some packages.
It is a diffucult issue with packages which don't have a maintainer. Nobody makes release versions of them. The Online Package Manager admin must take some revision from the commit history and "release" it. What if it crashes? Is it his responsibility to find a revision that does not crash?
This issue is a swamp.
Now initially it doesn't matter because everybody is enthusiastic. After some years however people get bored with such admin work. The packages without a maintainer are forgotten even if their master source got some fixes.
The goal is that all packages have a maintainer but it is not realistic. I was surprised to learn that even well-known packages like Synapse have no recent releases. It has a maintainer after all.

We are now solving the same problem that CCR repo tried to solve years ago. The idea was that all Lazarus packages can be found easily in one place. They are under revision control, thus their maintainers can easily develop them.
Well, it has not worked very well. Most package authors have fleed.
There is a danger that we end up with a similar outdated system. Most packages are not maintained and to get their latest fixes you must use revision control directly anyway.
Maintained packages are our hope. The Online Package Manager should be easy and attractive for their maintainers. That's why the remote URL should be supported, so that trusted sources can be updated as easily as possible.
How to make it more attractive? How to get people to maintain packages? Uhhh...

I am not criticizing GetMem's efforts obviously. I am looking to the future years ahead. The maintenance cannot depend on one person eternally, it must be made sustainable.
Title: Re: Online Package Manager
Post by: lainz on November 01, 2016, 01:00:11 pm
If size needs to be really low I must trim test projects that are most the size of bgracontrols, a lot of tests  ::) nothing more than 4 or 3 mb  :D

About the mantainer I agree, I want to be myself that uploads the new versions directly. Then aproved or not it needs more people than only one that reviews, if is the case.

Why don't use GitHub directly, SF can't be used without the trick of the redirection (is in the wiki to get direct downloads from sf) or picking manually the mirror.

You can create a new repository called packages, create releases and put there the files and that's all. Then instead of having to upload to your server the file uses the github one (you miss the server load in this case, all goes to github) and link it to the package as is currently beign doing for the alternate download, well instead of beign alternate just main.
Title: Re: Online Package Manager
Post by: JuhaManninen on November 01, 2016, 01:24:57 pm
You can create a new repository called packages, create releases and put there the files and that's all. Then instead of having to upload to your server the file uses the github one (you miss the server load in this case, all goes to github) and link it to the package as is currently beign doing for the alternate download, well instead of beign alternate just main.
I think the remote URL feature already supports that.
Later a support for all GitHub repositories can be added, like Delphinus has now. It is a different concept however, it allows any package without any checks to pass. The user must be informed about it when he enables that option.
Title: Re: Online Package Manager
Post by: GetMem on November 01, 2016, 01:44:27 pm
@Juha
Quote
Earlier there was discussion about including snapshots of development versions of some packages.
It is a diffucult issue with packages which don't have a maintainer. Nobody makes release versions of them. The Online Package Manager admin must take some revision from the commit history and "release" it. What if it crashes? Is it his responsibility to find a revision that does not crash?
This issue is a swamp.
Now initially it doesn't matter because everybody is enthusiastic. After some years however people get bored with such admin work. The packages without a maintainer are forgotten even if their master source got some fixes.
The goal is that all packages have a maintainer but it is not realistic. I was surprised to learn that even well-known packages like Synapse have no recent releases. It has a maintainer after all.
For the first time, I'm not following you  :D. If somebody doesn't want to maintain his package, there is little you can do about it.

Quote
We are now solving the same problem that CCR repo tried to solve years ago. The idea was that all Lazarus packages can be found easily in one place. They are under revision control, thus their maintainers can easily develop them.
Well, it has not worked very well. Most package authors have fleed.
There is a danger that we end up with a similar outdated system. Most packages are not maintained and to get their latest fixes you must use revision control directly anyway.
Maintained packages are our hope. The Online Package Manager should be easy and attractive for their maintainers. That's why the remote URL should be supported, so that trusted sources can be updated as easily as possible.
The update feature will be ready tomorrow. Next week I plan to add SVN support. It shouldn't be to difficult. In a month or so the ''personal repository" feature will be also available. Everyone can create a whole repository, store it on GitHub for example. LazOpkman will be able to load multiple repositories at the same time.

Quote
@lainz
If size needs to be really low I must trim test projects that are most the size of bgracontrols, a lot of tests  ::) nothing more than 4 or 3 mb  :D
Don't trim anything. Although bgracontrols is one of the biggest package, still way below the upper limit. After all is a complex package.
Title: Re: Online Package Manager
Post by: wp on November 01, 2016, 02:28:10 pm
Here one more ccr package: lazbarcodes. I don't use it, but I checked it with Laz trunk/1.6 / fpc3.0/trunk: compiling fine, and a simple test runs without any issues. And I remember that there are a few users here in the forum using it, I had fixed one or two issues for them. So, I think, lazbarcodes should be included.

I uploaded a "release" snapshot to the Files section of SourceForce: https://sourceforge.net/projects/lazarus-ccr/files/LazBarcodes/ (https://sourceforge.net/projects/lazarus-ccr/files/LazBarcodes/)
Title: Re: Online Package Manager
Post by: GetMem on November 01, 2016, 05:39:54 pm
Thank you @wp!
I added lazbarcodes to the main repository.
Title: Re: Online Package Manager
Post by: minesadorada on November 02, 2016, 12:56:23 pm
I will have some packages to add, but first I am testing them with Laz 1.7 and fpc3.1.1, Windows and Linux.
There are UTF8 issues to tweak on some of them for instance.
Title: Re: Online Package Manager
Post by: minesadorada on November 03, 2016, 03:21:33 pm
In the ccr svn repositry https://sourceforge.net/p/lazarus-ccr/svn/HEAD/tree/components/
There are:
1. lazautoupdate
2. longtimer
3. playsoundpackage
4. poweredby
5. scrolltext


All are currently maintained.


I was unable to package them in OPM.  I downloaded & installed today, but it crashed itself and Lazarus (Laz1.7/fpc3.1.1 Win64)  I assume it's still under development.

Title: Re: Online Package Manager
Post by: wp on November 03, 2016, 04:14:04 pm
@minesadorada: Thank you for bringing the attention to these nice components (the playsoundpackage has already been included).

A few questions/remarks to these components

lazautoupdate:
I saw that you distribute synapse along with this component. I don't know, but wouldn't it be better if you just provide the download link?

Or even better: As I can see (I may be wrong) you use synapse only to download files from their web sites. You should be able to do this also with fphttpclient that comes along with fpc (see folder packages/fcl-web/base). Reducing external depencies is always a good objective.

In the package file, you add the synapse.lpk package file to the "files" section of the lazupdate section. Why? I fear this will sooner or later cause trouble for those users who already have synapse installed (like myself), because it will end up in a version mixup of the already installed files and the files added by your package.

All packages:
All packages have a requirement of the ideintf package. I think this is not good because it pulls all the designtime code into your application. The correct way, in my opinion, would be to split the stuff into two packages, the runtime package containing all the runtime stuff (your unit code plus LCL), and the designtime package for the work with form editor and object inspector (ideintf plus your aboutbox)

P.S. I know that "my" tvplanit does the same - but these components were written a long time ago by TurboPower when nobody was aware of this issue, it will be fixed in one of the next releases.

Title: Re: Online Package Manager
Post by: minesadorada on November 03, 2016, 04:34:32 pm
@minesadorada: Thank you for bringing the attention to these nice components (the playsoundpackage has already been included).

A few questions/remarks to these components

lazautoupdate:
I saw that you distribute synapse along with this component. I don't know, but wouldn't it be better if you just provide the download link?

Or even better: As I can see (I may be wrong) you use synapse only to download files from their web sites. You should be able to do this also with fphttpclient that comes along with fpc (see folder packages/fcl-web/base). Reducing external depencies is always a good objective.

In the package file, you add the synapse.lpk package file to the "files" section of the lazupdate section. Why? I fear this will sooner or later cause trouble for those users who already have synapse installed (like myself), because it will end up in a version mixup of the already installed files and the files added by your package.

All packages:
All packages have a requirement of the ideintf package. I think this is not good because it pulls all the designtime code into your application. The correct way, in my opinion, would be to split the stuff into two packages, the runtime package containing all the runtime stuff (your unit code plus LCL), and the designtime package for the work with form editor and object inspector (ideintf plus your aboutbox)

P.S. I know that "my" tvplanit does the same - but these components were written a long time ago by TurboPower when nobody was aware of this issue, it will be fixed in one of the next releases.


Hi wp,

lazautoupdate:  I include synapse so that the component installs and works smoothly.  In my own case, if a package complains about missing stuff when I first install it, then I look for an alternative, and I suspect others do the same.  The synapse version is 40 stable, and proven to work with the component.  I have no plans to rewrite the component to use fphttpclient without a powerful reason for doing so.

All Packages:  You are probably right about designtime/runtime.  I don't worry about filesizes too much nowadays, but maybe some folk do.

In all cases, it's up to you if you want to include the packages as is in OPM.  I simply offer them as freely-licensed code which is maintained.
Of course, you are free to change/patch them if you have the time or inclination.  They are all LGPL2.

I appreciate the feedback - it's all constructive.
Title: Re: Online Package Manager
Post by: wp on November 03, 2016, 05:31:24 pm
They are all LGPL2.
This confuses me: You say here that the license is LGPL2, but in the license header of ulazautoupdate you write: "This library is free software; you can redistribute it and/or modify it  under the terms of the GNU Library General Public License as published by the Free Software Foundation; either version 2 of the License..." - this is GPL v2, LGPL would be "GNU Lesser Library General Public License".

So what is valid? Is there a special reason why you don't select the modified LGPL (with linking exception) which is used for all (?) LCL code?
Title: Re: Online Package Manager
Post by: minesadorada on November 03, 2016, 05:41:09 pm
They are all LGPL2.
This confuses me: You say here that the license is LGPL2, but in the license header of ulazautoupdate you write: "This library is free software; you can redistribute it and/or modify it  under the terms of the GNU Library General Public License as published by the Free Software Foundation; either version 2 of the License..." - this is GPL v2, LGPL would be "GNU Lesser Library General Public License".

So what is valid? Is there a special reason why you don't select the modified LGPL (with linking exception) which is used for all (?) LCL code?
Ah - well spotted!  I wrote the code before I properly understood licensing.  Being linkable components, they should all be modified GPL of course.
I'll edit the component source and update the svn - it's a one-line change for each component.

== done ==
Title: Re: Online Package Manager
Post by: wp on November 03, 2016, 05:51:49 pm
Ah - well spotted!  I wrote the code before I properly understood licensing.
Lucky man - I still don't...
Title: Re: Online Package Manager
Post by: minesadorada on November 03, 2016, 06:21:50 pm
All packages now ModifiedGPL and updated in svn.  Also gave me a chance to clean up the Package options, versions etc.
note:: playsound has also been updated in svn

Thanks @wp  :)
Title: Re: Online Package Manager
Post by: GetMem on November 03, 2016, 08:38:37 pm
@minesadorada
Thank you! I will add the packages to the main repository:
Quote
1. lazautoupdate
2. longtimer
3. playsoundpackage
4. poweredby
5. scrolltext
In a few days the main repository will be updated again. We have 40+ packages.
Title: Re: Online Package Manager
Post by: wp on November 03, 2016, 09:46:53 pm
GetMem, playsoundpackage is already included, you may have to update it though.
Title: Re: Online Package Manager
Post by: GetMem on November 03, 2016, 10:04:23 pm
Quote
GetMem, playsoundpackage is already included, you may have to update it thought.
True. Playsoundpackage is already in the list. Luckily opkman won't allow me to add the same lpk twice. I will update the existing one.
Title: Re: Online Package Manager
Post by: kapibara on November 04, 2016, 12:39:49 am
Sorry I have not read the whole thread, but installed the package manager and got Invalid JSON file on Packages -> Online Package Manager. That happens after the attempt to download package list.

Closing the manager dialog also gave exception in Lazarus.

Latest 64 bit lazarus trunk / Debian
Title: Re: Online Package Manager
Post by: wp on November 04, 2016, 01:05:30 am
Another component from ccr: https://sourceforge.net/projects/lazarus-ccr/files/CalLite/ (see also http://forum.lazarus.freepascal.org/index.php/topic,21093.msg227791.html#msg227791)
Title: Re: Online Package Manager
Post by: minesadorada on November 04, 2016, 06:47:05 am
Sorry I have not read the whole thread, but installed the package manager and got Invalid JSON file on Packages -> Online Package Manager. That happens after the attempt to download package list.

Closing the manager dialog also gave exception in Lazarus.

Latest 64 bit lazarus trunk / Debian
That happened to me.
64-bit Lazarus 1.7 /Win 10 64-bit
Title: Re: Online Package Manager
Post by: minesadorada on November 04, 2016, 06:54:15 am
@minesadorada
Thank you! I will add the packages to the main repository:
Quote
1. lazautoupdate
2. longtimer
3. playsoundpackage
4. poweredby
5. scrolltext
In a few days the main repository will be updated again. We have 40+ packages.
So what is the procedure if I update a package?  Up to now I just used TortoiseSVN to commit to the ccr repository.
Title: Re: Online Package Manager
Post by: GetMem on November 04, 2016, 07:18:41 am
@kapibara, @minesadorada
The package manager is still under development. To support update from external link, I had to change the json structure again(hance the invalid json file error). The source code is updated regularly, however I cannot update the repository each day since I' m not hosting it. The next update will be Monday(40+ pacakges, no more json error). If somebody is interested, I can pm a version which works with the current repository(8 packages).
Quote

@minesadorada
So what is the procedure if I update a package?  Up to now I just used TortoiseSVN to commit to the ccr repository.
You don't have to do anything. I will download each package with a SVN client and add them manually to the main repository. In a week or so the SVN feature will be ready. Meaning: we provide the SVN link, on request the package manager will checkout, compile, build install the package(see attachment). This is the plan anyway.
Title: Re: Online Package Manager
Post by: lainz on November 04, 2016, 10:12:45 pm
Hi GetMem, I've updated BGRA Controls v4.3.2 and BGRA Controls FX v0.1.6. Both works with BGRA Bitmap v9.2.1.
Title: Re: Online Package Manager
Post by: kapibara on November 04, 2016, 10:50:17 pm
@GetMem I see, will try again next week then. Great work btw!
Title: Re: Online Package Manager
Post by: GetMem on November 05, 2016, 05:10:33 am
Ok. I updated:
   - playsoundpackage
   - BGRA Controls
   - BGRA Controls FX
Added:
   - lazautoupdate
   - longtimer   
   - poweredby   
   - scrolltext   
   - callite     
The list looks like this(40 packages):
Quote
bgrabitmap9.2.1
bgracontrols-4.3.2
bgracontrolsfx-0.1.6
callite
cmdline
colorpalette
csvdocument
dcpcrypt-2.0.4.1
epiktimer
eyecandycontrols_0-9-6
FileMenuHandler
fortes4lazarus 3.24
fortesreport-ce4
fpspreadsheet
HistoryFiles
indy10.6.2.0
jujiboutils
kcontrols_1.7
lazautoupdate
lazbarcodes
lclextensions-0.6
longtimer
mplayer0.1.2
OnlinePackageManager
playsoundpackage
plotpanel-lazarus-0.97.1
poweredby
powerpdf_0.9.14
richmemo1.0
ringwatch
scrolltext
SMNetGradient2.0.6
spktoolbar
synapse40
synapse41
tvplanit
uniqueinstance
virtualtreeview-4.8.7-R4
virtualtreeview-5.5.3-R1
zeosdbo-7.1.4

@minesadorada
I removed synapse from lazautoupdate since it's already in the repository. The package manager recursively search/resolve dependencies when needed. For example if you try to install lazautoupdate the pm. will warn you about the missing dependency and install it automatically(see attachment) unless you explicitly ask not to.
Title: Re: Online Package Manager
Post by: minesadorada on November 05, 2016, 07:38:07 am
@getmem
All my packages have a wiki page - as do most others.  Do you think it would be a good idea to have an [info] link next to the ones that do?
http://wiki.freepascal.org/LazAutoUpdater (http://wiki.freepascal.org/LazAutoUpdater)
http://wiki.freepascal.org/LongTimer (http://wiki.freepascal.org/LongTimer)
http://wiki.freepascal.org/Poweredby (http://wiki.freepascal.org/Poweredby)
http://wiki.freepascal.org/ScrollingText (http://wiki.freepascal.org/ScrollingText)
http://wiki.freepascal.org/playsound (http://wiki.freepascal.org/playsound)
Liking the dependencies feature.  It's going to make Windows users feel like Linux users :)
Title: Re: Online Package Manager
Post by: GetMem on November 05, 2016, 07:44:11 am
Quote
@minesadorada
All my packages have a wiki page - as do most others.  Do you think it would be a good idea to have an [info] link next to the ones that do
Yes. I will add the wiki pages.

@all
From now on, as an extra check, each package is compiled before install. If an error occurs during compile, the package will be removed from the install list. Unfortunately this feature only works with Lazarus trunk, since @Juha just recently added the compile procedure to the package interface.
I made a short video. It's kinda boring but illustrates how easy the install process will become(at least in theory).
https://youtu.be/4Sf1yz_aEA8

Ps: If somebody has a package please post the link. Thank you!
Title: Re: Online Package Manager
Post by: rvk on November 05, 2016, 07:56:15 am
I see you called the synapse trunk synapse41. I thought it was version 40.1.0.0. Calling it synapse41 might be confusing as it is not synapse 41.0.
Title: Re: Online Package Manager
Post by: eric on November 05, 2016, 01:39:20 pm
In the confirmation dialog shown in this post: http://forum.lazarus.freepascal.org/index.php/topic,34297.msg227870.html#msg227870
there are some typos. It should read
Quote
Package: "lazupdate.lpk" depends on package: "laz_synapse.lpk". Resolve dependency?
Title: Re: Online Package Manager
Post by: GetMem on November 05, 2016, 07:46:07 pm
@minesadorada
Done! Each package default hompage is linked to your wiki page. Thanks.

@rvk
Fixed. "synapse 41.0" is now "synapse40.1".  Thanks.

@eric
Thanks. I fixed the typos.
Title: Re: Online Package Manager
Post by: minesadorada on November 07, 2016, 08:16:51 pm
Hi Getmem,
Can you hold fire on LazAutoUpdate for a bit?  SourceForge changed their downloading protocol, (grr) and the current version of lazautoupdate doesn't cope with it yet.
Title: Re: Online Package Manager
Post by: GetMem on November 07, 2016, 11:33:47 pm
Quote
@minesadorada
Can you hold fire on LazAutoUpdate for a bit?
The repository is already updated. I can hide LazAutoUpdate if you like.
Title: Re: Online Package Manager
Post by: GetMem on November 07, 2016, 11:37:18 pm
The repository is up and running again(40 packages). Before testing, please download and install lazopkman from the following link: https://github.com/getmem/LazOpkMan
What's new:
  - compile packages before install(available in Lazarus trunk r53263+), in version 1.6 the compile process is skipped
  - no more annoying explorer windows
  - force download, force extract implemented
  - remember last download directory, remember package source and package destination directory
  - a lot of bugix
  - update packages from external link(test with bgrabitmap9.2.1 - it has a valid link). I will fix the others soon. In my opinion this was a bad idea, since very few maintainers provide snapshots of development versions.

To do:
 - update packages from SVN
 - documentation
 - voting system

Please test. Thank you!
Title: Re: Online Package Manager
Post by: josh on November 08, 2016, 01:21:59 am
Download New version from above link
Went to install package and get error
opkman_installer.pas(166,40) Error: identifier idents no member "DoCompilePackage"

in line
function TPackageInstaller.CompilePackage(const AIDEPackage: TIDEPackage;
  APackageFile: TPackageFile): Integer;
begin
  Result := -1;
  {$if declared(lcl_version)}
   {$if (lcl_major >= 1) and (lcl_minor >= 7)}
     //DoCompilePackage function is only available with Laz 1.7 +
     DoOnPackageInstallProgress(imCompilePackage, APackageFile);
     Result := PackageEditingInterface.DoCompilePackage(AIDEPackage, [pcfCleanCompile, pcfDoNotSaveEditorFiles], False);  <--------- HERE
   {$endif}
  {$endif}
end;             


I am using Laz 1.7 SVN 51308, FPC 3.1.1.
Previous version .9  Worked ok (only 8 packages thoug)
Title: Re: Online Package Manager
Post by: halim on November 08, 2016, 01:35:37 am
i get same error
i am using Newpascal ver 1.034 Laz Ver.1.7 and FPC 3.1.1
Title: Re: Online Package Manager
Post by: GetMem on November 08, 2016, 06:28:15 am
@Josh, @halim
I edited my post. DoCompilePackage function is only available in Lazarus trunk r53263+. I cannot distinguish with ifdef between different SVN versions, only major/minor/release versions. So you need Lazarus trunk r53263 or change the following line from:
Quote
{$if (lcl_major >= 1) and (lcl_minor >= 7)}
to
Quote
{$if (lcl_major >= 1) and (lcl_minor >= 8)}
The package manager will work fine(no compiling feature though). Of course none of this will be a problem after the next release.
Title: Re: Online Package Manager
Post by: minesadorada on November 08, 2016, 08:35:13 am
Excellent job @getmem!  Works like a charm.
Laz 1.7/fpc 3.1.1 Windows v10 x86_64
Title: Re: Online Package Manager
Post by: minesadorada on November 08, 2016, 07:56:24 pm
Quote
@minesadorada
Can you hold fire on LazAutoUpdate for a bit?
The repository is already updated. I can hide LazAutoUpdate if you like.
Ok LazAutoUpdate is working now (thanks Sourceforge for changing your API!)
There is another issue.

LazAutoUpdate is an updating system.  The component takes care of most of the updating, but there is a console app that is a companion, and a GUI for developers to manage the update/versioning process.  The question is: How much should OPM do?
Ideally it would download the component; the companion non-GUI (source) and the GUI developer maintainer (source).  Is this possible?  With all 3 elements, making an updateable Lazarus app hosted on SourceForge is pretty easy (which was the reason I wrote it) so I think it has value to developers.

I really don't know the answer, and would ask your advice.
Title: Re: Online Package Manager
Post by: GetMem on November 08, 2016, 11:12:18 pm
Quote
@minesadorada
The question is: How much should OPM do? Ideally it would download the component; the companion non-GUI (source) and the GUI developer maintainer (source).
Ideally only download the component, but I guess source is always ok. The user should decide the appropriate action.
Title: Re: Online Package Manager
Post by: GetMem on November 09, 2016, 07:51:31 am
@minesadorada
Your package is visible again. Please test it. You should download the source first: https://github.com/getmem/LazOpkMan
Title: Re: Online Package Manager
Post by: minesadorada on November 09, 2016, 09:18:56 am
@minesadorada
Your package is visible again. Please test it. You should download the source first: https://github.com/getmem/LazOpkMan
At svn the version is 0.1.28.0  O.P.M. has 0.1.27.0 as the downloadable.  Can you update please?
Title: Re: Online Package Manager
Post by: GetMem on November 09, 2016, 09:29:56 am
Quote
At svn the version is 0.1.28.0  O.P.M. has 0.1.27.0 as the downloadable.  Can you update please?
Consider it done. However I don't know exactly when the next update will be. As I told you previously I'm not hosting the repository.
Title: Re: Online Package Manager
Post by: minesadorada on November 09, 2016, 09:43:16 am
Quote
At svn the version is 0.1.28.0  O.P.M. has 0.1.27.0 as the downloadable.  Can you update please?
Consider it done. However I don't know exactly when the next update will be. As I told you previously I'm not hosting the repository.
I notice OPM has 'Update Link'. The update link for lazautoupdate is https://sourceforge.net/projects/lazautoupdate/files/lazautoupdatesource/ - I keep that synchronised with svn.  In future, I'll include the version in the package file name.
Title: Re: Online Package Manager
Post by: GetMem on November 09, 2016, 10:07:20 am
Quote
I notice OPM has 'Update Link'. The update link for lazautoupdate is https://sourceforge.net/projects/lazautoupdate/files/lazautoupdatesource/ - I keep that synchronised with svn.  In future, I'll include the version in the package file name.
Thanks! This would be helpful.
Please note:
The update link should point to a zip file. The zip structure should be the same as the one in the repository. I mean you can add new files to your project but the path to the lpk file, the main directory name should stay the same, otherwise OPM will fail to load the package.


 
Title: Re: Online Package Manager
Post by: minesadorada on November 09, 2016, 10:42:31 am
Quote
I notice OPM has 'Update Link'. The update link for lazautoupdate is https://sourceforge.net/projects/lazautoupdate/files/lazautoupdatesource/ (https://sourceforge.net/projects/lazautoupdate/files/lazautoupdatesource/) - I keep that synchronised with svn.  In future, I'll include the version in the package file name.
Thanks! This would be helpful.
Please note:
The update link should point to a zip file. The zip structure should be the same as the one in the repository. I mean you can add new files to your project but the path to the lpk file, the main directory name should stay the same, otherwise OPM will fail to load the package.


 
Full link to zipfile with correct path (\lazautoupdate) : https://sourceforge.net/projects/lazautoupdate/files/lazautoupdatesource/lazautoupdate.zip (https://sourceforge.net/projects/lazautoupdate/files/lazautoupdatesource/lazautoupdate.zip)
Title: Re: Online Package Manager
Post by: GetMem on November 09, 2016, 11:08:06 am
To test the link I updated the JSON file in my local computer. Works great. The package is successfully updated:
https://youtu.be/Mi3bhavwxMI
Please do this to all your five packages.

In a week or so the SVN update will be also available(hopefully).
Title: Re: Online Package Manager
Post by: minesadorada on November 09, 2016, 11:44:45 am
@getmem:

https://sourceforge.net/projects/lazautoupdate/files/otherpackages/longtimer.zip
https://sourceforge.net/projects/lazautoupdate/files/otherpackages/poweredby.zip
https://sourceforge.net/projects/lazautoupdate/files/otherpackages/scrolltext.zip
https://sourceforge.net/projects/lazautoupdate/files/otherpackages/playsoundpackage.zip
Title: Re: Online Package Manager
Post by: GetMem on November 09, 2016, 12:14:00 pm
@minesadorada
Done!
Title: Re: Online Package Manager
Post by: minesadorada on November 09, 2016, 12:22:53 pm
@minesadorada
Done!
Thanks.
Where's the update link for OnlinePackageManager? :)
Title: Re: Online Package Manager
Post by: wp on November 09, 2016, 02:02:57 pm
As I can see the package list contains also CSVDocument. I don't know if this is a good idea: CSVDocument is included in the fpc installation since version 3.0. At least there should be a comment that the ccr version should be used only for older fpc versions.

Another wish: Many packages don't contain a description of what the package is good for, but for those which do the description usually runs across several lines. In the package list this desciption, however, is squeezed into a single line. Wouldn't it be better to provide a separate memo for the package description, or to increase the row height of this line to show all lines as a wordwrapped text? The minimum would be a popup hint showing the entire text. The same probably for the license.
Title: Re: Online Package Manager
Post by: minesadorada on November 09, 2016, 03:00:08 pm
...
Another wish: Many packages don't contain a description of what the package is good for, but for those which do the description usually runs across several lines. In the package list this desciption, however, is squeezed into a single line. Wouldn't it be better to provide a separate memo for the package description, or to increase the row height of this line to show all lines as a wordwrapped text? The minimum would be a popup hint showing the entire text. The same probably for the license.
Each package should ideally have a Wiki page (or one constructed for it) as it's 'Home Page'.  In that way, OPM can keep it compact, and the wiki page is easily updated by the package author with more information and/or links.
It's the Lazarus way to separate functionality for ease of maintainance :)

I agree the license type should be displayed in the OPM tree display.  That's important.

@getmem is doing a magnificent job so far - let's not discourage him with loads of 'suggestions/improvements'  8)

[hypocrite mode]
It would be nice if OPM showed which IDE tab the component was being installed to. (though sometimes 'hunt the newly-installed component' can be a diversion from an otherwise boring day)
[/hypocrite mode]

 
Title: Re: Online Package Manager
Post by: lainz on November 09, 2016, 03:11:55 pm
Installed spktoolbar with this tool and it's working well in Trunk. Good job.
Title: Re: Online Package Manager
Post by: GetMem on November 09, 2016, 04:29:13 pm
Quote
As I can see the package list contains also CSVDocument. I don't know if this is a good idea: CSVDocument is included in the fpc installation since version 3.0. At least there should be a comment that the ccr version should be used only for older fpc versions.
Thanks! I removed CVSDocument from my local repository. It will take a few days though until the main repository is updated.

Quote
Another wish: Many packages don't contain a description of what the package is good for, but for those which do the description usually runs across several lines. In the package list this desciption, however, is squeezed into a single line. Wouldn't it be better to provide a separate memo for the package description, or to increase the row height of this line to show all lines as a wordwrapped text? The minimum would be a popup hint showing the entire text. The same probably for the license.
I will add a popup hint, because a variable height row(with a lot of info) would look ugly in my opinion. The tree is already displaying to much info.


@minesadorada
Quote
Let's not discourage him with loads of 'suggestions/improvements'
No no! Please let the suggestions/improvements coming. I like that. I will try to implement most of them(at least the reasonable ones).

Quote
It would be nice if OPM showed which IDE tab the component was being installed to. (though sometimes 'hunt the newly-installed component' can be a diversion from an otherwise boring day)
Use the: Lazarus Menu-->View-->Components(Ctrl+Alt+P). It takes just a few seconds to locate the newly installed component.

Quote
@lainz
Installed spktoolbar with this tool and it's working well in Trunk. Good job.
Thanks. You should send me the upload links for the BGRA components. See BgraBitmap9.2.1 and the previous discussion with @minesadorada.
This one
https://github.com/bgrabitmap/bgracontrols/archive/v4.3.2.zip
and this one
https://github.com/bgrabitmap/bgracontrolsfx/archive/v0.1.6.zip
?
Title: Re: Online Package Manager
Post by: lainz on November 09, 2016, 05:03:47 pm
@GetMem
These links are the right ones for these versions. But I'm not mantaining an update link, the bgrabitmap one is also not an update link, when a new release arrives that will not be the same zip file.

Maybe we can create a special release called OnlinePackageManager where you always can grab a zip file we upload when we release the newest version. (If I understand it right). Of course it will have always the same name.

Or you can use the master repository (git) with has the latest sources always (we need to ensure that always compile and is working well). Or we can create a separate branch wich is updated only when we release a new version.

I need to talk with circular.
Title: Re: Online Package Manager
Post by: GetMem on November 09, 2016, 08:29:53 pm
@wp
I implemented the popup hint: https://github.com/getmem/LazOpkMan

@lainz
Quote
Maybe we can create a special release called OnlinePackageManager where you always can grab a zip file we upload when we release the newest version. (If I understand it right). Of course it will have always the same name.
Yes, you understood correctly. That special release would be great!

Quote
Or you can use the master repository (git) with has the latest sources always (we need to ensure that always compile and is working well). Or we can create a separate branch wich is updated only when we release a new version.
I plan to add this feature. The package manager will be able to checkout/update/pull whatever from a SVN repository. The thing is most of the packages out there don't have a zip with the latest version, only a link to a SVN repository. This feature seems more logical then the "update from external zip".
Title: Re: Online Package Manager
Post by: lainz on November 09, 2016, 10:23:50 pm
@GetMem
We will use the Master branch as release. I talked with circular.

BGRA Bitmap
There will be a new repository for BGRA Bitmap tomorrow so stay tuned.

BGRA Controls
https://github.com/bgrabitmap/bgracontrols/archive/master.zip

BGRA Controls FX
https://github.com/bgrabitmap/bgracontrolsfx/archive/master.zip
Title: Re: Online Package Manager
Post by: GetMem on November 10, 2016, 07:01:59 am
@lainz
Great! I will add the links to the JSON file. One more thing, I noticed that the description and author fields are missing from "bgracontrols.lpk" and "bgracontrolsfx.lpk". Is this by accident?
Title: Re: Online Package Manager
Post by: wp on November 10, 2016, 01:48:48 pm
There's a new release version of CalLite (https://sourceforge.net/projects/lazarus-ccr/files/CalLite/) and a wiki page (http://wiki.lazarus.freepascal.org/CalLite).
Title: Re: Online Package Manager
Post by: lainz on November 10, 2016, 03:39:55 pm
@lainz
Great! I will add the links to the JSON file. One more thing, I noticed that the description and author fields are missing from "bgracontrols.lpk" and "bgracontrolsfx.lpk". Is this by accident?

I will update them today.

This is the new link for bgrabitmap
https://github.com/bgrabitmap/bgrabitmap/

This is the update link
https://github.com/bgrabitmap/bgrabitmap/archive/master.zip
Title: Re: Online Package Manager
Post by: GetMem on November 11, 2016, 07:07:01 am
Quote
@wp
There's a new release version of CalLite (https://sourceforge.net/projects/lazarus-ccr/files/CalLite/) and a wiki page (http://wiki.lazarus.freepascal.org/CalLite).
I updated the package. Thanks.
Title: Re: Online Package Manager
Post by: josh on November 14, 2016, 09:44:06 pm
Hi
Just put a clean Laz Trunk on Clean Installation and thought I would try this to add in some packages; all was going well until I instal synapse 40.1. This failed with the screen shot attached, i checked installed packages and nothing is installed. So then I tried to install another package and this generates the same error. I cleaned sources, and rebuild ide, and installed the other package and all seems ok, so I installed lazsynapse 40. and this seems ok.

Is this a known problem?

Also When Error occurred, I clicked the install updates button again, and it failed again, when I checked installed packages There were 2 instances of each of the installed packages.
Title: Re: Online Package Manager
Post by: GetMem on November 14, 2016, 10:46:24 pm
Hi josh,

Quote
Is this a known problem?
Not until now. It was a bug. Please test it again: https://github.com/getmem/LazOpkMan
Thank you!

PS: Both synapse40 and synapse40.1 are runtime packages, after "install" you won't be able to see them in the installed package list. However the packages can be added to the required package list of a project.

PS1: Today(2016.11.15) I had to change the JSON structure again. Please test with: https://drive.google.com/open?id=0B9Me_c5onmWoaldvMndnTFNxTTQ
Title: Re: Online Package Manager
Post by: josh on November 16, 2016, 03:26:41 pm
Hi Getmem,

As the new release 1.6.2 was released; i thought I would test it out; I now have a 1.6.3 (Fixes) and FPC 3.0.1 (Fixes) installed; when I try to compile I get

opkman_oleutils.pas(110,64) Error: Call by var for arg no. 3 has to match exactly: Got "Int64" expected "QWord"

Which points to here.

Res:=FSrcStream.Seek(Int64(liOffset), Origin, Int64(liResult));

Any ideas?
Title: Re: Online Package Manager
Post by: GetMem on November 16, 2016, 05:17:28 pm
@josh
Fixed! Download link: https://drive.google.com/open?id=0B9Me_c5onmWoY2xCZEdONERqa28
Hopefully the main repository will be synchronized again in a few days, then you can use the github version.

Title: Re: Online Package Manager
Post by: GetMem on November 18, 2016, 08:10:01 am
Hi all,

The main repository has been moved to: http://packages.lazarus-ide.org/ The download speed is quite exceptional. Thanks @Marc! 
From now on, I can update the repository more frequently, multiple times per day if necessary. I already synchronized the repository with latest changes from my local computer. In order to get the latest version:
   1. Download the source from: https://github.com/getmem/LazOpkMan
   2. Compile/Build/Install
   3. Go to package manager option and change the remote repository address to: http://packages.lazarus-ide.org/
In a few days the package manager will be moved to Lazarus trunk, so updating the source will become much easier than it is now.

PS: Feedback, bug report, package update is more then welcome.

Title: Re: Online Package Manager
Post by: lainz on November 18, 2016, 02:03:41 pm
Hi, when I try to rebuild the ide it says "can't find unit opkman_optionsfrm.pas".

This line in the file causes the problem:

Code: Pascal  [Select]
  1.  *   Wide Web at <http://www.gnu.org/copyleft/gpl.html>. You can also      *

Seems that it thinks that's an html file? Is a bug in the compiler.

Edit: I can't reproduce it. It happened only one or two times, then I edited the file to remove that line, then It compiled. Then I reverted the line and deleted the lib folder and then recompiled and now compiles  %)
Title: Re: Online Package Manager
Post by: GetMem on November 18, 2016, 03:06:32 pm
@lainz
I did try to reproduce myself without success.

Good news! Thanks to @varianus the package manager has proxy support. Please test.
Title: Re: Online Package Manager
Post by: lainz on November 18, 2016, 03:29:05 pm
@lainz
I did try to reproduce myself without success.

Good news! Thanks to @varianus the package manager has proxy support. Please test.

Well that's these runtime errors that it's supossed to be reported to the bugtracker, but if we can't reproduce there's nothing to report, I've not taken a screenshot not written the number down to a text file, so nothing can be done. It was strange, why the compiler will try to find the file in the comments  :o

I don't have proxy but seems that it will put some people happy.
Title: Re: Online Package Manager
Post by: minesadorada on November 18, 2016, 09:21:14 pm
Downloaded latest version and changed the repository URL.  Glad to report its working great - and very fast, too.
Environment: NewPascal Laz1.7/fpc 3.1.1 Win 10 64bit
I'll test it tomorrow in Laz 1.6/fp c3.0 32-bit Ubuntu 14.x =update= Complete Success!

Minor annoyance: I installed the DCrypt package, which in itself is pretty cryptic, so I clicked on the help link on OPM, and it points to an obsolete Delphi page which doesn't help at all.  I would suggest changing the link to http://wiki.lazarus.freepascal.org/DCPcrypt (http://wiki.lazarus.freepascal.org/DCPcrypt)

Thanks for this useful add-on.
Title: Re: Online Package Manager
Post by: GetMem on November 18, 2016, 11:23:01 pm
Quote
Minor annoyance: I installed the DCrypt package, which in itself is pretty cryptic, so I clicked on the help link on OPM, and it points to an obsolete Delphi page which doesn't help at all.  I would suggest changing the link to http://wiki.lazarus.freepascal.org/DCPcrypt
Although the obsolete delphi page is the original one, I agree http://wiki.lazarus.freepascal.org/DCPcrypt is more helpful for Lazarus user. Link changed. Thanks!
Title: Re: Online Package Manager
Post by: Groffy on November 21, 2016, 09:31:04 am
Hello,

just saw, that in Lazarus trunk the onlinepackagemanager moved into the \components folder. This version allows to use a proxy server. Great! After installing and open the manager form, there is a message about an invalid JSON ini file. How to avoid that?



Title: Re: Online Package Manager
Post by: GetMem on November 21, 2016, 09:35:45 am
Quote
Just saw, that in Lazarus trunk the onlinepackagemanager moved into the \components folder. This version allows to use a proxy server. Great! After installing and open the manager form, there is a message about an invalid JSON ini file. How to avoid that?
This could only happen if you connect to the old repository with outdated JSON. Open options, then change Remote repository to: http://packages.lazarus-ide.org/
Title: Re: Online Package Manager
Post by: Groffy on November 21, 2016, 09:39:49 am
Hello GetMem

Quote
This could only happen if you connect to the old repository with outdated JSON. Open options, then change Remote repository to: http://packages.lazarus-ide.org/

Thank you for your quick response, I forgot about that. Again thank you for your great work!

Title: Re: Online Package Manager
Post by: GetMem on November 21, 2016, 09:44:06 am
Quote
@Groffy
Thank you for your quick response, I forgot about that. Again thank you for your great work!
You're more then welcome! Feel free to post any bugs you find or suggestions/feature requests.
Title: Re: Online Package Manager
Post by: Groffy on November 21, 2016, 01:03:34 pm
Feel free to post any bugs you find or suggestions/feature requests.

Well, its fun to follow the progress, you put a lot of work in it to make it work withing 6weeks (first posting on October 05.). Just a few things/ideas :

1) Filter functionality seems not fully functional. I just tested to filter by category. With any of the defined categories, the result is an empty listbox. (would be nice to store the last assigned filter)
Furthermore there is a filtering by Author, but it seems that the repositiory items do not contain the authors name (at least not visible)

2) Makes it sense to manage more than one repository address?

3) Makes it sense to enable an automatic message when for one of the installed packages an update is available?



My best regards
Title: Re: Online Package Manager
Post by: GetMem on November 21, 2016, 01:37:08 pm
@Groffy
Quote
Filter functionality seems not fully functional. I just tested to filter by category. With any of the defined categories, the result is an empty listbox.
This is a bug,  fixed in r53396.  Please test.

Quote
would be nice to store the last assigned filter
This can be easily implemented, but then the list gets filtered by default. It may lead to confusion. What do you think?

Quote
Furthermore there is a filtering by Author, but it seems that the repositiory items do not contain the authors name (at least not visible)
Each repository package can contain multiple lpk files. The Author field is located in the lpk node(see attachment).

Quote
Makes it sense to manage more than one repository address?
Yes. This is in the todo list.

Quote
Makes it sense to enable an automatic message when for one of the installed packages an update is available?
I can implement this feature for the main repository, however checking updates from external webpage(see update feature) can be dangerous(malware, license issues, etc...). I don't want to automate the process.
Title: Re: Online Package Manager
Post by: JuhaManninen on November 21, 2016, 02:19:21 pm
I could not install SpkToolbar package because character case in a unit name did not match its file name. They must match exactly when a file name is not all lowercase.

I fixed it in r5371 in Lazarus-CCR.

Index: components/spktoolbar/SpkToolbar/SpkToolbar.pas
===================================================================
--- components/spktoolbar/SpkToolbar/SpkToolbar.pas     (revision 5370)
+++ components/spktoolbar/SpkToolbar/SpkToolbar.pas     (working copy)
@@ -1,4 +1,4 @@
-unit spktoolbar;
+unit SpkToolbar;

It has changed at some point. I remember installing SpkToolbar on Linux a long time ago.

Otherwise the package manager looks good!

For future changes in Options format and default values, a "Version" attribute can be added.
Other config files of the IDE have it.

About the filtering: saving and using the previous filter is not a good idea. Filter could be selected from a history list maybe.
I think the whole filter GUI must be revamped. Now it is clumsy and counter-intuitive.
There should be a separate window where many filter conditions can be set and they are 'AND'ed together.

I also think many packages now included in Lazarus components/ directory should be moved to CCR and provided through the online package manager.
Title: Re: Online Package Manager
Post by: lainz on November 21, 2016, 02:50:59 pm
The problem with the CCR is that doesn't support the 'update' option, to get the latest sources.

Why you (Lazarus mantainers?) don't create a GitHub organization and create a separate repository for each package, so all benefit from it: getting the latest with the Online Package Manager is a download of the master.zip

Edit: I mean only for those in active development, else it has no sense at all.
Title: Re: Online Package Manager
Post by: Groffy on November 21, 2016, 04:13:25 pm
@GetMem

This can be easily implemented, but then the list gets filtered by default. It may lead to confusion. What do you think?

Right, I was just thinking about filtering into categories, which would be my prefered filter. I agree thats not really helpful for other filter settings. Could lead to confusions.

I like JuhaManninnen's idea (later posting) about saving filter settings into a history

Quote
Each repository package can contain multiple lpk files. The Author field is located in the lpk node(see attachment).

Right, did not see/understand it on the first view.

Quote
I can implement this feature for the main repository, however checking updates from external webpage(see update feature) can be dangerous(malware, license issues, etc...). I don't want to automate the process.

Hm, good point. However I was just thinking of an automatic information, not to start an update process for a package.


Best regards
Title: Re: Online Package Manager
Post by: minesadorada on November 21, 2016, 04:34:36 pm
The problem with the CCR is that doesn't support the 'update' option, to get the latest sources.

Why you (Lazarus mantainers?) don't create a GitHub organization and create a separate repository for each package, so all benefit from it: getting the latest with the Online Package Manager is a download of the master.zip

Edit: I mean only for those in active development, else it has no sense at all.
All my packages are updateable and downloadable via ccr SVN (https://svn.code.sf.net/p/lazarus-ccr/svn (https://svn.code.sf.net/p/lazarus-ccr/svn)) and also zips maintained at SourceForge that OPM currently uses.  I also try to keep the associated Wiki pages up to date.  I like Sourceforge and SVN.

I regularly svn-update to my local disk. - the ccr repository is currently at Revision 5371 with lots of new stuff.
Title: Re: Online Package Manager
Post by: JuhaManninen on November 21, 2016, 04:39:42 pm
The problem with the CCR is that doesn't support the 'update' option, to get the latest sources.
'svn' command can be used to update latest sources of individual packages there.

Quote
Why you (Lazarus mantainers?) don't create a GitHub organization and create a separate repository for each package, so all benefit from it: getting the latest with the Online Package Manager is a download of the master.zip
I have a gap in my knowledge. Who makes a master.zip file?

I understood earlier that GetMem planned for a direct revision control access in the package manager. I am not sure if I undertstood right.
It would allow loading latest development versions of selected packages by a svn, git or hg etc. command. Such command takes longer than loading a ready zipped package of course.

Maintained packages that have releases, the latest release version must be added to the JSON config file manually now.
I am not sure how GetMem planned to automate the process. There must be a way for the package maintainer to pass the version info without changing download URL (and file name).
This is quite important in future. The online package manager maintainer should have as little admin work as possible. Otherwise we will have an obsolete and outdated package repository in future, years from now.
[Edit] ... or maybe it is not too much maintenance to copy a new package and update the config. Don't know.
In any case it requires action from 2 people: the maintainer of a package and the maintainer of the online package manager.
Title: Re: Online Package Manager
Post by: wp on November 21, 2016, 05:15:36 pm
The problem with the CCR is that doesn't support the 'update' option, to get the latest sources.
I do it all the time. In Windows, using TortoiseSVN: Right-click on local ccr repository, "SVN Update" - that's it.
Title: Re: Online Package Manager
Post by: lainz on November 21, 2016, 05:26:07 pm
The problem with the CCR is that doesn't support the 'update' option, to get the latest sources.
I do it all the time. In Windows, using TortoiseSVN: Right-click on local ccr repository, "SVN Update" - that's it.

I know wp, I know, I was talking about the update option of this Online package manager that expects a zip file to get the latest sources with it. Else I don't will use it, for example I installed the spktoolbar, then you added some new themes, if I want them with the online package manager I can't, I must use svn (that of course I have and know to update with it) but that's not the point of using the package manager?
Title: Re: Online Package Manager
Post by: minesadorada on November 21, 2016, 06:17:00 pm
The problem with the CCR is that doesn't support the 'update' option, to get the latest sources.
I do it all the time. In Windows, using TortoiseSVN: Right-click on local ccr repository, "SVN Update" - that's it.

I know wp, I know, I was talking about the update option of this Online package manager that expects a zip file to get the latest sources with it. Else I don't will use it, for example I installed the spktoolbar, then you added some new themes, if I want them with the online package manager I can't, I must use svn (that of course I have and know to update with it) but that's not the point of using the package manager?
Ideally OPM will eventually use SVN preferentially for downloads and updates.  There is a case to be made for only providing packages with an active svn (or git) repository,  however I like the fallback option of using a hosted zip for developers that don't use svn or git - but only for selected packages i.e. ones where the developer is actively maintaining it.

For easier auto-maintainance, there need to be instructions for developers to make their components 'OPM-friendly'.  Some kind of standard versions.txt file that OPM can check (on user request) as well as including/updating the json file when committing/uploading newer versions.  The instructions should be in the wiki (with a link from OPM)

my 2€
Title: Re: Online Package Manager
Post by: lainz on November 21, 2016, 06:30:46 pm
Quote
'svn' command can be used to update latest sources of individual packages there.

Good point.

Quote
I have a gap in my knowledge. Who makes a master.zip file?

Is automatically done, like this:
https://github.com/bgrabitmap/bgrabitmap/archive/master.zip

So if you have any GitHub repository the zip is already available. It contains the latest sources.

Quote
Maintained packages that have releases, the latest release version must be added to the JSON config file manually now.
I am not sure how GetMem planned to automate the process. There must be a way for the package maintainer to pass the version info without changing download URL (and file name).
This is quite important in future. The online package manager maintainer should have as little admin work as possible. Otherwise we will have an obsolete and outdated package repository in future, years from now.
[Edit] ... or maybe it is not too much maintenance to copy a new package and update the config. Don't know.
In any case it requires action from 2 people: the maintainer of a package and the maintainer of the online package manager.

That's the main problem of having a single JSON made by hand. If for example we have a website to upload the packages, then a database will do the trick, you do a query and get the list of packages in a generated JSON, that no one needs to mantain.

Quote
For easier auto-maintainance, there need to be instructions for developers to make their components 'OPM-friendly'.  Some kind of standard versions.txt file that OPM can check (on user request) as well as including/updating the json file when committing/uploading newer versions.  The instructions should be in the wiki (with a link from OPM)

That's true. a versions.txt pointing:
- The latest download source of release in zip (an URL)
- The latest version number
- All the other neccessary stuff

So the package manager database will be updated automatically, say one time per month? Of course there's no one testing the quaility of the packages with this way. Is always a responsability of the user what to trust.

So in fact you need only a list of URL's to get the data from (list of versions.txt only), process them and update the database, it will take time to process all, but not too much, since these are a few kb files.

So the submit new app will be something like:

Name of your package: Something
URL of updates: version.txt url containing all the data for the json (generated with the online package manager of course)
Title: Re: Online Package Manager
Post by: JuhaManninen on November 21, 2016, 07:07:37 pm
Ideally OPM will eventually use SVN preferentially for downloads and updates.  There is a case to be made for only providing packages with an active svn (or git) repository,  however I like the fallback option of using a hosted zip for developers that don't use svn or git - but only for selected packages i.e. ones where the developer is actively maintaining it.
Oops, now the idea went upside down.
The main purpose of the package manager must be to provide stable versions of packages provided by their authors/maintainers.
The same way that any production quality SW is delivered.

In a perfect world all packages have an active maintainer but in real world they don't.
A fallback option must be to provide direct access to a development SVN / Git / etc. server of  the unmaintained packages. Another option is that the maintainer of our new online package manager picks and tests good revisions and "releases" them. He essentially then partly maintains those packages which is not realistic in long term.

People who participate in testing or developing a certain package will continue to use its revision control access directly.
It is not the main scope of our new package manager.

So if you have any GitHub repository the zip is already available. It contains the latest sources.
Ok, the same thing as getting directly using git command.

Quote
That's true. a versions.txt pointing:
- The latest download source of release in zip (an URL)
- The latest version number
- All the other neccessary stuff

So the package manager database will be updated automatically, say one time per month? Of course there's no one testing the quaility of the packages with this way. Is always a responsability of the user what to trust.
Updating automatically once per month sounds clumsy. Why not weekly or daily then?
Actually this works best with external URLs. The maintenance and hosting is moved to a package maintainer then.
I think the process of acknowledging new versions and copying to package repo should be made smoother but maybe not fully automatic.
Title: Re: Online Package Manager
Post by: minesadorada on November 21, 2016, 07:15:30 pm
@GetMem - I have a new package for you.  It's a runtime-only non-installable one (maybe a good test for OPM?)
OPM package with JSON:
https://sourceforge.net/projects/lazautoupdate/files/otherpackages/cryptini_opm.zip (https://sourceforge.net/projects/lazautoupdate/files/otherpackages/cryptini_opm.zip)

I hope I configured it OK.
Updates in: https://sourceforge.net/projects/lazautoupdate/files/otherpackages/cryptini.zip (https://sourceforge.net/projects/lazautoupdate/files/otherpackages/cryptini.zip)
Wiki Page http://wiki.lazarus.freepascal.org/CryptINI (http://wiki.lazarus.freepascal.org/CryptINI)
Testing app/demo:
Source: https://sourceforge.net/projects/lazautoupdate/files/otherpackages/cryptinitest_source.zip (https://sourceforge.net/projects/lazautoupdate/files/otherpackages/cryptinitest_source.zip)
Ready-compiled (Win32/64 + Linux 32/64): https://sourceforge.net/projects/lazautoupdate/files/otherpackages/testcryptini_compiled.zip (https://sourceforge.net/projects/lazautoupdate/files/otherpackages/testcryptini_compiled.zip)

Note: When using  the OPM 'Create Repository Package' I noticed that it still puts unnecessary stuff in the zipfile.  I manually removed them.

 
Title: Re: Online Package Manager
Post by: lainz on November 21, 2016, 07:35:15 pm
Quote
Ok, the same thing as getting directly using git command.

Yes, of course it requires no git, but is the same.

Quote
Updating automatically once per month sounds clumsy. Why not weekly or daily then?
Actually this works best with external URLs. The maintenance and hosting is moved to the package maintainer then.
I think the process of acknowledging new versions and copying to package repo should be made smoother but maybe not fully automatic.

Yes the update can be done daily.

Of course we have control about our packages with no need of a unique mantainer, a bit of work for everyone doesn't harm. Then also saves the repository bandwidth, that's used only for the database.

Well is ok, maybe:
- Check if the source is reliable, then accept or decline the submission. Then we need a mantainer, but only for the first time the package is accepted or declined. That will prevent spam, but for the rest can be made automatically.
Title: Re: Online Package Manager
Post by: JuhaManninen on November 21, 2016, 08:35:46 pm
Yes, of course it requires no git, but is the same.
Ok, I forgot that somehow. Subversion / Git must be installed before they can be used.
It is a requirement that should not be forced to everybody.
The package manager can check if "svn" executable is available and provide package links accordingly.

Yes, it means the master.zip file is useful, development versions from GitHub can be downloaded without Git installed.
Title: Re: Online Package Manager
Post by: lainz on November 21, 2016, 08:38:16 pm
If you want I can create the organization and all the repositories, then also I can add every mantainer by mail or by username.

It will be called the same, Lazarus CCR but in github, and instead of only a repository, separate repos for each package.
Title: Re: Online Package Manager
Post by: JuhaManninen on November 21, 2016, 10:01:04 pm
If you want I can create the organization and all the repositories, then also I can add every mantainer by mail or by username.

It will be called the same, Lazarus CCR but in github, and instead of only a repository, separate repos for each package.
Moving the whole CCR to GitHub is not something I can decide or want to decide.
There are many people involved in the current CCR's maintenance. Discussion in mailing list would reach more of those people.

Another question is if the master.zip file is enough reason to move things there.
People who want development versions can easily install svn. The main purpose of the package manager would remain the released packages.

A GitHub organization can be created in any case. How much and how quickly things should move there from CCR, must be discussed with others.
Title: Re: Online Package Manager
Post by: lainz on November 21, 2016, 10:05:54 pm
Ok, I've created the organization. Unfortunately, the lazarus organization is already created by someone. So I created lazarusccr one.

Please send me your github username or mail to add you as administrator, then you can add all the other people.
Title: Re: Online Package Manager
Post by: GetMem on November 21, 2016, 10:07:59 pm
Nice discussion!  :D
I will try to address all the issues raised.

@Juha @minesadorada
Thanks for the packages. I will add them soon to the repository.

@lainz, @wp, @minesadorada
Both github and sourceforge are perfectly capable to store the zipfiles needed for the update feature as @lainz proved for github and @minesadorada for sourceforge. The only problem with the ccr(sourceforge) is that one cannot download an individual package, only the whole repository, it's not practical. Of course this is nobodies fault and this is why I decided to add SVN/Git support to package manager(it's not yet implemented).

@lainz
Your idea with external JSON is good, but in order to get there we needed a stable version first. Without a central repository and a working package manager we would only have an abstract, theoretical discussion. Luckily @Juha supported the whole project from the beginning.
If I understood you correctly I need to add another JSON file which will point to external JSONs, created by the package maintainers. The external JSON contains all the details about the package, along with a zipfile. If this true, it's already done in 95%.

PS: I would like to see the the ccr moved to github, in my opinion is more flexible the source forge.

Title: Re: Online Package Manager
Post by: minesadorada on November 22, 2016, 01:17:25 pm
A suggestion:
Sometimes, package writers make a demo/example app (source) to go with a package.  Could the .json file include a link field pointing to the location of one, if it exists?

I find it very helpful when first installing a package, both to test compatibility and to look at the source and gain some tips on using the package.  Most of the built-in packages have example code, and it's very helpful.
Title: Re: Online Package Manager
Post by: GetMem on November 22, 2016, 01:30:00 pm
@minesadorada
Quote
Sometimes, package writers make a demo/example app (source) to go with a package.  Could the .json file include a link field pointing to the location of one, if it exists?
I find it very helpful when first installing a package, both to test compatibility and to look at the source and gain some tips on using the package.  Most of the built-in packages have example code, and it's very helpful
Usually the demo applications are located in the root directory of the package, namely in: demo/example or sample folder. Currently almost every repository package has a demo subfolder. Would you like to add a link to an external demo zip file? 
Title: Re: Online Package Manager
Post by: minesadorada on November 22, 2016, 01:41:23 pm
@minesadorada
Quote
Sometimes, package writers make a demo/example app (source) to go with a package.  Could the .json file include a link field pointing to the location of one, if it exists?
I find it very helpful when first installing a package, both to test compatibility and to look at the source and gain some tips on using the package.  Most of the built-in packages have example code, and it's very helpful
Usually the demo applications are located in the root directory of the package, namely in: demo/example or sample folder. Currently almost every repository package has a demo subfolder. Would you like to add a link to an external demo zip file? 
Ahh, I see.  I assumed OPM only zipped the package files in order to save space.  So, such a link is unnecessary.
Title: Re: Online Package Manager
Post by: lainz on November 22, 2016, 01:47:22 pm
Hi GetMem, here is a copy of the lazarus ccr in GitHub, you already are a member
https://github.com/lazarusccr

Actually I'm mantaining it commiting regularly the changes from the original ccr.

Juha Manninen is administrator and if is possible or if they want, can add the original authors that can keep updating his repositories. I already created everything, 60 repos.

I'm adding the URL (wiki article) and description for each one, that's the only thing missing. Also maybe I need to create or update the wiki articles to point the mirror.
Title: Re: Online Package Manager
Post by: GetMem on November 22, 2016, 01:50:08 pm
@minesadorada
Feel free to add as many demo application you like(source only). I will add the demo projects to your repository packages. Just let me know which ones are changed.
Title: Re: Online Package Manager
Post by: GetMem on November 22, 2016, 02:09:23 pm
@lainz
Great! I will pause the development and update the JSON to point to the new ccr(where needed).

Quote
Actually I'm mantaining it commiting regularly the changes from the original ccr.
Manually or do you use some automated tool?

Quote
I'm adding the URL (wiki article) and description for each one, that's the only thing missing. Also maybe I need to create or update the wiki articles to point the mirror.
Ok. Perhaps we should also make some experiment with the external JSON. Can you add a JSON file along with the zip? The content of the JSON is irrelevant for now.

Thank you for your contribution. This helps a lot.
Title: Re: Online Package Manager
Post by: lainz on November 22, 2016, 02:20:09 pm
Quote
Manually or do you use some automated tool?

By hand. For that I need to contact the original authors / mantainers to get moved (if they want) to GitHub.

Quote
Ok. Perhaps we should also make some experiment with the external JSON. Can you add a JSON file along with the zip? The content of the JSON is irrelevant for now.

You can do it yourself so you can get more control over it. Pick a package of your choice, you have currently permissions to commit in every package.
Title: Re: Online Package Manager
Post by: wp on November 22, 2016, 02:41:47 pm
I am totally against movement of ccr to github without agreement of all component contributors (at least of those who actively work on their components). We will totally get lost in a mixture of thousands of derived versions, and we will be overwhelmend with issues of users selecting outdated versions from the wrong repository. With some developers transferring their work from ccr to some private github repositories without deleting the ccr folders we have a mess already now.
Title: Re: Online Package Manager
Post by: JuhaManninen on November 22, 2016, 02:46:57 pm
Hi GetMem, here is a copy of the lazarus ccr in GitHub, you already are a member
https://github.com/lazarusccr

Actually I'm mantaining it commiting regularly the changes from the original ccr.

Juha Manninen is administrator and if is possible or if they want, can add the original authors that can keep updating his repositories. I already created everything, 60 repos.
As I wrote to lainz in a PM, that is a bad idea, a kind of a brain-fart.
It would only add useless maintenence work for lainz himself and confusion for everybody else, without any benefit.
He should have discussed the issue before forking the whole CCR!

The new lazarusccr in GitHub is good for projects with a maintainer, if the maintainer wants to do his development in GitHub.
The biggest problem is that most packages and apps in CCR have no active maintainers. Forking them to GitHub does not improve the maintenance in any way.
CCR in SourceForge works pretty well for project that are maintained.

Quote from: GetMem
Great! I will pause the development and update the JSON to point to the new ccr(where needed).
Don't change anything yet!
Title: Re: Online Package Manager
Post by: lainz on November 22, 2016, 02:50:38 pm
I get it.

Please sorry. For the actively developed and those will not move I will delete the repositories.

Again please tell me those what don't want to be moved and I will delete them.
Title: Re: Online Package Manager
Post by: minesadorada on November 22, 2016, 02:56:57 pm
The lazarusccr GIT repository currently isn't a faithful copy of https://svn.code.sf.net/p/lazarus-ccr/svn (https://svn.code.sf.net/p/lazarus-ccr/svn) - it seems to be missing /components

I'm not familiar with GIT so I'm finding it hard to make a simple repository that syncs with my local files.  TortoiseSVN is a snap to use.

Why all the enthusiasm for GIT?  I understand the 'download Zip' feature, but with OPM any user can download a single package or many, whilst developers can easily set up SVN on their dev machines to commit updates to their component code (and already do)

Lazarusccr SVN has been very free and open up to now, but with OPM perhaps some rules for developers would be in order.  For instance, maintaining a ´stable' branch that OPM can rely on.

I agree that duplication is not generally a good idea - it normally leads to versioning issues.

If the svn ccr could be automatically cloned (daily perhaps) than perhaps users who prefer GIT would benefit.

my 2€
Title: Re: Online Package Manager
Post by: GetMem on November 22, 2016, 03:00:02 pm
I thought this is just a clone like this one: https://github.com/fpc-svn/lazarus-ccr
Apparently it's possible to make an automated tool that syncs with the original repository. The development still remains at sourforge. What's wrong with that? All we have to do is separate each package on the github repository.
Title: Re: Online Package Manager
Post by: JuhaManninen on November 22, 2016, 03:16:34 pm
Again please tell me those what don't want to be moved and I will delete them.
You should delete them all except if a maintainer explicitly tells you otherwise.
Remember, most projects in CCR have no maintainers. How could they tell you anything?

Any project in CCR that is currently not maintained, should be moved to GibHub only by a person who takes over its maintenance. Then the original project directory in CCR should be deleted.

I quote some pieces of your PM because this is of common interest:
Quote
Again, I will keep it up to date if no one moves to GitHub. At least it works as a mirror for the Online Package Manager.
No, we don't need a mirror for the Online Package Manager. We just got a nice and fast server for its packages.
I don't know why the whole idea was misunderstood so badly.

For worries about people preferring Git over SVN, I can say it is a non-issue. Both have easy commands. A SVN repo can be accessed using git tools (git-svn link) which also I use for my Lazarus development.
So, that is a very bad excuse to make useless forks.
Title: Re: Online Package Manager
Post by: lainz on November 22, 2016, 03:24:09 pm
I got it. I deleted it.

Please sorry if I wasted your time (and other users) with this.

Keep the discussion in the package manager from now, and I will do it the same.
Title: Re: Online Package Manager
Post by: JuhaManninen on November 22, 2016, 03:26:43 pm
I thought this is just a clone like this one: https://github.com/fpc-svn/lazarus-ccr
Apparently it's possible to make an automated tool that syncs with the original repository. The development still remains at sourforge. What's wrong with that? All we have to do is separate each package on the github repository.
Ok, a fully automatic mirror makes sense. Actually there are now multiple such mirrors for FPC and Lazarus projects.
That was not the purpose of our new lazarusccr in GitHub. Its idea was to be a master repo for some projects.
Then some projects would continue in SourceForge but be merged manually to GitHub. No good.

@lainz, if you can create an automatic SourceForge CCR mirror update, that would be great. However it must be separated from the projects hosted in GitHub.
Title: Re: Online Package Manager
Post by: JuhaManninen on November 22, 2016, 03:35:07 pm
I got it. I deleted it.
Please sorry if I wasted your time (and other users) with this.
Keep the discussion in the package manager from now, and I will do it the same.
Sorry I expressed my opinion so strongly. You deleted the whole lazarusccr organization. Actually it was a good idea for new hosted projects. Only forking everything from another repo was a problem.

Projects can still be moved to GitHub if their current (or new) maintainer so decides.
Also the automatic mirror makes sense.
Title: Re: Online Package Manager
Post by: lainz on November 22, 2016, 03:36:50 pm
Quote
@lainz, if you can create an automatic SourceForge CCR mirror update, that would be great. However it must be separated from the projects hosted in GitHub.

Hi, that's a great idea! But I don't have the knowledge to do this. Maybe Graeme? can help you because he has a lazarus repository in his account that is updated with commits and everything.

Quote
Sorry I expressed my opinion so strongly. You deleted the whole lazarusccr organization. Actually it was a good idea for new hosted projects. Only forking everything from another repo was a problem.

Don't worry I can create it again. I will add you again as administrator. Please accept the invitation soon.
Title: Re: Online Package Manager
Post by: lainz on November 22, 2016, 06:25:25 pm
I found a script, seems that's for linux?

https://github.com/dpocock/sync2git
Title: Re: Online Package Manager
Post by: JuhaManninen on November 22, 2016, 06:42:31 pm
I found a script, seems that's for linux?
https://github.com/dpocock/sync2git
Typically servers are Linux or Unix systems.
Anyway you need admin rights to a computer which is on all the time. Then add a cron job there.
Title: Re: Online Package Manager
Post by: minesadorada on November 22, 2016, 06:52:34 pm
I prefer to look at the future rather than the past.  There's been a lot of discussion about historical components and code which is no longer maintained.

The ccr has been somewhat anarchic up to now, but OPM offers the Lazarus community an opportunity to 'get it right'.  I suggest we embrace GetMem's initiative to make a long-term solution that benefits users.  @lainz and @JuhaMannien are valuable partners.  I think @wp could make valuable contributions too.

The issues seem to be versioning and maintenance.

Both need standards that don't rely on a maintainer person to continually oversee them.

The team have some good brains and analysis skills - I'm sure they can together come up with something great.

The eventual solution needs to be attractive to future contributors and users alike.

Wouldn't it be fantastic if OPM was included in the Lazarus 1.8 distribution?

Once OPM is sorted, I would love an equivalent online repository for code snippets.  The wiki and this forum have loads of useful code which gets swallowed up by time and obscurity.  I suspect like lots of users, I keep a local cache of snippets which are really useful.
Title: Re: Online Package Manager
Post by: JuhaManninen on November 22, 2016, 07:25:37 pm
The ccr has been somewhat anarchic up to now, but OPM offers the Lazarus community an opportunity to 'get it right'.
No, actually the CCR repository and the new OPM have different purposes.

CCR is a code hosting place for developers and active testers, just like other revision control servers are. Changes in source code happen there.

OPM is for end users to easily find, download and install packages. It is similar to CPAN which partly made Perl so popular years ago. (I believe you remember that time.)
An end user can trust that he gets a well tested bug-free production quality package. That's the idea anyway ...
OPM is not tied to CCR anyhow. Code for the packages can be hosted anywhere. CCR is just one of those places.

Quote
I suggest we embrace GetMem's initiative to make a long-term solution that benefits users.  @lainz and @JuhaMannien are valuable partners.
GetMem's initiative is already embraced. The package manager is in Lazarus trunk SVN sources and a server for package files is established.

Quote
The issues seem to be versioning and maintenance.
Both need standards that don't rely on a maintainer person to continually oversee them.
Well, it depends very much on a person who fixes bugs, improves code and makes stable releases when the time is right.
How exactly will "standards" help with that?

Quote
The eventual solution needs to be attractive to future contributors and users alike.
What solution do you propose?

Title: Re: Online Package Manager
Post by: minesadorada on November 22, 2016, 07:43:22 pm
No, actually the CCR repository and the new OPM have different purposes.

CCR is a code hosting place for developers and active testers, just like other revision control servers are. Changes in source code happen there.

OPM is for end users to easily find, download and install packages. It is similar to CPAN which partly made Perl so popular years ago. (I believe you remember that time.)
An end user can trust that he gets a well tested bug-free production quality package. That's the idea anyway ...
OPM is not tied to CCR anyhow. Code for the packages can be hosted anywhere. CCR is just one of those places.
OPM (if its included in Laz 1.8 ) will lend respectability to the ccr and other packages.  Is that a good or wanted thing?  Do Lazarus core developers want to field queries about packages they have no control over?  If not, who will take responsibility over new additions?

Quote
GetMem's initiative is already embraced. The package manager is in Lazarus trunk SVN sources and a server for package files is established.
Embraced by the Lazarus release team? OPM is kind-of pointless if it's 'just another 3rd-party package'. Will it be in the Lazarus 1.8 distribution? (spoiler: I think it should be)

Quote
Well, it depends very much on a person who fixes bugs, improves code and makes stable releases when the time is right.
How exactly will "standards" help with that?
No need for the scare-quotes.  Standards are necessarily arbitrary and help collaboration.  How is that scary?  Standards enable testing and validation - and if they are well-designed they can be automated.  Standards are a way to invalidate badly-designed packages that are unsupportable.

Quote
What solution do you propose?
I don't pretend to have the brains nor expertise that you guys have.  However I'll continue to make suggestions as long as that doesn't irritate you all too much :)
Writing a package specifically for OPM (Cryptini) taught me a lot about the issues from the developer's POV.  As it stands, it's somewhat cumbersome - but I have faith that OPM can be slick, robust, futureproof and simple for all, given the talents of the team.
 
Title: Re: Online Package Manager
Post by: JuhaManninen on November 22, 2016, 10:33:54 pm
OPM (if its included in Laz 1.8 ) will lend respectability to the ccr and other packages.  Is that a good or wanted thing?  Do Lazarus core developers want to field queries about packages they have no control over?  If not, who will take responsibility over new additions?
Lazarus core developers have no part in (most of) the packages in CCR. They are 3rd party packages.
The idea is to deliver 3rd party packages in an easy way.
The OPM maintainer, currently GetMem, is responsible for adding new packages. The criteria for adding should be relaxed. A maintainer's duty is not to judge the quality or usefulness of packages.
Later there will be a server side SW allowing user voting and comments. Judgement will come through user feedback.

Quote
Embraced by the Lazarus release team? OPM is kind-of pointless if it's 'just another 3rd-party package'. Will it be in the Lazarus 1.8 distribution? (spoiler: I think it should be)
Yes it will be in the next distribution. It is not a 3rd-party package, it is now in Lazarus trunk. Everything in Lazarus trunk will be in the next major release. The package manager author, GetMem, is now officially a Lazarus developer, meaning that he has commit access and communicates in the developers mailing list.

Quote
No need for the scare-quotes.  Standards are necessarily arbitrary and help collaboration.  How is that scary?  Standards enable testing and validation - and if they are well-designed they can be automated.  Standards are a way to invalidate badly-designed packages that are unsupportable.
I had no intention to insult you. I only wanted to show what is realistic and what is not.
Open source development works a little differently than commercial development.
A standard may say: "package xxx maintainer must fix bugs and make releases until he is told otherwise". In a company it works! A payed employee really does what he is told to do.
In open source, if a package's author/maintainer flees, as happened to many CCR packages, no standards requirement helps with that.

Quote
I don't pretend to have the brains nor expertise that you guys have.  However I'll continue to make suggestions as long as that doesn't irritate you all too much :)
Writing a package specifically for OPM (Cryptini) taught me a lot about the issues from the developer's POV.  As it stands, it's somewhat cumbersome - but I have faith that OPM can be slick, robust, futureproof and simple for all, given the talents of the team.
No irritation here. Suggestions are welcome. Even better, patches for the package manager in Lazarus sources are welcome throught Mantis bug tracker!
In open source a suggestion with a patch always works better.
Title: Re: Online Package Manager
Post by: wp on November 23, 2016, 12:08:26 am
This is a long thread, and I did not read everything - so excuse me if this issue already has been reported...

Now that Luiz has disabled the ccr version of VirtualTreeView which I did not want to bypass by adding comment braces around the very efficient installation stopper lines I decided to go the future way and get myself the "official" VirtualTreeView 4.8 using the OPM. I uninstalled the old VTV version (which was in a folder shared between a variety of Lazarus installations), then installed the current OPM from Lazarus trunk, checked virtualtreeview-4.8.7-R4 in the OPM and clicked Install. Fine. But the recompilation of the IDE stopped because of "The unit registervirtualtreeview exists twice in the unit path of the IDE." - the message lists the paths of the old and new installations. Using "Package" / "Package Links" I removed the old path from the list of known packages - still the same error.

What to do? Where do I have access to the "unit path of the IDE" so that I could delete the old directory?

Another point - maybe this already has been suggested, excuse me in this case: When clicking "Install" in the OPM there is no way to control the destination folder. With the VTV issue I know that the installation goes into the profile folder. This is fine in most cases, but I could imagine there are users (like myself) who want to put all third-party components into a common folder which is shared between several Lazarus installations. So, I would appreciate if there were a selection box for the destination directory in the OPM Options dialog.

Title: Re: Online Package Manager
Post by: GetMem on November 23, 2016, 06:55:32 am
@wp

Thanks for testing the package manager.

Quote
Now that Luiz has disabled the ccr version of VirtualTreeView which I did not want to bypass by adding comment braces around the very efficient installation stopper lines I decided to go the future way and get myself the "official" VirtualTreeView 4.8 using the OPM. I uninstalled the old VTV version (which was in a folder shared between a variety of Lazarus installations), then installed the current OPM from Lazarus trunk, checked virtualtreeview-4.8.7-R4 in the OPM and clicked Install. Fine. But the recompilation of the IDE stopped because of "The unit registervirtualtreeview exists twice in the unit path of the IDE." - the message lists the paths of the old and new installations. Using "Package" / "Package Links" I removed the old path from the list of known packages - still the same error.
Unfortunately I don't have an answer to your question. I repeatedly reinstalled VST over an existing installation, without errors. The compile/install/rebuild process is done by the IDE through the "PackageIntf" interface, OPM only "asks" the IDE to compile the packages and rebuild itself when needed, if the IDE says there is a duplicate path, probably is somewhere. This is the kind of error message I was afraid of when I decided to add VST internally to the package manager.

Quote
What to do? Where do I have access to the "unit path of the IDE" so that I could delete the old directory?
Deleting or renaming the old VST directory should solve the problem.

Quote
Another point - maybe this already has been suggested, excuse me in this case: When clicking "Install" in the OPM there is no way to control the destination folder. With the VTV issue I know that the installation goes into the profile folder. This is fine in most cases, but I could imagine there are users (like myself) who want to put all third-party components into a common folder which is shared between several Lazarus installations. So, I would appreciate if there were a selection box for the destination directory in the OPM Options dialog.
Consider it done. I will try to implement it today.

Title: Re: Online Package Manager
Post by: minesadorada on November 23, 2016, 09:14:11 am
A Suggestion:
In the Create Package dialog, a button 'Submit Package?' - meaning 'Submit Package for consideration for the public Online Repository' (wouldn't fit on a button!)

This would perhaps trigger a stock dialog, where the submitter is asked various questions (incl. contact info), and the info then gets zapped to the OPM maintainer (currently GetMem)
Title: Re: Online Package Manager
Post by: wp on November 23, 2016, 09:29:20 am
As for the VirtualTreeView installation, it is working now. The problem was that I still had installed other components which depend on VTV. After uninstalling them the IDE compiles again, and I finally could install the OPM version of VTV.
Title: Re: Online Package Manager
Post by: GetMem on November 23, 2016, 11:10:14 am
Quote
@minesadorada
In the Create Package dialog, a button 'Submit Package?' - meaning 'Submit Package for consideration for the public Online Repository' (wouldn't fit on a button!)
This would perhaps trigger a stock dialog, where the submitter is asked various questions (incl. contact info), and the info then gets zapped to the OPM maintainer (currently GetMem)
It's already in the to do list for a long time, but there is a problem. Where to submit the packages/JSON/contact Info?
1. To http://packages.lazarus-ide.org/
   We need a folder with public access, the risk of spam is high
2. Send by mail to me or somebody else
   In order to send a mail directly from OPM, the user must install the OpenSSL library.
3. Upload to a FTP server
  This is my favorite solution. But upload where? We need a public FTP somewhere.

I'm open to suggestions.

Quote
@wp
As for the VirtualTreeView installation, it is working now. The problem was that I still had installed other components which depend on VTV. After uninstalling them the IDE compiles again, and I finally could install the OPM version of VTV.
Ok. I'm glad it's working.
Title: Re: Online Package Manager
Post by: JuhaManninen on November 23, 2016, 12:11:58 pm
3. Upload to a FTP server
  This is my favorite solution. But upload where? We need a public FTP somewhere.
You could ask Marc and Vincent if they can start an FTP server in the same machine where http://packages.lazarus-ide.org/ is.
The disk space could be limited to prevent huge files uploaded either by accident or intentionally.
Title: Re: Online Package Manager
Post by: wp on November 23, 2016, 01:19:28 pm
@GetMem: I just uploaded a new zipped release version of CalLite (version 0.3). In addition, I also added a file callite-current-release.zip which is identical to the 0.3 version file. In future releases the most recent file will always be duplicated under this name. Therefore, if you link the OPM to callite-current-release.zip (https://sourceforge.net/projects/lazarus-ccr/files/CalLite/callite-current-release.zip/download), it will always catch the most recent file (except for the short interruption it takes me to replace the old file by the new one - btw: how does OPM behave if a zip file is not available?).

I am planning to keep this naming scheme with all the repositories that I am maintaining.
Title: Re: Online Package Manager
Post by: JuhaManninen on November 23, 2016, 01:40:36 pm
@GetMem: I just uploaded a new zipped release version of CalLite (version 0.3). In addition, I also added a file callite-current-release.zip which is identical to the 0.3 version file. In future releases the most recent file will always be duplicated under this name. Therefore, if you link the OPM to callite-current-release.zip (https://sourceforge.net/projects/lazarus-ccr/files/CalLite/callite-current-release.zip/download), it will always catch the most recent file
Sorry but it does not solve anything. Information about the new version is not passed to the package manager.
When you update to version 0.4, keeping the same callite-current-release.zip file name, the package manager's JSON file still shows version 0.3. It would have to download and extract the zip file to get its real version, but that is too much traffic.

Earlier I (and others, too) suggested an external JSON file for the version etc. info.
It should be supported to move admin work from GetMem to maintainers of individual packages.
Title: Re: Online Package Manager
Post by: wp on November 23, 2016, 02:16:10 pm
Thanks. I guess I should read all 276 posts...

So, I'll remove the callite-current-release.zip again, and wait for instructions how to setup the JSON file. The zip file for OPM is callite-0.3.zip then (https://sourceforge.net/projects/lazarus-ccr/files/CalLite/callite-0.3.zip/download).
Title: Re: Online Package Manager
Post by: minesadorada on November 23, 2016, 02:30:11 pm
@GetMem: I just uploaded a new zipped release version of CalLite (version 0.3). In addition, I also added a file callite-current-release.zip which is identical to the 0.3 version file. In future releases the most recent file will always be duplicated under this name. Therefore, if you link the OPM to callite-current-release.zip (https://sourceforge.net/projects/lazarus-ccr/files/CalLite/callite-current-release.zip/download (https://sourceforge.net/projects/lazarus-ccr/files/CalLite/callite-current-release.zip/download)), it will always catch the most recent file
Sorry but it does not solve anything. Information about the new version is not passed to the package manager.
When you update to version 0.4, keeping the same callite-current-release.zip file name, the package manager's JSON file still shows version 0.3. It would have to download and extract the zip file to get its real version, but that is too much traffic.

Earlier I (and others, too) suggested an external JSON file for the version etc. info.
It should be supported to move admin work from GetMem to maintainers of individual packages.

When I made the external cryptini.zip 'update file' I put 'cryptini.json' in the same server location. (it's also in the zip of course)
I assumed that was the way to do maintenance updates.  I'm obviously ahead of myself! :)

Hey - wouldn't it be nice if the json file could have a 'What's new in this version' entry?
 
Title: Re: Online Package Manager
Post by: lainz on November 23, 2016, 02:49:18 pm
Hi, I'm also waiting for GetMem response to upload the Json file.

Where I must download the package manager from now, it is still beign updated on github or I must switch for the lazarus components one? I want to create the json with the latest of course.

If I remember well I also don't response a question of GetMem, where I can upload the json file on GitHub, well this can be done in two places:
- Along with the sources, then you can get it thanks to the raw.githubusercontent website with no need of git, so is served like a text file.
- In a release, but this must be a special release that I must update every time I change the json. So delete the previous and upload it again.

I think I will use the first.

I don't know how you GetMem will test it, if you need a new release of some of the packages I mantain I can do one, since I have some changes in the repositories that can apply for a new release, at least for a minor one.
Title: Re: Online Package Manager
Post by: GetMem on November 23, 2016, 02:58:54 pm
Quote
@wp
So, I would appreciate if there were a selection box for the destination directory in the OPM Options dialog.
Implemented in r.53424

@Juha
Quote
You could ask Marc and Vincent if they can start an FTP server in the same machine where http://packages.lazarus-ide.org/ is.
The disk space could be limited to prevent huge files uploaded either by accident or intentionally.
Ok. I will send a mail to Marc.

@Juha
Quote
Earlier I (and others, too) suggested an external JSON file for the version etc. info.
It should be supported to move admin work from GetMem to maintainers of individual packages.
Yes, but the external json idea is not fully implemented + I always thought that the packages from the main repository will be manually added/updated for security reasons. The main repository will contain the most 25 popular packages(or whatever number you find appropriate). If not the whole central repository is pointless. Without a central repository all we need is a master json(make it a txt), that contains the links to the developer's JSON.

Again I'm open for suggestions.
Quote
@minesadorada
When I made the external cryptini.zip 'update file' I put 'cryptini.json' in the same server location. (it's also in the zip of course)
That JSON is concatenated to the main one from the central repository. It was designed for me, it's not the "external json" that @Juha and @lainz talks about.
Title: Re: Online Package Manager
Post by: lainz on November 23, 2016, 03:05:04 pm
Quote
Yes, but the external json idea is not fully implemented...

Ok that's the response I was looking. So from now uploading it is not neccessary.
Title: Re: Online Package Manager
Post by: JuhaManninen on November 23, 2016, 03:22:10 pm
Where I must download the package manager from now, it is still beign updated on github or I must switch for the lazarus components one? I want to create the json with the latest of course.
The master is in Lazarus sources now. It is best the download from there (together with the whole Lazarus trunk maybe).
I have updated the wiki page, too.
Title: Re: Online Package Manager
Post by: lainz on November 23, 2016, 04:06:35 pm
Where I must download the package manager from now, it is still beign updated on github or I must switch for the lazarus components one? I want to create the json with the latest of course.
The master is in Lazarus sources now. It is best the download from there (together with the whole Lazarus trunk maybe).
I have updated the wiki page, too.

I've downloaded it with Fpcupdeluxe with also trunk of fpc, but bgrabitmap doesn't compile, seems that they're working in changes of assembler and that code is not working in the package.

So I need to download FPC 3.0 + lazarus trunk. It will work? I need bgrabitmap for my projects and packages so is the only thing I can do.
Title: Re: Online Package Manager
Post by: GetMem on November 23, 2016, 04:18:55 pm
Quote
@lainz
I've downloaded it with Fpcupdeluxe with also trunk of fpc, but bgrabitmap doesn't compile, seems that they're working in changes of assembler and that code is not working in the package.So I need to download FPC 3.0 + lazarus trunk. It will work? I need bgrabitmap for my projects and packages so is the only thing I can do.
While testing the package manager, I installed bgrabitmap/bgracontrols/bgracontrolsfx several times today. It works fine. I also have FPC 3.0.0 + Lazarus trunk. So the answer is yes.
Title: Re: Online Package Manager
Post by: minesadorada on November 23, 2016, 05:05:15 pm
So, if an FTP server is set up, and there is a public area for new package uploads (which would be subject to approval) - why not a password-protected 'updates' area for developers who already have approved packages?

To update a package, just ftp the zip and json file to the protected 'updates' area.  No need for external zips with such a simple system.

In the meantime, OPM (Or a separate cron maintenance app) regularly and automatically checks the protected area (just the jsons, so not too heavy) and if it finds a newer version number for an existing package, it transfers the zip to the OPM server and deletes it from the FTP 'updates' area.

This would make maintenance of existing packages a breeze for both devs and the OPM manager.  A weakness is that it would rely on devs not producing a 'bad' update which would propagate to the main server, but it is probably OK because otherwise it means the maintainer has to approve all updates as well as new stuff, which seems unnecessary work.

The only manual maintenance of OPM would be vetting new packages.  Once approved, the dev gets r/w access to the protected 'updates' ftp directory and ongoing maintainence is then up to the dev.

A viable system?

Whatever system is adopted, I have a point update to CryptINI (v0.0.9) ready to test whatever is decided.

On a different note, is Lazarus svn trunk no longer synchronised? I get revision 41451 from the svn server (with no OPM)
Title: Re: Online Package Manager
Post by: lainz on November 23, 2016, 06:37:17 pm
You downloaded from sourceforge that is not updated anymore :)

Download from here
http://svn.freepascal.org/svn/lazarus/trunk
Title: Re: Online Package Manager
Post by: minesadorada on November 23, 2016, 06:41:02 pm
You downloaded from sourceforge that is not updated anymore :)

Download from here
http://svn.freepascal.org/svn/lazarus/trunk
Thanks.
Title: Re: Online Package Manager
Post by: minesadorada on November 23, 2016, 06:57:54 pm
An older post in this thread:

Quote
Download New version from above link
Went to install package and get error
opkman_installer.pas(166,40) Error: identifier idents no member "DoCompilePackage"

in line
Code: Pascal  [Select]
  1. function TPackageInstaller.CompilePackage(const AIDEPackage: TIDEPackage;
  2.   APackageFile: TPackageFile): Integer;
  3. begin
  4.   Result := -1;
  5.   {$if declared(lcl_version)}
  6.    {$if (lcl_major >= 1) and (lcl_minor >= 7)}
  7.      //DoCompilePackage function is only available with Laz 1.7 +
  8.      DoOnPackageInstallProgress(imCompilePackage, APackageFile);
  9.      Result := PackageEditingInterface.DoCompilePackage(AIDEPackage, [pcfCleanCompile, pcfDoNotSaveEditorFiles], False);  <--------- HERE
  10.    {$endif}
  11.   {$endif}
  12. end;            

This error is still there.  (Downloaded from trunk)

I have found that with the combination fpc 3.1.1 + Laz 1.7
Code: Pascal  [Select]
  1. {$if something >= somethingelse}
doesn't work. I don't know if it's a bug or intentional.

It only seems to accept
Code: Pascal  [Select]
  1. {$if something > somethingelse}
or
Code: Pascal  [Select]
  1. {$if something < somethingelse}

The fix is:
Code: Pascal  [Select]
  1. {$if (lcl_major > 0) and (lcl_minor > 6)}

Title: Re: Online Package Manager
Post by: minesadorada on November 23, 2016, 07:10:51 pm
I'm all over this thread today!

Yet another suggestion:

In the new OPM Options/Folders page, How about the defaults in the Tedit:
Local Repository: {$I LazarusDir}\components
Archive Directory: GetTempDir
Update Directory GetTempDir

Title: Re: Online Package Manager
Post by: GetMem on November 23, 2016, 07:22:29 pm
@minesadorada
About the fpc 3.1.1 bug: It's weird that ">=" is not supported. I applied your suggestion in r.53428 since logically it's the same as the old one. Thanks!

About the folder page: There is a default value for each folder, just press the "Restore Defaults" button, or delete the config file. You need to do this once, since I recently changed the config file structure.

About the update feature: The "external json" seems more easier to implement. I believe I found a solution, we can keep the central repository and update the packages from external json. I need 1-2 days to implement the feature.
Title: Re: Online Package Manager
Post by: wp on November 23, 2016, 07:41:11 pm
I do understand the Local Repository folder, but what are the Archive and Update directories for?
Title: Re: Online Package Manager
Post by: minesadorada on November 23, 2016, 07:53:59 pm
@minesadorada
About the fpc 3.1.1 bug: It's weird that ">=" is not supported. I applied your suggestion in r.53428 since logically it's the same as the old one. Thanks!

About the folder page: There is a default value for each folder, just press the "Restore Defaults" button, or delete the config file. You need to do this once, since I recently changed the config file structure.

About the update feature: The "external json" seems more easier to implement. I believe I found a solution, we can keep the central repository and update the packages from external json. I need 1-2 days to implement the feature.
Keep with it @Getmem!  I am relying on the team's brains and expertise :)

Please review all of this thread regularly for all suggestions and failures.
Title: Re: Online Package Manager
Post by: GetMem on November 23, 2016, 07:58:23 pm
Quote
@wp
I do understand the Local Repository folder, but what are the Archive and Update directories for?
The packages in the main repository are stored as a zip file(to minimize server load), the OPM first downloads the zip to the "Archive" directory, then unzip it to the "Local Repository" , from here the packages are installed into the IDE. The same is true for zip files downloaded from the package maintainer site, but instead of the "Archive" it's downloaded to the "Update" folder.
I choose this multiple folder solution to keep the "Local repository" clean.
Title: Re: Online Package Manager
Post by: lainz on November 23, 2016, 08:00:39 pm
Quote
About the update feature: The "external json" seems more easier to implement. I believe I found a solution, we can keep the central repository and update the packages from external json. I need 1-2 days to implement the feature.

Amazing. Thanks GetMem, is nice to hear that the ideas of the community are heard and implemented for the advance of this service.
Title: Re: Online Package Manager
Post by: GetMem on November 23, 2016, 11:12:55 pm
I did some initial testing...In order to get/display the latest version from the maintainer site, the OPM must connect to each server separately and read the external JSON. The average connection time(depends on the server) is 1.5-2 sec. The data read is almost instantaneous, since the file is small. However multiply 1.5-2 with 50 packages and you got the idea. Not a practical solution. Conclusions:
   1. I'm glad we did not drop the main repository, otherwise OPM speed would be painfully slow
   2. I'm forced to use a worker thread to read the external data from time to time(in the background), plus remembering the latest info somewhere for a quick display on startup

PS: To be honest I like the current solution better. The package maintainer keeps the latest version in a zip file(with a predefined name), on request the zip is downloaded/extracted and after a confirmation dialog, where the version is displayed, the package gets installed. The traffic is between the user and the external server.
This worked well until now as @minesadorada and @lainz proved with their packages. @wp came up with the same idea today.

Or even better along with the zip, the maintainer can save the actual version of the package in the first line of a text file(4.7.9.0 for example). When the users press the update button, the OPM reads then displays the version. After a confirmation dialog, the zip is downloaded or the update is canceled. What is wrong with this solution? We don't have to go through each server just to get a version info. The users are only interested in a few preferred package. I add once the package to the main repository, the maintainers update the txt file. Easy for everyone.

Title: Re: Online Package Manager
Post by: lainz on November 23, 2016, 11:25:56 pm
Quote
Or even better along with the zip, the maintainer can save the actual version of the package in the first line of a text file(4.7.9.0 for example). When the users press the update button, the OPM reads then displays the version. After a confirmation dialog, the zip is downloaded or the update is canceled. What is wrong with this solution? We don't have to go through each server just to get a version info. The users are only interested in a few preferred package. I add once the package to the main repository, the maintainers update the txt file. Easy for everyone.

Seems the best, so we must (at least in my case) only update a text file. The url to the text file will be in the generated json?
Title: Re: Online Package Manager
Post by: GetMem on November 23, 2016, 11:37:02 pm
Quote
Seems the best, so we must (at least in my case) only update a text file.
Exactly. This will be true for everyone, not just you.

Quote
The url to the text file will be in the generated json?
Yes it's already generated for the zip. The txt should be the same name, in your case "master".
https://github.com/bgrabitmap/bgrabitmap/archive/master.zip
https://github.com/bgrabitmap/bgrabitmap/archive/master.txt

Or perhaps I only need the a link to the text file, with two lines: line1 ---> version; line2---> link to zip. This way the zip and the txt can be in two different directories or servers for that matter.
Title: Re: Online Package Manager
Post by: lainz on November 23, 2016, 11:41:42 pm
Ok when you have it working tell us to upload it. Maybe you can add that to the wiki too.
Title: Re: Online Package Manager
Post by: minesadorada on November 24, 2016, 10:25:22 am
@GetMem: to help you test..
I have CryptINI V0.0.9 update on my external server (.zip + json)
How can OPM show that it is available?  Currently it shows 0.0.8 as 'Available'
Title: Re: Online Package Manager
Post by: GetMem on November 24, 2016, 10:33:27 am
@minesadorada
For now the OPM only shows the version available in the main repository. Currently we try to figure it out, what would be the best approach for updates located at the developers site(see my previous post #295).
Title: Re: Online Package Manager
Post by: josh on November 24, 2016, 11:02:47 am
Hi Getmem,

Just curious how the updater will look and how hard it will be navigate when 100's of packages get added?

Would a collapseable tree structure be needed; where the primary list is category; this may also make navigation easier and also possibly adding in a submitters package description as the displayed name.
ie
Graphics Libraries
   BGRABitmap ( Cross platform graphics and Image Manipulations Routines, Including GL)
Visual Controls
  BGRAControls (Requires BGRABitmap, Blah Blah Blah)
   Eye Candy (Track Bars, Buttons etc)
   ueControls (Requires BGRABITMAP, LED, visual knobs blah blah blah)

Networking Libraries
  Synapse (cross platform routines for email ftp etc.....)

Title: Re: Online Package Manager
Post by: minesadorada on November 24, 2016, 11:41:35 am
@minesadorada
For now the OPM only shows the version available in the main repository. Currently we try to figure it out, what would be the best approach for updates located at the developers site(see my previous post #295).
Any comments on mu post#285 as a viable system?  It would have picked up my update without you having to do anything.
Title: Re: Online Package Manager
Post by: minesadorada on November 24, 2016, 11:46:39 am
Hi Getmem,

Just curious how the updater will look and how hard it will be navigate when 100's of packages get added?

Would a collapseable tree structure be needed; where the primary list is category; this may also make navigation easier and also possibly adding in a submitters package description as the displayed name.
ie
Graphics Libraries
   BGRABitmap ( Cross platform graphics and Image Manipulations Routines, Including GL)
Visual Controls
  BGRAControls (Requires BGRABitmap, Blah Blah Blah)
   Eye Candy (Track Bars, Buttons etc)
   ueControls (Requires BGRABITMAP, LED, visual knobs blah blah blah)

Networking Libraries
  Synapse (cross platform routines for email ftp etc.....)

With an extra layer:

DesignTime and Runtime
  Graphics Libraries
     BGRABitmap ( Cross platform graphics and Image Manipulations Routines, Including GL)
    Visual Controls
    BGRAControls (Requires BGRABitmap, Blah Blah Blah)
     Eye Candy (Track Bars, Buttons etc)
     ueControls (Requires BGRABITMAP, LED, visual knobs blah blah blah)

  Networking Libraries
     IPControls   
Runtime Only
  Networking Libraries
    Synapse (cross platform routines for email ftp etc.....)
  Graphics
    BGRA runtime
  ...etc.
Title: Re: Online Package Manager
Post by: lainz on November 24, 2016, 11:52:28 am
@minesadorada
For now the OPM only shows the version available in the main repository. Currently we try to figure it out, what would be the best approach for updates located at the developers site(see my previous post #295).

About that, I just remember that a single version number will not work for packages with 2 or more lpk with different versions each one. For example bgrabitmap.
Title: Re: Online Package Manager
Post by: wp on November 24, 2016, 12:00:20 pm
With an extra layer:

DesignTime and Runtime
  Graphics Libraries
     BGRABitmap ( Cross platform graphics and Image Manipulations Routines, Including GL)
    Visual Controls
    BGRAControls (Requires BGRABitmap, Blah Blah Blah)
     Eye Candy (Track Bars, Buttons etc)
     ueControls (Requires BGRABITMAP, LED, visual knobs blah blah blah)

  Networking Libraries
     IPControls   
Runtime Only
  Networking Libraries
    Synapse (cross platform routines for email ftp etc.....)
  Graphics
    BGRA runtime
  ...etc.

No. There are libraries which consist of a standalone runtime-only package (FPSpreadsheet --> laz_fpspreadsheet.lpk) and a visual addon  package (laz_fpspreadsheet_visual.lpk). Your suggestion would force the user to look for FPSpreadsheet under both headers.

Sometimes libraries do not fit into a single category. Therefore it should be possible to have the same package in several categories.
Title: Re: Online Package Manager
Post by: GetMem on November 24, 2016, 12:01:59 pm
@josh
The tree is already contains too much info. Adding another node would make things worst. To locate a package you can use the filter.

@minesadorada
Yes I saw your post, it's a good idea with one major flaw: somebody with malicious intent can overwrite the whole repository in ten minutes.

@lainz
True! I believe we can overcome that.

@wp
With a small modification I can achieve that, I mean adding the same package in several categories. Added to the to do list.
Title: Re: Online Package Manager
Post by: JuhaManninen on November 24, 2016, 12:04:41 pm
For now the OPM only shows the version available in the main repository. Currently we try to figure it out, what would be the best approach for updates located at the developers site(see my previous post #295).
It will not work! Let's dump it. We now try to mix many ideas and create a half-baked hybrid which is not good for anyone.
A master package must be either in OPM's own repository or in an external repository.
Now we are planning a system where the master package is maybe in OPM repository but maybe not. It may be elsewhere but OPM does not know it before reading some external files. Uhhh ... no good!

One fact remains: the OPM admin work must be minimized to keep it well maintained also in far future.
My original idea, before other ideas got mixed, was having 3 types of sources for the packages:
They all would be configured in the JSON config file of OPM.
(Number 4. could be a Delphinus style thing but it is not important now.)

Now I start to think we should concentrate in number 1. which is also GetMem's original idea.
Just updating the packages in OPM's repo should be made as easy as possible.
We need the FTP server.
We also need an admin tool which allows copying the packages and updating the configuration with few mouse clicks.
Title: Re: Online Package Manager
Post by: josh on November 24, 2016, 12:08:49 pm
Adding on
A hint box appears over the component that displays More information or possible the contents of the readme.txt for the component; this would be handy if the new version has additions or a particular bug fix that is required as i do not always keep components up to date if changes to component require major rework of code.

Maybe the addition of an OPM.TXT file would be good use for this; if the file exists in the package the file layout could be.
line 1 : Name and basic description of package ie  BGRABitmap ( Cross platform graphics and Image Manipulations Routines, Including GL)
The subsequent lines then contain the hint text to display.

This way the package owner can update the contents of his components opm.txt  file when needed; with little or no interaction with OPM Maintainer.
 


Title: Re: Online Package Manager
Post by: minesadorada on November 24, 2016, 12:12:16 pm
@minesadorada
Yes I saw your post, it's a good idea with one major flaw: somebody with malicious intent can overwrite the whole repository in ten minutes.
Surely easily solved with ftp permissions, whereby each developer only has write access to their own area?  Only needs to be set up once, then it's self-maintaining.
Title: Re: Online Package Manager
Post by: wp on November 24, 2016, 12:16:33 pm
Quote
@wp
I do understand the Local Repository folder, but what are the Archive and Update directories for?
The packages in the main repository are stored as a zip file(to minimize server load), the OPM first downloads the zip to the "Archive" directory, then unzip it to the "Local Repository" , from here the packages are installed into the IDE. The same is true for zip files downloaded from the package maintainer site, but instead of the "Archive" it's downloaded to the "Update" folder.
I choose this multiple folder solution to keep the "Local repository" clean.

Sorry to overflow you with suggestions. But this one came to my mind when reading your response: What about a checkbox "Delete downloaded zip files after installation/update"? I think the downloaded zip files are not needed any more for anything else, aren't they?
Title: Re: Online Package Manager
Post by: lainz on November 24, 2016, 12:19:14 pm
Quote
OPM repository.
External repository with an external URL.
Development version got directly from a revision control system (SVN, Git, automatic master.zip, whatever).

Well, the first works, so we only need:
- the automatic generation of the main json
- a submission website for developers

The second I don't know.

The third already works for some packages. Only svn/git is missing. Edit: I have for example Fpcupdeluxe that finds and uses svn from my path, else it automatically downloads it to a folder, that code can be used.
Title: Re: Online Package Manager
Post by: JuhaManninen on November 24, 2016, 12:33:07 pm
Well, the first works, so we only need:
- the automatic generation of the main json
- a submission website for developers
Why a website? It would require server side SW and is completely useless for this task.
Anybody can install an FTP client or the OPM client could send the generated package to the server with a button click.

Quote
The second I don't know.
It may not be needed if number one works well.

Quote
The third already works for some packages. Only svn/git is missing.
I think you misunderstood my idea. Now the update may work but it is kind of a hack and a hybrid. A master package is in OPM server but then it is not used, the real package is updated from somewhere else.
My idea is that a package could be defined in OPM configuration to be only a development version copied directly from revision control system.
It is useful for packages which are not maintained actively (yes, there are many).
The OPM GUI must indicate they are development versions.
Note: development versions don't have version numbers thus they are not an issue.
Title: Re: Online Package Manager
Post by: lainz on November 24, 2016, 12:39:44 pm
Quote
...or the OPM client could send the generated package to the server with a button click.

I like this part.

Quote
My idea is that a package could be defined in OPM configuration to be only a development version copied directly from revision control system.

That is possible using svn, the same I see in Fpcupdeluxe, it makes a fpc and lazarus folder and keeps them updated. Also works with GitHub, since GitHub supports getting the sources with svn tools too.
Title: Re: Online Package Manager
Post by: JuhaManninen on November 24, 2016, 12:47:36 pm
That is possible using svn, the same I see in Fpcupdeluxe, it makes a fpc and lazarus folder and keeps them updated.
Yes.

Quote
Also works with GitHub, since GitHub supports getting the sources with svn tools too.
With GitHub we don't need any extra tools because it generates master.zip files automatically. That's why you wanted to move everything to GitHub, remember? :)
Title: Re: Online Package Manager
Post by: lainz on November 24, 2016, 12:51:51 pm
Quote
With GitHub we don't need any extra tools because it generates master.zip files automatically. That's why you wanted to move everything to GitHub, remember? :)

Yes, but if you say that it will get a repository, I think is best instead of getting the master.zip, is less time downloading in each update.

Edit: for those with bad internet connections like me will be a lot of help.
Title: Re: Online Package Manager
Post by: JuhaManninen on November 24, 2016, 01:02:20 pm
Yes, but if you say that it will get a repository, I think is best instead of getting the master.zip, is less time downloading in each update.
Ok yes, true.
Actually SourceForge supports Git and GitHub supports SVN. The differences are exaggerated.
Title: Re: Online Package Manager
Post by: lainz on November 24, 2016, 01:07:42 pm
Nice, so just GetMem need to pick one and will work in both places.
Title: Re: Online Package Manager
Post by: GetMem on November 24, 2016, 09:12:26 pm
Quote
@wp
Sorry to overflow you with suggestions. But this one came to my mind when reading your response: What about a checkbox "Delete downloaded zip files after installation/update"? I think the downloaded zip files are not needed any more for anything else, aren't they?
A package lifecycle has five stages: Repository-->Downloaded-->Extracted(Local repository)-->Installed(Up To Date) the fifth one is the Error stage, assigned when one of the steps fails. In the screenshot you can see various packages in different stages.
Let's say a package is downloaded, extracted then the installation fails for some reason(as happened to you with VST). Next time when you press the "Install" button, the OPM will skip the first two steps(download and extract) and will resume the package installation, this way saving time and band with. The archive folder has its purpose, however I would like to make this optional:
Very soon I will add two checkboxes to the option dialog:
   - Force download and extract(if checked will overwrite the above mentioned behaviour, and the package is always re-downloaded/extracted)
   - Delete downloaded zip files after installation/update(only enabled when the previous checkbox is checked). It make sense to delete the old archive/extracted package, because it will be downloaded/extracted again anyways)

PS:  I hope it make sense what I wrote.  :-\
PS1: "Delete downloaded zip files after installation/update" option is always enabled, the user decides the appropriate behaviour
Title: Re: Online Package Manager
Post by: josh on November 26, 2016, 02:39:35 am
@getmem
Any chance you could add the download link into your signature; it would make locating this package much easier.
Title: Re: Online Package Manager
Post by: JuhaManninen on November 26, 2016, 10:56:32 am
Any chance you could add the download link into your signature; it would make locating this package much easier.
A download link for the online package manager. Is it necessary? It is now part of Lazarus sources and everybody knows how to get them. (?)
To get it separately:
Code: [Select]
$ svn co http://svn.freepascal.org/svn/lazarus/trunk/components/onlinepackagemanager OPM
The old repository should be clearly marked as outdated, for example by adding a source line "This is outdated source" that prevents compilation.
Other option is to delete the GitHub version as it only confuses people. Who would need it?
Title: Re: Online Package Manager
Post by: GetMem on November 26, 2016, 11:07:21 am
@josh
The package manager is in Lazarus trunk now.

@Juha
I did synchronized the github version until now, but I will drop the support.

PS: Marked as outdated on github
Title: Re: Online Package Manager
Post by: Xor-el on November 26, 2016, 12:22:36 pm
@josh
The package manager is in Lazarus trunk now.

@Juha
I did synchronized the github version until now, but I will drop the support.

PS: Marked as outdated on github

please don't drop the github support.
not everyone uses Lazarus trunk.
Title: Re: Online Package Manager
Post by: JuhaManninen on November 26, 2016, 12:43:33 pm
please don't drop the github support.
not everyone uses Lazarus trunk.
You don't need to checkout the whole Lazarus trunk as I tried to show earlier.
Having the sources in many places adds extra administrative work and also confusion. It is especially useless here because Lazarus trunk already has many GitHub mirrors.
Uhhh... :(
Title: Re: Online Package Manager
Post by: GetMem on November 26, 2016, 02:35:19 pm
@Xor-el
Ok. I will synchronize the github repository, basically it will be a clone of ($LazarusDir)/components/onlinepackagemanager. 
I added to the readme.txt that the main development has moved to: http://svn.freepascal.org/svn/lazarus/trunk/components/onlinepackagemanager
See: https://github.com/getmem/LazOpkMan
Title: Re: Online Package Manager
Post by: Xor-el on November 26, 2016, 03:15:49 pm
@Xor-el
Ok. I will synchronize the github repository, basically it will be a clone of ($LazarusDir)/components/onlinepackagemanager. 
I added to the readme.txt that the main development has moved to: http://svn.freepascal.org/svn/lazarus/trunk/components/onlinepackagemanager
See: https://github.com/getmem/LazOpkMan

thanks  :D
Title: Re: Online Package Manager
Post by: JanRoza on November 28, 2016, 10:57:36 am
Just downloaded the latest trunk version of Online Package Manager but when I try to install the package in Lazarus 1.7 (FPC 3.1.1) I get the following error:

Quote
opkman_installer.pas(166,40) Error: identifier idents no member "DoCompilePackage"

Any ideas how to solve this? Is there a minimum trunk version of Lazarus I need?
Thanks!
Title: Re: Online Package Manager
Post by: GetMem on November 28, 2016, 11:03:41 am
Hi JanRoza,

You need Lazarus trunk r53263+ or change the following line(opkman_installer.pas(163) from:
Quote
{$if (lcl_major > 0) and (lcl_minor > 6)}
to
Quote
{$if (lcl_major > 0) and (lcl_minor > 7)}
Title: Re: Online Package Manager
Post by: JuhaManninen on November 28, 2016, 11:34:08 am
Just downloaded the latest trunk version of Online Package Manager but when I try to install the package in Lazarus 1.7 (FPC 3.1.1) I get the following error:

Quote
opkman_installer.pas(166,40) Error: identifier idents no member "DoCompilePackage"

Any ideas how to solve this? Is there a minimum trunk version of Lazarus I need?
When using trunk the minimum version is the latest trunk.
According to your error message you are doing something stupid now. I think you download OPM from the GitHub mirror and try to install it in an old Lazarus trunk. Am I right?
Hey, OPM is already part of Lazarus trunk!
Still synchronizing the old repository is counter-productive. It only creates confusion (as happened now).
I still think the old repo should be either removed or deprecated.
There are already at least 2 mirrors for the code in GItHub. Explicitly synchronizing a 3rd one makes no sense!
I honestly don't understand why Xor-el suggested it and why GetMem agreed.
Title: Re: Online Package Manager
Post by: minesadorada on November 28, 2016, 11:44:24 am
Hi JanRoza,

You need Lazarus trunk r53263+ or change the following line(opkman_installer.pas(163) from:
Quote
{$if (lcl_major > 0) and (lcl_minor > 6)}
to
Quote
{$if (lcl_major > 0) and (lcl_minor > 7)}
So that only becomes TRUE when Lazarus is at V1.8.  How is that testable?
Title: Re: Online Package Manager
Post by: JuhaManninen on November 28, 2016, 11:59:31 am
So that only becomes TRUE when Lazarus is at V1.8.  How is that testable?
See, more confusion...
GetMem, please remove the GitHub mirror or add a line to its sources that prevent compilation.
OPM is now part of Lazarus sources. Period!
It should be easy enough to understand for everybody.
Title: Re: Online Package Manager
Post by: minesadorada on November 28, 2016, 12:30:08 pm
In which case the revised {$IFDEF} is correct, since trunk will presumably be eventually released as V1.8.
Title: Re: Online Package Manager
Post by: JanRoza on November 28, 2016, 12:52:18 pm
@JuhaManninen: No need to immediately call met stupid.  :-[ 
I indeed had an older trunk and looking at the source of the offending opk module it looked as if lazarus 1.7 was sufficient.

Just to stop people calling me stupid  :D  I made a second Lazarus installation with the latest trunk and in this installation everything seems to be working perfectly.
Title: Re: Online Package Manager
Post by: JuhaManninen on November 28, 2016, 01:31:45 pm
@JuhaManninen: No need to immediately call met stupid.  :-[ 
I indeed had an older trunk and looking at the source of the offending opk module it looked as if lazarus 1.7 was sufficient.
Actually I wrote: "you are doing something stupid". Clearly you were misguided by the confusing OPM mirror situation. Even clever people can be misguided by it. :)

Quote
Just to stop people calling me stupid  :D  I made a second Lazarus installation with the latest trunk and in this installation everything seems to be working perfectly.
Why a second Lazarus installation? Why not just "svn up"?
Title: Re: Online Package Manager
Post by: JanRoza on November 28, 2016, 01:41:02 pm
No offence taking, I was just teasing a bit.
I'm always a bit cautious, when I have a working situation I leave that intact and create a new folder with a secondary installation (and configuration).
Once I'm satisfied i delete the old installation.
Maybe not the simplest way but I never loose my working environment this way in case I really do something stupid.  ;)
 
Title: Re: Online Package Manager
Post by: Xor-el on November 28, 2016, 03:02:41 pm
Just downloaded the latest trunk version of Online Package Manager but when I try to install the package in Lazarus 1.7 (FPC 3.1.1) I get the following error:

Quote
opkman_installer.pas(166,40) Error: identifier idents no member "DoCompilePackage"

Any ideas how to solve this? Is there a minimum trunk version of Lazarus I need?
When using trunk the minimum version is the latest trunk.
According to your error message you are doing something stupid now. I think you download OPM from the GitHub mirror and try to install it in an old Lazarus trunk. Am I right?
Hey, OPM is already part of Lazarus trunk!
Still synchronizing the old repository is counter-productive. It only creates confusion (as happened now).
I still think the old repo should be either removed or deprecated.
There are already at least 2 mirrors for the code in GItHub. Explicitly synchronizing a 3rd one makes no sense!
I honestly don't understand why Xor-el suggested it and why GetMem agreed.

I understand your worries, but not everybody wants to download the whole repo just to get a few files.
Title: Re: Online Package Manager
Post by: JuhaManninen on November 28, 2016, 04:19:41 pm
I understand your worries, but not everybody wants to download the whole repo just to get a few files.
This will get you only the OPM sources:
Code: [Select]
$ svn co http://svn.freepascal.org/svn/lazarus/trunk/components/onlinepackagemanager OPM
Actually I already wrote it earlier. Later update it with:
Code: [Select]
$ svn up
just like with any repository.
Title: Re: Online Package Manager
Post by: GetMem on November 28, 2016, 10:22:07 pm
1. Ok. No more github
2. I extended the category list. You can assign multiple categories to a package(attachment 1)
3. Description and license can be viewed in a separate form(attachment 2)
Title: Re: Online Package Manager
Post by: wp on November 28, 2016, 10:44:39 pm
Today I uploaded a new release (=zip) version of tvplanit: https://sourceforge.net/projects/lazarus-ccr/files/tvplanit/tvplanit-1.06.zip/download.
Title: Re: Online Package Manager
Post by: GetMem on November 29, 2016, 08:09:16 am
@wp
I updated callite and tvplanit.
Title: Re: Online Package Manager
Post by: minesadorada on November 29, 2016, 02:05:53 pm
Updated CryptINI to V0.1.0.  Demo (included) updated to V0.1.2
https://sourceforge.net/projects/lazautoupdate/files/otherpackages/cryptini.zip (https://sourceforge.net/projects/lazautoupdate/files/otherpackages/cryptini.zip)

BTW, when you get to do SVN links in OPM:
svn checkout svn://svn.code.sf.net/p/cryptini/code/ cryptini-code

or: https://sourceforge.net/p/cryptini/code/HEAD/tree/trunk/latest_stable
Title: Re: Online Package Manager
Post by: GetMem on November 29, 2016, 05:19:36 pm
Quote
Updated CryptINI to V0.1.0.  Demo (included) updated to V0.1.2
https://sourceforge.net/projects/lazautoupdate/files/otherpackages/cryptini.zip
Done. Please test it.

Quote
BTW, when you get to do SVN links in OPM:
svn checkout svn://svn.code.sf.net/p/cryptini/code/ cryptini-code
or: https://sourceforge.net/p/cryptini/code/HEAD/tree/trunk/latest_stable
I don't know the exact date. I still have to implement:
  - update from external zip(this will be ready soon)
  - update from SVN
  - voting system
Patches are always welcome!
Title: Re: Online Package Manager
Post by: JuhaManninen on November 29, 2016, 07:19:36 pm
  - update from external zip(this will be ready soon)
I think the whole idea is fishy. Which one is the master file, OPM repo or external? Very confusing. Please see my earlier writing about the same thing.
There should be one place for the master file with no ambiguity.

Yes, I also wrote about an external JSON and a package. Then my idea was a totally external package with no copy in the repo.


Quote
  - voting system
Patches are always welcome!
The server side requires more than patches. It must be deployed to a server machine. I think you should ask about it from Marc and Vincent in advance.
Anyway, nice to see you have energy also for that part.
Title: Re: Online Package Manager
Post by: minesadorada on November 29, 2016, 07:24:01 pm
Done. Please test it.
Looking good.  Version shows as 0.1.0.  Thanks.
Title: Re: Online Package Manager
Post by: minesadorada on November 29, 2016, 07:29:34 pm
I think the whole idea is fishy. Which one is the master file, local or external? Very confusing. Please see my earlier writing about the same thing.
There should be one place for the master file with no ambiguity.
I agree with you about the system 'as is' but let's wait and see what GetMem has in mind for package maintainers.  There are quite a few options available.

SVN could only work if package authors maintain a 'stable branch' for instance.  There needs to be a protocol established for packages offered by OPM both for new authors and maintainers that avoids versioning issues.

Once Laz 1.8 is released, a packaging versioning mistake in OPM could propagate to many many users simultaneously.

Re: External json - why can't an external json be periodically checked then downloaded and  merged with the master copy (along with the package) if its version is better?  This could be (semi)-automatic.
Title: Re: Online Package Manager
Post by: GetMem on November 29, 2016, 08:02:12 pm
@Juha
Fishy in what way? I saw your earlier comment but perhaps we misunderstood each other. As you can see there is already a new "Update" column(screenshot) in the trunk. This column will display the latest version of the package, loaded from an external JSON. To trigger an update, the package maintainer can:
  1. Modify the version number to be bigger then the one in the central repository
  2. Set a boolean flag to true(ForceUpdate for example)
The external json, will contain a link to a zip(also provided by the package maintainer). The zip can be anything as far as I'm concern(the trunk version of the package, the master version, the next release 4.2.1.0-->4.2.1.1, whatever). My job is to add the initial package to the central repository + the link to the external json. After that it's up to the user and package maintainer. Still consider it fishy? If yes please explain why.

PS: The OPM won't update automatically, just displays a message(balloon hint, fleshing line, etc...). The user decides the appropriate action.
Title: Re: Online Package Manager
Post by: minesadorada on November 29, 2016, 08:32:40 pm
Looks like a workable system.

It would be nice for package maintainers if there were a simple 'external json editor' app.  IMHO it would make the process of deploying an external update zip + json a lot easier and safer.  It doesn't need to be fancy - just a bunch of TLabel + TEdits and 'create', 'load' and  'save as' buttons.  A fancier version would check the given URLs were valid before saving.

I know the 'create package' has similar functionality for new packages, but it's not really suitable for maintenance updates (as I have found)
Title: Re: Online Package Manager
Post by: minesadorada on November 29, 2016, 08:53:07 pm
While we are on the subject of external zips, how about looking at security?

Let's say as a user, I download and update a component from OPM in my Lazarus 1.8, but...
..it turns out that the external update zip wasn't hosted on a secure version control server (GitHub, SourceForge etc) and a malaicious person has replaced the update zip with a malware version (without the maintainer's knowledge)  Components once installed, have a lot of power.  An 'Execute' method (or any method) could do some bad stuff, and quite invisibly too.  Few ordinary component users (like me) check all the source code before using an updated component, particularly if it appears 'built-in' to Lazarus (as OPM's packages will appear)

This is why I suggested a protected area on a known secure server for update zips.  Upload access can be as secure as needed (like commit access to the ccr)  I would favour Sourceforge because I'm used to it, it uses SVN and it does automatic virus checks on all file uploads.  Even better would be an ftp server with multiple folders tied to individual logins to avoid 'cross-contamination' between component commits/uploads (as sourceforge ccr allows)  With only a few dozen components, it doesn't take long to set up (for example) ftp://cryptini.secureserver.com (ftp://cryptini.secureserver.com) to point to the /components/cryptini folder with a unique login to upload - and it's mostly a one-off process.   These would then be the unique 'master' versions @JuhaMannien was talking about.  OPM/GetMem would have access to the whole /components tree (OPM=read-only, GetMem=read + write) via separate logins.  Having the read-only login /URL in OPM sourcecode would not be a security risk.
Title: Re: Online Package Manager
Post by: GetMem on November 29, 2016, 10:04:23 pm
@minesadorada

Quote
It would be nice for package maintainers if there were a simple 'external json editor' app.  IMHO it would make the process of deploying an external update zip + json a lot easier and safer.  It doesn't need to be fancy - just a bunch of TLabel + TEdits and 'create', 'load' and  'save as' buttons.  A fancier version would check the given URLs were valid before saving.
The JSON will be simple, it can be edited manually. Something like this:
Code: XML  [Select]
  1. {
  2.  "Package" : {
  3.     "Name" : "whatever.zip"
  4.     "ForceUpdate" : False,
  5.     "DownloadURL" : "https:\\....",    
  6.   },
  7.   "PackageFiles" : [
  8.     {
  9.       "Name" : "aaaa.lpk",
  10.       "Version" : "1.2.5.0"
  11.     },
  12.     {
  13.       "Name" : "bbbb.lpk",
  14.       "Version" : "0.0.5.0"
  15.     }
  16.   ]
  17. }
  18.  
I'm not against a GUI, but somebody has to do it.

Quote
I know the 'create package' has similar functionality for new packages, but it's not really suitable for maintenance updates (as I have found)
Well I'm using it and apparently works well since the whole central repository was built with it + one more tool. Still room for improvements, a lot of it. I agree!

Quote
Let's say as a user, I download and update a component from OPM in my Lazarus 1.8, but...
..it turns out that the external update zip wasn't hosted on a secure version control server (GitHub, SourceForge etc) and a malaicious person has replaced the update zip with a malware version (without the maintainer's knowledge)  Components once installed, have a lot of power.  An 'Execute' method (or any method) could do some bad stuff, and quite invisibly too.  Few ordinary component users (like me) check all the source code before using an updated component, particularly if it appears 'built-in' to Lazarus (as OPM's packages will appear)
This is why I wanted to avoid external updates, since it's allowed, there is no such things as security, especially under windows and I can prove it. You can convert an exe to a txt file(it will pass any virus detection),  when needed the malicious package converts it back to byte array then injects it directly to a process memory, without saving the exe file to disk. The last step(memory injection) can be detected by some of the AV's out there, but most of the time will pass. Anyway we don't have to be paranoid...
Title: Re: Online Package Manager
Post by: JuhaManninen on November 29, 2016, 10:22:51 pm
My job is to add the initial package to the central repository + the link to the external json. After that it's up to the user and package maintainer. Still consider it fishy? If yes please explain why.
The fishy part is that you add an initial package to central repo but it will not be used after the package author decides to update for the first time.
Then comes a question: why to add the initial version at all? Why not just provide an URL for the external place? Then it would be a purely external package source, not a half-baked mixture of initial OPM repo package + the real package from external source.
Many ideas were thrown here by many people, including myself, and apparently they got mixed together badly.

My idea still is:
If you want to support external packages, add support for an external URL without an initial OPM repo package. Then the OPM repo would contain only information about a package and its source but not the package itself.

Another option is to make updating the OPM repo packages so easy that loading from an external URL is not needed.
Then the whole update feature could be dumped.

minesadorada writes about security which is also an important issue.
However the idea of FTP user accounts for every package maintainer has technical difficulties. It requires a server admin (Marc, Vincent or anybody) manage those accounts. We want this to be easy for any new package author, a new step of bureaucracy for every new author is not good. I doubt the admins would accept the whole process idea. So, the FTP must be writable without a password or with a common password which is given to anybody who asks.
Title: Re: Online Package Manager
Post by: GetMem on November 29, 2016, 10:37:41 pm
Quote
The fishy part is that you add an initial package to central repo but it will not be used after the package author decides to update for the first time.
Then comes a question: why to add the initial version at all? Why not just provide an URL for the external place? Then it would be a purely external package source, not a half-baked mixture of initial OPM repo package + the real package from external source.
Because at least half of the packages in the main repository don't have a maintainer, there is no external URL. Another reason: all the packages inside the main repository are safe to use, the update feature is optional.
The well known package(synapse) is maintained, but the author rarely visits this forum(if ever) should I approach him to create a JSON? Or if I create the JSON myself, I have to check all the time if the link to the zip has changed. If I start to think about it OPM without a repo is the half-baked solution.

Quote
Another option is to make updating the repo packages so easy that loading from an external URL is not needed.Then the whole update feature could be dumped.
How? You have to provide credentials for that. Without username/password someone with malicious intent will overwrite the whole repo  in five minutes. A login system can be done, but it's not a one man job.

Quote
So, the FTP must be writable without a password or with a common password which is given to anybody who asks.
Exactly, then bad things start to happen immediately.
Title: Re: Online Package Manager
Post by: minesadorada on November 29, 2016, 10:40:18 pm
minesadorada writes about security which is also an important issue.
However the idea of FTP user accounts for every package maintainer has technical difficulties. It requires a server admin (Marc, Vincent or anybody) manage those accounts. We want this to be easy for any new package author, a new step of bureaucracy for every new author is not good. I doubt the admins would accept the whole process idea. So, the FTP must be writable without a password or with a common password which is given to anybody who asks.

Well second-best is still better then nothing.  How about a Sourceforge OPM Components repository?  After all, the original source for GetMem was the SourceForge CCR repository.

For example; I just installed Richmemo from OPM, and the update URL is https://havefunsoft.com/share/richmemo.zip (https://havefunsoft.com/share/richmemo.zip).  I have no idea if 'havefunsoft.com' is OK or not.

I think you overcook the difficulty.  It's not as though you're going to get new packages every week - or even every 6 months probably.  I agree it's a pain to set up an FTP server in the way I suggest but once done; server maintenance would be infrequent and trivial IMO, and there would be no admin for existing package maintenance, which is a big bonus.  New package authors could use a public access staging area with a common login (so no delay) - once trusted (vetted, effectively) they would get their ftp area/login for maintenance/updates.

And it does solve most of the security worries - which are important.
Title: Re: Online Package Manager
Post by: minesadorada on November 29, 2016, 10:52:00 pm
@minesadorada

Quote
It would be nice for package maintainers if there were a simple 'external json editor' app.  IMHO it would make the process of deploying an external update zip + json a lot easier and safer.  It doesn't need to be fancy - just a bunch of TLabel + TEdits and 'create', 'load' and  'save as' buttons.  A fancier version would check the given URLs were valid before saving.
The JSON will be simple, it can be edited manually. Something like this:
Code: XML  [Select]
  1. {
  2.  "Package" : {
  3.     "Name" : "whatever.zip"
  4.     "ForceUpdate" : False,
  5.     "DownloadURL" : "https:\\....",    
  6.   },
  7.   "PackageFiles" : [
  8.     {
  9.       "Name" : "aaaa.lpk",
  10.       "Version" : "1.2.5.0"
  11.     },
  12.     {
  13.       "Name" : "bbbb.lpk",
  14.       "Version" : "0.0.5.0"
  15.     }
  16.   ]
  17. }
  18.  
I'm not against a GUI, but somebody has to do it.
Once you have the format settled I'm willing to give it a go.  Perhaps I'll learn about JSON on the way :)   I guess http://wiki.lazarus.freepascal.org/fcl-json is the page to start with.

BTW, what about scaleability?  Is it practical for OPM to offer 200 packages for example?  If not - what would be a practical maximum?
Title: Re: Online Package Manager
Post by: JuhaManninen on November 29, 2016, 11:20:02 pm
How? You have to provide credentials for that. Without username/password someone with malicious intent will overwrite the whole repo  in five minutes. A login system can be done, but it's not a one man job.
I had the FTP server in mind + a manual check of new packages aided by some admin tools maybe. Yes I know this solution has issues.
It is better I leave this to you people to decide. Maybe the mixture of OPM repo + external package actually works if the repo version is also updated sometimes.
I am not very experienced with server SW configs. You should ask from the server admin about FTP, if it is still needed.

On a positive side: the basic functionality of OPM is solid. The manually added packages are safe and good. It is good to build on that.
Title: Re: Online Package Manager
Post by: lainz on November 30, 2016, 12:05:39 am
Quote
It would be nice for package maintainers if there were a simple 'external json editor' app.  IMHO it would make the process of deploying an external update zip + json a lot easier and safer.  It doesn't need to be fancy - just a bunch of TLabel + TEdits and 'create', 'load' and  'save as' buttons.  A fancier version would check the given URLs were valid before saving.
The JSON will be simple, it can be edited manually. Something like this:
Code: XML  [Select]
  1. {
  2.  "Package" : {
  3.     "Name" : "whatever.zip"
  4.     "ForceUpdate" : False,
  5.     "DownloadURL" : "https:\\....",    
  6.   },
  7.   "PackageFiles" : [
  8.     {
  9.       "Name" : "aaaa.lpk",
  10.       "Version" : "1.2.5.0"
  11.     },
  12.     {
  13.       "Name" : "bbbb.lpk",
  14.       "Version" : "0.0.5.0"
  15.     }
  16.   ]
  17. }
  18.  
I'm not against a GUI, but somebody has to do it.

Here you have the JSON editor for the external json, feel free to update it as it needs more or less fields.

This is the generated json:

Code: Pascal  [Select]
  1. { "Package" : { "DownloadURL" : "google.com", "ForceUpdate" : true, "Name" : "BGRAControls" }, "PackageFiles" : [{ "Name" : "bgra.lpk", "Version" : "1.0.0.0 3" }, { "Name" : "gl.lpk", "Version" : "2.0.0.0 3" }] }
  2.  

It does not output a pretty print version, but it works.
Title: Re: Online Package Manager
Post by: minesadorada on November 30, 2016, 01:20:18 am

It does not output a pretty print version, but it works.
Thanks @lainz - nice neat code!
Useable as is, but I'll mess around with the GUI if that's OK with you.
Title: Re: Online Package Manager
Post by: lainz on November 30, 2016, 01:43:40 am

It does not output a pretty print version, but it works.
Thanks @lainz - nice neat code!
Useable as is, but I'll mess around with the GUI if that's OK with you.

Thanks. Play with it, but please keep it working for High DPI screens when you change it! (http://wiki.lazarus.freepascal.org/High_DPI)
Title: Re: Online Package Manager
Post by: minesadorada on November 30, 2016, 02:34:53 am
Thanks. Play with it, but please keep it working for High DPI screens when you change it! (http://wiki.lazarus.freepascal.org/High_DPI (http://wiki.lazarus.freepascal.org/High_DPI))
No worries - just tweaking captions and adding data checks to make it more idiot-proof for folk like me :)

@GetMem - Is the DownloadURL  fully formed? i.e. does the DownloadURL field contain the full zipfile name?
Title: Re: Online Package Manager
Post by: lainz on November 30, 2016, 02:58:57 am
Also change the title to something like OPM External JSON Editor, the name I set is really bad.
Title: Re: Online Package Manager
Post by: minesadorada on November 30, 2016, 03:25:49 am
Also change the title to something like OPM External JSON Editor, the name I set is really bad.
I've just been tidying up the window-dressing, project options and other stuff.  I've attached what I have so far, but I'll have another go tomorrow (it's late here)
Title: Re: Online Package Manager
Post by: lainz on November 30, 2016, 04:52:39 am
You're from Spain? From your mail address I guess..

Here in south america is late too.

Attached a new version:
- corrected executable file name: now it generates only a single .po file for all platforms
- corrected memory leak
- corrected LazFileUtils position to use their functions instead of FileUtils ones
- translated into spanish
Title: Re: Online Package Manager
Post by: GetMem on November 30, 2016, 05:40:29 am
@minesadorada
Quote
BTW, what about scaleability?  Is it practical for OPM to offer 200 packages for example?  If not - what would be a practical maximum?
I tested with 500+ packages(reused the same ones, multiple times). Works well.

Quote
Is the DownloadURL  fully formed? i.e. does the DownloadURL field contain the full zipfile name?
Yes it should contain the full zipfile name.

@lainz
Quote
It does not output a pretty print version, but it works.
Code: Pascal  [Select]
  1. ...
  2. Streamer.Options := Streamer.Options + [jsoUseFormatString];
  3. ...
The JSON structure will change, but your demo is good and helps a lot. Thanks!
Title: Re: Online Package Manager
Post by: minesadorada on November 30, 2016, 01:18:47 pm
You're from Spain? From your mail address I guess..

Here in south america is late too.

Attached a new version:
- corrected executable file name: now it generates only a single .po file for all platforms
- corrected memory leak
- corrected LazFileUtils position to use their functions instead of FileUtils ones
- translated into spanish
Thanks.  The \locale folder wasn't in the zip - could you attach it so we can keep synchronised please?
- Added optional popup help (that's why I need the .po files)
- Added 'About' dialog
- Added [...] to fetch update zipfile name
- ToDo: Improve validation
- Improving error-checking

I have some time today in sunny Tenerife to fiddle about :)
Title: Re: Online Package Manager
Post by: cryptkeeper on November 30, 2016, 01:23:29 pm
hello folks, very nice job with Online Package Manager, thank you.
I have a small suggestion:  including line or two description what is in the package will be usefull for the user.
now we have to know, what we want to download/install with OPM.
e.g. package name callite. is category information Date/Time but it's visual components or only library of routing for dealing with date/time?
it would be usefull especially with category Other

thank you
Title: Re: Online Package Manager
Post by: lainz on November 30, 2016, 02:45:33 pm
Code: Pascal  [Select]
  1. ...
  2. Streamer.Options := Streamer.Options + [jsoUseFormatString];
  3. ...
  4. The JSON structure will change, but your demo is good and helps a lot. Thanks!

Thanks! Yes that's the power of the streamer, is really easy to use.

Sorry I missed the .po files, here it is again with them.
Title: Re: Online Package Manager
Post by: GetMem on November 30, 2016, 06:37:06 pm
@cryptkeeper
Just expand the *.lpk node(callight_pkg.lpk in your case) then check the description. It's set by the package maintainer, we cannot modify it.
Title: Re: Online Package Manager
Post by: minesadorada on November 30, 2016, 07:17:34 pm
@lainz
Did a bit more today.  The app is much too fancy for its purpose, but it was an interesting exercise for me.

More/changed resourcestrings, so the .es.po file is now out of whack (I still need to tidy them up so hold off translation for now)

I changed DefaultTranslator to LCLTranslator so the app could deal with languages in a more interactive way. Another reason was the DefaultTranslator chooses 'es' on my system, despite an English-language system.  It's probably fooled by my Spanish keyboard.

- Better interactive error-checking on inputs
- Language menu (only English/Español as yet)
- Hint system that runs on first use
- cfg file to store options
- 'About' dialog that uses app VersionInfo
- ..and other stuff
- I expect it could be refactored, but it seems to work in a user-friendly way and the code is maintainable.

Attached: current version 0.1.3.0

@GetMem I'm not sure anything needs to be added to the JSON - it seems enough for its purpose and it will be quick to download.
I have uploaded a test update JSON for you to test 'live'
https://sourceforge.net/projects/lazautoupdate/files/otherpackages/update_cryptini.json (https://sourceforge.net/projects/lazautoupdate/files/otherpackages/update_cryptini.json)
and https://sourceforge.net/projects/lazautoupdate/files/otherpackages/update_cryptini_force.json  (force update)
(A good protocol is that update JSONs must always have a filename which is distinct from regular OPM-generated JSONs. You could use a custom name-protocol for update JSONs such as 'update_<packagename>.json' - this avoids filename clashes with package JSON files created by OPM and stored in the same folder)
Title: Re: Online Package Manager
Post by: lainz on November 30, 2016, 08:33:02 pm
I like the new error messages, these are good to upload the json with no errors.

I need to read the code, but it works as far as I tested.
Title: Re: Online Package Manager
Post by: minesadorada on November 30, 2016, 09:46:07 pm
I like the new error messages, these are good to upload the json with no errors.

I need to read the code, but it works as far as I tested.
here's something I'd like you to look at:
If I am in 'en' mode and execute
Code: Pascal  [Select]
  1. SetDefaultLang('es', 'locale', TRUE);
It correctly switches to Spanish.

If I am in 'es' mode, and execute
Code: Pascal  [Select]
  1.   SetDefaultLang('en', 'locale', TRUE);
It doesn't switch back to English until the app is restarted (when the SetDefaultLang('en', 'locale', TRUE) is executed in Form.Create().)  Why so?

I have the jsoneditor.es.po and the jsoneditor.en.po in the appropriate /locale folder.
 
Title: Re: Online Package Manager
Post by: lainz on November 30, 2016, 10:18:47 pm
I don't know exactly, but maybe once the translation is set you need to restart the application.

Lazarus needs a restart. You can force a restart with:

Code: Pascal  [Select]
  1. OpenURL(ProgramDirectory + Application.ExeName);
  2. Application.Terminate;

I think once you change the constant strings from english to something else then the translator doesn't know anymore how to translate?

Like:

Open -> Abrir: this translates open to abrir
Abrir -> ?: doesn't know how to translate Abrir to something else, because the .po are in english to something else, not in spanish to something else.
Title: OPM update JSON Editor
Post by: minesadorada on December 01, 2016, 11:06:17 am
@lainz I think we have an alpha (attached V0.1.5.0)  Please review.
I would really appreciate it if you would revisit the Spanish translation and bring it up to date.

Switching Spanish -> English: It might work better with an .en.po file that matches the es.po in all entries
Title: Re: Online Package Manager
Post by: lainz on December 01, 2016, 01:58:09 pm
Here it is the updated translation.

I see some bugs:
- Remove: it removes the last item not the selected one
- Change language: it always display 'es' as language changed no matter what language I select (so no matter what I click I can't get back to english, even with a restart)
Title: Re: Online Package Manager
Post by: minesadorada on December 01, 2016, 03:16:19 pm
Here it is the updated translation.

I see some bugs:
- Remove: it removes the last item not the selected one
- Change language: it always display 'es' as language changed no matter what language I select (so no matter what I click I can't get back to english, even with a restart)
Bugs:
1.  I haven't changed the original remove code.  I'll have a look at it.
2. Spanish -> English works for me after a restart of the app. (Testing in Linux 64-bit)
2a. Does your cfg file have an entry 'Language=en'?  Does your /locale folder have jsoneditor.en.po (renamed from jsoneditor.po)?
2b If  Spanish -> English doesn't work in a Spanish system. then we can produce multiple distributions with different Project/options/Version Info/Language Options settings.

Thanks a million for the translation and review.  My Spanish still isn't good enough to do the job :)
Are you OK with GPL license?  Your (excellent) code is the core of the app, so you call the license.

I've done a bit more refactoring.  V0.1.6.0 to follow...
Title: Re: Online Package Manager
Post by: lainz on December 01, 2016, 04:16:04 pm
Quote
1.  I haven't changed the original remove code.  I'll have a look at it.

I know. The thing is that I created it to remove the last item, but the translation says that it removes the selected item.

Quote
2a. Does your cfg file have an entry 'Language=en'?  Does your /locale folder have jsoneditor.en.po (renamed from jsoneditor.po)?

No, I don't have one. But since the default translation is in english, why don't use the default po? That can be done?

Quote
Are you OK with GPL license?  Your (excellent) code is the core of the app, so you call the license.

Sure, GPL is OK.

We need to wait for GetMem, he said that maybe the external JSON structure changes, so we must wait for it.

Then upload it to sourceforge or github, or where you want, is the same for me.
Title: Re: Online Package Manager
Post by: lainz on December 01, 2016, 06:38:54 pm
Hi, this is the Spanish translation for the OPM
http://bugs.freepascal.org/view.php?id=31047

Please test and add to the Lazarus svn.
Title: Re: Online Package Manager
Post by: GetMem on December 02, 2016, 06:14:00 am
@lainz
Thanks for the translation. I see that @maxim already applied it.

@minesadorada, @lainz
Thanks for the application. It's very helpful.

@all
I have some personal issues to take care off. As soon as possible I will finish the update feature.
Title: Re: Online Package Manager
Post by: JuhaManninen on December 02, 2016, 12:43:47 pm
OPM now has temporary forks of FPC's lib units zipper and httpclient.
Somebody should create a patch for FPC project with the extensions and changes needed.
The forks are maybe from FPC 3.0. (GetMem, confirm?)
A diff should be made against the original code, then all formatting changes should be removed. Only real functional changes should go to the patch.
Then the changes must be applied to latest FPC trunk code and the final patch made against it.
FPC trunk has many changes for Unicode etc.
This would be a good task for somebody else than GetMem to share the work-load.
The code can be used directly from FPC libs only when it is included in a version that is the minimum requirement for Lazarus. So this is a long term plan.

There is another fork, VirtualTreeView.
Its master version may be added and maintained in Lazarus sources later. Now there are many forks and versions, the situation is messy.
Title: Re: Online Package Manager
Post by: minesadorada on December 02, 2016, 12:50:53 pm
Here it is the updated translation.

I see some bugs:
- Remove: it removes the last item not the selected one
- Change language: it always display 'es' as language changed no matter what language I select (so no matter what I click I can't get back to english, even with a restart)
1. Simpler to change the hint text :)
2. I have the answer to the Spanish -> English translation difficulty.
It's all about the .en.po file.
By default, Lazarus makes a .po file with the msgstr entries empty (which is correct behaviour)  However, when you want to make an .en.po file, you have to copy over the (English) msgid entries to the msgstr entries.  Then you can switch to English 'on-the-fly' just as you can with Spanish.  You learn something every day....
For the next version, I'll do that and compile them into .mo files ready for distribution.
3. The cfg file is now in the GetAppConfigFile(false) folder - which is better for Linux and Windows permissions.

A bit busy, but it should be ready by Monday.
Title: Re: Online Package Manager
Post by: lainz on December 02, 2016, 01:18:45 pm
@minesadorada

Wow, never imagined that =) And you solved it, excelent!

@JuhaManninen

Quote
The code can be used directly from FPC libs only when it is included in a version that is the minimum requirement for Lazarus. So this is a long term plan.

So this depends on when next FPC will be released and the changes on these libraries must be approved also for the next release of FPC.

If OPM will be available in the next lazarus, also depends if the new FPC for that date is also released, so we must "stay tuned"  ::)
Title: Re: Online Package Manager
Post by: JuhaManninen on December 02, 2016, 02:04:08 pm
If OPM will be available in the next lazarus, also depends if the new FPC for that date is also released, so we must "stay tuned"  ::)
Everything in Lazarus trunk, including OPM, will be included in next major version release.
Even if FPC had a major release before that and the to-be-made patch was included there, OPM must still have the forked units because Lazarus should compile with 2 latest compiler versions.
FPC release cycle is slow so this will take time.
Anyway, who will make the patches? It may require some refactoring, too. It would also make sense to update the forked units to latest FPC trunk versions.
I can do it later but I also have many things to do.
Title: Re: Online Package Manager
Post by: GetMem on December 02, 2016, 02:43:48 pm
@Juha
Quote
The forks are maybe from FPC 3.0. (GetMem, confirm?)
Yes.

"opkman_httpclient.pas" was recently synchronized with "fphttpclient.pp"(FPC trunk) , when proxy support was added. So the only unit that needs to be synchronized is "opkman_zip.pas" with "zipper.pp"", but please be careful not to reintroduce the overflow bug, since it wasn't fixed in FPC trunk.

Quote
There is another fork, VirtualTreeView.
Its master version may be added and maintained in Lazarus sources later. Now there are many forks and versions, the situation is messy.
PS: I wouldn't worry about VirtualTreeView, according to @Luiz we can add any version we like. Once it's added to Lazarus sources, I can remove my fork and link OPM to the one in the Lazarus trunk(10 min. of work).
Title: Re: Online Package Manager
Post by: JuhaManninen on December 02, 2016, 03:06:31 pm
... "zipper.pp"", but please be careful not to reintroduce the overflow bug, since it wasn't fixed in FPC trunk.
Yes, this one:
 http://bugs.freepascal.org/view.php?id=30800
I don't know why it takes so long for FPC patches to get applied. It is annoying.
Yes, I know some patches for Lazarus have been ignored for a long time as well in past. Now the situation is better, most patches are either applied or rejected in reasonable time.
Title: Re: Online Package Manager
Post by: minesadorada on December 02, 2016, 03:08:55 pm
Version 0.1.6.1 of the Update JSON Editor

Now bi-lingual on demand.  Spanish and English translations
Compiles to Win32/64 Linux32/64 Darwin32 Tested in Win64 and Linux64
Title: Re: Online Package Manager
Post by: GetMem on December 02, 2016, 03:54:19 pm
@minesadorada

Please add this line to the SaveToFileMethod:
Code: Pascal  [Select]
  1.   Streamer := TJSONStreamer.Create(nil);
  2.   Streamer.Options := Streamer.Options + [jsoUseFormatString]; //<--this one
  3.   s.AddText(Streamer.ObjectToJSONString(Self));
Title: Re: Online Package Manager
Post by: minesadorada on December 02, 2016, 04:30:15 pm
Done.  To 0.1.7.0

Windows 32/64-bit Installer (Inno Setup) https://sourceforge.net/projects/lazautoupdate/files/otherpackages/setup_jsoneditor.exe (https://sourceforge.net/projects/lazautoupdate/files/otherpackages/setup_jsoneditor.exe)
I can do a DEB file when I next boot into Linux
Title: Re: Online Package Manager
Post by: wp on December 03, 2016, 12:13:33 pm
I am not sure if the git repository of the OPM still exists. If yes I assume that it is a mirror of the version included in Laz trunk.

In this case, there is a problem that people using older Lazarus versions are able to compile the package. I just compiled the trunk version with Laz 1.6, and ran into several issues:
Title: Re: Online Package Manager
Post by: JuhaManninen on December 03, 2016, 12:28:04 pm
I am not sure if the git repository of the OPM still exists. If yes I assume that it is a mirror of the version included in Laz trunk.
I think it is now disabled or deprecated. It already created confusion as you can see earlier in this thread.
OPM sources can be downloaded separately from Lazarus trunk SVN.

Quote
...
With these modifications, the current version of OPM can be compiled with Laz 1.6
Sounds good.
As you are now testing the issue, you can as well commit those changes yourself. You have commit rights in components dir, don't you?
Title: Re: Online Package Manager
Post by: wp on December 03, 2016, 12:47:49 pm
Done. I always forget this because I am hesitant to modify other people's code.
Title: Re: Online Package Manager
Post by: GetMem on December 03, 2016, 02:28:14 pm
Thanks @wp. Feel free to modify the code whenever you like.

PS: I deleted the github repository.
Title: Re: Online Package Manager
Post by: minesadorada on December 04, 2016, 09:38:39 am
Bugfix for json_packager - to 0.1.8
- V0.1.7 had a bad config file name.  Fixed.
- If app is moved to a new folder, it makes a unique config filename (so each component can have its own stored paths)
- When saving the json, it suggests as the filename '.\updates\update_<component zipfilename>.json'
- Added 'New' to file menu (todo: translate -> Español)

Windows installer updated: https://sourceforge.net/projects/lazautoupdate/files/otherpackages/setup_jsoneditor.exe (https://sourceforge.net/projects/lazautoupdate/files/otherpackages/setup_jsoneditor.exe)
Title: Re: Online Package Manager
Post by: lainz on December 04, 2016, 03:11:20 pm
Hi, here it is the updated spanish translation for the external json editor.

Maybe we must upload it in a revision control? I have both accounts in sourceforge and github, we have also the new Lazarus CCR in GitHub, we can upload the first code in there.
Title: Re: Online Package Manager
Post by: minesadorada on December 04, 2016, 03:28:22 pm
Hi, here it is the updated spanish translation for the external json editor.

Maybe we must upload it in a revision control? I have both accounts in sourceforge and github, we have also the new Lazarus CCR in GitHub, we can upload the first code in there.
Thanks for the translation.  I have access to the SVN ccr repository (which I use for other components), so if I commit to there, the GitHub will mirror it OK?
Title: Re: Online Package Manager
Post by: lainz on December 04, 2016, 03:42:35 pm
Hi, here it is the updated spanish translation for the external json editor.

Maybe we must upload it in a revision control? I have both accounts in sourceforge and github, we have also the new Lazarus CCR in GitHub, we can upload the first code in there.
Thanks for the translation.  I have access to the SVN ccr repository (which I use for other components), so if I commit to there, the GitHub will mirror it OK?

If you can upload it in the SVN ccr is a good place. About the github mirror, I don't have a server to create the automatic mirroring, so is not available. But don't worry about that. The idea is that the source don't get lost :)
Title: Re: Online Package Manager
Post by: GetMem on December 04, 2016, 03:58:57 pm
@minesadorada
1. Try to feed invalid JSONs to your application, then make sure the exceptions are handled properly(see attachment for invalid JSON). This is crucial for me(OPM), since the update process will run in a worker thread. Unhandled exceptions are not desired. I'm gonna use the same code base(more or less...I don't need the GUI part).
2. I noticed that you allow multiple lpk's with the same name. All package names must be unique.
Can you please fix this issues?

+1 for the revision conrol.

PS: I may add one or more fields in the future(objects to JSON), hope this is not a problem.
Title: Re: Online Package Manager
Post by: minesadorada on December 04, 2016, 04:15:48 pm
@GetMem.  OK  - will do (not today, because its the UK Snooker Final, though I will get a gap in the mid-session interval) :)

Committed to CCR as V0.1.8.0
Title: Re: Online Package Manager
Post by: GetMem on December 04, 2016, 05:03:40 pm
@minesadorada
Take your time, it's not urgent.
Title: Re: Online Package Manager
Post by: lainz on December 04, 2016, 05:22:42 pm
Well it's really simple to add the exception. You can do the same as SaveToFile(): boolean, change LoadFromFile to a function with a boolean.

I suggest to remove the ShowMessage from the exception code, and move it to the else part of the SaveToFile, so it looks like

if SaveToFile() then
  ShowMessage('ok')
else
  ShowMessage('not ok');

And with the Load the same.

if LoadFromFile() then
begin
 // Put everything in the controls
end
else
  // do nothing, just show an error message.

And the exception with no code will be usable by GetMem. But he must add a LoadFromURL or something like that.
Title: Re: Online Package Manager
Post by: minesadorada on December 04, 2016, 06:51:07 pm
All good suggestions lainz.

0.1.11.0 committed to ccr svn (https://svn.code.sf.net/p/lazarus-ccr/svn/applications/json_packager (https://svn.code.sf.net/p/lazarus-ccr/svn/applications/json_packager))
- Changes as suggested by lainz
- Error-check for duplicate .lpk entries implemented as requested

@GetMem - if you add any fields, can you specify any relevant error checks with them please?
Title: Re: Online Package Manager
Post by: lainz on December 04, 2016, 09:57:05 pm
Thanks for implementing the suggestions. I already implemented them when I wrote the message, but by some unknown reason I get the spanish translation broken, something about ANSI? And then every special character inside was broken. So I better let you made the changes, I don't know if was a problem with lazarus or with the po editor.
Title: Re: Online Package Manager
Post by: GetMem on December 05, 2016, 07:54:14 am
@minesadorada
Quote
if you add any fields, can you specify any relevant error checks with them please?
You don't have to do any further error checks, it will work fine with the code suggested by @lainz, however please rename DownloadURL to DownloadZipURL(or something similar). I already have a DownloadURL, which now points to the "external JSON". To prevent confusion we should give a different, unique name.

PS: Today/tomorrow the update feature will be ready.
PS1: I also renamed a few class since I already have similar names. Please download the zip, and replace your main form. Other then a few names is the same unit.


Title: Re: Online Package Manager
Post by: minesadorada on December 05, 2016, 09:58:31 am
@minesadorada
Quote
if you add any fields, can you specify any relevant error checks with them please?
You don't have to do any further error checks, it will work fine with the code suggested by @lainz, however please rename DownloadURL to DownloadZipURL(or something similar). I already have a DownloadURL, which now points to the "external JSON". To prevent confusion we should give a different, unique name.

PS: Today/tomorrow the update feature will be ready.
PS1: I also renamed a few class since I already have similar names. Please download the zip, and replace your main form. Other then a few names is the same unit.



A global rename from DownloadURL -> DownloadZipURL seemed to do the trick.

- To V0.1.12.0 in svn ccr
- Updated .po files
- InnoSetup install updated

Updated test update: https://sourceforge.net/projects/lazautoupdate/files/otherpackages/update_cryptini.json to new fieldname
Title: Re: Online Package Manager
Post by: GetMem on December 05, 2016, 10:25:00 am
@minesadorada

Apparently you missed PS1 from my previous post. I attach the zip again, since I changed something. Your project also have a memory leak somewhere:
Project Options-->Debugging-->Use HeapTrc unit.

Thank you!

Title: Re: Online Package Manager
Post by: minesadorada on December 05, 2016, 11:01:46 am
Hi Getmem,
Thank you for the review.
Have you got write access to the ccr svn?  I missed your attached zip because I assumed dev was now under version control lock/checkouts from the svn.

After a quick look,I see only 2 changes - can you confirm?
1- Commenting out in form.create
Code: Pascal  [Select]
  1. {  Self.AutoAdjustLayout(lapAutoAdjustForDPI, Self.DesignTimeDPI,
  2.     Screen.PixelsPerInch, Self.Width, ScaleX(Self.Width, Self.DesignTimeDPI));}
2- Changing fieldname from DownloadURL -> DownloadZipURL
If they are the only 2 changes, I can update the svn version with (1) easily enough.  It's a good idea to add to the Version History section in umain with any changes.

Re:Memory leak
It seems to come from this recursive procedure within Form.Create (which I'd like to keep)
Code: Pascal  [Select]
  1.  
  2. procedure TfrmMain.FormCreate(Sender: TObject);
  3. var
  4.   sLang, INIFilePath: string;
  5.  
  6.   procedure CreateUniqueINI(aCount: integer);
  7.   // Recursively loop until correct INI found, or new one created
  8.   begin
  9.     INIFilePath := GetAppConfigFile(False) + IntToStr(aCount);
  10.     CFG := TIniFile.Create(INIFilePath);
  11.     if CFG.ReadString('Options', 'AppPath', ProgramDirectory) <> ProgramDirectory then
  12.     begin
  13.       CFG.Free; // Ditch the old one
  14.       Inc(aCount);
  15.       CreateUniqueINI(aCount); // Make a new one
  16.     end;
  17.   end;
  18. //... more code
  19.  
Any suggestions to solve it? CFG is a private form member.

I'll hold fire until I see your thoughts.
Title: Re: Online Package Manager
Post by: GetMem on December 05, 2016, 11:17:33 am
Quote
After a quick look,I see only 2 changes - can you confirm?
   TPackageFiles --> TUpdatePackageFiles
   TPackageData --> TUpdatePackageData
  etc...
I only changed names, since I already have TPackageFiles in OPM and it's very confusing. Compare the two files with WinMerge or whatever tool you use for comparing. SVN clients already have built in tools for that.

Quote
Have you got write access to the ccr svn?  I missed your attached zip because I assumed dev was now on lock/checkouts from the svn.
No.

Quote
It seems to come from this recursive procedure within Form.Create (which I'd like to keep)
I did not have time to check it, since I'm busy with OPM. I wil take a look later.
       
Title: Re: Online Package Manager
Post by: minesadorada on December 05, 2016, 12:06:55 pm
To:  0.1.13.0: Renamed TPackageData ->  TUpdatePackageData (GetMem)
            Renamed TPackageFiles -> TUpdatePackageFiles (GetMem)
            Comment out Self.AutoAdjustLayout line in Form.Create (GetMem)
            Removed StrUtils from uses (minesadorada)
            Fixed memory leaks with CFG and slErrorList (minesadorada)
            Moved inline procedure CreateUniqueINI to separate function (minesadorada)
            Added Const C_DEBUGMESSAGES=TRUE/FALSE (minesadorada)

Memory leaks were weird.  I had to manually free 2 form members (a TiniFile and TStringList) in the QueryClose method to fix them.  I'd assumed all owned TForm objects were automatically freed on exit. (Laz1.7/fpc3.1.1)
Title: Re: Online Package Manager
Post by: JuhaManninen on December 05, 2016, 12:34:30 pm
Memory leaks were weird.  I had to manually free 2 form members (a TiniFile and TStringList) in the QueryClose method to fix them.
It is not weird as Pascal has no garbage collection.
FormDestroy may be more logical place to free them. It makes kind of a pair with FormCreate.
Title: Re: Online Package Manager
Post by: lainz on December 05, 2016, 01:57:27 pm
@minesadorada

If you commented the AutoAdjustLayout I will see the application badly or not scaled. There is a Scaled property in 1.7 you can use to do the High DPI stuff, but is under construction AFAIK.
Title: Re: Online Package Manager
Post by: GetMem on December 05, 2016, 02:24:23 pm
You don't have to comment out that line, only change a few names...I attached the whole project fixed.
Title: Re: Online Package Manager
Post by: GetMem on December 05, 2016, 02:55:56 pm
Ok. The update will look like this(scrennshot1). When expanded(screenshot2).
Do we need color/background color? I have tried a lot of combination, it looks best with a bold/black text.
Popup balloons, memo with more detailed text?
Title: Re: Online Package Manager
Post by: minesadorada on December 05, 2016, 03:29:17 pm
You don't have to comment out that line, only change a few names...I attached the whole project fixed.
Thanks for that @GetMem
Committed to ccr svn as V0.1.14.0

json packager code updaters: Please update the umain Version History and project versioninfo to save me time doing diffs and merges..  The project is under svn version control.

@Getmem - opm: Good work! When trunk is updated I'll have a look.
I wouldn't worry about details of appearance until the update mechanism is tested for functionality.  How does the 'force update' look?  Perhaps when true, then 'New' could be different text?
Title: Re: Online Package Manager
Post by: minesadorada on December 05, 2016, 04:38:31 pm
Memory leaks were weird.  I had to manually free 2 form members (a TiniFile and TStringList) in the QueryClose method to fix them.
It is not weird as Pascal has no garbage collection.
FormDestroy may be more logical place to free them. It makes kind of a pair with FormCreate.

Hold on.
So you're saying that some member objects of TForm instantiated in TForm.Create have to explicitly freed in TForm.Destroy?
I always assumed that objects in a TForm derivative class were freed when the TForm derivative class was destroyed - automatically (garbage collection within a class)  Mind you, the Tstrings.Create method doesn't specify a parent; nor TIniFile.Create whcih I thought was curious.
Thanks for that info - I'll have to revisit some old code...

Committed to ccr svn as V0.1.15.0
- Objects freed in Form.Destroy as per JuhaMannien's advice
Title: Re: Online Package Manager
Post by: GetMem on December 05, 2016, 05:47:11 pm
Quote
@lainz
A question, the zip must be generated with the OPM or it can be any zip (like master.zip from github)?
No. You just simply zip your source. It can be the master.zip generated by github or your next stable version zipped. It's up you.

Quote
The zip must contain inside the normal .json or not?
No again. You generate the json with the tool created by you and minesadorada and send me the link.  That's it. In fact to make it easier OPM will generate the initial JSON(it's not yet imlemented), this can be later edited by your tool.

The zip/json generated by the OPM was for me, to make my work easier. It's not needed at all. If somebody wants to add a new package to the repository, just send me the link to the zip file.

I will add all the info to wiki soon.
Title: Re: Online Package Manager
Post by: lainz on December 05, 2016, 05:59:41 pm
Thanks. Well I think once you made the first external json you don't need the tool if you don't change the url or the zip names / package names. Just changing the version information and nothing more.
Title: Re: Online Package Manager
Post by: minesadorada on December 05, 2016, 06:09:33 pm
Thanks. Well I think once you made the first external json you don't need the tool if you don't change the url or the zip names / package names. Just changing the version information and nothing more.
I can't understand the lack of automation here.
If GetMem is unavailable, currently the update system fails.  Am I right there?

Why can't OPM regularly poll the http://<update_url>/<update_componentname>.json list automatically and look for version updates?  This could even be in a background thread when OPM loads (as an user-chosen option)  All that's needed is a standardised json filename for updates, and json packager provides that as a default.
Title: Re: Online Package Manager
Post by: lainz on December 05, 2016, 06:19:21 pm
If I understand it well, the external json replaces the external zip that was working before. Now it's better because you actually know the new version you will try to download and install.

GetMem already said that is not only the external json because downloading all the json takes more time than downloading the main json from the OPM server.

About updating the main server, that's not resolved yet, so we must wait until they decide how it will be managed. Currently you're right, GetMem is the only updating the packages. But when the package already contains an extenal json you can update it by yourself how many times you want.
Title: Re: Online Package Manager
Post by: minesadorada on December 05, 2016, 06:22:13 pm
If I understand it well, the external json replaces the external zip that was working before. Now it's better because you actually know the new version you will try to download and install.

GetMem already said that is not only the external json because downloading all the json takes more time than downloading the main json from the OPM server.

About updating the main server, that's not resolved yet, so we must wait until they decide how it will be managed. Currently you're right, GetMem is the only updating the packages. But when the package already contains an extenal json you can update it by yourself how many times you want.
That makes no sense.  The external json is trivial in size and quick to download and check. 41 (currently) update json files are no great burden. They point to the location of the external update zipfile, which could be of considerable size and only downloaded on user request.  Currently AFAIK the 'master' json is downloaded each time OPM shows,
Title: Re: Online Package Manager
Post by: lainz on December 05, 2016, 06:25:55 pm
Well he tested it and says that takes time (I've readed it somewhere here in this thread If I remember it well), so there is a kind of problem here, real one, or the library for downloading is not as fast, or something else is happening :)
Title: Re: Online Package Manager
Post by: minesadorada on December 05, 2016, 06:29:41 pm
Well he tested it and says that takes time (I've readed it somewhere here in this thread If I remember it well), so there is a kind of problem here, real one, or the library for downloading is not as fast, or something else is happening :)
Let me see if I understand it correctly:
1. As a server cron job, the update jsons are polled (i.e. weekly) and merged into the master json file on the server when a new version is detected.  This is invisible to users and requires no interaction by GetMem.
2. The master json file is downloaded when OPM starts on the user's system - which is current behaviour.  This is the speed bottleneck, if there is any.  It could be eased by downloading a small Hash file and checking against the local version.

I'm not seeing any issues with updating and speed.

All the component maintainer has to do is update their previously-specified UpdateZipfile and modify the corresponding update_<componentname>.json file  (that's what the json editor does).  No notifications nor human interaction are necessary.
Title: Re: Online Package Manager
Post by: lainz on December 05, 2016, 06:40:01 pm
Ok well we must see if they like that idea. I personally think is better, only we need a mantainer for the first commit, then the responsability is of the mantainers. On the other hand, if I don't mantain the package anymore and the download get's broken, the repository will not have that component anymore.

So is not only merging the main json, but the zip files. This was talked some days or weeks ago in this thread, and there is no consent about how to manage this.

I like your idea, but I'm not the one that will do it real or takes the decision of making it real. And positions against the automated stuff are always on.
Title: Re: Online Package Manager
Post by: minesadorada on December 05, 2016, 06:43:59 pm
I like your idea, but I'm not the one that will do it real or takes the decision of making it real. And positions against the automated stuff are always on.
I totally agree.  "The man who pays the piper calls the tune" :)

Let's see which sustainable system for package maintainers @GetMem settles on.

No harm in proposing ideas - @GetMem is a reasonable person; open to ideas AFAIK.

Quote
On the other hand, if I don't mantain the package anymore and the download get's broken, the repository will not have that component anymore.
With the system as proposed: the auto-checking would automatically remove the component from the master tree if the URL went bad.  That's correct behavior for unmaintainable/obselete components, surely?  That's why many components from the ccr were not included in the initial master list AFAIK.

Quote
So is not only merging the main json, but the zip files. This was talked some days or weeks ago in this thread, and there is no consent about how to manage this.
Why merge the zip files on the OPM server by default? The initial (vetted) zips can be canonical. The user chooses to update their local repository or not (with a suitable warning).  The 'Available' column reflects the 'available update version'.  That makes sense for a user. I thought that was the point of the 'force update' option in the update json - where the canonical version is somehow faulty or out-of date and has to be replaced on the server - not default behaviour.

I'm trying to look 3 - 5 years ahead with Laz 2.x/OPM when active maintenance of OPM may not be so available as now.

I'm very enthusiastic about @GetMem's initiative - it was an needed idea for Lazarus waiting to happen, and makes Lazaus look more professional to new and old users alike.
Title: Re: Online Package Manager
Post by: GetMem on December 05, 2016, 07:05:21 pm
@minesadorada
Quote
I can't understand the lack of automation here.
If GetMem is unavailable, currently the update system fails.  Am I right there?
No. You completely misunderstood the update system, but that's ok since nobody knew how it should work or what would be the optimal solution.

Quote
Why can't OPM regularly poll the http://<update_url>/<update_componentname>.json list automatically and look for version updates?  This could even be in a background thread when OPM loads (as an user-chosen option)  All that's needed is a standardised json filename for updates, and json packager provides that as a default.
A pull in every 6 sec is ok? Currently is 6000 ms. :D Though I plan to extend a little bit, say 1 minute to minimize traffic load.

@lainz
Quote
Well he tested it and says that takes time (I've readed it somewhere here in this thread If I remember it well), so there is a kind of problem here, real one, or the library for downloading is not as fast, or something else is happening :)
True. I said that once, but the problem is now solved. I save the last update value, then from time to time OPM pull the json. For example after you update the external JSON, the user won't see the changes instantaneously, it will take 1-2 min(max). I believe this is more then ok. The pull is done in a background thread, so it won't interfere with the GUI.


The basics steps:
1. I add the package to the main repository. This step is not yet automatized, I have to do it manually. This is why you guys created that zip/json("Create repository package")
2. Once the package is added to the main repository you can generate the "external json", then change the version, link to external zip, etc..whenever, however you like
3. Send me the link to the external JSON, so I can add to the package again(see: screenshot). This has to be done only once.
4. After this step everything is automatic, the OPM from time to time(1 min or so) will check the external json, is something has changed, OPM notify the user, on request download the zip, install the new package.

Hope is clear now. I believe this is how @Juha, @lainz and others envisioned from the beginning or at least something similar.
Title: Re: Online Package Manager
Post by: lainz on December 05, 2016, 07:20:27 pm
Thanks. Now seems everything clear. When you consider it ready tell us so we can create the external json.
Title: Re: Online Package Manager
Post by: minesadorada on December 05, 2016, 07:24:11 pm
+1 Thanks for the clarification.

One question: What do you envision are the consequences of checking the 'force update' option in the update_json file?  What are the guidelines?
Title: Re: Online Package Manager
Post by: GetMem on December 05, 2016, 08:52:34 pm
Quote
@minesadorada
One question: What do you envision are the consequences of checking the 'force update' option in the update_json file?  What are the guidelines?
It's useful when you change something in the source, but don't want to increase the version number yet. This is specially true for trunk, a lot of commits are done before the version number is modified. By setting ForceUpdate to true, you ensure that the user gets the latest updates, even if the version number stays the same.
Title: Re: Online Package Manager
Post by: minesadorada on December 05, 2016, 09:10:36 pm
Quote
@minesadorada
One question: What do you envision are the consequences of checking the 'force update' option in the update_json file?  What are the guidelines?
It's useful when you change something in the source, but don't want to increase the version number yet. This is specially true for trunk, a lot of commits are done before the version number is modified. By setting ForceUpdate to true, you ensure that the user gets the latest updates, even if the version number stays the same.

Now I'm really puzzled.

So, If I set ForceUpdate=TRUE what does the user see?
According to your scenario, I can make a minor update (with no version number increase) and that can force the user to update.  I can make a major update, with many new features and a new version number, but no forceupdate=true, and the user can ignore it.

I just don't get it.

Surely any change to the source would trigger a version update - even if it's 0.0.n+1.0?  Maybe trunk Lazarus developers don't do that, but why should that paradigm be adopted by independent component developers by default in the OPM system?  Lazarus developers use 'svn revision#' but OPM only allows for Version Quads.

When should a package maintainer set forceupdate=true, and why? 
That is my real question, and I don't understand your previous answer sufficiently.
Title: Re: Online Package Manager
Post by: lainz on December 05, 2016, 10:02:23 pm
Quote
So, If I set ForceUpdate=TRUE what does the user see?
According to your scenario, I can make a minor update (with no version number increase) and that can force the user to update.  I can make a major update, with many new features and a new version number, but no forceupdate=true, and the user can ignore it.

If you set forceupdate true but don't change the version number, the user sees nothing, as far as I understand the system.

If you do a major update with version number, and force update is false, the user will see the new version number.

Updates are not forced never I think, is just to 'force the download of the latest zip, doesn't matter what it contains'.

Quote
When should a package maintainer set forceupdate=true, and why?


I don't know. If I will use the master.zip of github it will gives you always the latest sources, doesn't matter what the external.json says!

So it's a bad idea to use master.zip of github as I understand it now. Or I must always ensure that it will work, or always increase the last number in the release builds, causing the user to download every day my package  ::)

Force updates will download per user request, OPM can't see if we made changes or not in the zip. Notifications will not be triggered only by setting forceupdate to true.
Title: Re: Online Package Manager
Post by: GetMem on December 05, 2016, 10:25:50 pm
Perhaps ForceUpdate is the wrong word. I mean you cannot force anything, in the end the user decides if he/she want to update or not. However if you want to notify the user that a new version is available, without actually increasing the version number, you should set ForceUpdate to true. From the users point of view there is no difference between ForceUpdate = true(same version number) or an increased version number, they will see the same "alert" message, namely that a new version is available for download/install. Force update is for the package maintainers, I explained in my previous post why I consider it useful. If you don't like the idea, no problem, we can come up with something better.

However we need a system that makes possible to trigger an update message, without increasing the version number. I' m open to suggestions.
Title: Re: Online Package Manager
Post by: lainz on December 05, 2016, 10:30:46 pm
Ok now I get it.  I was thinking it wrong.

For example you download now the latest zip right? Then force update is still true. So the dialog keeps saying that there's a new version available?
Title: Re: Online Package Manager
Post by: GetMem on December 05, 2016, 10:39:30 pm
Quote
For example you download now the latest zip right? Then force update is still true. So the dialog keeps saying that there's a new version available?
Yes. I believe this is specially useful for your master.zip. You don't have to change the version number, just leave ForceUpdate(ForceNotify) to true. Perhaps we can combine with a date? So the OPM will only notify the user when the date also changes?
Title: Re: Online Package Manager
Post by: lainz on December 05, 2016, 10:43:36 pm
Oh I see now the utility.

Well can be a date time. Since can be several updates in a day? Or only one per day. As you think is best. Maybe only once per day is the best so user don't loose too much time updating again and again in the same day.
Title: Re: Online Package Manager
Post by: GetMem on December 05, 2016, 11:00:39 pm
Quote
Well can be a date time.
Sure, no problem. Datetime is perfectly fine. Let's wait and see what other people has to say about this. The collective brain power is always better the the individual one.  8-)
Title: Re: Online Package Manager
Post by: lainz on December 05, 2016, 11:05:14 pm
Sure!  8)
Title: Re: Online Package Manager
Post by: minesadorada on December 06, 2016, 10:23:08 am
Couple of small bugfixes for the json Editor.  To v0.1.15.0
Please update from svn

There's an update file for Cryptini here that you can use for testing:
https://sourceforge.net/projects/lazautoupdate/files/otherpackages/update_cryptini.json

P.S. Just updated OPM from trunk, and tried updating OPM to 'V4.8.7.4' but no luck (see attached).  Work-in-progress, so nothing to worry about?
2nd screenshot was after clicking OK.  It froze at that point.
Title: Re: Online Package Manager
Post by: GetMem on December 06, 2016, 10:36:06 am
Quote
P.S. Just updated OPM from trunk, and tried updating OPM to 'V4.8.7.4' but no luck (see attached).  Work-in-progress, so nothing to worry about?
2nd screenshot was after clicking OK.  It froze at that point.
It's not ready yet...
Title: Re: Online Package Manager
Post by: minesadorada on December 06, 2016, 10:39:54 am
Understood.  I was too eager to see it!
Title: Re: Online Package Manager
Post by: minesadorada on December 06, 2016, 10:42:56 am
Perhaps ForceUpdate is the wrong word. I mean you cannot force anything, in the end the user decides if he/she want to update or not. However if you want to notify the user that a new version is available, without actually increasing the version number, you should set ForceUpdate to true. From the users point of view there is no difference between ForceUpdate = true(same version number) or an increased version number, they will see the same "alert" message, namely that a new version is available for download/install. Force update is for the package maintainers, I explained in my previous post why I consider it useful. If you don't like the idea, no problem, we can come up with something better.

However we need a system that makes possible to trigger an update message, without increasing the version number. I' m open to suggestions.

Now you've explained it, I think it's a good feature as it is.  The hint text on the GUI editor is "'This option should only be used for crucial updates or bug-fixed packages. Are you OK with that?'"

I can see the difficulty with expiring the notification though.
Title: Re: Online Package Manager
Post by: CharlyTango on December 06, 2016, 10:56:40 am
beeing eager to user OPM i checked it out from the trunk and tried it with Lazarus 1.6

OPM was compilable with Lazarus 1.6 , showed his main window and tried to load the list of available packages.

But it stoppes with an information messagebox

Code: Pascal  [Select]
  1. Invalid JSON File
  2. "LazSerial": Could not Convert variant of type (Null) into type (String)

furthermore an access violation occured when closing the OPM window

Hope that Info helps getting it stable.

Q: ist there any possibility of a formal bugtracking (let's say in http://bugs.freepascal.org (http://bugs.freepascal.org)) ?
Title: Re: Online Package Manager
Post by: GetMem on December 06, 2016, 10:57:52 am
@minesadorada
You should rename ForceUpdate to ForceNotify both in GUI and in the JSON class(TUpdatePackageData),  ForceNotify is more suggestive. You can also send me the link to a json(one of your packages), so we can run a few test.
Title: Re: Online Package Manager
Post by: GetMem on December 06, 2016, 11:00:49 am
Hi CharlyTango,

Thanks for testing OPM.

Quote
But it stoppes with an information messagebox
furthermore an access violation occured when closing the OPM window
Please take the latest trunk version again(I just commited a few changes). After you reinstall the package, go to the OPM Options, then press the "Restore Defaults" button. Your config file points to an older, obsolete repository(hence the error). 
Title: Re: Online Package Manager
Post by: minesadorada on December 06, 2016, 12:09:31 pm
@minesadorada
You should rename ForceUpdate to ForceNotify both in GUI and in the JSON class(TUpdatePackageData),  ForceNotify is more suggestive. You can also send me the link to a json(one of your packages), so we can run a few test.
OK. Done.
0.1.16.0: Renamed ForceUpdate to ForceNotify
The translation has somehow messed up - its a work in progress.

I made 2 update jsons; one with forcenotify=true using the new format:

https://sourceforge.net/projects/lazautoupdate/files/otherpackages/update_cryptini.json
https://sourceforge.net/projects/lazautoupdate/files/otherpackages/update_cryptini_force.json
Title: Re: Online Package Manager
Post by: GetMem on December 06, 2016, 02:55:40 pm
@minesadorada
Please update to the latest Lazarus trunk then test cryptini update. Thanks!

@lainz
Can you give me the link to your JSON? I will add it later to the main repository.
Title: Re: Online Package Manager
Post by: minesadorada on December 06, 2016, 03:04:28 pm
@GetMem I have Lazarus 1.7 SVN Revision 52081M - is that OK or do I need to update?
Title: Re: Online Package Manager
Post by: lainz on December 06, 2016, 03:48:05 pm
BGRA Controls
https://raw.githubusercontent.com/bgrabitmap/bgracontrols/master/update_bgracontrols.json
https://raw.githubusercontent.com/bgrabitmap/bgracontrols/master/update_bgracontrols_force.json

BGRA Controls FX
https://raw.githubusercontent.com/bgrabitmap/bgracontrolsfx/master/update_bgracontrolsfx.json
https://raw.githubusercontent.com/bgrabitmap/bgracontrolsfx/master/update_bgracontrolsfx_force.json

About BGRA Bitmap, I need to tell circular for what is this first, then we will have one also for bgrabitmap.

When you test that the two works well, I will keep only the first (update_bgracontrols.json) and remove the force one, but I will add the force option to the first.
Title: Re: Online Package Manager
Post by: GetMem on December 06, 2016, 04:43:43 pm
@ minesadorada
Please take the latest version. I'm afraid it's buggy, but at least you can run a few tests.

@lainz
No worries! This is just for testing purposes, still a long way to go, you can tell @circular later. I will add the link as soon as possible.
PS: Done. I added the ForceNotify version just to trigger a message. Please update to the latest trunk.

Thank you guys for you help!
Title: Re: Online Package Manager
Post by: CharlyTango on December 06, 2016, 05:18:28 pm
Hi CharlyTango,

Thanks for testing OPM.

Quote
But it stoppes with an information messagebox
furthermore an access violation occured when closing the OPM window
Please take the latest trunk version again(I just commited a few changes). After you reinstall the package, go to the OPM Options, then press the "Restore Defaults" button. Your config file points to an older, obsolete repository(hence the error).

OK, THX -- that worked fine.
OPM main window started and after pressing "Restore Defaults" the package List was retrieved and showed. -- great work - thx

Using OPM shows a "NEW" mark at onlinepackagemanager indicates that version 1.0.1.0 would be available.
Installing the new version via OPM failed -- it seems that an old version is downloaded and it does not work.
The version directly from the trunk is ok.

my 3cents...:

The OP seem to have the potential to a crucial part of Lazarus.
One of the most time consuming tasks ist the safe installation and updating of packages.
Therefore OPM should have the best support -- Did you thought about a category in bugs.freepascal.org (http://bugs.freepascal.org) ?

The first impression of OPM: Looks fine, but what i miss is a description of the package because i don't know what the seldom used packages do.

Filtering of installed packages and their update status would be nice too.

Good Job -- thx

Title: Re: Online Package Manager
Post by: GetMem on December 06, 2016, 05:35:37 pm
@CharlyTango
Quote
Using OPM shows a "NEW" mark at onlinepackagemanager indicates that version 1.0.1.0 would be available.
Installing the new version via OPM failed -- it seems that an old version is downloaded and it does not work.
The version directly from the trunk is ok.
Unfortunately the update feature is not yet implemented(we are working on it), for now please ignore it. However you can test all the other functionality.

Quote
first impression of OPM: Looks fine, but what i miss is a description of the package because i don't know what the seldom used packages do.
See screenshot1, however not all package has a detailed description, depends on the package maintainer.

Quote
Filtering of installed packages and their update status would be nice too.
You can find a lot of filters in the top of the form, I can add more if necessary. See screenshot2

Quote
Therefore OPM should have the best support -- Did you thought about a category in bugs.freepascal.org ?
Please report at bugs.freepascal.org(category Lazarus/Packages) or here.

Quote
Good Job
Thanks.

Title: Re: Online Package Manager
Post by: lainz on December 06, 2016, 06:22:13 pm
Quote
@lainz
No worries! This is just for testing purposes, still a long way to go, you can tell @circular later. I will add the link as soon as possible.
PS: Done. I added the ForceNotify version just to trigger a message. Please update to the latest trunk.

I've installed bgrabitmap, bgracontrols and bgracontrolsfx with your tool in a few minutes (that's by my slow connection). So it works fine. Then it keeps displaying that there's a new version: for bgracontrols, bgracontrolsfx, cryptini and onlinepackagemanager.

cryptini seems that is not showing forceupdate in the json, but it still shows that there's a new version available (because is different the external version from the repository version), but I don't have it installed, that's ok?
Title: Re: Online Package Manager
Post by: minesadorada on December 06, 2016, 07:06:43 pm
@ minesadorada
Please take the latest version. I'm afraid it's buggy, but at least you can run a few tests.
Sorry @GetMem - I've been out playing golf this afternoon in the Tenerife sunshine. (par+2 - yay!)  I'll catch up tomorrow as I'm currently celebrating with a very large Scotch :)
Title: Re: Online Package Manager
Post by: lainz on December 06, 2016, 08:32:13 pm
@minesadorada

When you have time change the AutoAdjustLayout lines with this:

Quote
Self.AutoAdjustLayout(lapAutoAdjustForDPI, 96, Screen.PixelsPerInch, Self.Width, ScaleX(Self.Width, 96), False);

That will work only under Lazarus 1.7+, because the ammount of parameters and some more things about High DPI has changed recently.

Edit: Or we can remove that and use the Scaled property, but it has (currently) a disadvantage: you must compile it with a setting of 96 dpi to work well.
Title: Re: Online Package Manager
Post by: GetMem on December 06, 2016, 10:14:12 pm
@lainz
Yes. This is how it should work. If you install cryptini for example the alert message will disappear, since ForceNotify is false.

@minesadorada
Enjoy your time.  :)
Title: Re: Online Package Manager
Post by: lainz on December 07, 2016, 12:11:45 am
Hi GetMem, seems that is working really well. I hope you can talk with the lazarus developers (well, with the others, you are one of them now) and include this package as one of the pre-installed with Lazarus, so with that it will become more visible to all developers that open the package menu in the IDE, and more people will use it.

Edit:
@minesadorada

Hi, when you have time remove the AutoAdjustLayout from OnCreate and just check the option Scaled in the object inspector for the form, now is working really well and is the best from now using Lazarus 1.7 to get HighDPI working well.
Title: Re: Online Package Manager
Post by: minesadorada on December 07, 2016, 07:50:03 am
@minesadorada

Hi, when you have time remove the AutoAdjustLayout from OnCreate and just check the option Scaled in the object inspector for the form, now is working really well and is the best from now using Lazarus 1.7 to get HighDPI working well.
@lainz: On my Laz 1.7 Rev 52081M I don't see a Scaled property in the OI - only a PixelsPerInch property
or.. can Scaled:=True be set in Form.Create inside an {$IFDEF} version check, and if so, what is the fallback?
P.S. If I set the form.Autosize=True, and form.Borderwidth=8 the form looks OK on my system.

re trunk: I'm reluctant to mess with my cross-compiling dev environment, unless you can suggest a way to upgrade to trunk in a simple way. (not fpcup)  I have the latest source of trunk - can it be patched in by simply merging with my existing setup somehow?

@GetMem: Just installed latest OPM and the updating goes flawlessly.  Thank you!
1. CryptINI updated OK√
2. I tried to install bgracontrols.  It flagged the dependency OK√ and started the install, but compilation errored out, and OPM correctly offered to cancel OK√

GUI suggestions for future:
I find that what is missing in OPM is an 'UnInstall' button for (an) installed package(s). I know the IDE can do this, but OPM would look better for it.
It would be nice if, in the uncollapsed treeview, the 'Installed' column indicated 'Installed' (or not) for single packages. (some other text for partly-installed multi-packages like bgraBitmap)  It can be a bit of a fiddle expanding/collapsing the treeview all the time for simple info.

This project already exceeds early expectations..
Title: Re: Online Package Manager
Post by: GetMem on December 07, 2016, 10:26:34 am
Quote
re trunk: I'm reluctant to mess with my cross-compiling dev environment, unless you can suggest a way to upgrade to trunk in a simple way. (not fpcup)  I have the latest source of trunk - can it be patched in by simply merging with my existing setup somehow?
Since you're under windows, just download Tortoise SVN. To upgrade to trunk just press the "Update" button. Initially you have to do a SVN checkout though. This is why version control was introduced in the first place.

Quote
@GetMem: Just installed latest OPM and the updating goes flawlessly.  Thank you!
1. CryptINI updated OK√
2. I tried to install bgracontrols.  It flagged the dependency OK√ and started the install, but compilation errored out, and OPM correctly offered to cancel OK√
I'm glad it works. I need to fix a few bugs though.

Quote
I find that what is missing in OPM is an 'UnInstall' button for (an) installed package(s). I know the IDE can do this, but OPM would look better for it.
First of all there is no interface implemented yet for uninstalling packages, somebody has to do it first. Secondly OPM was not meant to replace all the functionality of "Install/Uninstall packages" dialog, just a quick way to (re)install your favorite packages.
Quote
It would be nice if, in the uncollapsed treeview, the 'Installed' column indicated 'Installed' (or not) for single packages. (some other text for partly-installed multi-packages like bgraBitmap)  It can be a bit of a fiddle expanding/collapsing the treeview all the time for simple info
Ok. This is a good idea. I will implement it today.
Title: Re: Online Package Manager
Post by: minesadorada on December 07, 2016, 11:02:03 am
Since you're under windows, just download Tortoise SVN. To upgrade to trunk just press the "Update" button. Initially you have to do a SVN checkout though. This is why version control was introduced in the first place.
A quick follow-up (I have Tortoise svn)
1. What is the best svn repository URL for trunk?
Title: Re: Online Package Manager
Post by: GetMem on December 07, 2016, 11:13:25 am
Quote
1. What is the best svn repository URL for trunk?
http://svn.freepascal.org/svn/lazarus/trunk
Title: Re: Online Package Manager
Post by: minesadorada on December 07, 2016, 01:04:43 pm
All good now.
JsonEditor to V0.1.18.0
- Compiled with Trunk (form.scaled=true)
- .po files are now bundled into the exe as resources and unpack on first run. 2 methods of packing them at compile time depending on Laz version.
  - Only the executable needs to be deployed, and it goes to any folder with write access
  - Bugfix from V0.1.17.0 - Linux locale creation fixed
- Updated Windows installer (https://sourceforge.net/projects/lazautoupdate/files/otherpackages/setup_jsoneditor.exe (https://sourceforge.net/projects/lazautoupdate/files/otherpackages/setup_jsoneditor.exe))
Title: Re: Online Package Manager
Post by: minesadorada on December 07, 2016, 04:33:48 pm
@lainz I need an enormous favour.
When learning about i8n I committed the cardinal error of editing the .po files in Windows Notepad.  This messed up the .po files so they could no longer be edited in my BetterPo editor.  Lesson learned!

Since then, I have installed PoEdit, and rebuilt the jsoneditor.en.po and jsoneditor.es.po but using the machine translation to Spanish (which is less than ideal)

Now the .po files are correct, and POChecker also validates them.  I also found a way to include and unpack them as resources in Windows and Linux (which was a bonus)

The enormous favour is.. can you fix my awful Spanish translations in (attached) jsonedit.es.po just one last time please?   I would totally understand if you refused, as it was totally my fault your previous work was lost.  If it's any comfort, I've spent hours trying to fix my own mistake already.
Title: Re: Online Package Manager
Post by: lainz on December 07, 2016, 04:59:06 pm
Sure, I can translate that.

Edit: here it is. An awesome feature of Poedit is that it remembers the translations you did before, so I was just clicking on the already translated stuff I did before and it get's done in a few minutes. Of course there was new translations but these was easy too.

There is something not translated automatically and these are the headers of the TStringGrid. "lpk filename" and "version (n.n.n.n)". You maybe want to assign the translations by code.
Title: Re: Online Package Manager
Post by: minesadorada on December 07, 2016, 05:43:17 pm
Sure, I can translate that.

Edit: here it is. An awesome feature of Poedit is that it remembers the translations you did before, so I was just clicking on the already translated stuff I did before and it get's done in a few minutes. Of course there was new translations but these was easy too.

There is something not translated automatically and these are the headers of the TStringGrid. "lpk filename" and "version (n.n.n.n)". You maybe want to assign the translations by code.
@lainz  Thanks a million!
I feel suitably stupid for making unnecessary work for you.  I curse Bill Gates's stupid 'NotePad/SaveAs UTF8'!

In the meantime, I've learned a lot about Lazarus i8n deployment, including embedding .po files in the executable, using portable CFG files and and also using poedit and POChecker to ensure things are 'right' for deploying different languages with a runtime menu.

@all
Now I'm looking for German, French and Mandarin translations :)   Anyone?

If you are a package maintainer, I think jsoneditor is a good, user-friendly utility to make OPM-compliant update jsons.  How can I spread the word, now that I've put a disproportionate effort into making it?
Windows: https://sourceforge.net/projects/lazautoupdate/files/otherpackages/setup_jsoneditor.exe (https://sourceforge.net/projects/lazautoupdate/files/otherpackages/setup_jsoneditor.exe)
Title: Re: Online Package Manager
Post by: GetMem on December 07, 2016, 07:35:39 pm
Quote
@lainz
I hope you can talk with the lazarus developers (well, with the others, you are one of them now) and include this package as one of the pre-installed with Lazarus, so with that it will become more visible to all developers that open the package menu in the IDE, and more people will use it.
I don't think is possible since OPM depends on VirtualStringTree and third party components are not added to the pre-installed packages.

1. I implemented @minesadorada's request(screenshot)
Quote
It would be nice if, in the uncollapsed treeview, the 'Installed' column indicated 'Installed' (or not) for single packages. (some other text for partly-installed multi-packages like bgraBitmap)  It can be a bit of a fiddle expanding/collapsing the treeview all the time for simple info
2. The update is only visible if the package is installed. I hide it because a direct update, without install, although possible is not recommended. The update may fail if dependencies are not found.
3. Added Help button which points to OPM wikipage
4. Fixed a few bugs.

Please test(r 53591)!

We should add the LastNotifyDate(TDateTime) to the JSON + remove name.zip because it's not needed, just the name:
Quote
{
  "UpdatePackageData" : {
    "DownloadURL" : "https://sourceforge.net/projects/lazautoupdate/files/otherpackages/cryptini.zip",
    "ForceNotify" : false,
   "LastNotifyDate" : 4.2689000000000000E+004, <--add this
    "Name" : "cryptini"                         <--cryptini instead of cryptini.zip
  },
  "UpdatePackageFiles" : [
    {
      "Name" : "cryptini.lpk",
      "Version" : "0.1.1.0"
    }
  ]
Title: Re: Online Package Manager
Post by: lainz on December 07, 2016, 09:04:52 pm
@GetMem
Don't worry, maybe there's another way of making it popular. Do you have any kind of statistics of the downloads of each package? Or that need to be implemented in the server?
Title: Re: Online Package Manager
Post by: Rayvenhaus on December 07, 2016, 10:41:24 pm
Has this package moved? The link in the first message gives me a 404 error at github.
Title: Re: Online Package Manager
Post by: lainz on December 07, 2016, 10:51:55 pm
Yes it has been moved to Lazarus svn. See the wiki for the links.
http://wiki.freepascal.org/Online_Package_Manager
Title: Re: Online Package Manager
Post by: wp on December 08, 2016, 09:26:57 am
@GetMem: I am currently converting some more components of the jvcllaz project in ccr (JVCL components). For easier conversion I'll stick to the package structure of the JVCL Delphi version. This results in a multitude of packages which have to be compiled/installed in a given order.

How does OPM know the order in which packages must be compiled/installed?
Does OPM "know" that runtime packages only have to be compiled, while designtime packages must be installed?
Title: Re: Online Package Manager
Post by: GetMem on December 08, 2016, 09:57:07 am
@lainz
Quote
Don't worry, maybe there's another way of making it popular
I'm not worried at all. OPM it's just a tool, so nothing to worry about.

Quote
Do you have any kind of statistics of the downloads of each package? Or that need to be implemented in the server?
I have to ask @Marc. He maintains the server on which the repository is stored.

@wp
Quote
How does OPM know the order in which packages must be compiled/installed?
Recursively checks for dependencies both in central repository and  locally installed packages.  It will warn you before install if an unresolved dependency is found(see screenshot). The install order is decided up on the dependencies. If you are interested about the details, please check:
 - procedure GetPackageDependencies(unit opkman_serializablepackages)
 - procedure StartInstall(unit opkman_installer)

Quote
Does OPM "know" that runtime packages only have to be compiled, while designtime packages must be installed?
Yes. Runtime packages are only compiled, designtime packages compiled, installed then the IDE is rebuild.

I'm almost finished the update feature. After that I need a link to an external json file. When a new update is available, all you have to do is increase the version number and update the source zip.
Title: Re: Online Package Manager
Post by: wp on December 08, 2016, 09:59:38 am
Clever!
Title: Re: Online Package Manager
Post by: GetMem on December 08, 2016, 10:14:15 am
Quote
I am currently converting some more components of the jvcllaz project in ccr (JVCL components). For easier conversion I'll stick to the package structure of the JVCL Delphi version.
This is great! I always wondered why such a good component is not converted to Lazarus? Which version exactly are you converting? The newer one are more focused on XE2+ support.
Title: Re: Online Package Manager
Post by: minesadorada on December 08, 2016, 12:46:10 pm
1. I implemented @minesadorada's request(screenshot)
2. The update is only visible if the package is installed. I hide it because a direct update, without install, although possible is not recommended. The update may fail if dependencies are not found.
3. Added Help button which points to OPM wikipage
4. Fixed a few bugs.
Looks good.

We should add the LastNotifyDate(TDateTime) to the JSON + remove name.zip because it's not needed, just the name:
Quote
{
  "UpdatePackageData" : {
    "DownloadURL" : "https://sourceforge.net/projects/lazautoupdate/files/otherpackages/cryptini.zip (https://sourceforge.net/projects/lazautoupdate/files/otherpackages/cryptini.zip)",
    "ForceNotify" : false,
   "LastNotifyDate" : 4.2689000000000000E+004, <--add this
    "Name" : "cryptini"                         <--cryptini instead of cryptini.zip
  },
  "UpdatePackageFiles" : [
    {
      "Name" : "cryptini.lpk",
      "Version" : "0.1.1.0"
    }
  ]
I'd like to keep the '.zip' in if possible.  In jsoneditor it acts as a redundancy check on the URL - which is an easy field to get wrong.  If you think it's important, then I suppose the GUI could insist on the .zip but only store the name part in the json, but this would make it inconsistent for manual editing.

With the LastNotifyDate entry, I have a couple of issues:
1. What if I upload a json with LastNotifyDate set for 10 years from now? As I understand, OPM will keep polling the json but be unable to change the 'bad' entry.
2. With the date in TDateTime format (which it has to be), it becomes difficult and error-prone to manually edit the json file (which some users will do)

I assume what is wanted is really an 'expire-by' date (days).  If so, then it's probably better to store the current date (internally) and let the user specify a number of days for the NotifyUpdate to apply.  (The GUI can calculate and display the ExpiryDate for the user to confirm) This can be stored as a simple integer field.  If OPM can parse the filedate of the json, then it doesn't even need the problematical LastNotifyDate TDateTime field - only the ExpiryDays one.
Title: Re: Online Package Manager
Post by: GetMem on December 08, 2016, 01:50:22 pm
Quote
I'd like to keep the '.zip' in if possible.  In jsoneditor it acts as a redundancy check on the URL - which is an easy field to get wrong.  If you think it's important, then I suppose the GUI could insist on the .zip but only store the name part in the json, but this would make it inconsistent for manual editing.
The *.zip is not used internally by OPM, if it's useful for you then feel free to keep it.

Quote
With the LastNotifyDate entry, I have a couple of questions:
1. What if I upload a json with LastNotifyDate set for 10 years from now?
2. With the date in TDateTime format (which it has to be), it becomes difficult and error-prone to manually edit the json file (which some users will do)

I assume what is wanted is really an 'expire-by' date (days).  If so, then it's probably better to store the current date (internally) and let the user specify a number of days for the NotifyUpdate to apply.  This can be stored as a simple integer field.  If OPM can parse the filedate of the json, then it doesn't even need the problematical TDateTime field - only the ExpiryDays one.
First of all the "ForceNotify" feature will be rarely used, secondly we need a system that prevents continuous alert messages even when ForceNotify is set to true.  Consider this:
1. A package maintainer modifies the source, without increasing the version number(it's not unusual)
2. To trigger an alert message on the users computer, he/she sets ForceNotify to true(the other valid alert method is to increase the version number, which do not apply here)
3. OPM alerts the user, the package is downloaded/installed through the update feature
4. Now comes the confusing/annoying part, although the source is updated, the user will receive continuous alert messages since ForceNotify is true. To prevent this, I want to add a TDate/TDateTime field, which represents the date when the last update was released. After the update is done, OPM remembers internally the date, preventing further messages until the date changes again. We can replace the date with an integer, the effect would be the same. You're right about the TDate/TDatetime, it's very difficult to edit manually in the JSON file, however is more suggestive then an integer value, besides why did you developed the jsoneditor? To handle dates in the json  8). So what's gonna be: integer or date?
Title: Re: Online Package Manager
Post by: minesadorada on December 08, 2016, 02:24:27 pm
OK thanks for the explanation.  I'd made the wrong assumption that the user was setting the TDateTime.
Why can't OPM read the FileDate of the downloaded json and use that?  Am I missing something?
Title: Re: Online Package Manager
Post by: lainz on December 08, 2016, 02:27:03 pm
I think better is an integer. What for example I use is an integer with year-month-day, so today will be 20161208, tomorrow 20161209 and is easy to write. In the GUI editor just pick a date and then convert to this is really easy.

The time I don't know if it's neccessary, I think updating once a day is most that all users will do, more times in a day is not a good thing.

Edit: or just a regular int to increase. number of notify: 0, 1, 2, 3, 4, and so on.
Title: Re: Online Package Manager
Post by: GetMem on December 08, 2016, 03:15:19 pm
@minesadorada
Quote
Why can't OPM read the FileDate of the downloaded json and use that?  Am I missing something?
The JSON never "touch" the hard disk, it's loaded directly into a memory stream.

Quote
@lainz
Edit: or just a regular int to increase. number of notify: 0, 1, 2, 3, 4, and so on.
I like this better. What about a meaningful name? InternalVersion is ok?
Title: Re: Online Package Manager
Post by: lainz on December 08, 2016, 03:38:03 pm
InternalVersion is ok. The most important thing along with the name is the hint you can give to the package mantainer on what it does, and that is done in the wiki or in the json maker minesadorada keeps updating.
Title: Re: Online Package Manager
Post by: wp on December 08, 2016, 05:36:11 pm
Quote
I am currently converting some more components of the jvcllaz project in ccr (JVCL components). For easier conversion I'll stick to the package structure of the JVCL Delphi version.
This is great! I always wondered why such a good component is not converted to Lazarus? Which version exactly are you converting? The newer one are more focused on XE2+ support.
I am continuing with the version with which the initial porter started. Fortunately I found this in my archive, it is v 3.33, and I can install it in my Delphi 7 to see how things should work.

I was motivated by a bug report by Michał Gawrycki who converted the Validators and the HTcontrols (HT = hypertext) - this is really nice work done by him. I saw that the XPControls did not have a demo, and so I decided to continue with this package. XPControls are almost complete now, but this is more an exercise than an important addition since nobody cares about XP controls any more nowadays.

The currently ported components work in Windows and Linux. The JvNavigationPane still lacks the component editor for adding pages, i.e. you must use runtime code to add pages (like in the demo). 

I changed the folder structure in the ccr folder to facilitate adapting other JVCL components. And I also separated runtime and designtime code into separate packages. So, users of the JVCL components forgive me because this breaks your code - sorry. But it effectively avoids compilation issues due to the same compiled units found by several packages.

But conversion is quite a pain because numerous classes and units are involved before you can test a component on the component palette and because the code is heavily loaded with Windows-stuff. After being unsuccessful with the OfficeColorButton today after two days of work I don't know if I'll spend more time on this...
Title: Re: Online Package Manager
Post by: minesadorada on December 08, 2016, 07:16:54 pm
InternalVersion is ok. The most important thing along with the name is the hint you can give to the package mantainer on what it does, and that is done in the wiki or in the json maker minesadorada keeps updating.
So what is the consensus?  I can update jsoneditor on Sunday once I know:
1. Which new fields in the json
2. What can the GUI user visually change?
3. Validation rules for (2)
4. Hint text for (2)
Title: Re: Online Package Manager
Post by: lainz on December 08, 2016, 08:37:53 pm
Add InternalVersion that holds an arbitrary integer set by the package mantainer. Add a spin edit in the GUI. Is arbitrary, so there's no validation, only it must be an integer. The hint text is 'Increase internal version when you have made changes in the source. It  works only combined with ForceNotify.'
Title: Re: Online Package Manager
Post by: JuhaManninen on December 09, 2016, 12:10:55 am
HtmlViewer master project is ported to FPC/Lazarus.
 https://github.com/BerndGabriel/HtmlViewer
I realized this from a recent bug report:
 http://bugs.freepascal.org/view.php?id=31075
Should it be included in the repository? CCR only has an old version which maybe should be removed.
Title: Re: Online Package Manager
Post by: GetMem on December 09, 2016, 05:41:25 am
@wp
Quote
The currently ported components work in Windows and Linux.
After OPM is finished, I would like to contribute by porting those components to OSX. 

@minesadorada
See @lainz answer.

The update feature soon will be ready. The last missing, major feature is the voting system. Some of the reply on this thread suggest, that rating packages would be beneficial. What do you think about this?
My thoughts:
  - each package can have from 1 to 5 starts
  - a user can vote only once/package(we need a login system for this, it can be used later to directly update the main repository)
  - a vote can be later edited/modified
  - OPM will display an avarage for each package
Implementing the feaute will take time, since it's a complex issue.

PS: I will remove the SVN/GIT item from the tree, since the package maintainer now can push trunk version through the update feature.
 
Title: Re: Online Package Manager
Post by: molly on December 09, 2016, 06:06:03 am
Hi GetMem,

The voting as i envisioned it had more to do with the functionality of the package.

- does the package install correctly or are there any issues encountered
- does the package work/compile for specific targets or only for some of them
- is the package well maintained or does it take ages for reported bugs to be fixed
- etc.

The idea behind that, was that anyone can offer his/her package but users are able to provide information about the package so that good well written packages/components have perhaps a better chance to get integrated into ccr or similar (simply because they get noticed)

imho there is no use whatsoever to use a ranking system when all we get is response from those that are happy enough the package exist at all and support only one single platform (how well written it might be, and of course with exception of those that are only capable of targeting one platform).

It would also allow for newcomers to take a shot at things and get some positive feedback and packages might be able to get a classification (new, in development, official, etc)

Not a big issue if something like that would be missing but could perhaps help in getting things more automated on its own without too much human required interference when talking about OPM manager. It has nothing to do with the manager itself in that regards so, i was not complaining  :)

Thank you for the tremendous and excellent job already done.
Title: Re: Online Package Manager
Post by: minesadorada on December 09, 2016, 10:02:39 am
Add InternalVersion that holds an arbitrary integer set by the package mantainer. Add a spin edit in the GUI. Is arbitrary, so there's no validation, only it must be an integer. The hint text is 'Increase internal version when you have made changes in the source. It  works only combined with ForceNotify.'
I found an hour to do this this morning.
jsoneditor to V0.1.19.0 in svn ccr
Windows installer at https://sourceforge.net/projects/lazautoupdate/files/otherpackages/setup_jsoneditor.exe/download
Please check.
Title: Re: Online Package Manager
Post by: JuhaManninen on December 09, 2016, 11:11:39 am
The last missing, major feature is the voting system. Some of the reply on this thread suggest, that rating packages would be beneficial. What do you think about this?
My thoughts:
  - each package can have from 1 to 5 starts
  - a user can vote only once/package(we need a login system for this, it can be used later to directly update the main repository)
1 to 5 stars I guess. Sounds good.
Earlier you wrote that this forum user account could be used for authentication. Is it still the plan?
In any case you must discuss with the server admin about this SW and its needs.

Quote
  - a vote can be later edited/modified
  - OPM will display an avarage for each package
A good point, a user must be able to change his vote as the package improves or his opinion changes.
The average must show decimal stars.

- does the package install correctly or are there any issues encountered
- does the package work/compile for specific targets or only for some of them
- is the package well maintained or does it take ages for reported bugs to be fixed
- etc.

The idea behind that, was that anyone can offer his/her package but users are able to provide information about the package so that good well written packages/components have perhaps a better chance to get integrated into ccr or similar (simply because they get noticed)
A comment, together with the rating, should be supported. There a user can explain his reasoning.
No, packages don't need to be "integrated into ccr". CCR is just one hosting place among others.
Through voting/rating a new user gets an idea of a package's quality. Other classification like supported platforms is already done by its author.
Remember, a package can be single-platform by definition, for example some MS Office OLE-automation package.
Title: Re: Online Package Manager
Post by: minesadorada on December 09, 2016, 11:32:02 am
It would be useful if awarding stars meant the same thing for everyone:
i.e.
1 = Unuseable. Buggy and/or obselete
2 = Compiles/Installs, but has many unfixed bugs
3 = OK, but poor support for fixes and/or updates
4 = OK, but unsuitable for production
5 = Perfection!
Title: Re: Online Package Manager
Post by: molly on December 09, 2016, 12:16:06 pm
Remember, a package can be single-platform by definition, for example some MS Office OLE-automation package.
Of course. I tried to cover that in my writings but apparently that didn't came across as such   :-[
Title: Re: Online Package Manager
Post by: lainz on December 09, 2016, 01:06:20 pm
Hi GetMem, I've updated the external json of bgracontrols and bgracontrolsfx with the changes minesadorada made to the editor (InternalVersion).
Title: Re: Online Package Manager
Post by: minesadorada on December 09, 2016, 02:49:18 pm
Hi GetMem, I've updated the external json of bgracontrols and bgracontrolsfx with the changes minesadorada made to the editor (InternalVersion).
Update CryptINI jsons also updated (including cryptini_force.json, with InternalVersion > 1)
Title: Re: Online Package Manager
Post by: Rayvenhaus on December 09, 2016, 04:44:33 pm
I'm curious. Why does the Online Package Manager show me that a version is out of date, but says that it is up to date?  As shown in the attached image, it shows me that bgrabitmappack, Installed Version is 9.2.1.0 and the repository version is 9.2.3.0, yet states that the package is up to date.

Title: Re: Online Package Manager
Post by: GetMem on December 09, 2016, 04:59:39 pm
@All
Thanks for the feedback. I will try two answer as soon as possible.

@Rayvenhaus
It's a bug. I will fix it soon.
Title: Re: Online Package Manager
Post by: minesadorada on December 10, 2016, 09:58:43 am
Added a Linux distribution to the compiled OPM Update jsonpackager (V0.1.19):

Zipfile of 32bit + 64-bit Linux versions
https://sourceforge.net/projects/lazautoupdate/files/otherpackages/linux_jsonpackager.zip/download (https://sourceforge.net/projects/lazautoupdate/files/otherpackages/linux_jsonpackager.zip/download)

Windows 32/64-bit setup
https://sourceforge.net/projects/lazautoupdate/files/otherpackages/setup_jsoneditor.exe/download (https://sourceforge.net/projects/lazautoupdate/files/otherpackages/setup_jsoneditor.exe/download)

Only the executable is needed and it can be located in any writable directory as all resources are bundled within.
Title: Re: Online Package Manager
Post by: minesadorada on December 10, 2016, 11:09:27 am
@GetMem - a suggestion

As part of the update/maintenance mechanism I'd like an extra field to temporarily disable users downloading the component (or better still make it invisible in OPM) Perhaps DisableInOPM=True/False? (default=false)

There's an entirely selfish reason for this, though I suspect the same situation may happen to other package maintainers in the future.

I have a component (LazAutoUpdate) that has become dysfunctional due to updated SourceForge download protocols, and another (PoweredBy) that has been adversely affected by recent changes to Lazarus trunk.

I'd like to update them both, but don't want users to download the 'old' versions until they are fixed, so in my update json I'd like to set DisableInOPM=True whilst I change and test them.  Once stable, I can set DisableInOPM=False and the component once again becomes available for users.

I think this would be a useful addition to the update spec - especially if a voting system is implemented, whereby an unfixed component could possibly be voted down whilst the maintainer is in the process of fixing it.

Is this possible?
Title: Re: Online Package Manager
Post by: GetMem on December 10, 2016, 04:37:48 pm
First of all sorry for the delay, I was very busy lately. I just commited the latest version of OPM, I believe the update feature now is fully functional(may contain a few bug though). I had to change the structure of the JSON one more time, but this is the last change(I promise). You can generate the JSON with OPM(see screenshot). The structure looks like this:
Quote
{
  "UpdatePackageData" : {
    "DownloadZipURL" : "",
    "Name" : "bgracontrols-master"
  },
  "UpdatePackageFiles" : [
    {
      "ForceNotify" : true,
      "InternalVersion" : 1,
      "Name" : "bgracontrols.lpk",
      "Version" : "4.3.1.0"
    },
    {
      "ForceNotify" : true,
      "InternalVersion" : 1,
      "Name" : "bgrapascalscriptcomponent.lpk",
      "Version" : "0.0.0.0"
    }
  ]
}
It make sense that each package has it's own ForceNotify, maybe the package maintainers only wants to update one of them not both.

@minesadorada
Please update your project, if you don't have time I will do it later. Again sorry for the inconvenience. Please also regenerate the json's for all your packages and send me links. Thanks.
Quote
As part of the update/maintenance mechanism I'd like an extra field to temporarily disable users downloading the component (or better still make it invisible in OPM) Perhaps DisableInOPM=True/False? (default=false)
Just rename the JSON temporary, OPM will reset every update data, like there is no update. I tested this a few times, works well.

@lainz
Please do the same for all your packages. I will update the main json today, so we can run a few tests. Thanks.

@Rayvenhaus
I fixed the bug. Please update to the latest version.
Title: Re: Online Package Manager
Post by: lainz on December 10, 2016, 04:59:16 pm
I updated both bgracontrols and bgracontrolsfx but seems that doesn't works "No valid download link found". You already implemented that? Also I'm not sure if the link must be like
Quote
https:\/\/github.com\/bgrabitmap\/bgracontrols\/archive\/master.zip
or like
Quote
https://github.com/bgrabitmap/bgracontrols/archive/master.zip
Title: Re: Online Package Manager
Post by: LuizAmérico on December 10, 2016, 05:07:25 pm
Quote
Also I'm not sure if the link must be like

Both are right see discussion: http://bugs.freepascal.org/view.php?id=30870
Title: Re: Online Package Manager
Post by: GetMem on December 10, 2016, 06:30:12 pm
As @Luiz said both are valid. I choosed the second one, because it's more easy to edit with a text editor. Let me check what is wrong.
Title: Re: Online Package Manager
Post by: lainz on December 10, 2016, 07:13:56 pm
As @Luiz said both are valid. I choosed the second one, because it's more easy to edit with a text editor. Let me check what is wrong.

Done, I've changed it to the second style. Maybe related with HTTPS? There's no need of dll's to download from secure sites?

Edit: can't be https, because it says is a problem in the URL, also it downloads the json, else it will not know that there's a new version. BTW I keep the second question, wich package you use to download from secure sites with no need of extra dll's?
Title: Re: Online Package Manager
Post by: GetMem on December 10, 2016, 07:44:18 pm
It was a stupid mistake of mine. I left a "/" in a wrong please. I spent half an hour debugging the code.  :-[
Anyway please update to trunk then try again. Wait 1-2 min. until the url's are updated with the correct values. You can also delete the config file for updates, just to make sure:
   C:\Users\%user%\AppData\Local\lazarus\onlinepackagemanager\config\updates.xml
After the update is done, please increment the InternalVersion value, then test if OPM displays the "NEW" value.

PS: I did noticed that sometimes files on github cannot be downloaded. FPHTTPClient returns an error, but if I try to download again everything works fine. This has nothing to do with the above issue, which was clearly a bug.
Title: Re: Online Package Manager
Post by: lainz on December 10, 2016, 07:48:58 pm
I must wait to be synchronized with the lazarus trunk on GitHub first to test.

About the download from GitHub, I've noticed that also.

So you're using only FPHTTPClient for secure downloads? That doesn't need some dll's in the executable folder to work?

Edit: tested and working well.
Title: Re: Online Package Manager
Post by: GetMem on December 10, 2016, 08:23:43 pm
Quote
So you're using only FPHTTPClient for secure downloads? That doesn't need some dll's in the executable folder to work?
The main repository is http, so we only need the OpenSSL library for external updates. Under windows I have copied the files into the system32 directory. Under linux/osx everything works fine, probably *.so, *.dylib files are preinstalled.

I will add the info to the wiki. Anyway no matter what component we use(indy, synapse,  FPHTTPClient) for https those libraries are needed anyways.
Title: Re: Online Package Manager
Post by: lainz on December 10, 2016, 08:42:26 pm
That's ok.

I was asking because I though that I don't have these dll's installed. The reality is that I have a lot of them (doing a search with windows search was not enough, I downloaded voidtools everything search tool to find them), maybe some are in PATH folders, I have these dll's in software like Git, fpcupdeluxe, photoshop, easeus partition master, foxit reader, intel stuff, graphic tablet stuff, and some other applications.

What a problem, I don't even know wich version is running when I use your tool. That's the called 'dll hell' or something like that  :D
Title: Re: Online Package Manager
Post by: GetMem on December 10, 2016, 09:39:33 pm
Quote
What a problem, I don't even know wich version is running when I use your tool. That's the called 'dll hell' or something like that  :D
Yeah!  :D  Good you mentioned though, because I already forget it. Perhaps we should store those dll's on the main repository and download when needed.
Title: Re: Online Package Manager
Post by: lainz on December 10, 2016, 10:16:41 pm
Is something like Fpcupdeluxe does, it downloads the dll's and place in them in the right folder.

The main problem is that the lazarus folder is not always writable?

Edit: I attach some of the NetBeans Plugins window, something like the OPM.
Title: Re: Online Package Manager
Post by: GetMem on December 11, 2016, 11:31:05 am
@lainz
Quote
Is something like Fpcupdeluxe does, it downloads the dll's and place in them in the right folder.
The main problem is that the lazarus folder is not always writable?
When you install a component, Lazarus must rebuild itself so the folder is always writable(at least it shoud be). I added ssl support in r53631. Yet another directory with the same dll's.  :D Perhaps is redundant for us, but for those who don't have any ssl library installed is mandatory, without the dlls OPM would crash on update.
I also fixed that strange download error(github), since then I never saw the "range check"/"chunk to big"  errors, although I downloaded the zip files more then 20 times. Please test.

Quote
Edit: I attach some of the NetBeans Plugins window, something like the OPM.
Looks great and gives me some ideas about the interface. For example I check for updates in every 6 sec. It's too often, perhaps once/day is more then enough.
Title: Re: Online Package Manager
Post by: minesadorada on December 11, 2016, 12:35:01 pm
It make sense that each package has it's own ForceNotify, maybe the package maintainers only wants to update one of them not both.

@minesadorada
Please update your project, if you don't have time I will do it later. Again sorry for the inconvenience. Please also regenerate the json's for all your packages and send me links. Thanks.
NP. I'll get on it.

Quote
As part of the update/maintenance mechanism I'd like an extra field to temporarily disable users downloading the component (or better still make it invisible in OPM) Perhaps DisableInOPM=True/False? (default=false)
Just rename the JSON temporary, OPM will reset every update data, like there is no update. I tested this a few times, works well.
That would just affect whether an update is shown.  I was talking about temporarily changing the visibility of the component itself in OPM.
Title: Re: Online Package Manager
Post by: GetMem on December 11, 2016, 01:00:29 pm
@minesadorada
Quote
NP. I'll get on it.
Thank you!  :)

Quote
That would just affect whether an update is shown.  I was talking about temporarily changing the visibility of the component itself in OPM.
I can do it(in fact all I have to do is set the node visibility to false) but is confusing for the user. From the users point of view a package appears/disappears randomly, without any explanation. It looks like a bug. Are you sure this is what we want?
Title: Re: Online Package Manager
Post by: minesadorada on December 11, 2016, 01:12:16 pm
Quote
That would just affect whether an update is shown.  I was talking about temporarily changing the visibility of the component itself in OPM.
I can do it(in fact all I have to do is set the node visibility to false) but is confusing for the user. From the users point of view a package appears/disappears randomly, without any explanation. It looks like a bug. Are you sure this is what we want?
It would be an unusual and exceptional circumstance (which I explained in the OP) so the user won't see components appearing/disappearing randomly on a regular basis, or even occasionally.  But I do think it's a feature that could be important to a package maintainer for the reasons I gave.  If a serious bug is found in an OPM package (or external factors render it obselete)  do you think it's a good idea for users to keep downloading it whilst the maintainer fixes it?
I don't think it's an update feature that would be used often (like ForceNotify), but can we build it into the spec anyway?
Title: Re: Online Package Manager
Post by: GetMem on December 11, 2016, 01:22:34 pm
Quote
I don't think it's an update feature that would be used often (like ForceNotify), but can we build it into the spec anyway?
Ok. Please add it to your project JSON structure, I will add it to OPM:
Quote
{
  "UpdatePackageData" : {
    "DownloadZipURL" : "",
    "DisableInOPM" : false,
    "Name" : "bgrabitmap-master"
  },
  "UpdatePackageFiles" : [
    {
      "ForceNotify" : false,
      "InternalVersion" : 1,
      "Name" : "bglcontrols.lpk",
      "Version" : "1.1.0.0"
    },
    {
      "ForceNotify" : false,
      "InternalVersion" : 1,
      "Name" : "bgrabitmappack.lpk",
      "Version" : "9.2.3.0"
    }
  ]
}
I will implement the functionality soon(in worst case tomorrow).
Title: Re: Online Package Manager
Post by: minesadorada on December 11, 2016, 01:41:02 pm
@Getmem Thanks for the addition.  Redesigning the GUI for the new per-lpk structure is a bit fiddly - I'll refactor it I think.
Here are the Package Objects
Code: Pascal  [Select]
  1.   { TUpdatePackageFiles }
  2.  
  3.   TUpdatePackageFiles = class(TCollectionItem)
  4.   private
  5.     FName: string;
  6.     FVersion: string;
  7.     FForceNotify: boolean;
  8.     FInternalVersion: integer;
  9.   published
  10.     property Name: string read FName write FName;
  11.     property Version: string read FVersion write FVersion;
  12.     property ForceNotify: boolean read FForceNotify write FForceNotify;
  13.     property InternalVersion: integer read FInternalVersion write FInternalVersion;
  14.   end;
  15.  
  16.   TPackageFilesList = specialize TGenericCollection<TUpdatePackageFiles>;
  17.  
  18.   { TUpdatePackageData }
  19.  
  20.   TUpdatePackageData = class(TPersistent)
  21.   private
  22.     FDownloadZipURL: string;
  23.     FDisableInOPM: Boolean;
  24.     FName: string;
  25.   public
  26.     constructor Create;
  27.   published
  28.     property Name: string read FName write FName;
  29.     property DownloadZipURL: string read FDownloadZipURL write FDownloadZipURL;
  30.     property DisableInOPM:Boolean read FDisableInOPM write FDisableInOPM;
  31.   end;
  32.  
  33.   { TUpdatePackage }
  34.  
  35.   TUpdatePackage = class(TPersistent)
  36.   private
  37.     FUpdatePackageData: TUpdatePackageData;
  38.     FUpdatePackageFiles: TPackageFilesList;
  39.   public
  40.     constructor Create;
  41.     destructor Destroy; override;
  42.     function LoadFromFile(AFileName: string): boolean;
  43.     function SaveToFile(AFileName: string): boolean;
  44.   published
  45.     property UpdatePackageData: TUpdatePackageData
  46.       read FUpdatePackageData write FUpdatePackageData;
  47.     property UpdatePackageFiles: TPackageFilesList
  48.       read FUpdatePackageFiles write FUpdatePackageFiles;
  49.   end;
  50.  

Meanwhile I'll manually update the cryptini, cryptini_force jsons to the new format.  Also I'll make a update_lazautoupdate.json to test the DisableInOPM member (no rush!)

=edit= update_cryptini.json and update_cryptini_force.json both uploaded in new format (incl. DisableInOPM=false)
Title: Re: Online Package Manager
Post by: JuhaManninen on December 11, 2016, 02:25:27 pm
I can do it(in fact all I have to do is set the node visibility to false) but is confusing for the user. From the users point of view a package appears/disappears randomly, without any explanation. It looks like a bug. Are you sure this is what we want?
If this is really needed then at least don't hide the package but mark it clearly as disabled by its author.
Make its text font grey or use other colors to make it distinctive. Otherwise we will get bug reports and for a reason as it would be a confusing feature.
Title: Re: Online Package Manager
Post by: minesadorada on December 11, 2016, 02:41:40 pm
I can do it(in fact all I have to do is set the node visibility to false) but is confusing for the user. From the users point of view a package appears/disappears randomly, without any explanation. It looks like a bug. Are you sure this is what we want?
If this is really needed then at least don't hide the package but mark it clearly as disabled by its author.
Make its text font grey or use other colors to make it distinctive. Otherwise we will get bug reports and for a reason as it would be a confusing feature.

OK.  That suggestion makes sense - as long as the download/install functionality is disabled.
Title: Re: Online Package Manager
Post by: GetMem on December 11, 2016, 04:57:41 pm
@Juha
Quote
If this is really needed then at least don't hide the package but mark it clearly as disabled by its author.
Make its text font grey or use other colors to make it distinctive. Otherwise we will get bug reports and for a reason as it would be a confusing feature.
Good idea! I must add the info to the wiki. OPM now has a help button, when pressed the wiki page opens.

@minesadorada
Please try to increase the version number(cryptini) in the json to trigger a message alert.
Title: Re: Online Package Manager
Post by: Rayvenhaus on December 11, 2016, 05:16:33 pm
I see that the repository states that there is an update to the AutoUpdate component, yet I get the error shown in the attached image when I try to update. I'm also getting it on some other components shown in the OPM.
Title: Re: Online Package Manager
Post by: minesadorada on December 11, 2016, 05:24:23 pm
@minesadorada
Please try to increase the version number(cryptini) in the json to trigger a message alert.
I'm not sure if you mean Version or InternalVersion. Just attach the cryptini jsons you want for testing, and I'll upload them.
Title: Re: Online Package Manager
Post by: GetMem on December 11, 2016, 05:26:38 pm
@Rayvenhaus
To install a package from repository use the "Install" button.
To update an already installed package from the package maintainers webpage, use the update button.

It may sound a little bit confusing, but in the beginning there only was a "Repository" column and the "Install" button. To allow more freedom for the package maintainers, later the update feature was added("Update" button and "Update" column). By modifying an external JSON file, the package maintainers can trigger an alert/update message, and give the user the possibility to update to the latest version directly from external site(no central repository involved). Soon I will add all the info to the wiki:
http://wiki.freepascal.org/Online_Package_Manager
Title: Re: Online Package Manager
Post by: GetMem on December 11, 2016, 05:31:27 pm
@minesadorada
Quote
I'm not sure if you mean Version or InternalVersion. Just attach the cryptini jsons you want for testing, and I'll upload them.
Internal version must be used in combination with ForceNotify. We all agreed, that you don't like the forcenotify option and is set to false(cryptini). The other valid trigger method is to increase the version number. So please increase it in your json file and watch OPM for the new message(this is for testing purposes, you don't have to update to the new version). Alternatively leave the version as it is and set forcenotify to true. You should also see a message. In fact please try both methods.
Title: Re: Online Package Manager
Post by: GetMem on December 11, 2016, 06:17:29 pm
@minesadorada
Implemented the feature you requested in r53637. It looks like this(see screenshot), just set DisableInOPM to true.
Quote
If this is really needed then at least don't hide the package but mark it clearly as disabled by its author.
Make its text font grey or use other colors to make it distinctive. Otherwise we will get bug reports and for a reason as it would be a confusing feature.

Please test.
Title: Re: Online Package Manager
Post by: minesadorada on December 11, 2016, 06:23:04 pm
Uploaded this json:
{
  "UpdatePackageData" : {
    "DownloadZipURL" : "https://sourceforge.net/projects/lazautoupdate/files/otherpackages/cryptini.zip (https://sourceforge.net/projects/lazautoupdate/files/otherpackages/cryptini.zip)",
    "DisableInOPM" : false,
    "Name" : "cryptini.zip"
  },
  "UpdatePackageFiles" : [
    {
      "ForceNotify" : false,
      "InternalVersion" : 1,
      "Name" : "cryptini.lpk",
      "Version" : "0.1.2.0"
    }
  ]
}

1. Updated OPM from trunk Rev 53636
2. Compiled/Installed in Laz Trunk
3. Started OPM
4. CryptINI version reported as 0.1.1.0 (correct)
*5. Waited awhile
6. CryptIni Update column correctly shows 'New' and 0.1.2.0 'New Updates available'

Uploaded this json:
{
  "UpdatePackageData" : {
    "DownloadZipURL" : "https://sourceforge.net/projects/lazautoupdate/files/otherpackages/cryptini.zip (https://sourceforge.net/projects/lazautoupdate/files/otherpackages/cryptini.zip)",
    "DisableInOPM" : false,
    "Name" : "cryptini.zip"
  },
  "UpdatePackageFiles" : [
    {
      "ForceNotify" : true,
      "InternalVersion" : 2,
      "Name" : "cryptini.lpk",
      "Version" : "0.1.1.0"
    }
  ]
}

3. Started OPM
4. CryptINI version reported as 0.1.1.0 (correct)
*5. Waited awhile (a couple of minutes)
6. CryptIni Update column correctly shows 'New' and 0.1.1.0 'New Updates available'

..So all looking good.

*An obscure bug:  After uploading, starting OPM and before step (5) if I clicked 'Refresh' OPM locked up at 'Parsing JSON' forcing a restart of Lazarus. Once the new info was showing, 'Refresh' works without problems.
Title: Re: Online Package Manager
Post by: minesadorada on December 11, 2016, 06:34:31 pm
Uploaded this JSON:
{
  "UpdatePackageData" : {
    "DownloadZipURL" : "https://sourceforge.net/projects/lazautoupdate/files/otherpackages/cryptini.zip (https://sourceforge.net/projects/lazautoupdate/files/otherpackages/cryptini.zip)",
    "DisableInOPM" : true,
    "Name" : "cryptini.zip"
  },
  "UpdatePackageFiles" : [
    {
      "ForceNotify" : false,
      "InternalVersion" : 1,
      "Name" : "cryptini.lpk",
      "Version" : "0.1.1.0"
    }
  ]
}

CryptINI column shows Version at 0.1.1.0 and Stratus as 'Up to date'  but not greyed out.

Uploaded this JSON:
{
  "UpdatePackageData" : {
    "DownloadZipURL" : "https://sourceforge.net/projects/lazautoupdate/files/otherpackages/cryptini.zip (https://sourceforge.net/projects/lazautoupdate/files/otherpackages/cryptini.zip)",
    "DisableInOPM" : true,
    "Name" : "cryptini.zip"
  },
  "UpdatePackageFiles" : [
    {
      "ForceNotify" : true,
      "InternalVersion" : 1,
      "Name" : "cryptini.lpk",
      "Version" : "0.1.1.0"
    }
  ]
}
Same result: CryptINI column shows Version at 0.1.1.0 and Update column as 'New'  but.. not greyed out.
Title: Re: Online Package Manager
Post by: minesadorada on December 11, 2016, 06:47:32 pm
This is the version currently uploaded for your testing.
{
  "UpdatePackageData" : {
    "DownloadZipURL" : "https://sourceforge.net/projects/lazautoupdate/files/otherpackages/cryptini.zip",
    "DisableInOPM" : true,
    "Name" : "cryptini.zip"
  },
  "UpdatePackageFiles" : [
    {
      "ForceNotify" : false,
      "InternalVersion" : 1,
      "Name" : "cryptini.lpk",
      "Version" : "0.1.1.0"
    }
  ]
}
Title: Re: Online Package Manager
Post by: GetMem on December 11, 2016, 07:29:21 pm
@minesadorada
Quote
Same result: CryptINI column shows Version at 0.1.1.0 and Update column as 'New'  but.. not greyed out.
The JSON looks ok, you probably did not update to the latest trunk.

Quote
*An obscure bug:  After uploading, starting OPM and before step (5) if I clicked 'Refresh' OPM locked up at 'Parsing JSON' forcing a restart of Lazarus. Once the new info was showing, 'Refresh' works without problems.
I fixed this in r. 53640.

Please update again to the latest version and test(both the refresh bug, and DisableInOPM). Thank you!
Title: Re: Online Package Manager
Post by: minesadorada on December 11, 2016, 07:41:27 pm
If you could update the build number of OPM it would save me some double-checking.
Here is the situation as is:
Title: Re: Online Package Manager
Post by: GetMem on December 11, 2016, 08:11:36 pm
You're right! I forget to commit something. Please try one more time: r. 53641
Title: Re: Online Package Manager
Post by: minesadorada on December 11, 2016, 08:15:12 pm
You're right! I forget to commit something. Please try one more time: r. 53641

All good now.   I'll upload with "DisableInOPM" : false and see if it comes back.
Title: Re: Online Package Manager
Post by: minesadorada on December 11, 2016, 08:29:16 pm
That's really odd.  I expected it to return to normal.

Here's the uploaded json and the result after 15 minutes and 'Refresh'.  Does it show the same for you?
BTW I appreciate the effort you're putting into this.  I had a quick look at the code and can see it's not a trivial change.
{
  "UpdatePackageData" : {
    "DownloadZipURL" : "https://sourceforge.net/projects/lazautoupdate/files/otherpackages/cryptini.zip (https://sourceforge.net/projects/lazautoupdate/files/otherpackages/cryptini.zip)",
    "DisableInOPM" : false,
    "Name" : "cryptini.zip"
  },
  "UpdatePackageFiles" : [
    {
      "ForceNotify" : false,
      "InternalVersion" : 1,
      "Name" : "cryptini.lpk",
      "Version" : "0.1.1.0"
    }
  ]
}
Title: Re: Online Package Manager
Post by: GetMem on December 11, 2016, 08:36:58 pm
Just double click the link(see image), the json is still:
Quote
{
  "UpdatePackageData" : {
    "DownloadZipURL" : "https://sourceforge.net/projects/lazautoupdate/files/otherpackages/cryptini.zip",
    "DisableInOPM" : true,
    "Name" : "cryptini.zip"
  },
  "UpdatePackageFiles" : [
    {
      "ForceNotify" : false,
      "InternalVersion" : 1,
      "Name" : "cryptini.lpk",
      "Version" : "0.1.1.0"
    }
  ]
}
Title: Re: Online Package Manager
Post by: minesadorada on December 11, 2016, 08:52:29 pm
It had to be something simple!  I'd uploaded the latest update_cryptini.json to the parent folder by mistake.

All good now.  Thanks!
Title: Re: Online Package Manager
Post by: GetMem on December 11, 2016, 09:01:20 pm
I'm glad it works. Please see opkman_updates.pas unit for the changes in JSON.
Title: Re: Online Package Manager
Post by: lainz on December 11, 2016, 09:34:58 pm
There's a new version of bgrabitmap 9.3 that you can update in the repository. I tell circular about the external json but I did not receive a response from him.

I think I will use it as is:
- If I do a new version I will disable the ForceNotify and just increase the Version
- If I do some new commits with fixes I will enable the ForceNotify and increase the InternalVersion

A thing I see:
- When you install for the first time is not allowed to download directly from the updates button.
Title: Re: Online Package Manager
Post by: GetMem on December 11, 2016, 10:06:12 pm
Quote
There's a new version of bgrabitmap 9.3 that you can update in the repository. I tell circular about the external json but I did not receive a response from him.
Ok. I will do it tomorrow(It's late now here).

Quote
I think I will use it as is:
- If I do a new version I will disable the ForceNotify and just increase the Version
- If I do some new commits with fixes I will enable the ForceNotify and increase the InternalVersion
This is correct! Please note: even when you set ForceNotify to false, don't decrease/reset the InternalVersion , because it's remembered locally by OPM, so when you enable again ForceNotify, the internal version from the external json must be greater then the local version(to trigger an update alert).

Quote
A thing I see:
- When you install for the first time is not allowed to download directly from the updates button.
You're right again. In order to update you must install the package first. I made this restriction to prevent unresolved dependencies error. For example, if you try to update bgracontrols directly from the external link and bgrabitmap is not installed the update will fail. After the package is installed you can update any time, if not then it's a bug.

PS: Please try to update a few times, because I believe I fixed that strange download/update bug(github). Can you reproduce it, with the latest trunk?
PS1: How often do you think OPM should scan for updates? Every few second, once per day?
Title: Re: Online Package Manager
Post by: lainz on December 11, 2016, 10:16:31 pm
Ok. I got it.

I can't reproduce the github bug, I've installed the packages several times with no problems.

Maybe just once a day is enough.

Edit: How you fixed the github bug? I need to fix it to for some of my applications that auto-updates from GitHub.
Title: Re: Online Package Manager
Post by: minesadorada on December 12, 2016, 09:22:59 am
Today I did a genuine update to CryptINI, and uploaded the update json and Zipfile

OPM soon notified the Update as New and showed the update version correctly.  I clicked 'Update' and the update downloaded and installed.  All good.

A minor niggle: In the updated lpk, I had altered the description field to describe an updated feature, but after the update OPM still shows the 'old' description.  Is this intended behaviour?
An update should update lpk Description and especially License fields (untested by me) in OPM - a developer may change the license conditions in an update which is important.
Title: Re: Online Package Manager
Post by: GetMem on December 12, 2016, 12:17:11 pm
@lainz
Quote
Edit: How you fixed the github bug? I need to fix it to for some of my applications that auto-updates from GitHub.
TFPHTTPClient sometimes raise an exception even when the operation was successful. You should catch the exception first, then check if the ResponseStatusCode differs or not from 200(OK). If it's 200, just ignore the exception and continue.

I updated BGRABitmaps to 9.3.

@minesadorada
Quote
A minor niggle: In the updated lpk, I had altered the description field to describe an updated feature, but after the update OPM still shows the 'old' description.  Is this intended behaviour?
An update should update lpk Description and especially License fields (untested by me) in OPM - a developer may change the license conditions in an update which is important.
The package description is stored on the main repository(the main JSON). After update OPM still reads the description/license from the main repository not from the updated lpk. This feature must be implemented.
Title: Re: Online Package Manager
Post by: minesadorada on December 12, 2016, 12:21:03 pm
The package description is stored on the main repository(the main JSON). After update OPM still reads the description/license from the main repository not from the updated lpk. This feature must be implemented.
OK. Let me know when it's implemented and I'll do a ForceNotify update for cryptini.

Refactoring jsoneditor to use Control Arrays for each lpk file. A learning experience that will take a couple of days to implement.  Using anchor properties for dynamically-created controls is the new learning experience.
Title: Re: Online Package Manager
Post by: GetMem on December 12, 2016, 02:19:37 pm
Quote
OK. Let me know when it's implemented and I'll do a ForceNotify update for cryptini.
Implemented in r. 53650. You don't have to use ForceNotify for this. OPM will automatically read the package info(license/description) from lpk, after an update/install.

Title: Re: Online Package Manager
Post by: minesadorada on December 12, 2016, 02:25:07 pm
Quote
OK. Let me know when it's implemented and I'll do a ForceNotify update for cryptini.
Implemented in r. 53650. You don't have to use ForceNotify for this. OPM will automatically read the package info(license/description) from lpk, after an update/install.


Wow - impressive.
Checks out 100% - well done and thanks!

Just a reminder - please update VersionInfo in the OnlinePackager lpk file.  It makes it much easier to test.
Title: Re: Online Package Manager
Post by: GetMem on December 12, 2016, 02:37:05 pm
@all,

I updated the wikipage(http://wiki.freepascal.org/Online_Package_Manager):
  - new information about the update feature: how to create the external JSON,  the meaning of each item in the JSON, etc...
  - what is the difference between download/install/update(see screenshot). A lot of people don't understand the subtle difference between download/install/update and for good reason: OPM is a hybrid between a central repository and external updates. There was a a lot of discussion about this subject and currently this is the most reasonable solution we came up with(it might be confusing though). I did my best to explain it in the picture. Please suggest new names, design improvements to make it more clear.

 
Title: Re: Online Package Manager
Post by: minesadorada on December 12, 2016, 02:41:34 pm
Perhaps a link to my GUI app when it's finished?
It's guaranteed to produce a usable update_json file that will be compatible with OPM.

I can update the wiki with information as I understand it, if that's OK.  Things can look very different between a developer and a user sometimes.
Title: Re: Online Package Manager
Post by: Rayvenhaus on December 12, 2016, 03:17:54 pm
When I look at the Package info on a control in the OPM, I can see a like to the controls homepage. Since I can not highlight and copy that link to go to the homepage, can the link be made to launch the default browser?
Title: Re: Online Package Manager
Post by: minesadorada on December 12, 2016, 03:32:29 pm
This looks like a regression.  It used to work as a live link.
Title: Re: Online Package Manager
Post by: GetMem on December 12, 2016, 04:04:42 pm
@Rayvenhaus
Just move the mouse over link: http://www.cityinthesky.co.uk...., the color should change to blue then double click it. If another behavior would be more logical...I'm open to suggestions.
Title: Re: Online Package Manager
Post by: Rayvenhaus on December 12, 2016, 04:21:13 pm
It's working as designed. I thought I was double-clicking, but apparently not. Thank you for the quick reply!
Title: Re: Online Package Manager
Post by: minesadorada on December 12, 2016, 04:35:06 pm
@Rayvenhaus
Just move the mouse over link: http://www.cityinthesky.co.uk (http://www.cityinthesky.co.uk)...., the color should change to blue then double click it. If another behavior would be more logical...I'm open to suggestions.
Suggestion: Single-click?  Most linux users would expect it, and Windows users just get 2 instances of the browser if they double-click :)   I was fooled.
Title: Re: Online Package Manager
Post by: GetMem on December 12, 2016, 05:17:39 pm
@minesadorada
Quote
Perhaps a link to my GUI app when it's finished?
It's guaranteed to produce a usable update_json file that will be compatible with OPM.
I can update the wiki with information as I understand it, if that's OK.
Yes, the link to your project its a must. I will add it when you're ready. In fact you can edit anytime you like. All you need is wiki account or perhaps you already have one? 

Quote
Things can look very different between a developer and a user sometimes.
True.

Quote
Suggestion: Single-click?
With single click you select the current row. Every selection would open a new browser window. Not good.  :D
Title: Re: Online Package Manager
Post by: minesadorada on December 13, 2016, 01:54:08 pm
Functional alpha of jsoneditor V0.2.0.0 available in svn ccr and
Windows InnSetup: https://sourceforge.net/projects/lazautoupdate/files/otherpackages/setup_jsoneditor.exe/download (https://sourceforge.net/projects/lazautoupdate/files/otherpackages/setup_jsoneditor.exe/download)

By 'functional', I mean New, Load and Save/Save As work, but the tooltips, validation and translation are unfinished. (It had a GUI makeover)
I might be useful for testing update_.json files in its current state.
Title: Re: Online Package Manager
Post by: GetMem on December 13, 2016, 02:30:48 pm
@minesadorada
You cannot increase the height of the form with every single packagefile added. You need a scrollbox instead(see attachment).
Even better, you should drop the whole TGroupBox array, all the info is already stored in JSONPackage(TUpdatePackage) class. On form show loop through the JSONPackage and add all items to a TListView(4/5 columns). When you want to edit one of the item, just open a new form and display the selected items info. After the form is closed, save data back to JSONPackage and update the tree.
Title: Re: Online Package Manager
Post by: minesadorada on December 13, 2016, 02:43:57 pm
@GetMem  I agree, a better design with the scrollbox.  I like the control arrays, but if its a big problem for you I can redesign.  Most components will only have 1 or 2 packages.

I'll revisit later - off to golf now :)
Title: Re: Online Package Manager
Post by: GetMem on December 13, 2016, 02:46:28 pm
Quote
I like the control arrays, but if its a big problem for you I can redesign.  Most components will only have 1 or 2 packages.
No problem at all. Feel free to keep it.
Title: Re: Online Package Manager
Post by: lainz on December 14, 2016, 04:02:44 pm
Here is attached the latest translation for the json editor.
Title: Re: Online Package Manager
Post by: minesadorada on December 14, 2016, 04:18:39 pm
Here is attached the latest translation for the json editor.
Looks great - thanks!
Title: Re: Online Package Manager
Post by: josh on December 15, 2016, 03:55:16 pm
Hi
I have tried to install KControld on osx and it fails to compile
kfunctions.pas(1489,23) Error Incompatible types: got TThreadID expected Longword
kfunctions.pas(2307,30) Error Incompatible types: got LongWord expected TThreadID

Is this just a osx issue of does kcontrols not compile on win/nix as well.

Title: Re: Online Package Manager
Post by: JuhaManninen on December 15, 2016, 04:51:51 pm
I have tried to install KControld on osx and it fails to compile
kfunctions.pas(1489,23) Error Incompatible types: got TThreadID expected Longword
kfunctions.pas(2307,30) Error Incompatible types: got LongWord expected TThreadID

Is this just a osx issue of does kcontrols not compile on win/nix as well.
It compiles fine on Linux.
Anyway kfunctions.pas uses wrong types. "TThreadID" should be used consistently when a Thread ID is needed.
See how TThreadID is defined in rtl/macos/sysosh.inc:
Code: Pascal  [Select]
  1. type
  2. {$ifdef CPU64}
  3.   THandle = Int64;  
  4. {$else CPU64}
  5.   THandle = Longint;
  6. {$endif CPU64}
  7.   TThreadID = THandle;

On Linux it is:
Code: Pascal  [Select]
  1. TThreadID = PtrUInt;

Please create a patch for the original project.

GetMem, do all package maintainers know their packages are available from OPM and that they should provide new versions?
Title: Re: Online Package Manager
Post by: GetMem on December 15, 2016, 05:44:26 pm
Quote
GetMem, do all package maintainers know their packages are available from OPM and that they should provide new versions?
No. At least third of the packages don't have a maintainer, some of the packages are maintained but the authors don't visit this forum. When I add a package to the main repository, I always check if it compiles under windows. However checking for every single platform would be a tremendous task(I cannot do this alone).
If somebody notice that a particular packages is not compiling, I can modify the supported widget set entry, or perhaps that person can fix the package.
Title: Re: Online Package Manager
Post by: JuhaManninen on December 15, 2016, 08:27:04 pm
No. At least third of the packages don't have a maintainer, some of the packages are maintained but the authors don't visit this forum.
For packages that have a maintainer, this package manager should be advertised.
If a maintainer does not take care of updates then this system will be outdated in future.
Everybody should now advertise this online package manager to his favorite package's author.
Title: Re: Online Package Manager
Post by: GetMem on December 16, 2016, 10:44:14 am
Quote
@Juha
For packages that have a maintainer, this package manager should be advertised.
For approximately 10 out of 41 packages the update system is up and running(the external JSON we talked about in previous posts). Later hopefully more package maintainer will join in.

I implemented voting system partially(r. 53697), for now it only works locally. Please test!
What will be on server side, is not yet decided, at least not entirely(database server, etc). Communication between a desktop application(Lazarus/OPM) and a remote database is not safe, unless another layer of security is added. It can be done through a php server or free pascal based fastcgi. The second one would be a nice show case for free pascal, but is more difficult to implement. Somebody has experience with fastcgi? I can do the implementation with php.

Screenshot for the voting system(OSX, Linux, Windows):
Title: Re: Online Package Manager
Post by: josh on December 16, 2016, 11:13:44 am
Hi JuhaManninen,

Thanks for looking into the issue.

What would I need to change to get kcontrols to compile and run on  osx?
Title: Re: Online Package Manager
Post by: JuhaManninen on December 16, 2016, 11:35:45 am
What would I need to change to get kcontrols to compile and run on  osx?
I think it is obvious. Use the right type (TThreadID) for MainThreadID.
Title: Re: Online Package Manager
Post by: JuhaManninen on December 16, 2016, 12:22:55 pm
For approximately 10 out of 41 packages the update system is up and running(the external JSON we talked about in previous posts). Later hopefully more package maintainer will join in.
I didn't mean the external JSON thing only. I meant the authors and maintainers should also deliver new versions for the master repository. That is the "official" place after all.
The external JSON update still looks like a little confusing design oddity to me, but no worries, I believe it is needed.
The process of providing packages could still be simplified to make it more attractive for package authors.
Now the wiki says: "Upload somewhere the files and send the link to getmem1@gmail.com".
It should be even simpler.

Quote
I implemented voting system partially(r. 53697), for now it only works locally. Please test!
What will be on server side, is not yet decided, at least not entirely(database server, etc). Communication between a desktop application(Lazarus/OPM) and a remote database is not safe, unless another layer of security is added. It can be done through a php server or free pascal based fastcgi. The second one would be a nice show case for free pascal, but is more difficult to implement. Somebody has experience with fastcgi? I can do the implementation with php.
Hey, let's document things first for a change. What will the server SW do? Could you at least make a bullet point list please. I can try to make a sequence diagram after it.
And yes, I would like to see Free Pascal code there.
Title: Re: Online Package Manager
Post by: minesadorada on December 16, 2016, 01:01:40 pm
Re:Voting system
Could we have a TPopupNotifier or similar show over each star graphic showing what it means?  That way, the voting would be equitable.

..something like:

Popup text over 1 : "Unuseable. Buggy and/or obselete"
Popup text over 2 : "Compiles/Installs, but has many unfixed bugs"
Popup text over 3 : "OK, but poor support for fixes and/or updates"
Popup text over 4 : "OK, but unsuitable for production"
Popup text over 5 : "Perfection!"
Title: Re: Online Package Manager
Post by: GetMem on December 16, 2016, 03:11:42 pm
@Juha
Quote
I didn't mean the external JSON thing only. I meant the authors and maintainers should also deliver new versions for the master repository. That is the "official" place after all.
That was my original idea, a well maintained central repository, but then everyone wanted updates  :D. Anyway, once the database server is up and running(I already discussed with Marc about the details), we can give rights to package maintainers, so in the end a direct upload to the central repository will be possible in the future.

Quote
The external JSON update still looks like a little confusing design oddity to me, but no worries, I believe it is needed.
Every major package system works in a similar way. Maybe you didn't follow the latest post, but almost every limitation is lifted now. The package maintainer doesn't have to follow any naming convention, zip structure, etc. Just generate a JSON with OPM, fill in two items, and that's all there is to it. If someone can came up with more simple design I'm open to suggestions.

Quote
Hey, let's document things first for a change. What will the server SW do? Could you at least make a bullet point list please. I can try to make a sequence diagram after it.
I do not understand the question, I mean what list should I create? The server SW is only needed as a gateway between OPM and the database(it will be PostgreSQL), since a direct connection is not safe.
The database will contain the:
  - table with users(package maintainers) and hashed passwords
  - table with user rights
  - table with packages
  - table with votes
  etc.
With time I plan to replace the main JSON(from the central repository) with a table. It's much easier to maintain then a JSON file.

@minesadorada
Quote
Re:Voting system
Could we have a TPopupNotifier or similar show over each star graphic showing what it means?  That way, the voting would be equitable.

..something like:

Popup text over 1 : "Unuseable. Buggy and/or obselete"
Popup text over 2 : "Compiles/Installs, but has many unfixed bugs"
Popup text over 3 : "OK, but poor support for fixes and/or updates"
Popup text over 4 : "OK, but unsuitable for production"
Popup text over 5 : "Perfection!"
Ok, but we all must agree on the list, because it can be very subjective. For example in your list after 4(which is still a crapy level) comes 5 Pefection :D
Title: Re: Online Package Manager
Post by: JuhaManninen on December 16, 2016, 04:21:21 pm
That was my original idea, a well maintained central repository, but then everyone wanted updates  :D. Anyway, once the database server is up and running(I already discussed with Marc about the details), we can give rights to package maintainers, so in the end a direct upload to the central repository will be possible in the future.
Sometimes it makes sense to follow your vision and not listen to others (much).
Anyway, let's see how it will work out.

Quote
Every major package system works in a similar way.
Not really. They usually have only a central repository. Then you can see if your local package is up-to-date or not. Easy, logical and intuitive.
With our hybrid system a local package can be up-to-date with the central repository but not with an external update source, at the same time. Not logical nor intuitive.

Quote
I do not understand the question, I mean what list should I create? The server SW is only needed as a gateway between OPM and the database(it will be PostgreSQL), since a direct connection is not safe.
List the technical details of what the gateway does. Protocols used over the net, data formats, authentication methods (is the forum user account still the plan).
How do you ensure the secure connection? Protocols and libraries used for it?
Is asynchronous communication needed, in addition to what TCP/IP already does?
Etc...
Think you are delegating the job to me and you must explain what must be done.
Title: Re: Online Package Manager
Post by: JuhaManninen on December 16, 2016, 04:41:09 pm
Popup text over 1 : "Unuseable. Buggy and/or obselete"
Popup text over 2 : "Compiles/Installs, but has many unfixed bugs"
Popup text over 3 : "OK, but poor support for fixes and/or updates"
Popup text over 4 : "OK, but unsuitable for production"
Popup text over 5 : "Perfection!"
Such explanations are not needed. The quality is always a subjective matter and it must be so. There is an infinite number of criteria by which people can judge a package. Trying to fit them all into those short explanations is a swamp.
If you look at other sites that allow voting, they also don't try to explain the scale.
For example SourceForge allows voting for different properties:
each having from 1 to 5 starts. The star-scale however is not explained because (I guess) it is so obvious.
1 star means bad and 5 stars mean good.
That is a sufficient explanation for our case, too. A scale from bad to good.

BTW, your texts already raise questions. No package can reach "Perfection!", thus 5 stars cannot be used. Even 4 stars is still unsuitable for production, thus no Lazarus package can be used in production. :(
Title: Re: Online Package Manager
Post by: minesadorada on December 16, 2016, 04:49:26 pm
Popup text over 1 : "Unuseable. Buggy and/or obselete"
Popup text over 2 : "Compiles/Installs, but has many unfixed bugs"
Popup text over 3 : "OK, but poor support for fixes and/or updates"
Popup text over 4 : "OK, but unsuitable for production"
Popup text over 5 : "Perfection!"
Such explanations are not needed. The quality is always a subjective matter and it must be so. There is an infinite number of criteria by which people can judge a package. Trying to fit them all into those short explanations is a swamp.
If you look at other sites that allow voting, they also don't try to explain the scale.
For example SourceForge allows voting for different properties:
  • Ease
  • Features
  • Design
  • Support
each having from 1 to 5 starts. The star-scale however is not explained because (I guess) it is so obvious.
1 star means bad and 5 stars mean good.
That is a sufficient explanation for our case, too. A scale from bad to good.

BTW, your texts already raise questions. No package can reach "Perfection!", thus 5 stars cannot be used. Even 4 stars is still unsuitable for production, thus no Lazarus package can be used in production. :(

I'm willing to be convinced. Let's see how the voting system works out without explanations...
My original list was just a flag to raise - I expected it to be changed by consensus.
Title: Re: Online Package Manager
Post by: GetMem on December 16, 2016, 05:54:59 pm
Quote
Not really. They usually have a central repository. Then you can see if your local package is up-to-date or not. Easy, logical and intuitive.
With our hybrid system a local package can be up-to-date with the central repository but not with an external update source. Not logical nor intuitive.
The thing is we have an update system now, we can't put the toothpaste back to the tube.  :) Seriously it would be a shame to drop it.

Quote
List the technical details of what the gateway does. Protocols used over the net, data formats, authentication methods (is the forum user account still the plan). How do you ensure the secure connection? Protocols and libraries used for it? Etc...
Think you are delegating the job to me and you must explain what must be done.
OPM with TFPHtmlClient sends Post/Get requests to the web server like any web browser would do. The protocol is https. It's logical to use it, since the traffic is already encrypted + the database can be also accessed with a browser. The web server can be Apache or a full fpc based server. A third possibility is an Apache with a FastCGI coded in fpc. FastCGi basically it's a interface between a program and a web server. The data format is simple, just basic command for the database: "Ïnsert into TableName(...) values(...)". I cannot tell you more details about the actual implementation until it's not decided what to use: php vs. fpc fastCGI vs full fpc server(it's an overkill for a small project like this). If we choose php I can do all the coding alone, if not I need help because I have little experience in server side coding with fpc. It would be fun to learn, but I'm busy implementing other stuff in OPM. The database server is PostgreSQL, according to Marc is very efficient and secure. I use firebird in my applications. Regarding the forum user account, it can be used, but since we create a new db. with users and passwors for voting/uploding packages to central repository it make sense to use the new username and password.
Title: Re: Online Package Manager
Post by: Rayvenhaus on December 16, 2016, 06:15:42 pm
So, the best way to update OPM at this time is to use SVN?  It's not showing any updates from inside OPM for OPM.
Title: Re: Online Package Manager
Post by: GetMem on December 16, 2016, 06:23:18 pm
Quote
@Rayvenhaus
So, the best way to update OPM at this time is to use SVN?  It's not showing any updates from inside OPM for OPM.
Yes, please update from SVN. I almost every day commit something new, it would be an overkill to update the central repository each day. More over, I think I will remove OPM from the repository. It make no sense to keep it there since it's part of lazarus.
Title: Re: Online Package Manager
Post by: JuhaManninen on December 16, 2016, 06:37:30 pm
The thing is we have an update system now, we can't put the toothpaste back to the tube.  :) Seriously it would be a shame to drop it.
Yes. I am not even suggesting that.

Quote
Regarding the forum user account, it can be used, but since we create a new db. with users and passwors for voting/uploding packages to central repository it make sense to use the new username and password.
So your client GUI will have an option to create a new user account? That is OK.

I don't think the CGI / FastCGI makes much difference because voting does not stress the server heavily.
CGI would be easy to implement.
Anyway, someone with experience of FPC on server side should make skeleton code which can then be improved.
Or maybe an existing example can be used as a skeleton.
Title: Re: Online Package Manager
Post by: lainz on December 16, 2016, 08:11:13 pm
I have a question, once the system is ready, for example I create an account it counts for voting and uploading packages, what's the limit? For example I can upload multiple packages, upload duplicate packages, upload some malware. Or it will be like is now moderated?

Edit: A suggestion -> when there's a new version available in the repository show it in bold, the same as when there's a new external version.
Title: Re: Online Package Manager
Post by: JuhaManninen on December 16, 2016, 10:59:49 pm
I have a question, once the system is ready, for example I create an account it counts for voting and uploading packages, what's the limit? For example I can upload multiple packages, upload duplicate packages, upload some malware. Or it will be like is now moderated?
Direct upload must be allowed only for few and selected people. There will be some admin work involved when the write access is given to them.
Rating will be allowed for anybody who logs in.

Let's see how the rating works.
Typically the results are biased to the positive side because only people who use and like a certain package rate it.
People who don't use it, don't rate it either.
I just noticed that Tcl is the "Projects Of The Month" in SourceForge.
 https://sourceforge.net/projects/tcl/
It got 60 ratings and all of them full 5 stars. Uhhh! In reality it is a horrible language. I guess 60 people in the world like it and they all went to rate it.
Title: Re: Online Package Manager
Post by: GetMem on December 17, 2016, 09:46:11 am
Quote
@lainz
I have a question, once the system is ready, for example I create an account it counts for voting and uploading packages, what's the limit? For example I can upload multiple packages, upload duplicate packages, upload some malware. Or it will be like is now moderated?
Yes, it must be moderated to prevent spam, malware etc.

Quote
A suggestion -> when there's a new version available in the repository show it in bold, the same as when there's a new external version.
Done r. 53704. Please test.

Quote
@Juha
It got 60 ratings and all of them full 5 stars. Uhhh! In reality it is a horrible language. I guess 60 people in the world like it and they all went to rate it.
:D This is funny!
Title: Re: Online Package Manager
Post by: minesadorada on December 17, 2016, 09:54:59 am
Initial Wiki page for the External JSON Editor:
http://wiki.lazarus.freepascal.org/opmjsonupdateeditor (http://wiki.lazarus.freepascal.org/opmjsonupdateeditor)

Quote
It got 60 ratings and all of them full 5 stars. Uhhh! In reality it is a horrible language. I guess 60 people in the world like it and they all went to rate it.
This can be the problem.  In a 5-star system, voting stars 2,3,and 4 are pretty meaningless without explanation, so people usually vote nothing or 1 (bad) or 5 (good) - a bit like the FaceBook 'like' button.
If the displayed stars are an average of all votes this could be sort-of okay.  The best-of-all system would be a reviews list webpage with individual ratings and comments (like imdb) but I do understand this would be shooting too high at this stage.  Let's see how it goes.

Re Direct upload: Perhaps to a 'sandbox' repository; then the moderator vets before moving to the 'real' repository?  This has the advantage that the author can test it him/herself (changing OPM/Options/General/Remote Repository to the sandbox URL) before notifying the moderator.  Only the moderator would have write access to the 'real' repository, which is the secure and accountable option.

Re Direct 'Upload': IMO the easiest way is to manage it is via svn access to the sandbox repository.  It could even be hosted for free on SF, so logins/account management/malware scanning/version control etc  are already done for you. (just add the prospective author's SF accountname to the OPMSandbox project as a 'Developer')  It already works fine for ccr and svn is available for all platforms AFAIK.  In SF there are separate permissions available for file management and svn access.
Title: Re: Online Package Manager
Post by: minesadorada on December 17, 2016, 10:59:13 am
OPM Rev 53704:
Screenshot 1 - after a ForceNotify
Screenshot 2 - after a version change

Question: Is there a process planned for merging updates into the main repository after a suitable period?
Title: Re: Online Package Manager
Post by: GetMem on December 17, 2016, 11:23:18 am
@minesadorada
Quote
Initial Wiki page for the External JSON Editor:
http://wiki.lazarus.freepascal.org/opmjsonupdateeditor
You should add a link to your wikipage: http://wiki.freepascal.org/Online_Package_Manager#Create_JSON_for_upates

Quote
OPM Rev 53704:
Screenshot 1 - after a ForceNotify
Screenshot 2 - after a version change
The screenshots are OK. There should be no difference visually between ForceNotify and a Version change(except the version number).

Quote
Question: Is there a process planned for merging updates into the main repository after a suitable period?
Yes. Most likely will be an option to directly upload a package to main repository. Authentication needed.
Title: Re: Online Package Manager
Post by: lainz on December 17, 2016, 02:40:27 pm
About this article
http://wiki.freepascal.org/opmjsonupdateeditor

I only need to say that must be consistent the naming of the tool.

"opmjsonupdateeditor"
"Online Package Manager External Update json Editor"
"External OPM JSON package editor"

Please just choose one and only one.
Title: Re: Online Package Manager
Post by: minesadorada on December 17, 2016, 08:11:41 pm
About this article
http://wiki.freepascal.org/opmjsonupdateeditor (http://wiki.freepascal.org/opmjsonupdateeditor)

I only need to say that must be consistent the naming of the tool.

"opmjsonupdateeditor"
"Online Package Manager External Update json Editor"
"External OPM JSON package editor"

Please just choose one and only one.
What is your suggestion to choose? (One is the application title, the other is more descriptive.  The filename is jsoneditor.)

The wiki page name is terrible - I was in a hurry :) AFAIK I can't change it now.

The app needs a bit of testing - it's quite ambitious.  A bug/undocumented feature I am aware of; when changing languages 'on-the-fly' control array #0 doesn't update until a restart of the app.

Thanks for your help @lainz in starting the code and doing translations.
Title: Re: Online Package Manager
Post by: lainz on December 17, 2016, 09:22:45 pm
The one is more descriptive. You can change the application title easily.

You can copy everything in the article and make a new one with the right name. Then empty the other article and I'm not sure how but there's a setting like mark for deletion, if you don't find that ask an administrator.
Title: Re: Online Package Manager
Post by: minesadorada on December 18, 2016, 10:40:30 am
I settled on OPM External Update json Editor for wiki page and application title: http://wiki.freepascal.org/OPM_External_Update_json_Editor (http://wiki.freepascal.org/OPM_External_Update_json_Editor)

-edit- Added categories links etc.  All should be good now.
Title: Re: Online Package Manager
Post by: Alextp on December 19, 2016, 08:53:05 pm
Now i see listview with items captions:

-fortes4lazarus 3.24
-indy10.6.2
-eyecandycontrols_0-9-6

my wish is: show NICE titles in column-0:

-Fortes for Lazarus
-Indy
-EyeCandyControls
Title: Re: Online Package Manager
Post by: Alextp on December 19, 2016, 08:56:54 pm
And also i see bug on Today's trunk:
i just move mouse over listview, and IDE shows
"List index out of bounds (-1)"; before i resized header column-0 to wider size.
maybe caused by none selection in list.

Ubuntu 14.04 x64
Title: Re: Online Package Manager
Post by: Alextp on December 19, 2016, 09:12:22 pm
"OPM External Update json Editor" - I suggest shorter one "OPM Editor", "OPM Helper", "OPM Assistant"
Title: Re: Online Package Manager
Post by: minesadorada on December 19, 2016, 10:25:25 pm
"OPM External Update json Editor" - I suggest shorter one "OPM Editor", "OPM Helper", "OPM Assistant"

I thank you for the suggestions, but the app is very specific - it edits only external update jsons, and is not part of OPM itself.  If I was selling it then marketing and brand/product recognition via a snappy title would probably be more important...  Treat it as an ugly but useful gift.
Title: Re: Online Package Manager
Post by: Alextp on December 19, 2016, 10:27:36 pm
I found how to make my ATButtons.zip. How to send it to OPM?
Title: Re: Online Package Manager
Post by: GetMem on December 19, 2016, 10:38:04 pm
@Alextp
Quote
Now i see listview with items captions:
-fortes4lazarus 3.24
-indy10.6.2
-eyecandycontrols_0-9-6

my wish is: show NICE titles in column-0:
-Fortes for Lazarus
-Indy
-EyeCandyControls
The packages names are given by the package maintainers,  besides each package can have multiple versions(see virtual tree).

Quote
And also i see bug on Today's trunk:
i just move mouse over listview, and IDE shows
"List index out of bounds (-1)"; before i resized header column-0 to wider size.
maybe caused by none selection in list.
Ubuntu 14.04 x64
I don't have ubuntu,  but I did test it on kali linux and I cannot reproduce the error. Can you please debug it and tell me which line cause the error? Thanks.
 
Quote
I found how to make my ATButtons.zip. How to send it to OPM?
Just paste the link to your package(s), I will add them tomorrow.
Title: Re: Online Package Manager
Post by: Alextp on December 19, 2016, 10:51:24 pm
Anyway, it is bad looking.
-fortes4lazarus 3.24
-indy10.6.2
-eyecandycontrols_0-9-6
must be one scheme. caps of not caps. version or not. _ or - or dot or space.
Title: Re: Online Package Manager
Post by: GetMem on December 20, 2016, 08:25:03 am
@Alextp
I added ATButtons, ATGauge to the central repository. The update link for your packages is wrong. Please read this section:
http://wiki.freepascal.org/Online_Package_Manager#Create_JSON_for_updates
Basically use the "Create-->Create JSON for updates" menu item.

I think the version number after package name(Column0) it's not needed, since is visible in column1, 2, 3 anyway.
Title: Re: Online Package Manager
Post by: Alextp on December 20, 2016, 02:56:34 pm
thanks for Nice Column0 :). Notes:
1) You use OpenDialog (i see Open btn, not Save) for saving my zip file
2) Pls disallow Github ZIP links, which I filled before (url of  master zip)

edit
3) For CalLite, i click item in column Status/data, url at wiki, but Firefox dont open
Title: Re: Online Package Manager
Post by: Alextp on December 20, 2016, 03:04:27 pm
4) For my comp's I want to fill "wiki" infos: url of Laz wiki, not url of Github homepage (both needed, GH+wiki)
Title: Re: Online Package Manager
Post by: GetMem on December 20, 2016, 05:26:34 pm
Quote
Pls disallow Github ZIP links, which I filled before (url of  master zip)
Done.

Quote
For CalLite, i click item in column Status/data, url at wiki, but Firefox dont open
Click-->Select row
DbClick--> Open link

Quote
For my comp's I want to fill "wiki" infos: url of Laz wiki, not url of Github homepage (both needed, GH+wiki)
Paste here the home package link and I will add it to them main JSON, you should also fill the update link.

Title: Re: Online Package Manager
Post by: GetMem on December 21, 2016, 01:27:21 pm
I extended the options dialog. Now it's possible to set the update interval + filters for packages. When creating a repository package, the zipped files are filtered according to the settings from option dialog(r. 53748). Please test.
Title: Re: Online Package Manager
Post by: minesadorada on December 21, 2016, 07:44:56 pm
Excellent work - thanks!

Is there a regression? After an update, I note that the description and license fields are once again unchanged from the original.

For future?
- I was able to create a package with just the Category field filled out (no display name, home page nor update link)
- No fields yet to indicate compatibility (min fpc version, min laz version supported OS/WidgetSet)
- A component may be useless without a clear understanding of its license (which may not be specified in the lpk)  How can OPM handle this?
- No dialog as to what to do next once the zip and json have been created

I had to add some exclusion filters; here's the set that works for most packages in ccr  (note /updates which is created by the Json Updates editor)
Title: Re: Online Package Manager
Post by: lainz on December 21, 2016, 08:24:57 pm
You can add .git and .svn folders too.
Title: Re: Online Package Manager
Post by: minesadorada on December 21, 2016, 08:27:30 pm
You can add .git and .svn folders too.
+1
Title: Re: Online Package Manager
Post by: GetMem on December 21, 2016, 09:33:07 pm
@minesadorada
Quote
Excellent work - thanks!
Thank you.

Quote
Is there a regression? After an update, I note that the description and license fields are once again unchanged from the original.
Yes it is! Fixed in r. 53750.
Quote
I was able to create a package with just the Category field filled out (no display name, home page nor update link)
Well those field are not mandatory. OPM can work without them. I updated the wiki a few days ago: http://wiki.freepascal.org/Online_Package_Manager#Create_repository_package
Quote
No fields yet to indicate compatibility (min fpc version, min laz version supported OS/WidgetSet)
Click to each child node(*lpk files), you will see those fields and even correct them if necessary.
Quote
A component may be useless without a clear understanding of its license (which may not be specified in the lpk)  How can OPM handle this?
The license/description is loaded from the *.lpk file. Editing directly from OPM would be confusing. Which is the real license?
Quote
No dialog as to what to do next once the zip and json has been created
The plan is to upload the packages directly to the main repository, however in order for the upload to work we need somekind of gateway, accessing the database directly is not safe. A fpc based FastCGI implementation would be ideal.

Quote
I had to add some exclusion filters; here's the set that works for most packages in ccr  (note /updates which is created by the Json Updates editor)
Thank you! Please copy all your filters from your configuration file and paste here.
Config file: C:\Users\%user%\AppData\Local\lazarus\onlinepackagemanager\config\options.xml
Search for: ExcludedFiles, ExcludedFolders

PS: For now it would be great if I can send the zip + json through mail, unfortunately fpc doesn't have a native component for sending mail.
Title: Re: Online Package Manager
Post by: minesadorada on December 21, 2016, 10:03:08 pm
Quote
Yes it is! Fixed in r. 53750.
Thanks.  Tested OK.
Quote
The license/description is loaded from the *.lpk file. Editing directly from OPM would be confusing. Which is the real license?
If the lpk licence entry is empty then 'Create Package' could offer options, or at least a warning (it would be the package author creating the OPM entry)  I think it's important enough to implement.

Quote
Click to each child node(*lpk files), you will see those fields and even correct them if necessary
This needs to be in the update json spec, since an update could modify compatibility.


Quote
The plan is to upload the packages directly to the main repository, however in order for the upload to work we need somekind of gateway, accessing the database directly is not safe. A fpc based FastCGI implementation would be ideal.
Did you read my post about a 'sandbox' repository?  Possibly hosted by SourceForge and accessible via svn?  A sandbox repository addresses most security concerns.
I think it's an extremely bad idea to allow anyone but the OPM moderator access to the main repository.

Quote
Please copy all your filters from your configuration file and paste here.
I thought the screenshot +lainz's post said it all.  If not, I can do as you suggest.
Title: Re: Online Package Manager
Post by: GetMem on December 21, 2016, 10:26:50 pm
Quote
If the lpk licence entry is empty then 'Create Package' could offer options.  I think it's important enough to implement.
You forget that we talk about package maintainers here not beginners. Ok I'm not against it, if somebody implements it...

Quote
Did you read my post about a 'sandbox' repository?  Possibly hosted by SourceForge?  A sandbox repository addresses most security concerns.
I think it's an extremely bad idea to allow anyone but the OPM moderator access to the main repository.
We have a dedicated VM just for OPM on lazarus official page, why should we in involve sourceforge? Everyone(after a login) will be able to upload files,  the packages will be visible/available for download only after moderation. It's the same system as you described in your post(mostly).

Quote
I thought the screenshot +lainz's post said it all.  If not, I can do as you suggest.
The screenshot is clear but to prevent errors it would be much easier to copy/paste. Anyway I did it, I will commit it tomorrow. Thanks.
Title: Re: Online Package Manager
Post by: minesadorada on December 21, 2016, 10:30:12 pm
Thanks for the responses @Getmem.  I appreciate your patience and hard work.
Title: Re: Online Package Manager
Post by: GetMem on December 21, 2016, 10:56:50 pm
@minesadorada
Quote
This needs to be in the update json spec, since an update could modify compatibility.
Again I agree, but then we have to redesign the whole external json structure + retest it from scratch.  :) A quick shortcut is to read the lpk after install/update, this is exactly what OPM does now for description and license. I can extend it for other items too.
Title: Re: Online Package Manager
Post by: tk on December 22, 2016, 10:34:23 am
What would I need to change to get kcontrols to compile and run on  osx?

Use newest version from repository at https://bitbucket.org/tkweb/kcontrols.
It compiles and runs well on osx.
When sth. is not working, ask here: http://tkweb.eu/en/delphicomp/kcontrols.html.

I noticed this forum already before but did not try the package manager yet.
From the screenshot I can see there is KControls 1.7. This is last "stable", but very old already.
(I still don't have everything ready what I planned for KControls 1.8 )

Is there any option to add a link to the repository for the included package (to give the option to download newer version) or maybe to its forum as well?
Title: Re: Online Package Manager
Post by: JD on December 22, 2016, 11:02:19 am
I noticed this forum already before but did not try the package manager yet.
From the screenshot I can see there is KControls 1.7. This is last "stable", but very old already.
(I still don't have everything ready what I planned for KControls 1.8 )

@tk Hi there. Thanks for your good work. Is KControls 1.8 ready for production use?
Title: Re: Online Package Manager
Post by: minesadorada on December 22, 2016, 11:23:45 am
I found an obscure bug in Cryptini; fixed it, and using the JsonEditor and OPM the bugfixed version (0.1.3.0) was available to all within 5-10 minutes.  That's what I call quick deployment!

Quote
Is there any option to add a link to the repository for the included package (to give the option to download newer version)
Check out the wiki: http://wiki.lazarus.freepascal.org/Online_Package_Manager#Create_JSON_for_updates (http://wiki.lazarus.freepascal.org/Online_Package_Manager#Create_JSON_for_updates) It's really easy to deploy updates with OPM.
Title: Re: Online Package Manager
Post by: GetMem on December 22, 2016, 01:34:17 pm
@minesadorada
I uploaded the extensions provided by you and @lainz, the list are sorted now both for files and dirs. I also added two help buttons to the following forms: "Create repository package" and "Create exteranl JSON".

@tk
Quote
Is there any option to add a link to the repository for the included package (to give the option to download newer version) or maybe to its forum as well?
Yes, basically you have to:
1. Create a JSON file with OPM
2. Edit "DownloadZipURL" and "Version" fields
3. Upload somewhere the JSON
4. Paste here the link, so I can update the main repository
When you release a new version, just update the zip file and increase the version number.
More details here: http://wiki.lazarus.freepascal.org/Online_Package_Manager#Create_JSON_for_updates

Thank you!
Title: Re: Online Package Manager
Post by: lainz on December 22, 2016, 08:51:32 pm
Hi GetMem, you can add these packages please?

File Association for Windows - modified LGPL (lainz)
https://github.com/lainz/FileAssociation
http://wiki.freepascal.org/FileAssociation

BGRA Games (Some BGRABitmap games and related units) modified LGPL (lainz, circular)
https://github.com/bgrabitmap/bgragames

uE Controls (Another set of UI components) see the readme for the author and license
https://github.com/bgrabitmap/uecontrols

Kaaj Controls (Controls for fpGUI) modified LGPL (lainz)
https://github.com/bgrabitmap/kaajcontrols

Lainz Code Studio (Lua scripting units and small test project) modified LGPL (lainz)
https://github.com/lainz/lainzcodestudio

And about these packages: These maybe will not be updated a lot so for now I will not mantain an external json. That doesn't mean that these are not usefull. Some are not mine like ueControls but I've mirrored that in bgrabitmap organization, the original code is at sourceforge (https://sourceforge.net/projects/uecontrols/)
Title: Re: Online Package Manager
Post by: Alextp on December 22, 2016, 09:58:35 pm
Quote
>If the lpk licence entry is empty then 'Create Package' could offer options, or at least a warning

It is needed. And gut to show checklistbox, with items "LGPL, MPL 1.0, MPL 2.0, MIT......" to easy check one item.
Title: Re: Online Package Manager
Post by: JuhaManninen on December 23, 2016, 12:08:40 am
I noticed this forum already before but did not try the package manager yet.
From the screenshot I can see there is KControls 1.7. This is last "stable", but very old already.
(I still don't have everything ready what I planned for KControls 1.8 )
You can release a bug fix dot-release, say 1.7.5.

Quote
Is there any option to add a link to the repository for the included package (to give the option to download newer version) or maybe to its forum as well?
You should create a repository package of your new KControls 1.7.5 release and send it to GetMem.
See:
http://wiki.freepascal.org/Online_Package_Manager#Create_repository_package

It is important that package maintainers provide them. Otherwise this will not work well in years to come.
Please everybody advertise this to package authors.
Title: Re: Online Package Manager
Post by: GetMem on December 23, 2016, 09:39:21 am
@lainz
I added the packages. Thanks! I also fixe BGRABitmap in the central repository by adding LCLPlatformDef to the uses clauses for BGRAText unit. Apparently some things where moved around lately. Since BGRABitmap is used a lot, I thought I fix it.

Quote
These maybe will not be updated a lot so for now I will not mantain an external json.
That's ok. The update feature is a possibility not mandatory.


PS: I also updated KControls to the latest version.
Title: Re: Online Package Manager
Post by: tk on December 23, 2016, 10:38:46 am
@tk Hi there. Thanks for your good work. Is KControls 1.8 ready for production use?

Not yet. I still need some time to implement planned features.

@GetMem, @Juha, @minesadorada:
Thank you all, I'll try the OPM and see what I can do.
Title: Re: Online Package Manager
Post by: JD on December 23, 2016, 11:25:19 am
@tk Hi there. Thanks for your good work. Is KControls 1.8 ready for production use?

Not yet. I still need some time to implement planned features.

OK. Noted!
Title: Re: Online Package Manager
Post by: lainz on December 23, 2016, 04:04:55 pm
Thanks GetMem, almost 50 packages available as single click installation.
Title: Re: Online Package Manager
Post by: JuhaManninen on December 23, 2016, 06:46:23 pm
I also fixe BGRABitmap in the central repository by adding LCLPlatformDef to the uses clauses for BGRAText unit. Apparently some things where moved around lately. Since BGRABitmap is used a lot, I thought I fix it.
Yes.
 http://wiki.freepascal.org/Lazarus_1.8.0_release_notes#TLCLPlatform_definition_was_moved
Did you offer the fix to the master BGRABitmap sources, too?
I admire your enthusiasm but you are essentially forking the original package's code now. In some other situation it can create a mess and the changes in your fork are overwritten.
The online repository should be used for delivering, not forking, existing packages.

Quote
PS: I also updated KControls to the latest version.
I still see version 1.7. I understood it is the latest version so far.
Title: Re: Online Package Manager
Post by: JuhaManninen on December 23, 2016, 06:58:09 pm
The OPM client creates a 100% CPU load for one processor core. It looks like OnIdle load because it does not slow anything down, but it will kill laptop batteries quickly.
I did not study the code (yet) for possible reasons.
Title: Re: Online Package Manager
Post by: lainz on December 23, 2016, 07:00:06 pm
I can fix the bgrabitmap pack but I need some help, adding that unit (LCLPlatformDef) to uses clause makes the package don't compile in Lazarus 1.6.

Any ideas? It will work in Lazarus 1.8, so I need something in the uses clause that adds the unit only in 1.8 or 1.7 trunk, never in 1.6.x
Title: Re: Online Package Manager
Post by: GetMem on December 23, 2016, 07:43:20 pm
@Juha
Quote
The OPM client creates a 100% CPU load for one processor core. It looks like OnIdle load because it does not slow anything down, but it will kill laptop batteries quickly.
Fixed in r. 53769. Please test.

@lainz
Quote
Any ideas? It will work in Lazarus 1.8, so I need something in the uses clause that adds the unit only in 1.8 or 1.7 trunk, never in 1.6.x
Code: Pascal  [Select]
  1. {$IF LCL_FULLVERSION >= 1070000} LCLPlatformDef; {$ENDIF}
I also must add to the one in central repository.
Title: Re: Online Package Manager
Post by: lainz on December 23, 2016, 07:52:42 pm
Code: Pascal  [Select]
  1. {$IF LCL_FULLVERSION >= 1070000} LCLPlatformDef; {$ENDIF}

Hi GetMem, I tried with your code but it gives me an error, with FPC 3.0
Code: Pascal  [Select]
  1. bgratext.pas(35,7) Error: Incompatible types: got "AnsiString" expected "Int64"
Title: Re: Online Package Manager
Post by: GetMem on December 23, 2016, 08:02:53 pm
@lainz
I mean like this(BGRAtext.pas):
Code: Pascal  [Select]
  1. //...
  2. uses
  3.   Classes, Types, SysUtils, BGRAGraphics, BGRABitmapTypes, InterfaceBase, BGRAPen,
  4.   {$IF LCL_FULLVERSION >= 1070000} LCLPlatformDef,{$ENDIF} BGRAGrayscaleMask;  
  5. //...
  6.  
Title: Re: Online Package Manager
Post by: lainz on December 23, 2016, 08:09:41 pm
I know. I did that, but again the compiler gives me the same error.
Title: Re: Online Package Manager
Post by: JuhaManninen on December 23, 2016, 09:36:46 pm
Code: Pascal  [Select]
  1. {$IF LCL_FULLVERSION >= 1070000} LCLPlatformDef; {$ENDIF}
Hi GetMem, I tried with your code but it gives me an error, with FPC 3.0
Code: Pascal  [Select]
  1. bgratext.pas(35,7) Error: Incompatible types: got "AnsiString" expected "Int64"
I am getting the same error when the IF condition is in the interface section. It works when it is in the implementation section.
It must be a bug in FPC 3.0. Does anybody know about it?
Title: Re: Online Package Manager
Post by: GetMem on December 23, 2016, 09:41:47 pm
Add LCLVersion to the first uses, then it works in the second one:
Code: Pascal  [Select]
  1. uses GraphType, Math, BGRABlend, BGRAUTF8, {$IF LCL_FULLVERSION > 1060000} LCLPlatformDef{$ENDIF};
and you must use ">" instead of ">= " since ">=" won't work in FPC 3.1.1. I have no idea why.

http://forum.lazarus.freepascal.org/index.php/topic,34297.msg229826.html#msg229826
Title: Re: Online Package Manager
Post by: JuhaManninen on December 23, 2016, 09:55:08 pm
Add LCLVersion to the first uses, then it works in the second one:
Code: Pascal  [Select]
  1. uses GraphType, Math, BGRABlend, BGRAUTF8, {$IF LCL_FULLVERSION > 1060000} LCLPlatformDef{$ENDIF};
Yes, then it works in the implementation uses section.
I tried to have both LCLVersion and the IF condition in interface uses section. Doesn't work!

Quote
and you must use ">" instead of ">= " since ">=" won't work in FPC 3.1.1. I have no idea why.
It must be a bug. Somebody should report it.

Quote
Fixed in r. 53769. Please test.
Works well now. Thanks.
Title: Re: Online Package Manager
Post by: GetMem on December 23, 2016, 09:58:57 pm
@Juha
Can you please check if my latest commit fix the CPU load issue?

Quote
It must be a bug. Somebody should report it.
I can report it later, maybe they fixed already. I have to check fpc 3.1.1 again or perhaps somebody has 3.1.1 installed and can do a quick test?
Title: Re: Online Package Manager
Post by: lainz on December 23, 2016, 10:15:22 pm
I did as you say and I get another error:

Quote
bgratext.pas(119,76) Fatal: no puedo encontrar LCLPlatformDefutilizado por BGRAText de paquete BGRABitmapPack.

The unit in the implementation uses is grayed, but it says it (the error) doesn't matter the unit it's grayed. I'm on 1.6 FPC 3.0.

Code: Pascal  [Select]
  1. uses
  2.   Classes, Types, SysUtils, BGRAGraphics, BGRABitmapTypes, InterfaceBase, BGRAPen, BGRAGrayscaleMask,
  3.   LCLVersion;
  4.  
  5. ...
  6.  
  7. implementation
  8.        
  9. uses GraphType, Math, BGRABlend, BGRAUTF8 {$IF LCL_FULLVERSION > 1060000}, LCLPlatformDef{$ENDIF};
Title: Re: Online Package Manager
Post by: tk on December 23, 2016, 10:22:28 pm
Yes, basically you have to:
1. Create a JSON file with OPM
...

I installed the OPM now and read the wiki, it looks already very good at first look, but I still have some questions.

1. From what I've read, it still does not provide a direct connection to Bitbucket? There I maintain the main repository of KControls (locally via Mercurial). So without this support the OPM can now only work with "stable" or "bugfix" releases stored somewhere on my website?

2. Does it support private packages? I have several packages stored in my private cloud drive and I must always install them manually. It would be great to automate this task via the OPM.

Thank you and Merry Christmas to all




Title: Re: Online Package Manager
Post by: GetMem on December 23, 2016, 10:39:38 pm
@lainz
Please check the values in LCLVersion. Perhaps the release or patch version is greater then 0. Anyway this should work(I don't have 1.6 to test):
Code: Pascal  [Select]
  1. uses GraphType, Math, BGRABlend, BGRAUTF8 {$IF (lcl_major > 0) and (lcl_minor > 6)}, LCLPlatformDef {$ENDIF};
Title: Re: Online Package Manager
Post by: GetMem on December 23, 2016, 10:58:08 pm
Quote
I installed the OPM now and read the wiki, it looks already very good at first look.
Thank you.

Quote
1. From what I've read, it still does not provide a direct connection to Bitbucket? There I maintain the main repository of KControls (locally via Mercurial). So without this support the OPM can now only work with "stable" or "bugfix" releases stored somewhere on my website?
I'm not sure what you mean by direct connection. There is no implementation that support Bitbucket api, however OPM can download from Bitbucket. For example I see that the download link to the latest version is: https://bitbucket.org/tkweb/kcontrols/get/d35ac14451b2.zip. With the following JSON, OPM can update KControls whenever is necessary. What is in the zip file is up to you, you can feed stable or trunk version.
Quote
{
  "UpdatePackageData" : {
    "DisableInOPM" : false,
    "DownloadZipURL" : "https://bitbucket.org/tkweb/kcontrols/get/d35ac14451b2.zip",
    "Name" : "kcontrols_1.7"
  },
  "UpdatePackageFiles" : [
    {
      "ForceNotify" : false,
      "InternalVersion" : 1,
      "Name" : "kcontrolsbase.lpk",
      "Version" : "1.7.0.0"
    },
    {
      "ForceNotify" : false,
      "InternalVersion" : 1,
      "Name" : "kcontrolslaz.lpk",
      "Version" : "1.7.0.0"
    }
  ]
}
All you have to do is upload this JSON wherever you like(bitbucket, personal site etc) and send me the link so I can update the main repository. When a new version is out, just change "DownloadZipURL" and increase "Version". OPM will automatically notify the user about a new version.
Quote
2. Does it support private packages? I have several packages stored in my private cloud drive and I must always install them manually. It would be great to automate this task via the OPM.
Yes. Please send as many package you like. You can use the "Create repository package" menu item from OPM, or just paste the link to your package herre and I will add them manually. Make sure your lpk contains a good description and a license info, also please specify the supported widgetset.
Title: Re: Online Package Manager
Post by: minesadorada on December 23, 2016, 11:07:47 pm
I'm running trunk/trunk and no LCL version $DEFINE seems to work.  FPC $DEFINES are OK -  though <= and >= seem to be broken.  I'd like to see the solution because I need it in jsoneditor.
I've tried lclversion in the .lpr and as the first item in the form's implementation uses clause.
Title: Re: Online Package Manager
Post by: GetMem on December 23, 2016, 11:12:52 pm
@minesadorada
Thanks for confirming. In this case we should report it.
Title: Re: Online Package Manager
Post by: tk on December 23, 2016, 11:35:40 pm
So I started managing all 3rd party packages using OPM. Surprisingly none I've used so far was missing! Excellent work!

...however OPM can download from Bitbucket.

Now I understand and I'll send you the link to the JSON. I think this is sufficient for now. I just wanted to avoid the frequent uploading of the entire zip file somewhere to my web site. Still for future I would appreciate (and I am sure many other package authors as well) that the OPM notifies the users about a change (new commits) in the linked repository (not just for the case the package author forgets to update the JSON). This would need some additional work (add the repository type and url to the JSON + communicate with the repository webservice).

Yes. Please send as many package you like.

I meant private packages with proprietary code created/used in our company, I cannot disclose them. I meant something like adding another (private) remote repository to the OPM settings and add these packages to this repository. OPM would then manage two repositories eg.:
1. http://packages.lazarus-ide.org
2. http://ourprivaterepository.ourdomain.com
Title: Re: Online Package Manager
Post by: minesadorada on December 23, 2016, 11:53:19 pm
@tk In OPM/Options you can switch repositories at will.
If your package is under svn or similar; using the GUI http://wiki.freepascal.org/OPM_External_Update_json_Editor (http://wiki.freepascal.org/OPM_External_Update_json_Editor) and a Zip archiver means you get an update out to all OPM users in under 5 minutes.  It only makes sense to offer stable versions to OPM.

I would be good to have documentation about how to set up a private repository server, though.  Perhaps later?
Title: Re: Online Package Manager
Post by: tk on December 24, 2016, 12:39:04 am
@tk In OPM/Options you can switch repositories at will.

This I suppose because there is edit box for the repository URL. But still, in future, I would appreciate to manage more repositories concurrently. Switching them would be a mess. Either way, of course a wiki about private repository structure/setup would be necessary.

Title: Re: Online Package Manager
Post by: lainz on December 24, 2016, 02:17:15 am
@minesadorada
Thanks for confirming. In this case we should report it.

If there's a bug in that functionality better I don't add the unit fix to bgra sources until 1.8 gets real, else it will add more problems to these that are using 3.1 and trunk, only for making it compatible with 1.6 (something that already is).
Title: Re: Online Package Manager
Post by: minesadorada on December 24, 2016, 02:51:15 am
@lainz I don't know if this helps your situation:

It's possible to isolate fpc v3.1.1 (or other fpc versions) successfully with the current broken trunk functionality:
Code: Pascal  [Select]
  1. {$IF FPC_VERSION = 3}
  2.   {$IF FPC_RELEASE > 0}
  3.    {$IF FPC_PATCH > 0}
  4.      {$DEFINE FPC311}
  5.     {$ENDIF}
  6.   {$ENDIF}
  7. {$ENDIF}
  8.  

Then:
Code: Pascal  [Select]
  1. {$IFDEF FPC311}<code that breaks under 3.0.0>{$ENDIF}
Title: Re: Online Package Manager
Post by: minesadorada on December 24, 2016, 03:16:30 am
Re Laz version detection:

Fiddling around with Trunk/Trunk I found a workaround that works for the initialization section of my unit, where I needed it.  It might be useful for others coding with current trunk.

Here's the state of play with fpc 3.11/laz 1.7 (svn rev 53588m)

The highest scope this can go is after the unit's Interface Uses clause
Code: Pascal  [Select]
  1. interface
  2. uses lclversion, <other stuff> ;
  3. {$IF LCL_FULLVERSION >= 1070000}
  4.   {$DEFINE LAZ17}
  5. {$ENDIF}
  6.  

From that point on,
Code: Pascal  [Select]
  1. {$IFDEF LAZ17}<some laz 1.7+ code>{$ENDIF}
works throughout the unit's implementation, initialization (and finalization AFAIK) but obviously not in the Interface Uses clause
It does work in the implementation Uses clause.

The 'greying out' feature of the editor is misleading when using {$IFDEF LAZ17} in this way, but the compiler sees it correctly.

I have no idea if this is a regression, bug or intended behaviour as I haven't needed to test for Laz version before now.
Title: Re: Online Package Manager
Post by: GetMem on December 24, 2016, 08:26:26 am
@tk
Quote
I meant private packages with proprietary code created/used in our company, I cannot disclose them. I meant something like adding another (private) remote repository to the OPM settings and add these packages to this repository. OPM would then manage two repositories eg.:
1. http://packages.lazarus-ide.org
2. http://ourprivaterepository.ourdomain.com
Oh, I see! I misunderstood the question. Private repositories are not supported yet, I plan to add it later(most likely in the beginning of January).

@lainz
I was also surprised that {$IF LCL_FULLVERSION >= 1070000} not working in the interface section, however works well below. I used many time in OPM.
Quote
If there's a bug in that functionality better I don't add the unit fix to bgra sources until 1.8 gets real, else it will add more problems to these that are using 3.1 and trunk, only for making it compatible with 1.6 (something that already is).
Yes, that's a good idea. Those who use Lazarus trunk can tweak the source anyway.

@minesadorada
Quote
The highest scope this can go is after the unit's Interface Uses clause
We came to the same conclusion, but is a bug an should be reported. I will ask in the devlist later, maybe somebody knows more about this issue.
Title: Re: Online Package Manager
Post by: JuhaManninen on December 24, 2016, 09:50:48 am
Code: Pascal  [Select]
  1. interface
  2. uses lclversion, <other stuff> ;
  3. {$IF LCL_FULLVERSION >= 1070000}
  4.   {$DEFINE LAZ17}
  5. {$ENDIF}
  6.  
Do you mean the ">=" operator works also in FPC trunk after all?

Quote
The highest scope this can go is after the unit's Interface Uses clause
We came to the same conclusion, but is a bug an should be reported. I will ask in the devlist later, maybe somebody knows more about this issue.
I had understood the bug was ">=" operator not working.
The condition does not work inside the Interface Uses clause although it works in Interface section otherwise.
That may be a feature instead of a bug. Reading the unit references may be done in one chomp. LCL_FULLVERSION is not evaluated yet then.
Better ask in FPC list.
Title: Re: Online Package Manager
Post by: JuhaManninen on December 24, 2016, 10:57:53 am
Quote
I meant private packages with proprietary code created/used in our company, I cannot disclose them. I meant something like adding another (private) remote repository to the OPM settings and add these packages to this repository. OPM would then manage two repositories eg.:
1. http://packages.lazarus-ide.org
2. http://ourprivaterepository.ourdomain.com
Oh, I see! I misunderstood the question. Private repositories are not supported yet, I plan to add it later(most likely in the beginning of January).
You mean you add support for multiple repositories? Tk's second URL could have any domain, makes no difference.
Private directories / packages are not in the scope of an online package manager.
The next important step will be tight integration with Lazarus package system. Now it does not know about online packages until they are installed.
The Lazarus package graph already can resolve dependencies between packages it knows about. The goal is to let it know about available online packages which would then become part of that package graph.
Lazarus components/ already contains packages with external dependencies, for example TAChartBgra.  Now trying to install it gives an error. In future it would automatically install BgraBitmapPack from online repo.
In the same way an external package can depend on a package in Lazarus components/. Then you must dump your current dependency handling code and integrate it with the package graph.
Details must be discussed in Lazarus devel list. Mattias is the most knowledgeable person about package graph.
Title: Re: Online Package Manager
Post by: GetMem on December 24, 2016, 01:58:42 pm
Quote
You mean you add support for multiple repositories? Tk's second URL could have any domain, makes no difference.
Private directories / packages are not in the scope of an online package manager.
Yes, support for multiple repositories. If the repository owner decide to share it it's public otherwise is private.

Quote
The next important step will be tight integration with Lazarus package system. Now it does not know about online packages until they are installed.
The Lazarus package graph already can resolve dependencies between packages it knows about. The goal is to let it know about available online packages which would then become part of that package graph.
Lazarus components/ already contains packages with external dependencies, for example TAChartBgra.  Now trying to install it gives an error. In future it would automatically install BgraBitmapPack from online repo.
In the same way an external package can depend on a package in Lazarus components/. Then you must dump your current dependency handling code and integrate it with the package graph.
Details must be discussed in Lazarus devel list. Mattias is the most knowledgeable person about package graph.
I don't know about this. Although they have similar functionality the two system are completely different. Mixing them would be like mixing bananas with oranges. Mattias use critical sections when loading the packages syncing those with my threaded system is a non-trivial operation. When searching for dependencies the local package manager has all the info needed in lpks, OPM has to rely on limited info available in the main JSON. For this to work everything has to be moved inside the packager folder, OPM once again completely redesigned, the two dialogs somehow mixed to become one. The benefits? Some lazy user don't have to open a second dialog to check if a package is available online. OPM was not meant to replace the "Install/Uninstall packages" dialog, it's just a tool to install as many packages you like with a button click. Until now you had to open each package, press the compile then the install button. Another benefit is that all the packages are in one place, with the latest updates available you don't have to go through the maintainers site.

PS: Before anything somebody must implement that fpc based cgi/fastcgi, so the database server can be used for maintaining the main repository + implement the voting system.
Title: Re: Online Package Manager
Post by: JuhaManninen on December 24, 2016, 03:52:35 pm
I don't know about this. Although they have similar functionality the two system are completely different. Mixing them would be like mixing bananas with oranges.
No, they both deal with Lazarus packages and their dependencies and installation. Conceptually the same thing. OPM is a front-end to external packages, Lazarus has front-end code for packages in local file system. The common code installs them and maintains a package graph.
You forgot that OPM already integrates into the package system using IDEIntf API, but in a clumsy and limited way.

Quote
Mattias use critical sections when loading the packages syncing those with my threaded system is a non-trivial operation. When searching for dependencies the local package manager has all the info needed in lpks, OPM has to rely on limited info available in the main JSON.
What info is missing from the JSON? I think it is pretty complete.
The package system code must be refactored and changed obviously to understand online packages.

Quote
For this to work everything has to be moved inside the packager folder, OPM once again completely redesigned, the two dialogs somehow mixed to become one.
No no! The package-API must be extended to support it. Don't worry, I am not asking you to do it. :)

Quote
The benefits? Some lazy user don't have to open a second dialog to check if a package is available online. OPM was not meant to replace the "Install/Uninstall packages" dialog, it's just a tool to install as many packages you like with a button click. Until now you had to open each package, press the compile then the install button.
Now you confuse the package system (dependency management + installation etc.) and its GUI front-ends.
The "Install/Uninstall packages" is a GUI for local packages, OPM for external packages. They both can install many packages at one go but it is an implementation detail of a GUI, nothing to do with the underlying package graph code.
The benefit will be good and elegant SW design! The current integration is a poorly stitched-in afterthought.
Think of my example TAChartBgra package. A user gets an error for missing package. At the same time he can see it is available in OPM. He thinks "this system is poorly designed" and yes he is right.

Quote
Another benefit is that all the packages are in one place, with the latest updates available you don't have to go through the maintainers site.
Yes, it is the benefit of an online package manager. It is the whole point of having OPM.
Yet that benefit does not reduce the need for integration anyhow.
We can even create more GUIs for various tasks with packages if need be. They all must integrate through a proper API.
Again, I am not asking you to do it. I will look at it later.

Quote
PS: Before anything somebody must implement that fpc based cgi/fastcgi, so the database server can be used for maintaining the main repository + implement the voting system.
This is a separate and independent project from the package system integration.
Title: Re: Online Package Manager
Post by: GetMem on December 24, 2016, 06:26:48 pm
@Juha
I'm not confusing anything, just I know how OPM works and studied Mattias code enough to tell you the integration are you talking about is an extremely difficult one.  Feel free to implement whatever you like, but please leave the current OPM system intact so I can keep developing. Still a lot of features to implement.

PS: Merry Christmas to all.
Title: Re: Online Package Manager
Post by: tk on December 28, 2016, 12:31:07 pm
Tested the OPM under Win, Linux and OSX, works very well.

Ideas for future:
1. OPM already preinstalled in Lazarus (to save one Lazarus rebuild for installing the OPM).
2. Manage packages listed in 'available for installation' list box with OPM and delete them from Lazarus distribution (to keep Lazarus distribution smaller).
Title: Re: Online Package Manager
Post by: GetMem on December 28, 2016, 03:06:01 pm
@tk
Quote
1. OPM already preinstalled in Lazarus (to save one Lazarus rebuild for installing the OPM).
OPM contains a third party component(VirtualStringTree) so it cannot be preinstalled in Lazarus yet, VST must be added first.
Quote
2. Manage packages listed in 'available for installation' list box with OPM and delete them from Lazarus distribution (to keep Lazarus distribution smaller).
Good idea, though I'm not the one who decide this.
Title: Re: Online Package Manager
Post by: JuhaManninen on December 29, 2016, 02:13:01 pm
OPM contains a third party component(VirtualStringTree) so it cannot be preinstalled in Lazarus yet, VST must be added first.
You have the VirtualStringTree fork there already. It could be included in BigIde in future.
This becomes a question before the next major release ( 1.8 ). Trunk users will install it separately anyway.

Quote
Quote
2. Manage packages listed in 'available for installation' list box with OPM and delete them from Lazarus distribution (to keep Lazarus distribution smaller).
Good idea, though I'm not the one who decide this.
I moved the Industrial package. Other packages must be discussed.
Integration of available online packages with the Lazarus package system is equally important. Now the user experience is confusing.

GetMem, I tried to send you an Opkman package but mail servers rejected it as dangerous. I tested with 2 different mail accounts, one being GMail. We must find a reliable way to transfer the files ASAP!
To me the FTP server sounded very good. Write access to everybody, then an admin publishes packages that he approves. If there is too much garbage/spam then we think again. Easy peacy...

Now I attach the files here. I had to rename the .json file to .txt because it was rejected also here. Please add to online repository.
How did other people manage to send you files?
Title: Re: Online Package Manager
Post by: lainz on December 29, 2016, 02:20:08 pm
Quote
How did other people manage to send you files?

SourceForge, GitHub, personal website
Title: Re: Online Package Manager
Post by: JuhaManninen on December 29, 2016, 02:25:17 pm
SourceForge, GitHub, personal website
Ok, I read the instructions poorly in
 http://wiki.freepascal.org/Online_Package_Manager#Create_repository_package
It says: "Upload somewhere the files and send the link to getmem1@gmail.com".
Anyway, it should be made easier. Upload to a server directly from Opkman with a click.
Title: Re: Online Package Manager
Post by: minesadorada on December 29, 2016, 02:41:08 pm
I moved the Industrial package. Other packages must be discussed.
By sheer coincidence I used one of the Industrial package components for the first time today in a project.  Not much documentation, but nice controls.  Are they maintained?
Title: Re: Online Package Manager
Post by: JuhaManninen on December 29, 2016, 03:03:40 pm
Are they maintained?
Not very actively but I put my name on the wiki page as a maintainer.
Code for often used controls was collected from around and cleaned to make this package. Some more controls can be added but it should remain light.
Title: Re: Online Package Manager
Post by: GetMem on December 29, 2016, 06:17:07 pm
@Juha
Quote
You have the VirtualStringTree fork there already. It could be included in BigIde in future.
This becomes a question before the next major release ( 1.8 ). Trunk users will install it separately anyway.
Ok. This is good news. In the devlist there was a discussion about adding VST to Lazarus. After that I can remove the dependencies from OPM. This would be the best solution in my opinion.

Quote
GetMem, I tried to send you an Opkman package but mail servers rejected it as dangerous. I tested with 2 different mail accounts, one being GMail. We must find a reliable way to transfer the files ASAP!
To me the FTP server sounded very good. Write access to everybody, then an admin publishes packages that he approves. If there is too much garbage/spam then we think again. Easy peacy...
Marc is very strict about the upload policy, he does not allow any public upload to the main repository(I agree with him, it's too risky). First we have to implement that pascal cgi/fastcgi or it's php equivalent. Until then we can use my own personal ftp server, but here comes the annoying part, fpc does not have any native component to send ftp. In order to implement the feature, I have to add another third party component. The code is ready, I did it with indy, works well, but I really don't want to add one more component, so I did not commit the changes. Another solution is to upload the files to a free web server. This can be done with fphttpclient, the implementation is almost ready.

Quote
Now I attach the files here. I had to rename the .json file to .txt because it was rejected also here. Please add to online repository.
Thanks. I will add it soon to the main repository.

Also please write down in pm or here your idea about linking the two package system. Consider this: OPM is not loaded yet, however the package graph or another package functionality must access the online packages. To do this, it has to start one of OPMs  thread and through a callback function receive the data when arrives. This cannot be done only through packageintf. How do you plan to implement it, without moving a part of the code inside packager folder.

PS: I added IndustrialStuff to main repository.
Title: Re: Online Package Manager
Post by: minesadorada on December 29, 2016, 06:27:36 pm
Another solution is to upload the files to a free web server. This can be done with fphttpclient, the implementation is almost ready.
How about a newly-created SourceForge OPM_CCR project?  All the heavy lifting (including virus-scanning) is done for you.
Title: Re: Online Package Manager
Post by: GetMem on December 29, 2016, 06:35:35 pm
@minesadorada
How do you plan to upload the files(from OPM) to source forge? FTP? HTTP? Whatever method you choose you have to provide a username and password, since Lazarus is open source everyone can read the password. How do you plan to solve this issue? 
Title: Re: Online Package Manager
Post by: minesadorada on December 29, 2016, 06:48:18 pm
@minesadorada
How do you plan to upload the files(from OPM) to source forge? FTP? HTTP? Whatever method you choose you have to provide a username and password, since Lazarus is open source everyone can read the password. How do you plan to solve this issue? 
Simple.  In order to contribute a new component, you must register a SourceForge account.  This account is then added to the OPM_CCR project via an email to the OPM moderator then added (as 'developer') in order to upload files.
The OPM moderator has an 'admin-level' account in SF OPM_CCR so can control it.

The advantage is:  this already has worked for the Lazarus ccr SF project - that's where you got most of your components from.

The OPM_CCR repository acts as a 'sandbox' whereby the OPM moderator gets to decide what to include in the main repository.  It also acts as an 'update' URL for component developers to upload their update jsons/zips.

Why re-invent the wheel?

All of the above could be GitHub or Sourceforge - both are free and reliable.

The OPM instructions become: 'Make a Sourceforge account (free) and send me your SF username to add to the OPM_CCR project'

Juha's idea to directly upload from OPM is not a good one IMHO.  A component developer is perfectly capable of making a SF/GitHub account and using svn version control.
Component developers can use svn to update components via SF as I do.

I have practical experience with OPM now, and the only missing component (forgive the pun) in the chain is the initial component creation/upload to OPM.

Subsequent updating/version control is dealt with really well in the current incarnation.
Title: Re: Online Package Manager
Post by: lainz on December 29, 2016, 06:54:57 pm
I agree with minesadorada, GitHub or SourceForge is the place to share code.

Anyone can create a new repository in their own accounts and then send you the mail (like I did for all my packages).

The thing that you can automate is the sending of the mail inside the OPM.
Title: Re: Online Package Manager
Post by: minesadorada on December 29, 2016, 07:10:03 pm
Anyone can create a new repository in their own accounts and then send you the mail (like I did for all my packages).
My point is: they don't need to create a repository outside of the OPM_CCR one.  All they need is a SF username and OPM_CCR access for uploads and svn commit.  Their 'stable_release' code is held in OPM_CCR.  If they want to copy it from some other repository, that's not part of the OPM equation.

All the OPM 'action' is in the OPM_CCR SF repository under svn version control.  It is however a 'sandbox' and no-one but the OPM moderator can alter the official online repository.  Direct uploads would be a disaster-in-waiting.

I stress 'version control' because I feel this essential to successful OPM deployment in Laz 1.8, and @GetMem has rightly handled it well for installation and updates.
Title: Re: Online Package Manager
Post by: GetMem on December 29, 2016, 07:13:07 pm
We discussed this before, if a login system must be used then better implement our own and host it in the Lazarus server. Using a third party hosting doesn't look to good in my opinion, even if sourceforge and github are more then professional services. It's like microsoft servers running on linux not IIS.
Until somebody implements the login system we need a temporal solution, even if it's not 100% secure.
Title: Re: Online Package Manager
Post by: minesadorada on December 29, 2016, 07:18:30 pm
We discussed this before, if a login system must be used then better implement our own and host it in the Lazarus server. Using a third party hosting doesn't look to good in my opinion, even if sourceforge and github are more then professional services. It's like microsoft servers running on linux not IIS.
Until somebody implements the login system we need a temporal solution, even if it's not 100% secure.
Fair enough - I sort-of agree.  But a SF-based system can easily be migrated when resources are available.  I can't see your objection to a 3rd-party supplier like SF or GitHub as important enough to delay setting up an initial system.  As I have said before, CCR has been hosted by SF for ages without complaint and it formed the basis for the initial OPM packages.

The assumption I would question is:  Why is a Lazarus server more 'professional' than a SourceForge server?  Both are open-source providers.  Which would you trust more if you were new to the Lazarus project, and which could you be more sure to continue to host your component code in 10 year's time?
'More professional' is an assumption that doesn't apply to everyone.

All this only applies to a 'sandbox'.  The 'real' OPM server should be in-house, of course.
Title: Re: Online Package Manager
Post by: lainz on December 29, 2016, 07:20:23 pm
What do you need?

A place to upload all zip + json packages and then be reviewed?

A place to upload all zip + json and automatically added to OPM?

Also if is the first, you will be the only one reviewing?
Title: Re: Online Package Manager
Post by: GetMem on December 29, 2016, 07:27:30 pm
@lainz
The first one, personally I would prefer a database, it's also useful for the voting system + other stuff. After a login everyone can upload packages(zip + json + updatejson), but the packages are only visible/downloadable after moderation. A few, trusted users will have reviewing rights. I prefer not to reviewing at all, since I'm busy with other things.

minesadorada
Quote
The assumption I would question is:  Why is a Lazarus server more 'professional' than a SourceForge server?  Both are open-source providers.  Which would you trust more if you had never heard of the Lazarus project, and which could you be more sure to continue to host your code in 10 year's time?
Probably the servers are less professional since it's an open source project with limited founding, but that's the whole point, if we manage to make a pascal based, safe login/package managment system, lazarus/fpc would look more professional to other people
Title: Re: Online Package Manager
Post by: lainz on December 29, 2016, 07:31:34 pm
Ok. We know what you need.

Sorry for my lack of knowledge, a database can contain zip files and json files?

The language you want to do the server side is FPC or PHP for example? ok you edited your message, it should be FPC
Title: Re: Online Package Manager
Post by: GetMem on December 29, 2016, 07:44:31 pm
@lainz
Quote
Sorry for my lack of knowledge, a database can contain zip files and json files?
Blob fields can contain any binary format.

Quote
The language you want to do the server side is FPC or PHP for example?
If I had the possibility to choose php, the loging system would be long implemented. However some core developer insist to be fpc as a showcase. I have little experience coding server side stuff with fpc,  I hopping somebody with more experience will help me. Another solution is to learn it myself, but since my time is limited the progress will be slow.
Title: Re: Online Package Manager
Post by: minesadorada on December 29, 2016, 07:48:07 pm
Probably the servers are less professional since it's an open source project with limited founding, but that's the whole point, if we manage to make a pascal based, safe login/package managment system, lazarus/fpc would look more professional to other people
Well, I guess it's what I argued for many posts ago on this thread, so I do agree.   Good luck with implementing as secure and user-friendly a system in-house as SourceForge before Laz1.8 comes out.  You know that SF gives you a free online DB - not that it would be needed for the sandbox system as I proposed.

I keep stressing it's a 'dual' system.  The sandbox in Sourceforge, but the 'real' server written in pascal and hosted in-house (this would include the voting system and DB)  The best of both worlds, with all the security risks and version control faffle borne by SourceForge and administered by a trusted person who is the only one with access to the 'real' server.
Title: Re: Online Package Manager
Post by: GetMem on December 29, 2016, 07:57:03 pm
@minesadorada
As a temporary solution we can use sourceforge, but without a login system. We need a public account, with limited storage place(100 Mb), where everyone can upload files, but only 1-2 person can delete. The upload must be done through OPMs "Create repository package" dialog? Forget the secure part, I know is not secure as a temporary solution is ok.


Quote
I keep stressing it's a 'dual' system.  The sandbox in Sourceforge, but the 'real' server written in pascal and hosted in-house (this would include the voting system and DB)  The best of both worlds, with all the security risks and version control faffle borne by SourceForge and administered by a trusted person who is the only one with access to the 'real' server.
Yes, this can be done. Move all the possible crap to sourceforge, and keep just the links in a db located in lazarus server + voting system + users. 
Title: Re: Online Package Manager
Post by: minesadorada on December 29, 2016, 08:05:11 pm
@minesadorada
As a temporary solution we can use sourceforge, but without a login system. We need a public account, with limited storage place(100 Mb), where everyone can upload files, but only 1-2 person can delete. The upload must be done through OPMs "Create repository package" dialog? Forget the secure part, I know is not secure as a temporary solution is ok.
I think you are distracted by automation.  Why should OPM manage the upload?
The only automation OPM could usefully do is send an email with the SF username to add to the OPM_CCR project.

There is a HUGE disadvantage to the SF temporary solution which I have not mentioned.  If you are have 'developer' access, you could potentially alter other OPM_CCR menber's code.  But this has been true for the SF CCR repository for ages, and it hasn't resulted in chaos.

I could potentially mess up the entire CCR with a careless commit, and could all the others with a CCR SF account - but it has never happened!

Investigate the SourceForge permissions system for file upload and svn access - it's better than you think.
If you can make a component in Laz/fpc you can surely manage version control and uploading to a SF 'sandbox' project!  No need for OPM to hold your hand..
Title: Re: Online Package Manager
Post by: GetMem on December 29, 2016, 08:09:39 pm
Quote
The only automation OPM could usefully do is send an email with the SF username to add to the OPM_CCR project.
There is no native component to send mails in fpc/lazarus. I cannot use third party components since OPM is part of lazarus.
Title: Re: Online Package Manager
Post by: minesadorada on December 29, 2016, 08:14:24 pm
Quote
The only automation OPM could usefully do is send an email with the SF username to add to the OPM_CCR project.
There is no native component to send mails in fpc/lazarus. I cannot use third party components since OPM is part of lazarus.
@GetMem
Understood.  A dialog can do the job though, so not a showstopper.  We have to assume a component developer is not an idiot.  Too much automation is not always desirable :)

As a non cutting-edge Laz developer, I can handle getting a SF account, Installing SVN/TortoiseSVN and uploading my source with an OPM json for approval - I can even handle updates (thanks to my groovy GUI).

What I'm saying is - if I can do it - anyone can and if they can't, maybe they are not the kind of developer to reliably maintain an OPM component in Laz1.8.

There are dozens of components in SF ccr and some are in active development even today.  It works; so current active component developers seem to cope with SF svn.  The proof is in the pudding (as they say).  The change would be simple - instead of the SF CCR repository, set up an OPM_CCR repository and clone the current contents.  Then advertise via OPM - access to component developers as their definitive repository for stable versions to include in OPM.  A risk-free strategy.

I realise that the current 40+ components in OPM have been chosen for stability rather than maintainability, but the next 40 OPM components need to be judged by different criteriia.  I wrote CryptIni specifically to test the OPM system, so I have some experience to offer.

I would suggest that your main worry is being too busy to un-vet a crap component that doesn't compile and is full of bugs that will never be fixed.  This could happen with an over-automated system, and would reflect badly on Lazarus 1.8.  I have no idea what a solution for that would look like but my 'SF sandbox' proposal minimises the chance - proposed components 'sit there' until the OPM moderator has time and energy to vet them.

@Juha's ideas are sound if applied to the in-house server.  I am suggesting an extra 'sandbox' layer for security and maintainability and above all - to maintain the Lazarus reputation for solid bug-free built-in components.
Title: Re: Online Package Manager
Post by: lainz on December 29, 2016, 09:11:49 pm
@lainz
Quote
Sorry for my lack of knowledge, a database can contain zip files and json files?
Blob fields can contain any binary format.

Quote
The language you want to do the server side is FPC or PHP for example?
If I had the possibility to choose php, the loging system would be long implemented. However some core developer insist to be fpc as a showcase. I have little experience coding server side stuff with fpc,  I hopping somebody with more experience will help me. Another solution is to learn it myself, but since my time is limited the progress will be slow.

Thanks for the clarification, I will learn databases maybe the next year in university.

Of course PHP is designed specifically for that kind of websites.
Title: Re: Online Package Manager
Post by: JuhaManninen on December 30, 2016, 12:54:48 am
All the OPM 'action' is in the OPM_CCR SF repository under svn version control.  It is however a 'sandbox' and no-one but the OPM moderator can alter the official online repository.  Direct uploads would be a disaster-in-waiting.
I stress 'version control' because I feel this essential to successful OPM deployment in Laz 1.8, and @GetMem has rightly handled it well for installation and updates.
Now there is a misunderstanding somewhere. The only task now is to transfer a package generated by the Opkman to the server repository, through the admin's approval.
This transfer does not require version control. The original sources of those packages are typically under version control but that is a different topic.
Thus I don't see any reason why SourceForge or similar should be used as a "sandbox", a transfer area. DropBox or Google Docs or whatever would suit better if it must be a public commercial provider.
BTW, nobody has suggested direct upload to the final repository. The discussion is about a transfer area where Opkman could automatically move packages, to be inspected later by admin.

Quote from: minesadorada
I would suggest that your main worry is being too busy to un-vet a crap component that doesn't compile and is full of bugs that will never be fixed.  This could happen with an over-automated system, and would reflect badly on Lazarus 1.8.  I have no idea what a solution for that would look like but my 'SF sandbox' proposal minimises the chance - proposed components 'sit there' until the OPM moderator has time and energy to vet them.

@Juha's ideas are sound if applied to the in-house server.  I am suggesting an extra 'sandbox' layer for security and maintainability and above all - to maintain the Lazarus reputation for solid bug-free built-in components.
No, we don't need another layer for security and maintainability. We already have such a layer, namely a human inspection.
In the beginning many people wanted to have a fully automatic Delphinus-style system because only it would be flexible enough. Now you think even a controlled delivery through admin moderation is not enough?

The packages delivered are not part of Lazarus, they are 3rd party packages. The admin's duty is not to rate them. That's why there will be a rating system. Having some lower quality packages is part of reality.

Whatever server is used for the temporary transfer, a single public account / password should be enough initially. Later it can be improved.
I didn't know there is no FTP protocol in FPC libs, but indeed HTTP works equally well.

If I had the possibility to choose php, the loging system would be long implemented. However some core developer insist to be fpc as a showcase. I have little experience coding server side stuff with fpc,  I hopping somebody with more experience will help me. Another solution is to learn it myself, but since my time is limited the progress will be slow.
No, I don't really insist it. I only said I would like to see an FPC solution. I was hoping somebody else will join and do an initial version. Maybe not. :(
If nobody does it, please feel free to use PHP.
Title: Re: Online Package Manager
Post by: JuhaManninen on December 30, 2016, 01:12:50 am
Also please write down in pm or here your idea about linking the two package system. Consider this: OPM is not loaded yet, however the package graph or another package functionality must access the online packages. To do this, it has to start one of OPMs  thread and through a callback function receive the data when arrives. This cannot be done only through packageintf. How do you plan to implement it, without moving a part of the code inside packager folder.
PackageIntf must be extended to query / pass information about available online packages.
A possible strategy :
1. Initially OPM registers itself with a new API call and provides a callback function for further instructions.
2. The IDE later requests for a list of packages. This is an asynchronous call, no waiting.
3. OPM starts its thread and gets the list of packages. When ready, it passes it to the IDE using another new API call.

Another possible strategy :
Just do the phase number 3., get available packages without waiting instructions and pass them to IDE.

Maybe some code should be moved to Lazarus main sources but nothing related to web protocols or threading.

P.S. This thread has been read ~ 30200 times. OMG, we are famous! :)
Title: Re: Online Package Manager
Post by: GetMem on December 30, 2016, 09:46:31 am
Quote
Another possible strategy :
Just do the phase number 3., get available packages without waiting instructions and pass them to IDE.
Passing the online packages to the IDE through packageintf is not the problem. I already do something similar when I compile/install packages. The question is:
on which part of IDE loading do you plan to initialize OPM and how? Now OPM is dormant until you press the "Online package manager" menu item. The download of package info has to be done safely, in a separate thread, because if something goes wrong(main repository is offline for example) the main IDE gets blocked. This is my main concern + the incompatibility between the two package system. Yes, translating between the two is possible but is non-trivial operation.

PS: As an initial step please make a patch(don't commit it, just send it to me), where the SerializablePackages class(opkman_serializablepackages.pas) is created from the main IDE, so I can run a few tests. From then I will make all the other implementation since I'm familiar with OPM, help of course always welcomed.
Title: Re: Online Package Manager
Post by: GetMem on December 30, 2016, 10:07:24 am
@minesadorada
Quote
We have to assume a component developer is not an idiot.  Too much automation is not always desirable :)
As a non cutting-edge Laz developer, I can handle getting a SF account, Installing SVN/TortoiseSVN and uploading my source with an OPM json for approval - I can even handle updates (thanks to my groovy GUI).
What I'm saying is - if I can do it - anyone can and if they can't, maybe they are not the kind of developer to reliably maintain an OPM component in Laz1.8.
Believe it or not the vast majority of the feedback I got until now, is about simplifying the creation of repository packages/update JSON. According to some users the process is too complicated.
Title: Re: Online Package Manager
Post by: lainz on January 01, 2017, 01:00:11 pm
About the complexity of making a new package, is the idea that "you're doing it wrong" maybe one has, so I sent you all the links, and you made all of them  :)
Title: Re: Online Package Manager
Post by: GetMem on January 01, 2017, 07:51:05 pm
@lainz
Quote
About the complexity of making a new package, is the idea that "you're doing it wrong" maybe one has, so I sent you all the links, and you made all of them
I understand that feeling, but even if somethings goes wrong, no harm is done, so nothing to worry about.
Title: Re: Online Package Manager
Post by: lainz on January 01, 2017, 08:57:20 pm
Ok!

Another thing I see in lazarus is that is full of 'packages' menus and windows. I think the best is the online one you did, if in a future all of them can be combined in a single package utility will be the best. Hard isn't it? but the best =)
Title: Re: Online Package Manager
Post by: GetMem on January 04, 2017, 02:59:45 pm
With r. 53885 is possible to submit packages directly to a remote server(see screenshot). Please send any package you like, in the next 48 hours I will ignore them, because for now we only testing the new functionality. I also updated the wiki.


Title: Re: Online Package Manager
Post by: lainz on January 04, 2017, 04:27:28 pm
You work fast  8)
Title: Re: Online Package Manager
Post by: minesadorada on January 04, 2017, 05:55:45 pm
I gave it a try.  The process isn't yet intuitive.


I chose a folder, and the component was visible in the tree (good)
As soon as I clicked the component root, the [Create] and [Submit] buttons became active (not good, as I hadn't filled in any info yet)
There was no prompt nor information to click the component in order to submit compatibility information (i.e. I would not have known it was there)
I didn't test validation of entered fields

I clicked 'Create' and chose my folder, and that was it - OPM zipped the package then disappeared and I had no chance to [Submit] my created package.
What would have happened if I'd clicked 'Submit' before 'Create'? (see update below)

Ideally the Create Package process would be some kind of Wizard with validated inputs.  (That's why I did the GUI for Update jsons)

=update= I tried again with another package.
This time, after I selected the package folder I clicked [Submit].  OPM prompted me for a Category (good)
OK, so I selected a category and clicked [Submit].

OPM sent it on it's way, so you will now have a 'mystery' package with no information other than its category. (not good)
Title: Re: Online Package Manager
Post by: GetMem on January 04, 2017, 07:38:10 pm
@lainz
Quote
You work fast  8)
:D Still a lot of work to do.

@minesadorada
Thanks for testing.
After a while I'm gonna remove the "Create" button. I leave it there until the "Submit" functionality is tested enough(2-3 days max).  No more confusion after that. 

Quote
As soon as I clicked the component root, the [Create] and [Submit] buttons became active (not good, as I hadn't filled in any info yet)
There was no prompt nor information to click the component in order to submit compatibility information (i.e. I would not have known it was there)
Yes, "Create" and "Submit" becomes active, but you cannot send the info if any of the obligatory fields are not completed or the data is invalid. Ok, I can change it tomorrow so the buttons only become active when all the obligatory fields are completed. It's more intuitive.Thanks.

Quote
Ideally the Create Package process would be some kind of Wizard with validated inputs.  (That's why I did the GUI for Update jsons)
All the necessary fields are validated, you should check the source if you don't believe me. Yes the package can be send without description, author, license. But the lpk files also can exists without this fields. It tells you a lot about the package maintainer, but I cannot force a license or a description for a package that is not mine. A few posts ago you sad I'm to obsessed with automation(ok, maybe I'm  :D), but apparently you also want to validate everything.

Quote
OPM sent it on it's way, so you will now have a 'mystery' package with no information other than its category. (not good)
Here is the JSON I received, perfectly valid, it can be added to the main JSON.
Quote
{
  "PackageData0" : {
    "Name" : "cmdline",
    "DisplayName" : "",
    "Category" : "Other",
    "RepositoryFileName" : "cmdline.zip",
    "RepositoryFileSize" : 29026,
    "RepositoryFileHash" : "540b92403c061068a149a219b9140ee5",
    "RepositoryDate" : 4.2739000000000000E+004,
    "PackageBaseDir" : "cmdline\\/",
    "HomePageURL" : "",
    "DownloadURL" : "",
    "SVNURL" : ""
  },
  "PackageFiles0" : [
    {
      "Name" : "cmdbox.lpk",
      "Description" : "",
      "Author" : "",
      "License" : "",
      "RelativeFilePath" : "",
      "VersionAsString" : "0.0.0.0",
      "LazCompatibility" : "1.6, Trunk",
      "FPCCompatibility" : "2.6.4, 3.0.0",
      "SupportedWidgetSet" : "win32/64, gtk2, carbon",
      "PackageType" : 0,
      "DependenciesAsString" : "LCL, FCL(1.0.0.0)"
    }
  ]
}
Title: Re: Online Package Manager
Post by: minesadorada on January 04, 2017, 10:19:08 pm
Quote
All the necessary fields are validated, you should check the source if you don't believe me. Yes the package can be send without description, author, license. But the lpk files also can exists without this fields. It tells you a lot about the package maintainer, but I cannot force a license or a description for a package that is not mine. A few posts ago you sad I'm to obsessed with automation(ok, maybe I'm  (http://forum.lazarus.freepascal.org/Smileys/ExcellentSmileys1/biggrin.gif (http://forum.lazarus.freepascal.org/Smileys/ExcellentSmileys1/biggrin.gif))), but apparently you also want to validate everything.
Interesting point - there are two kinds of package 'creators'
Type 1. Someone who is simply submitting someone else's package for OPM.
Type 2. A package author.

There are fields which can be left blank - Homepage, DownloadURL, Description
But there are fields which shouldn't: License - what good is an online package without license info?
The compatibility fields shouldn't be guessed IMO.  The info for the package I sent you could be completely wrong - and it wouldn't be my fault, as I never saw what OPM guessed.

So whilst I see your points about the Description, Homepage and DownloadURL (for type 1 submitters)  I think the others need more scrutiny by OPM and can be proactively filled out by either kind of submitter.

Don't take the above as a criticism - I simply reviewed the create process 'as a user' to give you (hopefully) useful feedback. We're all on the same side of making OPM a deserved success.

It's a shame that many package authors in the past didn't bother with descriptions in the lpk for instance, but OPM has an opportunity to prompt for a description in the 'Create' process in an effort to improve things in future.  And.. it doesn't make sense to accept a package for distribution via Lazarus without a license type and guessed compatibility.

I'm not advocating more automation for the sake of it - just a little more 'hand holding' for users new to the interface.  It could save you a lot of grief in the future when people complain about wrong/missing information in OPM.
Title: Re: Online Package Manager
Post by: GetMem on January 05, 2017, 09:25:52 am
@minesadorada
Quote
Don't take the above as a criticism - I simply reviewed the create process 'as a user' to give you (hopefully) useful feedback. We're all on the same side of making OPM a deserved success.
Criticism is always good, especially constructive criticism, so feel free to express your views about OPM. I want to keep an open mind about new ideas.

Quote
Type 1. Someone who is simply submitting someone else's package for OPM.
So whilst I see your points about the Description, Homepage and DownloadURL (for type 1 submitters)  I think the others need more scrutiny by OPM and can be proactively filled out by either kind of submitter.
Yes this can happen, but editing the package without the original authors agreement is wrong, especially editing the license. More over, even if I add the possibility to edit the license, the user still can submit wrong data by accident or intentionally. The bottom line is somebody must check each package manually before adding it to the main repository.
I'm not against implementing such a feature, but please note all the changes has to be saved back to the lpk file and this is not implemented yet in OPM. It's not particularly hard to implement, but somebody has to do it first. If you ready to create a patch, I'm more then happy to apply it.

Title: Re: Online Package Manager
Post by: minesadorada on January 05, 2017, 11:59:00 am
No I wouldn't advocate altering a license file.  Instead I advocate OPM not accepting an unlicensed component.

As for description, updating that seems OK if the lpk description is blank.
I'd assumed that the OPM description field was read from the lpk, and then stored elsewhere (in the DB) - this is where the updated version would be - leaving the original lpk unaltered.

As for Laz compatibility, if a component is compatible with (for example) V1.6, it should also be compatible with later versions, surely?  If so, then only the earliest version needs to be stored (as in >=1.6)

In any case I'll take a look at the code and stop making assumptions :)
Title: Re: Online Package Manager
Post by: ttomas on January 05, 2017, 07:51:37 pm
Error with BGRABitmap package LCLPlatformDef not found in BRGAText.pas.
I play with OPM for the first time with fpcupdeluxe/NewPascal
FPC 3.1.1 r52736, Lazarus 1.7
Changing lcl_minor > 6 to 7 solve the problem with manual compile package.

Code: Pascal  [Select]
  1. uses GraphType, Math, BGRABlend, BGRAUTF8 {$IF (lcl_major > 0) and (lcl_minor > 7)}, LCLPlatformDef {$ENDIF};
Title: Re: Online Package Manager
Post by: GetMem on January 05, 2017, 08:59:13 pm
@ttomas
Thanks for reporting the error. LCLPlatformDef was added in Lazarus revision 53711(Dec 8. 2016). So with Lazarus trunk, the condition:
{$IF (lcl_major > 0) and (lcl_minor > 6)} 
should work. Do you have the latest Lazarus? The error only occurs if you revision is smaller then 53711.
Title: Re: Online Package Manager
Post by: minesadorada on January 15, 2017, 09:24:45 am
@GetMem Can you update PoweredBy from ccr (or the update.zip) into the repository please?
It had the same issue with Trunk as LazAutoupdate.  It's fixed now (with LCLPlatformDef), but the repository OPM version won't compile in Trunk, so it's uninstallable.
Version remains at 1.0.4.0
Title: Re: Online Package Manager
Post by: GetMem on January 15, 2017, 10:01:37 am
Quote
@GetMem Can you update PoweredBy from ccr (or the update.zip) into the repository please?
It had the same issue with Trunk as LazAutoupdate.  It's fixed now (with LCLPlatformDef), but the repository OPM version won't compile in Trunk, so it's uninstallable.
Version remains at 1.0.4.0
Done.
Title: Re: Online Package Manager
Post by: GetMem on January 17, 2017, 01:10:07 pm
Added support for multiple repositories(r. 53961).
Title: Re: Online Package Manager
Post by: minesadorada on January 17, 2017, 01:49:35 pm
Good work!

One OR the other?  Can it not check both with a user-defined priority?

I'm thinking of a component author that keeps dev versions in a private repository, or a corporate user with proprietary modified versions of components in a private repository.

It would be a checkbox option whether to check both of course.
Title: Re: Online Package Manager
Post by: GetMem on January 17, 2017, 02:15:20 pm
Quote
One OR the other?  Can it not check both with a user-defined priority?
For now you cannot use more then one repository at the same time, and there is a reason for this: each package(lpk) name must be unique otherwise the system won't work(dependency check, etc). Since I don't have control on what goes into the other repositories, I decided to implement it this way. I could always check for name collision, warn the user, remove duplicates, etc..but for now is not worth the effort, since only one person asked for the possibility to add a personal repository, and the reason was to quickly install private packages. Later the feature can be extended and more then one repository loaded at the same time.
Title: Re: Online Package Manager
Post by: lainz on January 28, 2017, 05:59:37 pm
Hi, do you remember the compilation problems of BGRABitmap on trunk? Well circular fixed that in the master branch of the repository if you want to add the official fix, maybe it differs a bit for these changed lines.

There's still no new version..
Title: Re: Online Package Manager
Post by: GetMem on January 28, 2017, 07:03:17 pm
Quote
Hi, do you remember the compilation problems of BGRABitmap on trunk? Well circular fixed that in the master branch of the repository if you want to add the official fix, maybe it differs a bit for these changed lines.
Thanks @lainz, BGRABitmap updated. I also added a few interesting package lately.
Title: Re: Online Package Manager
Post by: lainz on January 28, 2017, 07:50:48 pm
Thanks. Maybe there's a way (of course) to make a website from the JSON file automatically. So there's a website in http://packages.lazarus-ide.org/ instead of that text. I will try to do something, if not at least you get the idea.
Title: Re: Online Package Manager
Post by: GetMem on January 28, 2017, 08:40:06 pm
Quote
Maybe there's a way (of course) to make a website from the JSON file automatically. So there's a website in http://packages.lazarus-ide.org/ instead of that text. I will try to do something, if not at least you get the idea
It is in my to do list, but first there are more important issues like:
  - integrate OPM into the package system, a very difficult issue by the way. Ideally it should be ready before 1.8(2.0).
  - implement the voting system
  - create personal repository(almost completed)
Regarding the website I plan to make a responsive page, with thumbnails. Every package will be downloadable via a browser.
Title: Re: Online Package Manager
Post by: lainz on January 28, 2017, 09:28:19 pm
Thankyou. You can try this small attached website?

The template is from here:
https://getmdl.io/templates/index.html (Text heavy webpage)

There's a simple website, is not what you need but you can use it now until you get your website done.
Title: Re: Online Package Manager
Post by: GetMem on January 28, 2017, 09:57:06 pm
Actually this is quite nice, thank you @lainz!  :)  I envisioned something similar. Now the packages are also available via browsers.

@all
Please take a look: http://packages.lazarus-ide.org/
 
Title: Re: Online Package Manager
Post by: lainz on January 28, 2017, 10:59:27 pm
Nice! I forget to add the URL's of the packages, you can add them easily if you want.
Title: Re: Online Package Manager
Post by: GetMem on January 29, 2017, 08:50:09 am
Quote
Nice! I forget to add the URL's of the packages, you can add them easily if you want.
Ok. I will make a few, small adjustment soon.
Title: Re: Online Package Manager
Post by: minesadorada on January 29, 2017, 10:01:11 am
Perhaps a forum admin will add a link to the website in the main sidebar once it's done.
Title: Re: Online Package Manager
Post by: wp on January 29, 2017, 10:26:50 am
@all
Please take a look: http://packages.lazarus-ide.org/ (http://packages.lazarus-ide.org/)

mbColorLib and jvcllaz are shown without a title.

Can you order by package name or by date of last change (latest change at top)?
Title: Re: Online Package Manager
Post by: GetMem on January 29, 2017, 11:00:47 am
Quote
mbColorLib and jvcllaz are shown without a title.
Fixed. Also renamed mbColorLib to MBColorLib and jvcllaz to JVCLLaz, since all the packages begins with upercase in the list. This is just the display name(webpage and OPM tree). I did not touch the zip file or any other part of the package. Hope it's ok, if not I can change it back.

Quote
Can you order by package name or by date of last change (latest change at top)?
I will implement it tomorrow.
Title: Re: Online Package Manager
Post by: wp on January 29, 2017, 11:05:14 am
Hope it's ok, if not I can change it back.
OK, no problem.
Title: Re: Online Package Manager
Post by: GetMem on January 30, 2017, 11:56:30 am
Quote
@wp
Can you order by package name or by date of last change (latest change at top)?
Now the packages are ordered by name. I will add more sorting option in the future + a search field: http://packages.lazarus-ide.org/

In the package tree, the newly added packages(<= 31 days) are highlighted by a different icon + the date.
 
Title: Re: Online Package Manager
Post by: wp on January 30, 2017, 12:41:29 pm
Great!
Title: Re: Online Package Manager
Post by: minesadorada on January 30, 2017, 04:41:28 pm
It just gets better...
Title: Re: Online Package Manager
Post by: lainz on February 04, 2017, 11:33:28 pm
Hi GetMem, if you let me I want to implement myself the search for the packages site, I have already a working solution for another project of mine using Material Design Lite that works just fine without using third party libraries.
Title: Re: Online Package Manager
Post by: GetMem on February 04, 2017, 11:53:04 pm
@lainz

Feel free to implement whatever you like, my time is very limited nowadays. Just let me know when I have to upload something to the server.
Thank you for your help, it's much appreciated!
Title: Re: Online Package Manager
Post by: lainz on February 05, 2017, 12:53:19 am
@lainz

Feel free to implement whatever you like, my time is very limited nowadays. Just let me know when I have to upload something to the server.
Thank you for your help, it's much appreciated!

Here it is. Feel free to change the About page or anything.
Title: Re: Online Package Manager
Post by: GetMem on February 05, 2017, 09:17:25 am
@lainz

Thank you! I uploaded the files to the main repository: http://packages.lazarus-ide.org/

Quote
Feel free to change the About page or anything.
I like the way it is.  ;)
Title: Re: Online Package Manager
Post by: lainz on February 05, 2017, 10:45:25 am
Thanks. One thing that can be optimized is the search, it searches each keyword (separated by single space) in each component element. Is fast, but maybe there's a better way like using a regular expression?
Title: Re: Online Package Manager
Post by: GetMem on February 05, 2017, 11:11:11 am
I ran a few test and the search is fast. We only have 58 packages and I seriously doubt we ever go beyond 150-200, for now I don't think we need a more optimized search. I'm not against it though, if somebody wants to implement it.
Title: Re: Online Package Manager
Post by: lainz on February 05, 2017, 11:20:41 am
OK. Is not so smart the search but it does the job of filtering at least.

A thing we should add is a list of the categories in a combobox to click and filter automatically. I will try to add that.
Title: Re: Online Package Manager
Post by: lainz on February 05, 2017, 12:41:10 pm
Here we go :)

Now it has the categories.
https://github.com/lazarusccr/LazarusPackagesRepository

Edit: You can add category 'Panels' to BGRAControls and BGRAControlsFX? These contains panel components.

To BGRAControls also 'Edit and Memos' because it contains edit controls too, 'Labels' if it's the standard label it contains it too. Also 'List and ComboBoxes' that contains too (both). 'Shapes' contains also TBGRAShape. Is a big package  :D

I need to improve the description of my packages, I will upgrade them and I will tell you when is ready.
Title: Re: Online Package Manager
Post by: GetMem on February 05, 2017, 02:26:43 pm
Quote
Here we go :)
Now it has the categories.
https://github.com/lazarusccr/LazarusPackagesRepository
Great! Thanks. Repository updated.  :)

Quote
Edit: You can add category 'Panels' to BGRAControls and BGRAControlsFX? These contains panel components.
To BGRAControls also 'Edit and Memos' because it contains edit controls too, 'Labels' if it's the standard label it contains it too. Also 'List and ComboBoxes' that contains too (both). 'Shapes' contains also TBGRAShape. Is a big package  :D
I need to improve the description of my packages, I will upgrade them and I will tell you when is ready.
Sure, just let me know when the packages are ready.


PS: I also post a list with available packages, if somebody knows a package that is not in the list, please let me know.
Code: Pascal  [Select]
  1. ACS - Audio Components Suite
  2. ATBinHex
  3. ATButtons/ATListbox
  4. ATFileNotif
  5. ATGauge
  6. ATScrollBar
  7. ATSynEdit
  8. BGRABitmap
  9. BGRAControls
  10. BGRAControlsFX
  11. BGRAGames
  12. CalLite
  13. CmdLine
  14. ColorPalette
  15. CryptINI - Crypted TIniFile
  16. DataPort
  17. DCPcrypt
  18. EpikTimer
  19. EyeCandyControls
  20. FileAssociation
  21. FileMenuHandler
  22. Fortes4Lazarus
  23. FortesReport-CE
  24. FPSpreadsheet
  25. HistoryFiles
  26. IndustrialStuff
  27. Indy
  28. JujiboUtils
  29. JVCLLaz
  30. KaajControls
  31. KControls
  32. LainzCodeStudio
  33. LazAutoUpdate
  34. LazBarcodes
  35. LazSerial
  36. LCLExtensions
  37. LNet
  38. LongTimer
  39. MBColorLib
  40. MPlayer
  41. PlaysoundPackage
  42. PlotPanel
  43. PoweredBy
  44. PowerPDF
  45. RichMemo
  46. RingWatch
  47. ScrollText
  48. SMNetGradient
  49. SpkToolbar
  50. Synapse 40
  51. Synapse 40.1
  52. TVPlanit
  53. UEControls
  54. UniqueInstance
  55. VirtualTreeView 4.8.7_R4
  56. VirtualTreeView 5.5.3_R1
  57. WebServiceToolkit
  58. ZeosDBO
Title: Re: Online Package Manager
Post by: lainz on February 05, 2017, 03:13:34 pm
There's UOS
https://fredvs.github.io/uos/

And Bare.Game
https://github.com/sysrpl/Bare.Game

Title: Re: Online Package Manager
Post by: GetMem on February 05, 2017, 03:24:00 pm
UOS is a great component, however it does not contain a package file(lpk). Believe it or not OPM cannot handle a package with no lpk files.  :D

I will add Bare.Game.
Title: Re: Online Package Manager
Post by: lainz on February 05, 2017, 03:27:43 pm
I know. We can contact FredVS and tell him if he can add a package file to his project.
Title: Re: Online Package Manager
Post by: GetMem on February 05, 2017, 10:53:44 pm
Hi FredVS,

Thank you for the package. I modified the lpk file(hope it's ok), added a description + author name. All the info is taken from your github page: https://github.com/fredvs/uos/. Please check attachment, if something is wrong, let me know.

regards,
GetMem
Title: Re: Online Package Manager
Post by: Fred vS on February 06, 2017, 11:24:58 am
Hi FredVS,

Thank you for the package. I modified the lpk file(hope it's ok), added a description + author name. All the info is taken from your github page: https://github.com/fredvs/uos/. Please check attachment, if something is wrong, let me know.

regards,
GetMem

Hello GetMem.

Ok, perfect, many thanks and wow for your great project.

Fre;D

Title: Re: Online Package Manager
Post by: lainz on February 14, 2017, 04:35:51 pm
We need to thank Martin_fr that added the 'Packages' link into the navigation bar of the forum at the left under 'Lazarus'.

Enjoy the website and OPM!
Title: Re: Online Package Manager
Post by: GetMem on February 14, 2017, 05:06:16 pm
Thank you @Martin_fr for adding the link and thank you @lainz for creating the page. I will also try to contribute as time permits.
Title: Re: Online Package Manager
Post by: howardpc on February 14, 2017, 05:08:35 pm
I noticed that the VirtualTreeView 4.3.7_R4 Description entry has two spurious "line1" and "line2" entries at the end of the sentence.
Title: Re: Online Package Manager
Post by: GetMem on February 14, 2017, 05:33:28 pm
Quote
@howarpc
I noticed that the VirtualTreeView 4.3.7_R4 Description entry has two spurious "line1" and "line2" entries at the end of the sentence.
Thank you. VirtualTreeView 4.8.7_R4 was one of the first packages added to OPM. I was testing the multi-line capabilities of JSON. Apparently I forgot to remove line1, line2 :D

PS: Fixed!
Title: Re: Online Package Manager
Post by: jacmoe on February 19, 2017, 05:54:28 am
I read the topic - well, the first 12 pages - before I decided to post this anyway:

The OPM does not correctly configure itself when used with Fpcupdeluxe.

I do have FPC installed globally. More or less because I couldn't be bothered to remove all FPC related packages.
And OPM thinks that that's all I've got.
No Lazarus.
Even though it should know that it can't be run outside of it.

I could - and I will - configure OPM manually - but using a self-compiled Lazarus/FPC is quite common, isn't it?
Besides, Lazarus knows how it's configured, so can't OPM just ask it?

Have anyone successfully installed and used this with Fpcupdeluxe?
Title: Re: Online Package Manager
Post by: GetMem on February 19, 2017, 06:43:14 am
Hi jacmoe,

I read your post more then once, still I'm not sure what are you after:
Quote
I do have FPC installed globally. More or less because I couldn't be bothered to remove all FPC related packages.
And OPM thinks that that's all I've got.
OPM does not deal with FPC packages only Lazarus packages.

Quote
I could - and I will - configure OPM manually - but using a self-compiled Lazarus/FPC is quite common, isn't it?
Besides, Lazarus knows how it's configured, so can't OPM just ask it?
That's exactly what OPM does. Asks lazarus where the config files are stored and save the newly downloaded/installed packages in the same folder. This way the primary and secondary installs are completely separated.

Quote
Have anyone successfully installed and used this with Fpcupdeluxe?
Yes, I just did. As far as I can tell it works as it should, unless I misunderstood something?
Title: Re: Online Package Manager
Post by: jacmoe on February 19, 2017, 07:00:25 am
Move along ..

I was totally wrong - posted about a wrong version of a package manager in the wrong topic.  :-[
Title: Re: Online Package Manager
Post by: GetMem on February 19, 2017, 07:09:14 am
As I said OPM doesn't deal with FPC packages and does not use Inet to download packages. At this point I'm completely lost and I have absolutely no idea what are you talking about.
Title: Re: Online Package Manager
Post by: GetMem on February 19, 2017, 07:41:44 am
Quote
Am I using the right thing?
No.  :)

OPM is this(see attachment). @jacmoe it's ok to be wrong, but next time please don't blame others for your own fault.

PS: I see you deleted your post. Anyway test OPM(the real one) and tell me what do you think.
Title: Re: Online Package Manager
Post by: jacmoe on February 19, 2017, 07:45:58 am
The wiki said that "This" sits in Lazarus, but I guess it sits in Trunk Lazarus and not 6.0.3 Lazarus.
Thanks for you patience!

I used the Blaszyk package manager.  ;)

I can definitely see how you can get confused by such irrelevant bug report.

Great to hear that it works with Fpcupdeluxe.
I will try it later.

it's ok to be wrong, but next time please don't blame others for your own fault.
Sometimes it takes time until you realise that you are at fault.
Sorry! Carry on the good work.  :-[ :-X
Title: Re: Online Package Manager
Post by: jacmoe on February 19, 2017, 08:21:18 am
So, because Fpcupdeluxe failed to install OPM, probably because it is supposed to be part of Lazarus, I just issued a svn checkout in lazarus/components ..
I opened the package, compiled it, and opened the package install dialog and installed it. Rebuilt Lazarus..
And it is installed!
Settings seems to be alright.
I haven't tried to install anything yet.
Really nice!  :D

Edit:
Castle Game Engine should be in it!
Title: Re: Online Package Manager
Post by: GetMem on February 19, 2017, 08:35:18 am
@jacmoe
I'm glad it's working :). If you find a bug please report it, I will do my best to fix the issue.
Title: Re: Online Package Manager
Post by: JuhaManninen on February 19, 2017, 09:06:57 am
I used the Blaszyk package manager.  ;)
I think I will remove it completely now.
I anticipated confusion (as now happened) and once removed its "global link" so it doesn't show in the Install/Uninstall Packages window, but it got restored.
Blaszyk package manager does not work and nobody has worked on it for many many years.

Quote
Castle Game Engine should be in it!
Please create a package and provide it to GetMem. I still hope in future it can be done with a single click.
Title: Re: Online Package Manager
Post by: jacmoe on February 19, 2017, 09:16:15 am
Sorry about the confusion and added drama. :)

I will look into creating OPM packages for Castle if the Castle project managers doesn't beat me to it.

I will also make sure to report bugs, now that I have triple-checked that I am indeed using the right online package manager.

Sounds good that the old Blaszyk package gets removed.
Less confusion is always a good thing.  ;)
Title: Re: Online Package Manager
Post by: GetMem on February 19, 2017, 09:33:31 am
Quote
@jacmoe
Castle Game Engine should be in it!
Agree. Great Engine/Component. I will add it soon to the main repository.

Quote
@Juha
I still hope in future it can be done with a single click.
What do you mean? I added a submit button at least a month ago. It can be done with a single click, or are you referring to something else? 
Title: Re: Online Package Manager
Post by: jacmoe on February 19, 2017, 09:34:40 am
He probably means that creating a package is going to take just one click in the future.
Title: Re: Online Package Manager
Post by: GetMem on February 19, 2017, 10:42:57 am
Quote
@jacmoe
He probably means that creating a package is going to take just one click in the future.
That's not possible, when you submit a package you have to do at least two things: select a category and check the lpk's. Anyway the whole process shouldn't take more then 1-2 minutes. I think @Juha talks about the fact that in the past you had to send the packages through mail. This is not true anymore. It can be done with a single click now.

I added Castle Game Engine to main repository.
Title: Re: Online Package Manager
Post by: JuhaManninen on February 19, 2017, 11:07:39 am
What do you mean? I added a submit button at least a month ago. It can be done with a single click, or are you referring to something else?
Oops, sorry. I haven't been following the development all the time.
Submit button sounds good. Makes contributing packages easier and thus more package maintainers will do it in future.

I removed LazarusPackageManager in r54189.
Title: Re: Online Package Manager
Post by: GetMem on February 19, 2017, 11:44:13 am
Quote
Oops, sorry. I haven't been following the development all the time.
Submit button sounds good. Makes contributing packages easier and thus more package maintainers will do it in future.
Things are looking good so far. I received at least 10 package in the last month. New packages are highlighted with a blue lightning icon(see attachment).

Quote
I removed LazarusPackageManager in r54189.
Thanks.
Title: Re: Online Package Manager
Post by: jacmoe on February 19, 2017, 06:03:57 pm
Thanks a lot for adding Castle! ;D

Castle and OPM and Fpcupdeluxe is a Ninja Combination!  :D

New packages are highlighted with a blue lightning icon.
Ah, that makes sense.
I have been trying to figure out the difference between the blue flash / regular package icons for a while.  ;D
Title: Re: Online Package Manager
Post by: lainz on February 19, 2017, 06:49:24 pm
Hi, such a big package the Castle Engine, how many rooms has this Castle  ::)

But seriously I can't install it with FPC trunk and Lazarus trunk.

I get errors when installing it, Internal error 2014081401. This line

Code: Pascal  [Select]
  1.     Handled := Items[I](Received) or Handled;    

at Castle Messaging unit.

"{ Message system to communicate between native code and Java on Android
  (TMessaging). }"

And maybe I need some android stuff to compile that unit?

Sorry if I'm posting it here, I did that because i'm installing it through OPM. Also I will copy this to the Castle Engine thread.
Title: Re: Online Package Manager
Post by: GetMem on February 19, 2017, 06:58:10 pm
Quote
@jacmoe
Thanks a lot for adding Castle! ;D
You're welcome.

Quote
Ah, that makes sense.
I have been trying to figure out the difference between the blue flash / regular package icons for a while.
Blue lightning icons appears in front of the packages not older then 31 days. You can configure the value from the options though.

@lainz
Quote
But seriously I can't install it with FPC trunk and Lazarus trunk.
I get errors when installing it, Internal error 2014081401. This line
Thanks for the feedback. I only tested with Lazarus Trunk/FPC 3.0/Win7. It works fine. You should contact the package maintainer about the issues in trunk.

Quote
Sorry if I'm posting it here, I did that because i'm installing it through OPM. Also I will copy this to the Castle Engine thread.
No problem. I will run a few test later with Laz/FPC trunk. I just installed a secondary Lazarus with fpcdeluxe, which is a great tool by the way.
Title: Re: Online Package Manager
Post by: lainz on February 19, 2017, 07:06:40 pm
Yes I also have trunk trunk with fpcupdeluxe, combined with OPM is the only way I use lazarus now.
Title: Re: Online Package Manager
Post by: Akira1364 on February 19, 2017, 09:22:29 pm
Hi, such a big package the Castle Engine, how many rooms has this Castle  ::)

But seriously I can't install it with FPC trunk and Lazarus trunk.

I get errors when installing it, Internal error 2014081401. This line

Code: Pascal  [Select]
  1.     Handled := Items[I](Received) or Handled;    

at Castle Messaging unit.

"{ Message system to communicate between native code and Java on Android
  (TMessaging). }"

And maybe I need some android stuff to compile that unit?

Sorry if I'm posting it here, I did that because i'm installing it through OPM. Also I will copy this to the Castle Engine thread.

It's just a compiler bug, as the way they wrote it should technically work (but obviously doesn't, sometimes.) I replied to your copied post in the announcement thread with a solution I've found to work fine.
Title: Re: Online Package Manager
Post by: esvignolo on February 20, 2017, 03:27:49 am
Very good work with the package manager, something that is essential to make it easier for new developers to know and simplify this task.
I just want to make a small suggestion, i just had an update of a package, but when I selected and installed it, the update still appeared to me even after i had recompiled.
Therefore, my humble suggestion, is to simplify in a single button to update/install and if the repository was different from the official, warn with a message like this:
You Has the latest version of the official repository, do you want to download a new version from the page of the developer?

Greetings and thanks.
Title: Re: Online Package Manager
Post by: lainz on February 20, 2017, 03:55:21 am
Hi, I agree with esvignolo, he was trying to update bgracontrols from external sources.

I know how it works and now he knows too. But for someone that never used it it can give you the impression that is not working.

A suggestion can be, if you already have that version installed, say

bgracontrols 4 installed, and you choose bgracontrols 4 to install again, and at the same time there's an external version available say bgracontrols 4.x (external update), when you click on install a prompt that says 'you want to re-install the same version or update to the external new version?, two buttons [Re-install bgracontrols 4] or [Update to bgracontrols 4.x]'

Or some better approach, I don't know  %)

Thanks again for this excelent tool.
Title: Re: Online Package Manager
Post by: GetMem on February 20, 2017, 08:29:39 am
@esvignolo, @lainz
     Thank you for the feedback. I know the update Column/Button is somewhat confusing. Initially, I only wanted a central repository + a tool to easily update the repository. Unfortunately this raises a few security issues. I don't want to go into the details since it was discussed many times before in this thread. So instead we choose the so called "Update" feature, which allows the package maintainer to easily notify a user about a new update. The update feature has at least two drawbacks, beside the confusion:
    1. You have to install the package from the central repository first. Although technically OPM can install directly from an update page, it's not allowed because of the possible unresolved dependency issues
    2. It's not 100% safe. I mean a package maintainer with bad intention, it can(at least in theory) install a malware, especially under windows
On the other hand the update feature also has a great advantage: it removes the workload from the central repository and pass it to the package maintainers, this way making the whole system sustainable in the long run.

Unifying the Install/Update button(@esvignolo suggestion) I'm not sure will solves the problem, in my opinion it will lead to even more confusion. The user will have no idea from where the package is installed(repository or update). Now at least you know(again I agree it's a little bit confusing), especially if your read the wiki page(section "Difference between download/install/update"): http://wiki.freepascal.org/Online_Package_Manager#Difference_between_download.2Finstall.2Fupdate
Title: Re: Online Package Manager
Post by: lainz on February 20, 2017, 07:45:41 pm
It's ok everything needs some user learning btw, is not too hard.

Edit: I know everything was discussed so keep going.
Title: Re: Online Package Manager
Post by: GAN on February 24, 2017, 12:48:05 am
Hi, I don't found On Line Package Manager on http://packages.lazarus-ide.org/ can you give me a direct download link?

Thank you.
Title: Re: Online Package Manager
Post by: jacmoe on February 24, 2017, 01:19:44 am
Check the wiki for how to get it: http://wiki.freepascal.org/Online_Package_Manager

Edit:
If it's not in ($LazarusDir)/components/onlinepackagemanager then you need to get it from SVN.
Go to ($LazarusDir)/components and run this command:
Code: Pascal  [Select]
  1. svn co http://svn.freepascal.org/svn/lazarus/trunk/components/onlinepackagemanager onlinepackagemanager

If it is there, skip the svn check out.

Now, open the package, compile it, and install it in the Lazarus Install Packages dialogue.

See the wiki  :)
Title: Re: Online Package Manager
Post by: GAN on February 24, 2017, 02:06:46 am
Yes, I saw the wiki, but I don't use SVN, so, I need a  link to a for a zip if it is posible.

Thanks.
Title: Re: Online Package Manager
Post by: molly on February 24, 2017, 02:11:06 am
Yes, I saw the wiki, but I don't use SVN, so, I need a  link to a for a zip if it is posible.
ftp://ftp.freepascal.org/pub/lazarus/snapshots/lazarus-1.7-54258-20170224-src.zip

or better yet (in order to not stress the ftp server), surf to https://github.com/graemeg/lazarus and press the download button.
Title: Re: Online Package Manager
Post by: GAN on February 24, 2017, 02:40:13 am
Yes, I saw the wiki, but I don't use SVN, so, I need a  link to a for a zip if it is posible.
ftp://ftp.freepascal.org/pub/lazarus/snapshots/lazarus-1.7-54258-20170224-src.zip

or better yet (in order to not stress the ftp server), surf to https://github.com/graemeg/lazarus and press the download button.

Thanks, but I only need Online Package Manager. I am using Lazarus 1.6 FCC 3.0.0.
Title: Re: Online Package Manager
Post by: molly on February 24, 2017, 02:49:15 am
Yes, I saw the wiki, but I don't use SVN, so, I need a  link to a for a zip if it is posible.
ftp://ftp.freepascal.org/pub/lazarus/snapshots/lazarus-1.7-54258-20170224-src.zip

or better yet (in order to not stress the ftp server), surf to https://github.com/graemeg/lazarus and press the download button.

Thanks, but I only need Online Package Manager. I am using Lazarus 1.6 FCC 3.0.0.
OPM is part of Lazarus. There is no standalone download.
Title: Re: Online Package Manager
Post by: jacmoe on February 24, 2017, 02:54:57 am
Yes, I saw the wiki, but I don't use SVN, so, I need a  link to a for a zip if it is posible.
ftp://ftp.freepascal.org/pub/lazarus/snapshots/lazarus-1.7-54258-20170224-src.zip

or better yet (in order to not stress the ftp server), surf to https://github.com/graemeg/lazarus and press the download button.

Thanks, but I only need Online Package Manager. I am using Lazarus 1.6 FCC 3.0.0.
OPM is part of Lazarus. There is no standalone download.
If 'Lazarus' means 'Lazarus trunk' then: yes.
If not - ie tagged or fixes ' then: no.
Not on my box, it isn't.

So, if the OP does not have trunk, then he must get the trunk of Lazarus, so that he can put OPM into his non-trunk Lazarus source tree. ;)
Title: Re: Online Package Manager
Post by: molly on February 24, 2017, 03:01:20 am
@jacmoe:
I'll be more explicit then  :)
OPM is part of Lazarus 1.7. There is no standalone download.
Title: Re: Online Package Manager
Post by: jacmoe on February 24, 2017, 03:06:49 am
@molly: I know that. But since GAN is using 1.6, he must grab the 'onlinepackagemanager' directory from the source of Lazarus 1.7 (trunk).
I think it is really great that OPM is part of Lazarus now. :D

I am using 1.6.3 and had to check out the opm package directory (from the trunk) too.

You are right: there is no standalone download. And there will never be one, because it is part of Lazarus 1.7 onwards.

The easiest for GAN would be to simply install subversion and do a check out of that directory.
That's the option that is smallest, and takes the least amount of bandwidth.
Title: Re: Online Package Manager
Post by: molly on February 24, 2017, 03:13:03 am
@jacmoe:
And... i understand that  :)

It's GAN who did not got the message.

It would be the same as asking for a separate download for TLabel component. It is part of lazarus so download lazarus in order to get it ;-)

(and in the particular case of OPM you would need lazarus trunk sources for that, exactly where my links pointed to).

fyi: OPM did start out with its own repository.
Title: Re: Online Package Manager
Post by: jacmoe on February 24, 2017, 03:20:40 am
I did exactly the same thing here -> http://forum.lazarus.freepascal.org/index.php/topic,34297.msg238533.html#msg238533 (http://forum.lazarus.freepascal.org/index.php/topic,34297.msg238533.html#msg238533)
I had Lazarus 1.6.3 (IIRC) and just checked out onlinepackagemanager from trunk into lazarus_1.6.3_source/components/onlinepackagemanager.
So, he must install subversion if he wants the easiest method.
Or, alternatively, download the zip ball from Github, extract it and copy the 'onlinepackagemanager' directory over to the components source directory.

Edit: I think we agree with each other, Molly  ;)
Title: Re: Online Package Manager
Post by: GAN on February 24, 2017, 05:27:17 am
OK now I understand, thanks for the explanation and sorry for my confusion.

Best regards.
Title: Re: Online Package Manager - own config settings?
Post by: CharlyTango on March 08, 2017, 02:35:15 pm
HI,

I'm facing the challenge to sync my the packages in different Lazarus installations  between two desktops and a notebook.
Furthermore there are several Lazarus versions per desktop or notebook to sync.
Is there a possibility to tell OPM to use some kind of configuration file to install the same Packages into all my Lazarus installations without chosing the Packages by hand ?

I was thinking of a config file which is synched by Dropbox or others and every Lazarus installation installs the fitting versions of the needed packages ;)

THX in advance
Title: Re: Online Package Manager
Post by: GetMem on March 08, 2017, 04:20:47 pm
1. You check all the packages you wish to install, save it to a config file.
2. Next time you load the checks from the same ini, so you don't have to check them again manually

 Did I understand correctly?
Title: Re: Online Package Manager
Post by: CharlyTango on March 09, 2017, 10:35:57 am
Yes you understood correctly.

But there should be a possibility to tell OPM where this (kind-of-)ini file is located. (name and path)

This way it would be possible to sync the ini-file automatically by Dropbox or GooglDrive etc.


BTW... is there a possibility that OPM updates itself in an other way than a SVN download.
Could OPM be a simple package which is updated by OPM itself?

Regards
Title: Re: Online Package Manager
Post by: GetMem on March 09, 2017, 09:51:20 pm
Quote
Yes you understood correctly.
But there should be a possibility to tell OPM where this (kind-of-)ini file is located. (name and path)
This way it would be possible to sync the ini-file automatically by Dropbox or GooglDrive etc.
It's implemented in trunk. Just right click the tree, then save/load the ini file.

Quote
BTW... is there a possibility that OPM updates itself in an other way than a SVN download.
Could OPM be a simple package which is updated by OPM itself?
Unfortunately no. It's in the todo list though.
Title: Re: Online Package Manager
Post by: minesadorada on March 20, 2017, 01:04:54 pm
I just updated Lazarus and FPC from SVN and rebuild IDE now hangs on the statements (in OPM.opkman_downloader)

Code: Pascal  [Select]
  1. FHTTPClient.Terminate;

Lazarus SVN build: 54442M
Title: Re: Online Package Manager
Post by: GetMem on March 20, 2017, 09:10:03 pm
Hi minesadorada,

OPM has changed a lot lately, however I cannot reproduce the error with Lazarus/FPC trunk(r. 54449 and r. 35631). I did try to download/install as many package as possible, but no hangs, no errors. I only tested on win7 though.
Title: Re: Online Package Manager
Post by: JuhaManninen on March 20, 2017, 09:48:51 pm
minesadorada, usually a clean build helps after code has been moved and reorganized.
Tools -> Configure Build Lazarus -> "Clean all".

I don't get errors either.
Title: Re: Online Package Manager
Post by: minesadorada on March 21, 2017, 09:14:02 am
OK Thanks guys - I'll investigate.
Title: Re: Online Package Manager
Post by: Alextp on March 24, 2017, 07:23:37 am
Hi
In mantis, you ll find n my fixes/patches:
http://mantis.freepascal.org/view.php?id=31495
http://mantis.freepascal.org/view.php?id=31497
http://mantis.freepascal.org/view.php?id=31498
http://mantis.freepascal.org/view.php?id=31499
http://mantis.freepascal.org/view.php?id=31501

Can you apply pls?
Title: Re: Online Package Manager
Post by: GetMem on March 24, 2017, 07:32:54 am
Hi Alextp,

Sorry for the delay, I was very busy lately. I will apply them as soon as possible. Thank you for the patch!

PS: Applied, except one where a little modification is needed.
Title: Re: Online Package Manager
Post by: Rayvenhaus on March 24, 2017, 01:41:50 pm
Running Lazarus 1.6.2, FPC 3.0.0, when compiling latest OPM from SVN, I get the following error:

Compile package OnlinePackageManager 1.0: Exit code 1, Errors: 1
opkman_mainfrm.pas(39,16) Fatal: Cannot find PackageLinkIntf used by opkman_mainfrm of package OnlinePackageManager.
Title: Re: Online Package Manager
Post by: Thaddy on March 24, 2017, 02:14:35 pm
You are using unsupported versions: Laz is at 1.6.4 and fpc is at 3.0.2.
Title: Re: Online Package Manager
Post by: Rayvenhaus on March 24, 2017, 02:47:24 pm
Ok, just upgraded to latest version, Lazarus 1.6.4 and FPC 3.0.2 and I still get the same error when compiling OpMan.  Where do I find the missing package needed to compile OpMan?
Title: Re: Online Package Manager
Post by: JuhaManninen on March 24, 2017, 03:57:21 pm
Where do I find the missing package needed to compile OpMan?
In Lazarus trunk. The OpMan is now part of Lazarus sources anyway, it is developed together with rest of the trunk.
Already next month Lazarus 1.8 may be forked. Then you can use also fixes_1_8 branch.

I know OpMan has supported older Lazarus versions with IFDEFs. I don't have objections if somebody wants to add IFDEFs but it is quite unusual. If somebody wants to use latest development from trunk, he can download trunk. This applies to ProjectGroups and any other package in trunk as well.
Title: Re: Online Package Manager
Post by: Rayvenhaus on March 24, 2017, 04:02:33 pm
I am using the latest released versions of Lazarus and FPC (1.6.4 & 3.0.2) and I still get the error when adding in OPM
Title: Re: Online Package Manager
Post by: GetMem on March 24, 2017, 04:15:49 pm
@Rayvenhaus

You have two choices:
1. Use a slightly older version, you can download from here(zip): https://drive.google.com/file/d/0B9Me_c5onmWoLUJmYUNwc0ItVkk/view?usp=sharing
2. Wait for Laz 1.8, it's just around the corner
Title: Re: Online Package Manager
Post by: GetMem on March 24, 2017, 04:20:18 pm
Quote
@Juha
I know OpMan has supported older Lazarus versions with IFDEFs. I don't have objections if somebody wants to add IFDEFs but it is quite unusual. If somebody wants to use latest development from trunk, he can download trunk. This applies to ProjectGroups and any other package in trunk as well.
I don't think it's worth ifdef-ing the latest changes. After Lazarus 1.8 is forked, we drop support for older versions. I can upload somewhere a zip with an older OPM, in case somebody insist to stay with the 1.6 series.
Title: Re: Online Package Manager
Post by: Rayvenhaus on March 24, 2017, 04:47:56 pm
So, the current version of OPM does not work with the latest released version of Lazarus? I either have to roll back a version of Lazarus or wait till a new version of Lazarus is released?
Title: Re: Online Package Manager
Post by: Thaddy on March 24, 2017, 04:54:38 pm
It's a matter of hours or days rather than weeks if I understand the Laz devs correctly.
Title: Re: Online Package Manager
Post by: JuhaManninen on March 24, 2017, 04:56:47 pm
@Rayvenhaus, the issue was quite clearly explained by myself and by GetMem.
GetMem even copied a compatible version to a server for you.

It's a matter of hours or days rather than weeks if I understand the Laz devs correctly.
What is a matter of hours or days?
Title: Re: Online Package Manager
Post by: GetMem on March 24, 2017, 05:00:34 pm
Quote
So, the current version of OPM does not work with the latest released version of Lazarus?

Yes.

Quote
I either have to roll back a version of Lazarus or wait till a new version of Lazarus is released?
No. Please see my previous post. You can download a version of OPM from march 14. It's basically the same version(no new function added). After march 14 OPM is heavily refactored in attempt to link the two package system together.
Title: Re: Online Package Manager
Post by: wp on March 29, 2017, 08:21:18 pm
I made a minor revision of the spktoolbar package today, uploaded a new zip file to sourceforge (https://sourceforge.net/projects/lazarus-ccr/files/SpkToolbar/spktoolbar-0.1.1.zip/download) and updated the json with the new zip file name and the new version number (0.1.1). I updated Lazarus trunk to get the current OPM and set it up to check for updates every few minutes. But now that half a day has passed OPM still displays version 0.1.0 as being the most up-to-date version - it should be 0.1.1. GetMem, did I do something wrong?

BTW: If I right-click in OPM on "package info" of any package I can select an option to "show json". I'd expect to see the json of this particular package, but the function shows the jsons of all packages. Would it be possible to filter the list such that only the json of the clicked package is displayed?
Title: Re: Online Package Manager
Post by: Thaddy on March 29, 2017, 08:58:48 pm
Just an aside: I am getting to like it... :)
Title: Re: Online Package Manager
Post by: GetMem on March 29, 2017, 09:12:16 pm
@wp
Quote
I made a minor revision of the spktoolbar package today, uploaded a new zip file to sourceforge (https://sourceforge.net/projects/lazarus-ccr/files/SpkToolbar/spktoolbar-0.1.1.zip/download) and updated the json with the new zip file name and the new version number (0.1.1). I updated Lazarus trunk to get the current OPM and set it up to check for updates every few minutes. But now that half a day has passed OPM still displays version 0.1.0 as being the most up-to-date version - it should be 0.1.1. GetMem, did I do something wrong?
After a quick debug it turn out that the JSON is invalid. I run it through a validator, I believe you accidentally deleted the coma after the "DownloadZipURL" line(see image).

Quote
BTW: If I right-click in OPM on "package info" of any package I can select an option to "show json". I'd expect to see the json of this particular package, but the function shows the jsons of all packages. Would it be possible to filter the list such that only the json of the clicked package is displayed?
Initially the "show json" was meant for debugging purposes, but I forget to remove it.  :-[ I don't know if it's useful for other things, but I can add the filter if you like.
Title: Re: Online Package Manager
Post by: GetMem on March 29, 2017, 09:14:50 pm
Quote
@Thaddy
Just an aside: I am getting to like it... :)
Thank you!
Title: Re: Online Package Manager
Post by: wp on March 29, 2017, 10:08:20 pm
I believe you accidentally deleted the coma after the "DownloadZipURL" line(see image).
:-[ The good thing is: I'll never make this error again

But the bad thing is: It does not seem to fix the problem. I uploaded the corrected update.json file about an hour ago, but OPM still displays the old version 0.1.0

BTW: What is the "repository filename"? The name of the zipped package in the Lazarus repository, or the name in the sourceforge repository? In the latter case the filename would be wrong: it would be spktoolbar-0.1.zip for the current version displayed (and spktoolbar-0.1.1.zip for the new version)
Title: Re: Online Package Manager
Post by: GetMem on March 29, 2017, 10:59:05 pm
Quote
But the bad thing is: It does not seem to fix the problem. I uploaded the corrected update.json file about an hour ago, but OPM still displays the old version 0.1.0
Yes, apparently the latest attempt to unify the two package system broke something. I'm gonna fix the issue tomorrow, because it's late here.

Quote
BTW: What is the "repository filename"?
The name of the zipped package in the Lazarus repository.
Title: Re: Online Package Manager
Post by: Fred vS on March 30, 2017, 12:18:38 am
Hello GetMem.

I get this message while clicking on Update: Downloader Inet not supported., then a access violation.

I use Lazarus 1.6.4 and fpc 3.0.2 on Linux Mint 64 bit.

Fre;D
Title: Re: Online Package Manager
Post by: jacmoe on March 30, 2017, 01:55:52 am
Looks like you've installed the wrong package manager? (Blaszyk)
That one has been removed from future Lazarus's.
Get the right one.
It's included with Lazarus trunk.
Title: Re: Online Package Manager
Post by: GetMem on March 30, 2017, 07:26:58 am
@wp
I fixed in r. 54500. Please test. Thank you!
Title: Re: Online Package Manager
Post by: wp on March 30, 2017, 09:41:45 am
No change still. Maybe I am looking at the false place? Or because the package is installed from my working repository (instead of the Lazarus repository) and therefore the package is "up to date"? In the attachment there's a screenshot of OPM. I am talking about the value shown in column "Repository": It should be 0.1.1.0 instead of 0.1.0.0
Title: Re: Online Package Manager
Post by: GetMem on March 30, 2017, 10:02:03 am
The value from column "Repository" 0.1.0.0 is correct(I did not had the chance to update the main repository yet), the "installed" version(0.1.1.0) is also correct since you installed from your working repository, however in the "update" column instead of "-" you should see the update version taken from "update_spktoolbar.json". It works fine for me, the json is downloaded, parsed and the value is displayed correctly in the tree(screenshot).
Title: Re: Online Package Manager
Post by: wp on March 30, 2017, 10:31:36 am
however in the "update" column instead of "-" you should see the update version taken from "update_spktoolbar.json". It works fine for me, the json is downloaded, parsed and the value is displayed correctly in the tree(screenshot).
And this happens because my system is "up to date"?
Title: Re: Online Package Manager
Post by: GetMem on March 30, 2017, 10:48:19 am
Quote
@wp
And this happens because my system is "up to date"?
If all you can see in the update column is "-" then is a bug. I thought I fixed today, but apparently not.

PS: Can somebody else install SpkToolbar? After a rebuild + a few minutes wait, is the update column empty? Thanks in advance for testing.
Title: Re: Online Package Manager
Post by: minesadorada on March 30, 2017, 01:01:20 pm
Quote
@wp
And this happens because my system is "up to date"?
If all you can see in the update column is "-" then is a bug. I thought I fixed today, but apparently not.

PS: Can somebody else install SpkToolbar? After a rebuild + a few minutes wait, is the update column empty? Thanks in advance for testing.
I have an older version of OPM. On opening it updated spktoolbar to V0.1.1.0.  So no problems with the package version, location, JSON etc.
HTH
Title: Re: Online Package Manager
Post by: GetMem on March 30, 2017, 01:10:29 pm
Thank you @minesadorada. In the past the update feature worked well. It was extensively tested by you and others. However after march 14, OPM went through a heavy refactoring which caused the latest bug. I fixed the bug today(or at least I thought I did), however still not works for @wp. I cannot decide is a specific setting in @wp's computer or a general bug. This is why I was hoping somebody else with the latest trunk can run a few tests. Anyway thanks for your feedback.
Title: Re: Online Package Manager
Post by: wp on March 30, 2017, 01:28:35 pm
I have another trunk installation which never had seen spktoolbar. I updated to current trunk and opened OPM: spktoolbar was marked as not being installed, repository version 0.1.0.0. Installed it --> OPM tells me that there is a new version. Installed that too, and it is displayed correctly as installed 0.1.1.0, update 0.1.1.0, status up to date.

My feeling is that there is some kind of conflict if a package is installed from another source and this one is up to date as well. The other source here is my local repository of my development vesion.
Title: Re: Online Package Manager
Post by: minesadorada on March 30, 2017, 02:40:25 pm
Thank you @minesadorada. In the past the update feature worked well. It was extensively tested by you and others. However after march 14, OPM went through a heavy refactoring which caused the latest bug. I fixed the bug today(or at least I thought I did), however still not works for @wp. I cannot decide is a specific setting in @wp's computer or a general bug. This is why I was hoping somebody else with the latest trunk can run a few tests. Anyway thanks for your feedback.
Sorry @GetMem, OPM still won't compile after I update fpc trunk and lazarus trunk to latest builds.   It will be a path conflict on my system I expect, but I can't seem to track it down.  I thought it might be useful to isolate the error to the OPM code rather than anything else.

Good luck with your bug-busting!
Title: Re: Online Package Manager
Post by: GetMem on March 30, 2017, 02:45:58 pm
@wp
To simulate your configuration, I did  the following:
1. Delete everything(lazarus and config dir)
2. Checkout, build IDE, install OPM etc...
3. Install SpkToolbar 0.1.1.0 from sourceforge: https://sourceforge.net/projects/lazarus-ccr/files/SpkToolbar/spktoolbar-0.1.1.zip/download, first I copied SpkToolbar to $(LazarusDir)\components folder, so it has nothing to do with OPM install dir.

After 1-2 minutes the update column show the correct value, so everything works fine. I'm really puzzled by this one.

Title: Re: Online Package Manager
Post by: GetMem on March 30, 2017, 02:53:11 pm
Quote
@minesadorada
Sorry @GetMem, OPM still won't compile after I update fpc trunk and lazarus trunk to latest builds.   It will be a path conflict on my system I expect, but I can't seem to track it down.  I thought it might be useful to isolate the error to the OPM code rather than anything else.
Good luck with your bug-busting!

I'm almost certain that the problem is with your fpc configuration. You're using older fpc source, because the "Terminate" procedure was recently added to fpc by @michael. If lazarus cannot find it, it means the fpc source are not the most recent one. However there is a temporary workaround, you can instruct OPM to use it's own internal fphttpclient. Just open "opkman_fpcdef.inc"and remove the second part so FPC311 is become undefined, this way force OPM tol use an internal version of httpclient.
Title: Re: Online Package Manager
Post by: wp on April 12, 2017, 10:19:45 am
GetMem, same problem again: Yesterday I uploaded a new version of fpspreadsheet 1.8.2 and updated the json file, but OPM keeps telling me that the most current version is 1.8.0 even on a Lazarus installation (today's r54612) which never had seen fpspreadsheet. I even downloaded one of the fpspreadsheet packages, and it IS 1.8.0.

The json looks correct to me now:
Code: [Select]
{
  "UpdatePackageData" : {
    "DisableInOPM" : false,
    "DownloadZipURL" : "https://sourceforge.net/projects/lazarus-ccr/files/FPSpreadsheet/fpspreadsheet-1.8.2.zip/download",
    "Name" : "fpspreadsheet"
  },
  "UpdatePackageFiles" : [
    {
      "ForceNotify" : false,
      "InternalVersion" : 1,
      "Name" : "laz_fpspreadsheet.lpk",
      "Version" : "1.8.2.0"
    },
    {
      "ForceNotify" : false,
      "InternalVersion" : 1,
      "Name" : "laz_fpspreadsheetexport_visual.lpk",
      "Version" : "1.8.2.0"
    },
    {
      "ForceNotify" : false,
      "InternalVersion" : 1,
      "Name" : "laz_fpspreadsheet_visual.lpk",
      "Version" : "1.8.2.0"
    }
  ]
}
Title: Re: Online Package Manager
Post by: GetMem on April 12, 2017, 12:00:45 pm
@wp
I'm aware that the update column/button is somewhat confusing. Initially I only wanted a central repository + a tool to easily update the repository. Unfortunately this raises a few security issues. I don't want to go into the details since it was discussed many times before in this thread. So instead we(forum users) decided to go with the so called "Update" feature, which allows the package maintainer to easily notify a user about a new update. The update feature has at least two drawbacks, beside the confusion:
    1. You have to install the package from the central repository first. Although technically OPM can install directly from an update page, it's not allowed because of the possible unresolved dependency issues
    2. It's not 100% safe. I mean a package maintainer with bad intention, it can(at least in theory) install a malware, especially under windows
On the other hand the update feature also has a great advantage: it removes the workload from the central repository and pass it to the package maintainers, this way making the whole system sustainable in the long run.

PS: I made a small video: https://youtu.be/wiN2GoCvgjc . Please watch it and tell me if OPM behaves the same at your side. Also please suggest another approach for the update system if you consider the current one flawed.


Title: Re: Online Package Manager
Post by: wp on April 12, 2017, 12:25:50 pm
OK, I understand now - I should have read the wiki, it explains everything correctly.

That Lazarus installation which never had seen fpspreadsheet exactly behaves like what you show in your video.

My fpspreadsheet folder has not been installed from repository, therefore, I don't get notified of updates. But why not? Look at the attached screenshof of OPM running on this Lazarus: OPM is detecting that
(1) fpspreadsheet is installed
(2) fpspreadsheet is up to date.

Why can't it also look for my update json file and get the current release version number and display it? I think this is the main source of my confusion.
Title: Re: Online Package Manager
Post by: GetMem on April 12, 2017, 02:07:24 pm
Quote
Why can't it also look for my update json file and get the current release version number and display it? I think this is the main source of my confusion.
After I saw your screen shot it hit me. The installed version is bigger then the one available for update, so OPM doesn't display it since technically it's not an update. I thought this is logical but I was wrong. Fixed in r54616. Thanks @wp for the usual help.
Title: Re: Online Package Manager
Post by: wp on April 12, 2017, 02:30:29 pm
Yes, sounds promising. But my development version still does not show the update version number.
Title: Re: Online Package Manager
Post by: GetMem on April 12, 2017, 02:53:45 pm
Quote
Yes, sounds promising. But my development version still does not show the update version number.
This is strange...I will investigate further. 
Title: Re: Online Package Manager
Post by: JuhaManninen on April 12, 2017, 03:10:45 pm
I'm aware that the update column/button is somewhat confusing. Initially I only wanted a central repository + a tool to easily update the repository. Unfortunately this raises a few security issues...
I was against this illogical mixture of different design ideas already before you implemented it.
The main design idea of this Online Package Manager was to have manual admin inspection for all included packages, as opposed to automatic systems like Delphinus.
It requires that package authors have an easy way to deliver their packages to be inspected and included. I think that goal is achieved.
The update feature becomes still less usefull when a new version can be easily delivered.
It is still not too late to remove this confusing update feature. The OPM would become logical, straightforward, simple and intuitive.
Title: Re: Online Package Manager
Post by: wp on April 12, 2017, 03:26:09 pm
No, I am perfectly fine with the OPM update system as it is now - now that I understood how it is working. And even if that particular issue with packages not originally installed from the main respository would not be resolved. The good thing is: it is almost no effort for the package maintainer to provide updates. And it is no effort at all for an "OPM manager". If we would give up this I'll bet that OPM will see the same fate as CCR: initially everybody is excited, but it will be devastated after a few years.
Title: Re: Online Package Manager
Post by: GetMem on April 12, 2017, 05:13:48 pm
Quote
@JuhaManninen
It is still not too late to remove this confusing update feature. The OPM would become logical, straightforward, simple and intuitive.
If needed, I can remove it in a few minutes. I only need to hide the update column/button, prevent the update thread to start, hide a few item in the option dialog and that's all there is to it. However it would be a shame, I spent way to much time to implement it and other people to test it. It works reasonably well, few issues still need to be fixed though. It's only confusing until you read the wiki page, I will also add a video soon where I will try to explain the update system in details. For now we should focus our effort to link the two package system together. This should be optional though, some of the developers did not like the idea. I also saw that @Joost is working on fppkg, it would be interesting to include some of the features of fppkg into OPM, like the possibility to install fpc packages too?

@wp
I've tried everything, unfortunately I cannot reproduce the issue. This is an elusive bug. As a last attempt please try the following:
1. Close lazarus
2. Delete everything from the config folder. Under windows, typically is located at: C:\Users\%USERNAME%\AppData\Local\lazarus\onlinepackagemanager\config\
3. Start Lazrus/OPM again
Title: Re: Online Package Manager
Post by: wp on April 12, 2017, 05:23:48 pm
As a last attempt please try the following...
No chance... Same issue.

But as I said above, I can live with it, and it looks as if I am the only one having this minor bug. So problem for me if you stop here.
Title: Re: Online Package Manager
Post by: CharlyTango on April 20, 2017, 11:29:38 am
Hi

i tried OPM with Lazarus trunk and it worked like i expected.

Synchronizing different Lazarus installations with the same packages works like a charm. Some clicks and a cup of coffee later its done.
But I've got a handful of unofficial or paid packages and want to install them in the same way. Is there any possibility to include my own packages in the OPM install process?
It would save a lot of time installing packages if there are some conveniant options.


Regards
K.

Title: Re: Online Package Manager
Post by: GetMem on April 20, 2017, 01:16:46 pm
@CharlyTango
Quote
i tried OPM with Lazarus trunk and it worked like i expected.
Synchronizing different Lazarus installations with the same packages works like a charm. Some clicks and a cup of coffee later its done.
I'm glad it's working.  :)

Quote
But I've got a handful of unofficial or paid packages and want to install them in the same way. Is there any possibility to include my own packages in the OPM install process?
It would save a lot of time installing packages if there are some conveniant options.
Support for private repositories it's not fully implemented yet. However if you don't mind concatenate a few JSONs, you can create your own private repository. Please do the following:
1. Create one or more repository package. OPM will generate for each package a .zip and a .json file.
2. Concatenate the JSONs and create a single big JSON. Rename it to packagelist.json
3. Copy all the zip files and the packagelist.json to your server.
4. Set up the address for the private repository
5. Select the remote repository in the option dialog

PS: I decided to make a small video, hope it helps: https://youtu.be/oxCArRUiKm8
Title: Re: Online Package Manager
Post by: CharlyTango on April 20, 2017, 02:41:29 pm

Support for private repositories it's not fully implemented yet. However if you don't mind concatenate a few JSONs, you can create your own private repository. Please do the following:
1. Create one or more repository package. OPM will generate for each package a .zip and a .json file.
2. Concatenate the JSONs and create a single big JSON. Rename it to packagelist.json
3. Copy all the zip files and the packagelist.json to your server.
4. Set up the address for the private repository
5. Select the remote repository in the option dialog

PS: I decided to make a small video, hope it helps: https://youtu.be/oxCArRUiKm8

Thanks a lot -- especially the video made it clear.
I will try it on the weekend.  :)
Title: Re: Online Package Manager
Post by: lainz on April 30, 2017, 05:31:12 am
Hi GetMem, I've optimized the speed and functionality of the Lazarus Packages website.

Now:
- It uses Bootstrap 4 instead of Material Design Lite (MDL is really slower)
- It uses AngularJS to fill the site and do the search / get the json file

Fixes:
- If any field is empty is not shown
- The categories that are not used are not displayed, and if you add new categories are added automatically (get categories from JSON)
- It shows the Home Page of each package
- The search and categories are separated, so you can pick a category and do a search in that category or simply filter by category or search independently

Download it here:
https://github.com/lazarusccr/LazarusPackagesRepository
Title: Re: Online Package Manager
Post by: x2nie on April 30, 2017, 08:25:16 am
Hi GetMem, I've optimized the speed and functionality of the Lazarus Packages website.

Download it here:
https://github.com/lazarusccr/LazarusPackagesRepository
I go to http://packages.lazarus-ide.org
and no any picture there.
If picture eat bandwidths too much , maybe we can store pictures somewhere else, such in github or sourceforge (used as pics storage).
I think one picture will talks more better than 500 words of description.
Title: Re: Online Package Manager
Post by: GetMem on April 30, 2017, 09:34:35 am
Hi lainz,

Quote
I've optimized the speed and functionality of the Lazarus Packages website.

Now:
- It uses Bootstrap 4 instead of Material Design Lite (MDL is really slower)
- It uses AngularJS to fill the site and do the search / get the json file

Fixes:
- If any field is empty is not shown
- The categories that are not used are not displayed, and if you add new categories are added automatically (get categories from JSON)
- It shows the Home Page of each package
- The search and categories are separated, so you can pick a category and do a search in that category or simply filter by category or search independently
Thank you! It's indeed faster.

I updated the files. I noticed the license is truncated in a few places(ex: ACS - Audio Components Suite). Is this intentional? I also left the old one for a few days for comparison:
New: http://packages.lazarus-ide.org/
Old:  http://packages.lazarus-ide.org/old

Please test!
Title: Re: Online Package Manager
Post by: GetMem on April 30, 2017, 09:40:18 am
Quote
I go to http://packages.lazarus-ide.org and no any picture there. If picture eat bandwidths too much , maybe we can store pictures somewhere else, such in github or sourceforge (used as pics storage). I think one picture will talks more better than 500 words of description.

Hi x2nie,

What pictures do you have in mind? I mean those are lazarus packages, what picture do you wish to associate whit each package so it will be more descriptive then the actual description? Or perhaps I misunderstand you?
Title: Re: Online Package Manager
Post by: lainz on April 30, 2017, 04:54:51 pm
Hi lainz,

Thank you! It's indeed faster.

I updated the files. I noticed the license is truncated in a few places(ex: ACS - Audio Components Suite). Is this intentional? I also left the old one for a few days for comparison:
New: http://packages.lazarus-ide.org/
Old:  http://packages.lazarus-ide.org/old

Please test!

HI, the truncation of license is intentional, up to 200 characters. But if is not so good we must find a way to don't display it fully or when the user requests it, else it does a bigger page height for each package just for license displaying.

I forget to add <meta name="viewport" content="width=device-width, initial-scale=1.0, minimum-scale=1.0">
So it looks fine on mobile devices, but should not be a problem since this website is not aimed for mobiles.

Another thing to be fixed is the mixed content, now seems that we have https, just change a line:

const packages_json = 'http://packages.lazarus-ide.org/packagelist.json';
to
const packages_json = 'packages.lazarus-ide.org/packagelist.json';
or
const packages_json = '/packagelist.json';

so it works on http and also on https

Edit: all these changes are now on the repository, just download it again. If the const doesn't work just change it in order to work with http and https.
Title: Re: Online Package Manager
Post by: GetMem on May 01, 2017, 07:09:14 am
Quote
Edit: all these changes are now on the repository, just download it again. If the const doesn't work just change it in order to work with http and https.
Updated. Thanks again for your help @lainz.
Title: Re: Online Package Manager
Post by: lainz on May 02, 2017, 01:26:23 am
Quote
Edit: all these changes are now on the repository, just download it again. If the const doesn't work just change it in order to work with http and https.
Updated. Thanks again for your help @lainz.

I learned to do that kind of stuff this last months :) I need to do something back to Lazarus, I learned a lot on this forum.
Title: Re: Online Package Manager
Post by: lainz on May 06, 2017, 11:56:50 pm
Hi GetMem, I've added a small but useful thing to this website, is search highlight, so the text anyone inputs in the search field will be highlighted in each component result:
https://github.com/lazarusccr/LazarusPackagesRepository

A note: the search will look somewhat slower than without the highlight, the plugin is not doing the page slow, but I've set 250 milliseconds between the last character you type before triggering the search, so it in fact uses less resources, and the highlighter can be used with no problems with words of any length.

If is not a great idea just keep the site as is now.
Title: Re: Online Package Manager
Post by: GetMem on May 07, 2017, 09:26:20 am
@lainz
Quote
Hi GetMem, I've added a small but useful thing to this website, is search highlight, so the text anyone inputs in the search field will be highlighted in each component result:
https://github.com/lazarusccr/LazarusPackagesRepository
I updated the files. Thank you!

Quote
A note: the search will look somewhat slower than without the highlight, the plugin is not doing the page slow, but I've set 250 milliseconds between the last character you type before triggering the search, so it in fact uses less resources, and the highlighter can be used with no problems with words of any length.
I see no noticeable difference in speed, so everything is ok. It's a useful feature.
Title: Re: Online Package Manager
Post by: GetMem on May 14, 2017, 10:09:48 am
From now on is possible to uninstall one or more packages directly from OPM(r. 54908).

PS: Did somebody noticed that laz2_xmlcfg.pas is leaking? Until now I wasn't able to locate the source of the leak.

Title: Re: Online Package Manager
Post by: avra on May 14, 2017, 12:41:58 pm
From now on is possible to uninstall one or more packages directly from OPM(r. 54908).
Much appreciated feature. Thank you!  ;)
Title: Re: Online Package Manager
Post by: Thaddy on May 14, 2017, 01:07:44 pm
It is getting better and better and I have it on by default (ok, because in 1.9 it's on by default  :D ).
Now THAT'S a feature!
Title: Re: Online Package Manager
Post by: avra on June 08, 2017, 02:36:31 am
Notes from testing OPM with latest NewPascal on 64-bit Win10:

LazSerial should be updated to v0.2 from https://github.com/JurassicPork/TLazSerial. The old v0.1 version found in OPM has conflict with Synapse (although it can be easily fixed by hand).

Web Service Toolkit has too many compilation errors to fix by hand. Solved using CodeTyphon version converted with ct2laz.

Everything else I was interested in seams to work out of the box.

Great work! I really like OPM and I am very happy to see it becoming part of Lazarus  8)
Title: Re: Online Package Manager
Post by: GetMem on June 08, 2017, 07:04:43 am
@avra

Quote
LazSerial should be updated to v0.2 from https://github.com/JurassicPork/TLazSerial. The old v0.1 version found in OPM has conflict with Synapse (although it can be easily fixed by hand).
Updated.

Quote
Web Service Toolkit has too many compilation errors to fix by hand. Solved using CodeTyphon version converted with ct2laz.
It works fine for me(Lazarus trunk/FPC 3.0.2). I don't mind replacing with your version if it compiles with Laz 1.8/Trunk. Can you please upload somewhere the zip file?

Quote
Great work! I really like OPM and I am very happy to see it becoming part of Lazarus  8)
Thank you.
Title: Re: Online Package Manager
Post by: JuhaManninen on June 08, 2017, 10:44:56 am
Web Service Toolkit has too many compilation errors to fix by hand. Solved using CodeTyphon version converted with ct2laz.
Doesn't compile here either. First error:
 semaphore.pas(83,18) Error: identifier idents no member "SemaphoreInit"

OPM has version 0.6.0 of WST which is old. It is known not to work.
WST trunk works:
 https://svn.code.sf.net/p/lazarus-ccr/svn/wst/trunk

In SWT's mailing list:
 http://groups.google.com/group/wst-list
I have asked for a new release already at late last year in thread
 "New version of WST needed, Lazarus online package manager".
Inoussa answered then but not any more after it.
He may be busy or his motivation has gone down. It is possible WST will not be actively maintained. He would need a co-maintainer for WST.

BTW, who made the WebServiceToolkit package for OPM? I don't think Inoussa made it.
Title: Re: Online Package Manager
Post by: GetMem on June 08, 2017, 11:18:42 am
Quote
@JuhaManninen
BTW, who made the WebServiceToolkit package for OPM? I don't think Inoussa made it.
I did. A few month ago I took the trunk version from Inoussa's webpage and created the package for OPM. I did not modify the version number(1.6) since officially there is no new release. The WST from OPM compiles with Laz 1.8/Trunk, FPC 3.0.2, Win7.

PS:I did not check if there are new commits since then.

Edit: A single commit(r. 5823) was made in march 28, 2017.
Title: Re: Online Package Manager
Post by: JuhaManninen on June 08, 2017, 12:17:28 pm
The WST from OPM compiles with Laz 1.8/Trunk, FPC 3.0.2, Win7.
I tested with FPC trunk. I will test with FPC 3.0.2 later.
Title: Re: Online Package Manager
Post by: avra on June 08, 2017, 08:52:12 pm
Quote
LazSerial should be updated to v0.2 from https://github.com/JurassicPork/TLazSerial. The old v0.1 version found in OPM has conflict with Synapse (although it can be easily fixed by hand).
Updated.
Thank you so much. Less manual work for me in the future.  ::)

Quote
Web Service Toolkit has too many compilation errors to fix by hand. Solved using CodeTyphon version converted with ct2laz.
It works fine for me(Lazarus trunk/FPC 3.0.2). I don't mind replacing with your version if it compiles with Laz 1.8/Trunk. Can you please upload somewhere the zip file?
Let me check first how that trunk that Juha mentioned behaves with what I got, and if I can catch some time next week then I will try to provide a patch. If FPC version is suspected to be guilty, then it would really help if someone determines what version breaks the compilation. I would like to maintain compatibility with current FPC 3.0.2 if possible with some ifdefs.

In case anyone is really curious then just use latest ct2laz and with it's default settings when you press on "CT => LAZ" button, it will download all CT examples and components and convert them to be ready for manual installation in Lazarus. WST will be in pl_WST directory. That works for me on 2 previously mentioned Lazarus installations (one FPC trunk and one FPC 3 months old).

Web Service Toolkit has too many compilation errors to fix by hand. Solved using CodeTyphon version converted with ct2laz.
Doesn't compile here either. First error:
 semaphore.pas(83,18) Error: identifier idents no member "SemaphoreInit"
Huh, this might turn to be harder then I suspected. I had different error...
Title: Re: Online Package Manager
Post by: GetMem on June 08, 2017, 10:36:45 pm
I downloaded then ifdef-ed the latest version. It should be ok now. Tested with:
- FPC 3.0.0, FPC 3.0.2, FPC trunk
- Lazarus 1.8, Lazarus trunk
- win32/64, gtk2, carbon

Please test.
Title: Re: Online Package Manager
Post by: JuhaManninen on June 09, 2017, 12:22:28 am
I downloaded then ifdef-ed the latest version. It should be ok now. Tested with:
- FPC 3.0.0, FPC 3.0.2, FPC trunk
- Lazarus 1.8, Lazarus trunk
- win32/64, gtk2, carbon
It does not work with FPC trunk on any OS except for Windows because unit semaphore in WST has {$DEFINE FPC_TM}.
For Windows it is not defined for some reason.
Then the code uses TThreadManager which I guess has changed a lot in FPC trunk resulting many errors.
This is not specific to the version in OPM, the same code is in WST trunk.
WST development has not been active recently, there is only 1 commit this year and the previous one is from October 2016.

Now I am worried that you people essentially fork the WST project and maintain a competing OPM version. It is waste of resources. In some situations a fork is good but not in this situation, as apparently the official maintainer is busy or tired. The OPM fork does not even use revision control which makes things worse.
WST sources are in Lazarus CCR repo. We can now take the liberty to commit bug fixes there. Then only the release version of WST for OPM would be outside the "official" releases. I guess all people involved (myself, GetMem, Avra) have commit rights to CCR. (?)
Fixes can be backported from CodeTyphon as much as needed. That is the beauty of FOSS (Free and Open Source Software).

We also must try to contact Inoussa more. He has not answered the WST mailing list.
Title: Re: Online Package Manager
Post by: GetMem on June 09, 2017, 07:05:31 am
Quote
@Juha
Now I am worried that you people essentially fork the WST project and maintain a competing OPM version. It is waste of resources. In some situations a fork is good but not in this situation, as apparently the official maintainer is busy or tired. The OPM fork does not even use revision control which makes things worse.
I'm not planning to fork or maintain any other package, it's indeed a waste of resources. However I don't like when packages from OPM does not work, it looks like OPM is broken. This is why sometimes(rarely) I do make small modification, especially when packages are no longer maintained. The modification are almost always IFDEFS to make the package compatible with the latest Lazarus/FPC versions. In this particular case(WST) time is in our side, since the issue affect only FPC trunk users.

Quote
I guess all people involved (myself, GetMem, Avra) have commit rights to CCR. (?)
AFAIK I don't have commit rights.

Quote
Fixes can be backported from CodeTyphon as much as needed. That is the beauty of FOSS (Free and Open Source Software).
I don't use CodeTyphon, so I hope @Avra will backport it. I will add the backported version to OPM.
Title: Re: Online Package Manager
Post by: JuhaManninen on June 09, 2017, 07:18:18 am
This is why sometimes(rarely) I do make small modification, especially when packages are no longer maintained.
Yes, that is already forking because the changes do not go to the original project.
It is an understandable fork and nothing serious but we should always think how to get the changes to the actual project.

Quote
I don't use CodeTyphon, so I hope @Avra will backport it. I will add the backported version to OPM.
Backports must go to the actual project in CCR. Then it can be used for OPM. I can commit to CCR if somebody without commit rights provides patches.
Title: Re: Online Package Manager
Post by: avra on June 09, 2017, 09:27:26 am
Quote
Quote
I don't use CodeTyphon, so I hope @Avra will backport it. I will add the backported version to OPM.
Backports must go to the actual project in CCR. Then it can be used for OPM.
That was my initial plan hopefully scheduled for next week.

Quote
I can commit to CCR if somebody without commit rights provides patches.
CCR is in Mantis, so maybe I should go that path? If not for any reason, then it is not a problem for me to send you patches.

I downloaded then ifdef-ed the latest version. It should be ok now. Tested with:
- FPC 3.0.0, FPC 3.0.2, FPC trunk
- Lazarus 1.8, Lazarus trunk
- win32/64, gtk2, carbon
It does not work with FPC trunk on any OS except for Windows because unit semaphore in WST has {$DEFINE FPC_TM}.
For Windows it is not defined for some reason.
Then the code uses TThreadManager which I guess has changed a lot in FPC trunk resulting many errors.
@GetMem: To avoid duplicate work please send me ifdefed version to avra at writeme dot com.
@Juha: It is not clear to me if you have these errors trying to install CT version on FPC trunk only on non Windows OS, or you have it also on Windows? It works for me on Windows. Haven't tested elsewhere.
Title: Re: Online Package Manager
Post by: GetMem on June 09, 2017, 09:46:32 am
Quote
@avra
To avoid duplicate work please send me ifdefed version to avra at writeme dot com.
It's already in the central repository. Just open OPM, check WST then click the download button. Apparently I only solved just part of the issues. It works on windows with FPC trunk, but not on the other platforms.   
Title: Re: Online Package Manager
Post by: avra on June 09, 2017, 10:32:03 am
Quote
@avra
To avoid duplicate work please send me ifdefed version to avra at writeme dot com.
It's already in the central repository. Just open OPM, check WST then click the download button.
Of course, silly me!  :D  %)  :D

Quote
Apparently I only solved just part of the issues. It works on windows with FPC trunk, but not on the other platforms.
Can anyone confirm that WST works on non-windows platforms with FPC 3.0.2 or older?
Title: Re: Online Package Manager
Post by: JuhaManninen on June 09, 2017, 11:52:19 am
CCR is in Mantis, so maybe I should go that path? If not for any reason, then it is not a problem for me to send you patches.
Both ways suit me.

Quote
@Juha: It is not clear to me if you have these errors trying to install CT version on FPC trunk only on non Windows OS, or you have it also on Windows? It works for me on Windows. Haven't tested elsewhere.
I tested only on Linux. By looking at code in semaphore.pas I guessed it works on Windows but not elsewhere:
Code: Pascal  [Select]
  1. {$IFDEF FPC}
  2.   {$IFDEF WINDOWS}
  3.     TSemaphoreHandle = THandle;
  4.   {$ELSE}
  5.     {$DEFINE FPC_TM}
  6.     TSemaphoreHandle = Pointer;
  7.   {$ENDIF}
  8. {$ENDIF}
Then see how FPC_TM is used.

I remember WST trunk worked well with FPC 3.0 on Linux at late last year. I believe it works with FPC 3.0.2, too.
I can test more sometime in near future.

This brings a question in my mind. Why WST needs such a FPC_TM define? FPC libs are cross-platform and lots of effort was used to make them so.
WST should be able to use TThreadManager on every platform, or then the alternative code on every platform, without IFDEFs.
semaphore.pas should not need this:
Code: Pascal  [Select]
  1. uses
  2. ...
  3. {$IFDEF FPC}
  4.   {$IFDEF WINDOWS}
  5.     ,Windows
  6.   {$ENDIF}
  7. {$ENDIF};
Anyway, any bigger changes should be consulted with Inoussa when he is available.
Title: Re: Online Package Manager
Post by: avra on June 15, 2017, 08:42:55 am
Can anyone confirm that WST works on non-windows platforms with FPC 3.0.2 or older?
Well, trunk WST will work only with FPC from Sep 5 2016 or newer because it uses at least TPasSourcePos introduced in this patch:
https://github.com/graemeg/freepascal/commit/37aa4c79e1ab81558cdfac05090309c2e2e1dc25
Does anyone have an idea how to at least aproximatelly detect FPC version for a compile message suggestion to install older WST?
Title: Re: Online Package Manager
Post by: JuhaManninen on June 15, 2017, 09:46:51 am
Well, trunk WST will work only with FPC from Sep 5 2016 or newer because it uses at least TPasSourcePos introduced in this patch:
https://github.com/graemeg/freepascal/commit/37aa4c79e1ab81558cdfac05090309c2e2e1dc25
No it doesn't. I just searched WST trunk sources and it was not there.
TPasSourcePos is part of Pascal parser fcl-passrc. WST does not parse Pascal code.

Quote
Does anyone have an idea how to at least aproximatelly detect FPC version for a compile message suggestion to install older WST?
Detecting FPC trunk / FPC 3.02 / FPC 3.0 should be enough. It is reasonable if the WST found in OPM supports them.

I can test WST trunk with FPC 3.0.2 on Linux later today if needed.
Title: Re: Online Package Manager
Post by: GetMem on June 15, 2017, 10:08:38 am
Quote
@avra
Does anyone have an idea how to at least aproximatelly detect FPC version?
I don't think it's possible to detect specific revision numbers, like the one from  Sep 5 2016(r. 34429), however a close approximation can be done like this(you can refine it more to detect 3.0.0, etc...):
Code: Pascal  [Select]
  1. {$IF FPC_VERSION = 3}
  2.   {$IF FPC_RELEASE = 0}
  3.     {$IF FPC_PATCH >= 1}
  4.       {$DEFINE FPC302}
  5.     {$ENDIF}
  6.   {$ELSE}
  7.     {$IF FPC_PATCH > 0}
  8.       {$DEFINE FPC311}
  9.