Recent

Author Topic: Online Package Manager  (Read 836010 times)

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4459
  • I like bugs.
Re: Online Package Manager
« Reply #630 on: December 24, 2016, 10:57:53 am »
Quote
I meant private packages with proprietary code created/used in our company, I cannot disclose them. I meant something like adding another (private) remote repository to the OPM settings and add these packages to this repository. OPM would then manage two repositories eg.:
1. http://packages.lazarus-ide.org
2. http://ourprivaterepository.ourdomain.com
Oh, I see! I misunderstood the question. Private repositories are not supported yet, I plan to add it later(most likely in the beginning of January).
You mean you add support for multiple repositories? Tk's second URL could have any domain, makes no difference.
Private directories / packages are not in the scope of an online package manager.
The next important step will be tight integration with Lazarus package system. Now it does not know about online packages until they are installed.
The Lazarus package graph already can resolve dependencies between packages it knows about. The goal is to let it know about available online packages which would then become part of that package graph.
Lazarus components/ already contains packages with external dependencies, for example TAChartBgra.  Now trying to install it gives an error. In future it would automatically install BgraBitmapPack from online repo.
In the same way an external package can depend on a package in Lazarus components/. Then you must dump your current dependency handling code and integrate it with the package graph.
Details must be discussed in Lazarus devel list. Mattias is the most knowledgeable person about package graph.
Mostly Lazarus trunk and FPC 3.2 on Manjaro Linux 64-bit.

balazsszekely

  • Guest
Re: Online Package Manager
« Reply #631 on: December 24, 2016, 01:58:42 pm »
Quote
You mean you add support for multiple repositories? Tk's second URL could have any domain, makes no difference.
Private directories / packages are not in the scope of an online package manager.
Yes, support for multiple repositories. If the repository owner decide to share it it's public otherwise is private.

Quote
The next important step will be tight integration with Lazarus package system. Now it does not know about online packages until they are installed.
The Lazarus package graph already can resolve dependencies between packages it knows about. The goal is to let it know about available online packages which would then become part of that package graph.
Lazarus components/ already contains packages with external dependencies, for example TAChartBgra.  Now trying to install it gives an error. In future it would automatically install BgraBitmapPack from online repo.
In the same way an external package can depend on a package in Lazarus components/. Then you must dump your current dependency handling code and integrate it with the package graph.
Details must be discussed in Lazarus devel list. Mattias is the most knowledgeable person about package graph.
I don't know about this. Although they have similar functionality the two system are completely different. Mixing them would be like mixing bananas with oranges. Mattias use critical sections when loading the packages syncing those with my threaded system is a non-trivial operation. When searching for dependencies the local package manager has all the info needed in lpks, OPM has to rely on limited info available in the main JSON. For this to work everything has to be moved inside the packager folder, OPM once again completely redesigned, the two dialogs somehow mixed to become one. The benefits? Some lazy user don't have to open a second dialog to check if a package is available online. OPM was not meant to replace the "Install/Uninstall packages" dialog, it's just a tool to install as many packages you like with a button click. Until now you had to open each package, press the compile then the install button. Another benefit is that all the packages are in one place, with the latest updates available you don't have to go through the maintainers site.

PS: Before anything somebody must implement that fpc based cgi/fastcgi, so the database server can be used for maintaining the main repository + implement the voting system.
« Last Edit: December 24, 2016, 02:08:51 pm by GetMem »

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4459
  • I like bugs.
Re: Online Package Manager
« Reply #632 on: December 24, 2016, 03:52:35 pm »
I don't know about this. Although they have similar functionality the two system are completely different. Mixing them would be like mixing bananas with oranges.
No, they both deal with Lazarus packages and their dependencies and installation. Conceptually the same thing. OPM is a front-end to external packages, Lazarus has front-end code for packages in local file system. The common code installs them and maintains a package graph.
You forgot that OPM already integrates into the package system using IDEIntf API, but in a clumsy and limited way.

Quote
Mattias use critical sections when loading the packages syncing those with my threaded system is a non-trivial operation. When searching for dependencies the local package manager has all the info needed in lpks, OPM has to rely on limited info available in the main JSON.
What info is missing from the JSON? I think it is pretty complete.
The package system code must be refactored and changed obviously to understand online packages.

Quote
For this to work everything has to be moved inside the packager folder, OPM once again completely redesigned, the two dialogs somehow mixed to become one.
No no! The package-API must be extended to support it. Don't worry, I am not asking you to do it. :)

Quote
The benefits? Some lazy user don't have to open a second dialog to check if a package is available online. OPM was not meant to replace the "Install/Uninstall packages" dialog, it's just a tool to install as many packages you like with a button click. Until now you had to open each package, press the compile then the install button.
Now you confuse the package system (dependency management + installation etc.) and its GUI front-ends.
The "Install/Uninstall packages" is a GUI for local packages, OPM for external packages. They both can install many packages at one go but it is an implementation detail of a GUI, nothing to do with the underlying package graph code.
The benefit will be good and elegant SW design! The current integration is a poorly stitched-in afterthought.
Think of my example TAChartBgra package. A user gets an error for missing package. At the same time he can see it is available in OPM. He thinks "this system is poorly designed" and yes he is right.

Quote
Another benefit is that all the packages are in one place, with the latest updates available you don't have to go through the maintainers site.
Yes, it is the benefit of an online package manager. It is the whole point of having OPM.
Yet that benefit does not reduce the need for integration anyhow.
We can even create more GUIs for various tasks with packages if need be. They all must integrate through a proper API.
Again, I am not asking you to do it. I will look at it later.

Quote
PS: Before anything somebody must implement that fpc based cgi/fastcgi, so the database server can be used for maintaining the main repository + implement the voting system.
This is a separate and independent project from the package system integration.
Mostly Lazarus trunk and FPC 3.2 on Manjaro Linux 64-bit.

balazsszekely

  • Guest
Re: Online Package Manager
« Reply #633 on: December 24, 2016, 06:26:48 pm »
@Juha
I'm not confusing anything, just I know how OPM works and studied Mattias code enough to tell you the integration are you talking about is an extremely difficult one.  Feel free to implement whatever you like, but please leave the current OPM system intact so I can keep developing. Still a lot of features to implement.

PS: Merry Christmas to all.
« Last Edit: December 24, 2016, 06:29:37 pm by GetMem »

tk

  • Sr. Member
  • ****
  • Posts: 361
Re: Online Package Manager
« Reply #634 on: December 28, 2016, 12:31:07 pm »
Tested the OPM under Win, Linux and OSX, works very well.

Ideas for future:
1. OPM already preinstalled in Lazarus (to save one Lazarus rebuild for installing the OPM).
2. Manage packages listed in 'available for installation' list box with OPM and delete them from Lazarus distribution (to keep Lazarus distribution smaller).

balazsszekely

  • Guest
Re: Online Package Manager
« Reply #635 on: December 28, 2016, 03:06:01 pm »
@tk
Quote
1. OPM already preinstalled in Lazarus (to save one Lazarus rebuild for installing the OPM).
OPM contains a third party component(VirtualStringTree) so it cannot be preinstalled in Lazarus yet, VST must be added first.
Quote
2. Manage packages listed in 'available for installation' list box with OPM and delete them from Lazarus distribution (to keep Lazarus distribution smaller).
Good idea, though I'm not the one who decide this.

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4459
  • I like bugs.
Re: Online Package Manager
« Reply #636 on: December 29, 2016, 02:13:01 pm »
OPM contains a third party component(VirtualStringTree) so it cannot be preinstalled in Lazarus yet, VST must be added first.
You have the VirtualStringTree fork there already. It could be included in BigIde in future.
This becomes a question before the next major release ( 1.8 ). Trunk users will install it separately anyway.

Quote
Quote
2. Manage packages listed in 'available for installation' list box with OPM and delete them from Lazarus distribution (to keep Lazarus distribution smaller).
Good idea, though I'm not the one who decide this.
I moved the Industrial package. Other packages must be discussed.
Integration of available online packages with the Lazarus package system is equally important. Now the user experience is confusing.

GetMem, I tried to send you an Opkman package but mail servers rejected it as dangerous. I tested with 2 different mail accounts, one being GMail. We must find a reliable way to transfer the files ASAP!
To me the FTP server sounded very good. Write access to everybody, then an admin publishes packages that he approves. If there is too much garbage/spam then we think again. Easy peacy...

Now I attach the files here. I had to rename the .json file to .txt because it was rejected also here. Please add to online repository.
How did other people manage to send you files?
« Last Edit: December 29, 2016, 02:16:43 pm by JuhaManninen »
Mostly Lazarus trunk and FPC 3.2 on Manjaro Linux 64-bit.

lainz

  • Hero Member
  • *****
  • Posts: 4460
    • https://lainz.github.io/
Re: Online Package Manager
« Reply #637 on: December 29, 2016, 02:20:08 pm »
Quote
How did other people manage to send you files?

SourceForge, GitHub, personal website

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4459
  • I like bugs.
Re: Online Package Manager
« Reply #638 on: December 29, 2016, 02:25:17 pm »
SourceForge, GitHub, personal website
Ok, I read the instructions poorly in
 http://wiki.freepascal.org/Online_Package_Manager#Create_repository_package
It says: "Upload somewhere the files and send the link to getmem1@gmail.com".
Anyway, it should be made easier. Upload to a server directly from Opkman with a click.
Mostly Lazarus trunk and FPC 3.2 on Manjaro Linux 64-bit.

minesadorada

  • Sr. Member
  • ****
  • Posts: 452
  • Retired
Re: Online Package Manager
« Reply #639 on: December 29, 2016, 02:41:08 pm »
I moved the Industrial package. Other packages must be discussed.
By sheer coincidence I used one of the Industrial package components for the first time today in a project.  Not much documentation, but nice controls.  Are they maintained?
GPL Apps: Health MonitorRetro Ski Run
OnlinePackageManager Components: LazAutoUpdate, LongTimer, PoweredBy, ScrollText, PlaySound, CryptINI

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4459
  • I like bugs.
Re: Online Package Manager
« Reply #640 on: December 29, 2016, 03:03:40 pm »
Are they maintained?
Not very actively but I put my name on the wiki page as a maintainer.
Code for often used controls was collected from around and cleaned to make this package. Some more controls can be added but it should remain light.
Mostly Lazarus trunk and FPC 3.2 on Manjaro Linux 64-bit.

balazsszekely

  • Guest
Re: Online Package Manager
« Reply #641 on: December 29, 2016, 06:17:07 pm »
@Juha
Quote
You have the VirtualStringTree fork there already. It could be included in BigIde in future.
This becomes a question before the next major release ( 1.8 ). Trunk users will install it separately anyway.
Ok. This is good news. In the devlist there was a discussion about adding VST to Lazarus. After that I can remove the dependencies from OPM. This would be the best solution in my opinion.

Quote
GetMem, I tried to send you an Opkman package but mail servers rejected it as dangerous. I tested with 2 different mail accounts, one being GMail. We must find a reliable way to transfer the files ASAP!
To me the FTP server sounded very good. Write access to everybody, then an admin publishes packages that he approves. If there is too much garbage/spam then we think again. Easy peacy...
Marc is very strict about the upload policy, he does not allow any public upload to the main repository(I agree with him, it's too risky). First we have to implement that pascal cgi/fastcgi or it's php equivalent. Until then we can use my own personal ftp server, but here comes the annoying part, fpc does not have any native component to send ftp. In order to implement the feature, I have to add another third party component. The code is ready, I did it with indy, works well, but I really don't want to add one more component, so I did not commit the changes. Another solution is to upload the files to a free web server. This can be done with fphttpclient, the implementation is almost ready.

Quote
Now I attach the files here. I had to rename the .json file to .txt because it was rejected also here. Please add to online repository.
Thanks. I will add it soon to the main repository.

Also please write down in pm or here your idea about linking the two package system. Consider this: OPM is not loaded yet, however the package graph or another package functionality must access the online packages. To do this, it has to start one of OPMs  thread and through a callback function receive the data when arrives. This cannot be done only through packageintf. How do you plan to implement it, without moving a part of the code inside packager folder.

PS: I added IndustrialStuff to main repository.
« Last Edit: December 29, 2016, 06:26:45 pm by GetMem »

minesadorada

  • Sr. Member
  • ****
  • Posts: 452
  • Retired
Re: Online Package Manager
« Reply #642 on: December 29, 2016, 06:27:36 pm »
Another solution is to upload the files to a free web server. This can be done with fphttpclient, the implementation is almost ready.
How about a newly-created SourceForge OPM_CCR project?  All the heavy lifting (including virus-scanning) is done for you.
GPL Apps: Health MonitorRetro Ski Run
OnlinePackageManager Components: LazAutoUpdate, LongTimer, PoweredBy, ScrollText, PlaySound, CryptINI

balazsszekely

  • Guest
Re: Online Package Manager
« Reply #643 on: December 29, 2016, 06:35:35 pm »
@minesadorada
How do you plan to upload the files(from OPM) to source forge? FTP? HTTP? Whatever method you choose you have to provide a username and password, since Lazarus is open source everyone can read the password. How do you plan to solve this issue? 

minesadorada

  • Sr. Member
  • ****
  • Posts: 452
  • Retired
Re: Online Package Manager
« Reply #644 on: December 29, 2016, 06:48:18 pm »
@minesadorada
How do you plan to upload the files(from OPM) to source forge? FTP? HTTP? Whatever method you choose you have to provide a username and password, since Lazarus is open source everyone can read the password. How do you plan to solve this issue? 
Simple.  In order to contribute a new component, you must register a SourceForge account.  This account is then added to the OPM_CCR project via an email to the OPM moderator then added (as 'developer') in order to upload files.
The OPM moderator has an 'admin-level' account in SF OPM_CCR so can control it.

The advantage is:  this already has worked for the Lazarus ccr SF project - that's where you got most of your components from.

The OPM_CCR repository acts as a 'sandbox' whereby the OPM moderator gets to decide what to include in the main repository.  It also acts as an 'update' URL for component developers to upload their update jsons/zips.

Why re-invent the wheel?

All of the above could be GitHub or Sourceforge - both are free and reliable.

The OPM instructions become: 'Make a Sourceforge account (free) and send me your SF username to add to the OPM_CCR project'

Juha's idea to directly upload from OPM is not a good one IMHO.  A component developer is perfectly capable of making a SF/GitHub account and using svn version control.
Component developers can use svn to update components via SF as I do.

I have practical experience with OPM now, and the only missing component (forgive the pun) in the chain is the initial component creation/upload to OPM.

Subsequent updating/version control is dealt with really well in the current incarnation.
« Last Edit: December 29, 2016, 06:56:26 pm by minesadorada »
GPL Apps: Health MonitorRetro Ski Run
OnlinePackageManager Components: LazAutoUpdate, LongTimer, PoweredBy, ScrollText, PlaySound, CryptINI

 

TinyPortal © 2005-2018