Recent

Author Topic: Online Package Manager  (Read 442549 times)

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3652
  • I like bugs.
Re: Online Package Manager
« Reply #45 on: October 07, 2016, 12:44:55 am »
GetMem you can host your package manager on GitHub ...

+1
I don't know how quickly we can get you commit access to Lazarus repo. I don't even know yet what other Lazarus developers say about this package manager.

I finally tested it. Works well! VirtualTreeview package was installed together with its dependency.
When this gets integrated with the current package management code, dependencies will be handled by it (I think).

You have implemented your own code for TZipper, TUnZipper etc. You should use the TZipper provided by FCL :
  http://wiki.freepascal.org/paszlib#TZipper
BTW, your zipper code has a memory leak somewhere.

You have also added a copy of the VirtualTreeview code into your package. Can't it use the VirtualTreeview package as a dependency? Code duplication is always bad for maintenance.

You used the UTF8 specific functions like DirectoryExistsUTF8(). They are not needed any more with Lazarus + FPC 3.
I think FPC 3+ can be made the minimum requirement for this new code.
Ok, maybe not yet. Don't change them.
« Last Edit: October 07, 2016, 01:22:30 am by JuhaManninen »

Leledumbo

  • Hero Member
  • *****
  • Posts: 8114
  • Programming + Glam Metal + Tae Kwon Do = Me
Re: Online Package Manager
« Reply #46 on: October 07, 2016, 06:24:11 am »
GetMem you can host your package manager on GitHub, so it's easy to get a newer version when you're developing it.
I kinda agree with this. Adding packages is a matter of approving merge request then, while maintaining code reviews and other things, with history. Otherwise, I can make my server a git server as well, but it will lack the convenience of github UI.

GetMem

  • Hero Member
  • *****
  • Posts: 3515
Re: Online Package Manager
« Reply #47 on: October 07, 2016, 07:29:38 am »
@Juha
Quote
When this gets integrated with the current package management code, dependencies will be handled by it (I think).
Yes.

Quote
You have implemented your own code for TZipper, TUnZipper etc. You should use the TZipper provided by FCL :
 http://wiki.freepascal.org/paszlib#TZipper
Not quite! I used the same unit provided by FCL, but with a few modifications, this is also true for httpclient. Neither one provides a timeout function. For example when you try to connect to an offline server with httpclient, sometimes it waits 25 - 30 sec before returns an error. The same is true for zip/unzip from Zipper.pas, once started you have to wait until the entire zip process is over, no possibility to abort the process. More over httpclient doesn't even show a download progress. I had to create a thread timer and a modified streamer class to be able to provide a feedback(see opkman_downloader.pas unit). The possibility to show a progress with TZipper is also limited, (although is there) so I had to enhance the function.
I know this is code duplication. Ideally I should create a patch for those units, but when will be accepted(if ever)? Since zipper.pas and httpclient.pas functionality is more then enough in 99% of the cases,  fpc core members would not accept a patch just to satisfy the online package manger needs(at least this is what I think).

Quote
BTW, your zipper code has a memory leak somewhere.
I always develop with "Debug IDE" profile to avoid leaks. I fixed many leaks in my package, obviously I missed something. Thanks.

Quote
You have also added a copy of the VirtualTreeview code into your package. Can't it use the VirtualTreeview package as a dependency?
Initially I wanted to avoid third party components. Unfortunately TTreeView is so limited in so many ways I decided to switch to VST. The only version which works well in every major widgetset is VST4.8.7R2. The newer one doesn't even compile under osx for example. What if somebody already has a newer version installed? It will lead to conflicts, multiple resource errors, etc. It's not a good practice to force the developers to use a specific version of VST, just to be able to install opkman. So I added VST internally to the package manager, even replaced the resource names with a resource editor to avoid conflicts. This way it can happily coexists with any version of VirtualStringTree. Code duplication again? Perhaps but the benefits outweigh the drawbacks.

Quote
@shobits1
It needs a filter for the packages with available new updates.
Ok. I can add this later.

@lainz, @Leledumbo
I will add the source to github soon.
« Last Edit: October 07, 2016, 07:50:30 am by GetMem »

rvk

  • Hero Member
  • *****
  • Posts: 3842
Re: Online Package Manager
« Reply #48 on: October 07, 2016, 07:58:04 am »
Since zipper.pas and httpclient.pas functionality is more then enough in 99% of the cases,  fpc core members would not accept a patch just to satisfy the online package manger needs (at least this is what I think).
If that is the case they should at least change the source so that you can inherit the code and enhance it. A lot of the time this is not possible due to unnecessary private variables.

I also hacked one of those http classes once because there was only a download progress option but none for upload (like for uploading a large file via a post). Luckily that could be done via inheritance but often this is not possible.

If a class is designed, so you would need to copy the whole source to add such functionality, I think it is designed poorly.

B.T.W. I thought httpclient did have a download progress option but it lacked an upload progress. (Can't look right now but I think it's the OnDataReceived in dodataread. But it lacks a OnDataSend)

GetMem

  • Hero Member
  • *****
  • Posts: 3515
Re: Online Package Manager
« Reply #49 on: October 07, 2016, 08:50:32 am »
Quote
@rvk
Can't look right now but I think it's the OnDataReceived in dodataread. But it lacks a OnDataSend
True! With my streamer class can do both download/upload progress threaded. @rvk if you find a way to abort a download/upload process via class inheritence let me know. The same is true for a timeout function.

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3652
  • I like bugs.
Re: Online Package Manager
« Reply #50 on: October 07, 2016, 11:02:03 am »
I know this is code duplication. Ideally I should create a patch for those units, but when will be accepted(if ever)? Since zipper.pas and httpclient.pas functionality is more then enough in 99% of the cases,  fpc core members would not accept a patch just to satisfy the online package manger needs(at least this is what I think).

Sounds like those features are valid for many use cases, not only for the online package manager.
SW always improves by iterations. This is true for FCL code, too.
I believe a patch for both zipper and httpclient would be accepted. Even more so because an essential Lazarus package must copy/paste FCL code otherwise.
[Edit] A patch may be included in FPC 3.2. It means your copied version is a temporary solution until that compiler version becomes the minimum requirement. We must always think long-term.

VirtualTreeview is more difficult. The porting process continues by Luiz Américo if I understand right.
The long term goal must however be to reuse code. It would mean to move VirtualTreeview to Lazarus codebase together with online package manager once everything works.
I personally like the VirtualTreeview component. I made some innovative SW with it and Delphi years ago.

A thought: if the management of local packages and online packages is integrated in Lazarus code, the non-GUI parts of your code must be moved to Lazarus code base. It means GUI must be separated from logic even more than it is now.
Another choice is to add hooks to the Lazarus package management interface so that available online packages can be registered by external code.
« Last Edit: October 07, 2016, 12:36:56 pm by JuhaManninen »

Graeme

  • Hero Member
  • *****
  • Posts: 1430
    • Graeme on the web
Re: Online Package Manager
« Reply #51 on: October 07, 2016, 11:26:07 am »
Good question. BGRA Bitmap has 4 packages on it, what is installed? the packages are one for lcl, fpgui, nogui and open gl.
And fpGUI has one for each GUI backend (GDI, X11, Carbon, OS/2 etc)
--
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

rvk

  • Hero Member
  • *****
  • Posts: 3842
Re: Online Package Manager
« Reply #52 on: October 07, 2016, 11:30:34 am »
Quote
@rvk
Can't look right now but I think it's the OnDataReceived in dodataread. But it lacks a OnDataSend
True! With my streamer class can do both download/upload progress threaded. @rvk if you find a way to abort a download/upload process via class inheritence let me know. The same is true for a timeout function.
I made a helper for TFPHTTPClient (for a WebDav client) in which I override the complete SendRequest() to call a special CopyFromStreamToStream() which has a Callback() function for upload-progress. Maybe you implemented it the same way for download (haven't looked yet). It can be easily changed that the Callback() function can receive an Abort-boolean in which case the CopyFromStreamToStream() can abort it's operation.

The only problem I had was that "socket" wasn't exposed in TFPHTTPClient. And that's a serious problem. So I had to change the original fphttpclient.pp. I just added the following line to my copied fphttpclient.pp and I could use socket in my helper and stream.
Code: Pascal  [Select]
  1.     Property Socket: TInetSocket read FSocket;

But I do agree with JuhaManninen. An upload-callback as well as Abort-option should be added to TFPHTTPClient. It would even be a very small patch. The Download-progress is already there in the DoDataRead (which calls OnDataReceived). So the FOnDataSent would simple to implement. Only 3 lines in TFPHTTPClient but with a complete CopyFromStreamToStream() because TStream.CopyFrom() doesn't have a progress-option. Or should that one be added too :)

I was also very surprised to see the download-progress option in TFPHTTPClient was added but upload-progress option was left out.

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3652
  • I like bugs.
Re: Online Package Manager
« Reply #53 on: October 07, 2016, 11:56:45 am »
BGRA Bitmap has 4 packages on it, what is installed? the packages are one for lcl, fpgui, nogui and open gl.

I think there should be only one package in a ZIP file.
Having many packages adds a useless extra complication.

GetMem

  • Hero Member
  • *****
  • Posts: 3515
Re: Online Package Manager
« Reply #54 on: October 07, 2016, 12:15:43 pm »
@Juha, @rvk
Great! Then we need a patch for httpclient/zipper. Right now I'm busy with opkman, but I will do it later. If somebody wants to help, I would be grateful.

Quote
@Juha
I think there should be only one package in a ZIP file.
Having many packages adds a useless extra complication.
That's true, however many big packages(zeos, ibobjects, bgrabitmap) just to mention a few, contains multiple lpk files. Which one should be installed by default? It would be a great help for me if only one package/zip is installed.
« Last Edit: October 07, 2016, 12:18:35 pm by GetMem »

rvk

  • Hero Member
  • *****
  • Posts: 3842
Re: Online Package Manager
« Reply #55 on: October 07, 2016, 12:34:03 pm »
Ok, did some more testing. Here is a list. Don't mind some of the points (they might seem like nitpicking) but they are just some second observations.
(SPM = Standard Package Manager)
  • When I install a package and choose NOT to rebuild and close opkman and open it again... the opkman shows the package as "Already installed" while it is not yet (it should still say "Downloaded, extracted, ready to install" like it does before closing opkman). The SPM shows green plus sign before the package in that case (to indicate it will be installed on the next build).
  • Would the "Downloaded" status be really necessary? When not installing a package and again pressing install, wouldn't re-downloading it and installing it be more logical (you would also have the updated version if any). Or is "Downloaded" also for non-installable packages (like synapse)?
  • "Already installed(Version: 1.0.0.0)". There is no space between installed and (Version). That said... maybe 2 columns with Current version and Online/updated version would be useful. Like the fppkg does with "Installed" and "Available" columns. (See at the bottom of this post)
  • Escape doesn't work to close the opkman. There should probably a Cancel button for that just like in the "Install/Uninstall Packages" to easily exit the dialog.
  • Maybe a button to go directly to "Install/Uninstall Packages". Not sure if this is possible (opkman is ShowModal and SPM too). But it can be handy to take a quick look in SPM and escape right back into opkman.
  • An extra column with the (short)description of the package in the folded-view. There is plenty room to move Status (and current and available version #) in a separate column.
  • Abbreviate "Status" to something simpler. In that case that extra column doesn't need to be that wide. It's hard to see what's available/installed or downloaded at the moment. Just Available, Downloaded and Installed will be sufficient. And it will be easier on the eyes.
  • After the Downloaded and Installed could be an update status (like " - Updated").
  • Only make the " - Updated" status bold. All others should be plain. The bold status also makes it hard to read. (Especially when my eyes are tired :) )
  • When those hundreds of packages are added (good luck with that  :P ) a category-pulldown will be useful (I think it was already suggested).
  • Regarding the multiple packages in one zip/download. Maybe there can be checkboxes per .lpk inside the treeview which to install. Otherwise there should be some dialog after choosing a package as to which package you really want installed. Moving each .lpk into it's own zip/source is not advised because all those .lpk use the same source and multiple "same sources" will eventually get you in trouble when compiling.
  • Moving the development of the opkman to SourceForge or GitHub is highly appreciated. We (I) can keep track of progress and automatically SVN a new version into my new trunk-compile scripts :) (unless you have Lazarus repo access real soon)
Code: [Select]
[~]$ fppkg list
Name                 Installed        Available
embweb               -                0.9.1-968
gecko                -                2.9.0-3
jquery               -                0.9.0-842
lazmkunit            -                0.9.3-1
lnet                 -                0.6.6-2606
webdesign            -                0.9.0-906

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3652
  • I like bugs.
Re: Online Package Manager
« Reply #56 on: October 07, 2016, 01:56:50 pm »
Maybe a button to go directly to "Install/Uninstall Packages". Not sure if this is possible (opkman is ShowModal and SPM too). But it can be handy to take a quick look in SPM and escape right back into opkman.

That would be clumsy. The goal is to integrate management of online package and local packages. The SPM would then show also online packages and this new VirtualTreeview based GUI would show local packages. In essence the new GUI will be an alternative to the old GUI. Competition is good.

Quote
Regarding the multiple packages in one zip/download. Maybe there can be checkboxes per .lpk inside the treeview which to install. Otherwise there should be some dialog after choosing a package as to which package you really want installed. Moving each .lpk into it's own zip/source is not advised because all those .lpk use the same source and multiple "same sources" will eventually get you in trouble when compiling.

Currently people must select from individual .lpk files anyway, don't they?
I don't know of all the complexities involved but maybe the package manager should add a new meta-package type which includes many actual packages. In the GUI those packages would show as sub-nodes under the meta-package parent node.
Just an idea ...

lainz

  • Hero Member
  • *****
  • Posts: 3303
    • Lainz
Re: Online Package Manager
« Reply #57 on: October 07, 2016, 02:53:07 pm »
BGRA Bitmap has 4 packages on it, what is installed? the packages are one for lcl, fpgui, nogui and open gl.

I think there should be only one package in a ZIP file.
Having many packages adds a useless extra complication.

Yes but why to duplicate the same source code and just for getting a different zip? Maybe for lpk only a file is changed to fit the widgetset, all other files remains the same.
« Last Edit: October 07, 2016, 03:42:18 pm by lainz »

GetMem

  • Hero Member
  • *****
  • Posts: 3515
Re: Online Package Manager
« Reply #58 on: October 07, 2016, 06:00:57 pm »
@rvk
Quote
When I install a package and choose NOT to rebuild and close opkman and open it again... the opkman shows the package as "Already installed" while it is not yet.
That's not true, it is installed. Close Opkman(without rebuild), then open "Install/Uninstall packages" and check the installed package list. More over, if you close Lazarus and delete the just installed package directory from the filesystem, on next startup Lazarus will complain about a missing package. You cannot use it until an IDE rebuild but it is installed.
Quote
Would the "Downloaded" status be really necessary? When not installing a package and again pressing install, wouldn't re-downloading it and installing it be more logical (you would also have the updated version if any). Or is "Downloaded" also for non-installable packages (like synapse)?
Search for ForceDownload and ForceExtract booleans. The user will choose the appropriate action. It is already implemented(false by default),  not yet visible in the option dialog though.
Quote
"Already installed(Version: 1.0.0.0)". There is no space between installed and (Version). That said... maybe 2 columns with Current version and Online/updated version would be useful. Like the fppkg does with "Installed" and "Available" columns. (See at the bottom of this post)
Escape doesn't work to close the opkman. There should probably a Cancel button for that just like in the "Install/Uninstall Packages" to easily exit the dialog.
Maybe a button to go directly to "Install/Uninstall Packages". Not sure if this is possible (opkman is ShowModal and SPM too). But it can be handy to take a quick look in SPM and escape right back into opkman.
An extra column with the (short)description of the package in the folded-view. There is plenty room to move Status (and current and available version #) in a separate column.
Abbreviate "Status" to something simpler. In that case that extra column doesn't need to be that wide. It's hard to see what's available/installed or downloaded at the moment. Just Available, Downloaded and Installed will be sufficient. And it will be easier on the eyes.
After the Downloaded and Installed could be an update status (like " - Updated").
Only make the " - Updated" status bold. All others should be plain. The bold status also makes it hard to read. (Especially when my eyes are tired :) )
Those are all valid and useful suggestions. I will implement them.
Quote
When those hundreds of packages are added (good luck with that  :P ) a category-pulldown will be useful (I think it was already suggested).
:D Ok fair enough, I won't find that many packages but adding one package to the repository will be a matter of seconds, once the automated tool is ready.
Quote
Regarding the multiple packages in one zip/download. Maybe there can be checkboxes per .lpk inside the treeview which to install. Otherwise there should be some dialog after choosing a package as to which package you really want installed. Moving each .lpk into it's own zip/source is not advised because all those .lpk use the same source and multiple "same sources" will eventually get you in trouble when compiling.
Each lpk will have a checkbox.
Quote
Moving the development of the opkman to SourceForge or GitHub is highly appreciated. We (I) can keep track of progress and automatically SVN a new version into my new trunk-compile scripts :) (unless you have Lazarus repo access real soon)
Done: https://github.com/getmem/LazOpkMan

@Juha, @lainz
I will implement the multiple lpk version. Each lpk will have it's own checkbox, description etc.

   

valdir.marcos

  • Hero Member
  • *****
  • Posts: 849
Re: Online Package Manager
« Reply #59 on: October 07, 2016, 06:56:51 pm »
What about big complex packages that fits simultaneously many versions of FPC (for instance 2.6.x, 3.0.x, 3.2.x, etc) or many versions of Lazarus (for example 1.0.x, 1.2.x, 1.4.x, 1.6.x, 1.8.x/2.0.x, etc)? Some of them might have different .lpk files for different versions of FPC or Lazarus.

BGRA Bitmap has 4 packages on it, what is installed? the packages are one for lcl, fpgui, nogui and open gl.

I think there should be only one package in a ZIP file.
Having many packages adds a useless extra complication.

Yes but why to duplicate the same source code and just for getting a different zip? Maybe for lpk only a file is changed to fit the widgetset, all other files remains the same.