Recent

Author Topic: Online Package Manager  (Read 836290 times)

balazsszekely

  • Guest
Re: Online Package Manager
« Reply #345 on: November 29, 2016, 08:02:12 pm »
@Juha
Fishy in what way? I saw your earlier comment but perhaps we misunderstood each other. As you can see there is already a new "Update" column(screenshot) in the trunk. This column will display the latest version of the package, loaded from an external JSON. To trigger an update, the package maintainer can:
  1. Modify the version number to be bigger then the one in the central repository
  2. Set a boolean flag to true(ForceUpdate for example)
The external json, will contain a link to a zip(also provided by the package maintainer). The zip can be anything as far as I'm concern(the trunk version of the package, the master version, the next release 4.2.1.0-->4.2.1.1, whatever). My job is to add the initial package to the central repository + the link to the external json. After that it's up to the user and package maintainer. Still consider it fishy? If yes please explain why.

PS: The OPM won't update automatically, just displays a message(balloon hint, fleshing line, etc...). The user decides the appropriate action.
« Last Edit: November 29, 2016, 08:19:27 pm by GetMem »

minesadorada

  • Sr. Member
  • ****
  • Posts: 452
  • Retired
Re: Online Package Manager
« Reply #346 on: November 29, 2016, 08:32:40 pm »
Looks like a workable system.

It would be nice for package maintainers if there were a simple 'external json editor' app.  IMHO it would make the process of deploying an external update zip + json a lot easier and safer.  It doesn't need to be fancy - just a bunch of TLabel + TEdits and 'create', 'load' and  'save as' buttons.  A fancier version would check the given URLs were valid before saving.

I know the 'create package' has similar functionality for new packages, but it's not really suitable for maintenance updates (as I have found)
« Last Edit: November 29, 2016, 08:59:31 pm by minesadorada »
GPL Apps: Health MonitorRetro Ski Run
OnlinePackageManager Components: LazAutoUpdate, LongTimer, PoweredBy, ScrollText, PlaySound, CryptINI

minesadorada

  • Sr. Member
  • ****
  • Posts: 452
  • Retired
Re: Online Package Manager
« Reply #347 on: November 29, 2016, 08:53:07 pm »
While we are on the subject of external zips, how about looking at security?

Let's say as a user, I download and update a component from OPM in my Lazarus 1.8, but...
..it turns out that the external update zip wasn't hosted on a secure version control server (GitHub, SourceForge etc) and a malaicious person has replaced the update zip with a malware version (without the maintainer's knowledge)  Components once installed, have a lot of power.  An 'Execute' method (or any method) could do some bad stuff, and quite invisibly too.  Few ordinary component users (like me) check all the source code before using an updated component, particularly if it appears 'built-in' to Lazarus (as OPM's packages will appear)

This is why I suggested a protected area on a known secure server for update zips.  Upload access can be as secure as needed (like commit access to the ccr)  I would favour Sourceforge because I'm used to it, it uses SVN and it does automatic virus checks on all file uploads.  Even better would be an ftp server with multiple folders tied to individual logins to avoid 'cross-contamination' between component commits/uploads (as sourceforge ccr allows)  With only a few dozen components, it doesn't take long to set up (for example) ftp://cryptini.secureserver.com to point to the /components/cryptini folder with a unique login to upload - and it's mostly a one-off process.   These would then be the unique 'master' versions @JuhaMannien was talking about.  OPM/GetMem would have access to the whole /components tree (OPM=read-only, GetMem=read + write) via separate logins.  Having the read-only login /URL in OPM sourcecode would not be a security risk.
« Last Edit: November 29, 2016, 09:43:46 pm by minesadorada »
GPL Apps: Health MonitorRetro Ski Run
OnlinePackageManager Components: LazAutoUpdate, LongTimer, PoweredBy, ScrollText, PlaySound, CryptINI

balazsszekely

  • Guest
Re: Online Package Manager
« Reply #348 on: November 29, 2016, 10:04:23 pm »
@minesadorada

Quote
It would be nice for package maintainers if there were a simple 'external json editor' app.  IMHO it would make the process of deploying an external update zip + json a lot easier and safer.  It doesn't need to be fancy - just a bunch of TLabel + TEdits and 'create', 'load' and  'save as' buttons.  A fancier version would check the given URLs were valid before saving.
The JSON will be simple, it can be edited manually. Something like this:
Code: XML  [Select][+][-]
  1. {
  2.  "Package" : {
  3.     "Name" : "whatever.zip"
  4.     "ForceUpdate" : False,
  5.     "DownloadURL" : "https:\\....",    
  6.   },
  7.   "PackageFiles" : [
  8.     {
  9.       "Name" : "aaaa.lpk",
  10.       "Version" : "1.2.5.0"
  11.     },
  12.     {
  13.       "Name" : "bbbb.lpk",
  14.       "Version" : "0.0.5.0"
  15.     }
  16.   ]
  17. }
  18.  
I'm not against a GUI, but somebody has to do it.

Quote
I know the 'create package' has similar functionality for new packages, but it's not really suitable for maintenance updates (as I have found)
Well I'm using it and apparently works well since the whole central repository was built with it + one more tool. Still room for improvements, a lot of it. I agree!

Quote
Let's say as a user, I download and update a component from OPM in my Lazarus 1.8, but...
..it turns out that the external update zip wasn't hosted on a secure version control server (GitHub, SourceForge etc) and a malaicious person has replaced the update zip with a malware version (without the maintainer's knowledge)  Components once installed, have a lot of power.  An 'Execute' method (or any method) could do some bad stuff, and quite invisibly too.  Few ordinary component users (like me) check all the source code before using an updated component, particularly if it appears 'built-in' to Lazarus (as OPM's packages will appear)
This is why I wanted to avoid external updates, since it's allowed, there is no such things as security, especially under windows and I can prove it. You can convert an exe to a txt file(it will pass any virus detection),  when needed the malicious package converts it back to byte array then injects it directly to a process memory, without saving the exe file to disk. The last step(memory injection) can be detected by some of the AV's out there, but most of the time will pass. Anyway we don't have to be paranoid...

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4459
  • I like bugs.
Re: Online Package Manager
« Reply #349 on: November 29, 2016, 10:22:51 pm »
My job is to add the initial package to the central repository + the link to the external json. After that it's up to the user and package maintainer. Still consider it fishy? If yes please explain why.
The fishy part is that you add an initial package to central repo but it will not be used after the package author decides to update for the first time.
Then comes a question: why to add the initial version at all? Why not just provide an URL for the external place? Then it would be a purely external package source, not a half-baked mixture of initial OPM repo package + the real package from external source.
Many ideas were thrown here by many people, including myself, and apparently they got mixed together badly.

My idea still is:
If you want to support external packages, add support for an external URL without an initial OPM repo package. Then the OPM repo would contain only information about a package and its source but not the package itself.

Another option is to make updating the OPM repo packages so easy that loading from an external URL is not needed.
Then the whole update feature could be dumped.

minesadorada writes about security which is also an important issue.
However the idea of FTP user accounts for every package maintainer has technical difficulties. It requires a server admin (Marc, Vincent or anybody) manage those accounts. We want this to be easy for any new package author, a new step of bureaucracy for every new author is not good. I doubt the admins would accept the whole process idea. So, the FTP must be writable without a password or with a common password which is given to anybody who asks.
« Last Edit: November 29, 2016, 10:29:53 pm by JuhaManninen »
Mostly Lazarus trunk and FPC 3.2 on Manjaro Linux 64-bit.

balazsszekely

  • Guest
Re: Online Package Manager
« Reply #350 on: November 29, 2016, 10:37:41 pm »
Quote
The fishy part is that you add an initial package to central repo but it will not be used after the package author decides to update for the first time.
Then comes a question: why to add the initial version at all? Why not just provide an URL for the external place? Then it would be a purely external package source, not a half-baked mixture of initial OPM repo package + the real package from external source.
Because at least half of the packages in the main repository don't have a maintainer, there is no external URL. Another reason: all the packages inside the main repository are safe to use, the update feature is optional.
The well known package(synapse) is maintained, but the author rarely visits this forum(if ever) should I approach him to create a JSON? Or if I create the JSON myself, I have to check all the time if the link to the zip has changed. If I start to think about it OPM without a repo is the half-baked solution.

Quote
Another option is to make updating the repo packages so easy that loading from an external URL is not needed.Then the whole update feature could be dumped.
How? You have to provide credentials for that. Without username/password someone with malicious intent will overwrite the whole repo  in five minutes. A login system can be done, but it's not a one man job.

Quote
So, the FTP must be writable without a password or with a common password which is given to anybody who asks.
Exactly, then bad things start to happen immediately.
« Last Edit: November 29, 2016, 11:01:51 pm by GetMem »

minesadorada

  • Sr. Member
  • ****
  • Posts: 452
  • Retired
Re: Online Package Manager
« Reply #351 on: November 29, 2016, 10:40:18 pm »
minesadorada writes about security which is also an important issue.
However the idea of FTP user accounts for every package maintainer has technical difficulties. It requires a server admin (Marc, Vincent or anybody) manage those accounts. We want this to be easy for any new package author, a new step of bureaucracy for every new author is not good. I doubt the admins would accept the whole process idea. So, the FTP must be writable without a password or with a common password which is given to anybody who asks.

Well second-best is still better then nothing.  How about a Sourceforge OPM Components repository?  After all, the original source for GetMem was the SourceForge CCR repository.

For example; I just installed Richmemo from OPM, and the update URL is https://havefunsoft.com/share/richmemo.zip.  I have no idea if 'havefunsoft.com' is OK or not.

I think you overcook the difficulty.  It's not as though you're going to get new packages every week - or even every 6 months probably.  I agree it's a pain to set up an FTP server in the way I suggest but once done; server maintenance would be infrequent and trivial IMO, and there would be no admin for existing package maintenance, which is a big bonus.  New package authors could use a public access staging area with a common login (so no delay) - once trusted (vetted, effectively) they would get their ftp area/login for maintenance/updates.

And it does solve most of the security worries - which are important.
« Last Edit: November 29, 2016, 10:59:19 pm by minesadorada »
GPL Apps: Health MonitorRetro Ski Run
OnlinePackageManager Components: LazAutoUpdate, LongTimer, PoweredBy, ScrollText, PlaySound, CryptINI

minesadorada

  • Sr. Member
  • ****
  • Posts: 452
  • Retired
Re: Online Package Manager
« Reply #352 on: November 29, 2016, 10:52:00 pm »
@minesadorada

Quote
It would be nice for package maintainers if there were a simple 'external json editor' app.  IMHO it would make the process of deploying an external update zip + json a lot easier and safer.  It doesn't need to be fancy - just a bunch of TLabel + TEdits and 'create', 'load' and  'save as' buttons.  A fancier version would check the given URLs were valid before saving.
The JSON will be simple, it can be edited manually. Something like this:
Code: XML  [Select][+][-]
  1. {
  2.  "Package" : {
  3.     "Name" : "whatever.zip"
  4.     "ForceUpdate" : False,
  5.     "DownloadURL" : "https:\\....",    
  6.   },
  7.   "PackageFiles" : [
  8.     {
  9.       "Name" : "aaaa.lpk",
  10.       "Version" : "1.2.5.0"
  11.     },
  12.     {
  13.       "Name" : "bbbb.lpk",
  14.       "Version" : "0.0.5.0"
  15.     }
  16.   ]
  17. }
  18.  
I'm not against a GUI, but somebody has to do it.
Once you have the format settled I'm willing to give it a go.  Perhaps I'll learn about JSON on the way :)   I guess http://wiki.lazarus.freepascal.org/fcl-json is the page to start with.

BTW, what about scaleability?  Is it practical for OPM to offer 200 packages for example?  If not - what would be a practical maximum?
« Last Edit: November 29, 2016, 11:11:24 pm by minesadorada »
GPL Apps: Health MonitorRetro Ski Run
OnlinePackageManager Components: LazAutoUpdate, LongTimer, PoweredBy, ScrollText, PlaySound, CryptINI

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4459
  • I like bugs.
Re: Online Package Manager
« Reply #353 on: November 29, 2016, 11:20:02 pm »
How? You have to provide credentials for that. Without username/password someone with malicious intent will overwrite the whole repo  in five minutes. A login system can be done, but it's not a one man job.
I had the FTP server in mind + a manual check of new packages aided by some admin tools maybe. Yes I know this solution has issues.
It is better I leave this to you people to decide. Maybe the mixture of OPM repo + external package actually works if the repo version is also updated sometimes.
I am not very experienced with server SW configs. You should ask from the server admin about FTP, if it is still needed.

On a positive side: the basic functionality of OPM is solid. The manually added packages are safe and good. It is good to build on that.
Mostly Lazarus trunk and FPC 3.2 on Manjaro Linux 64-bit.

lainz

  • Hero Member
  • *****
  • Posts: 4460
    • https://lainz.github.io/
Re: Online Package Manager
« Reply #354 on: November 30, 2016, 12:05:39 am »
Quote
It would be nice for package maintainers if there were a simple 'external json editor' app.  IMHO it would make the process of deploying an external update zip + json a lot easier and safer.  It doesn't need to be fancy - just a bunch of TLabel + TEdits and 'create', 'load' and  'save as' buttons.  A fancier version would check the given URLs were valid before saving.
The JSON will be simple, it can be edited manually. Something like this:
Code: XML  [Select][+][-]
  1. {
  2.  "Package" : {
  3.     "Name" : "whatever.zip"
  4.     "ForceUpdate" : False,
  5.     "DownloadURL" : "https:\\....",    
  6.   },
  7.   "PackageFiles" : [
  8.     {
  9.       "Name" : "aaaa.lpk",
  10.       "Version" : "1.2.5.0"
  11.     },
  12.     {
  13.       "Name" : "bbbb.lpk",
  14.       "Version" : "0.0.5.0"
  15.     }
  16.   ]
  17. }
  18.  
I'm not against a GUI, but somebody has to do it.

Here you have the JSON editor for the external json, feel free to update it as it needs more or less fields.

This is the generated json:

Code: Pascal  [Select][+][-]
  1. { "Package" : { "DownloadURL" : "google.com", "ForceUpdate" : true, "Name" : "BGRAControls" }, "PackageFiles" : [{ "Name" : "bgra.lpk", "Version" : "1.0.0.0 3" }, { "Name" : "gl.lpk", "Version" : "2.0.0.0 3" }] }
  2.  

It does not output a pretty print version, but it works.

minesadorada

  • Sr. Member
  • ****
  • Posts: 452
  • Retired
Re: Online Package Manager
« Reply #355 on: November 30, 2016, 01:20:18 am »

It does not output a pretty print version, but it works.
Thanks @lainz - nice neat code!
Useable as is, but I'll mess around with the GUI if that's OK with you.
« Last Edit: November 30, 2016, 01:24:13 am by minesadorada »
GPL Apps: Health MonitorRetro Ski Run
OnlinePackageManager Components: LazAutoUpdate, LongTimer, PoweredBy, ScrollText, PlaySound, CryptINI

lainz

  • Hero Member
  • *****
  • Posts: 4460
    • https://lainz.github.io/
Re: Online Package Manager
« Reply #356 on: November 30, 2016, 01:43:40 am »

It does not output a pretty print version, but it works.
Thanks @lainz - nice neat code!
Useable as is, but I'll mess around with the GUI if that's OK with you.

Thanks. Play with it, but please keep it working for High DPI screens when you change it! (http://wiki.lazarus.freepascal.org/High_DPI)

minesadorada

  • Sr. Member
  • ****
  • Posts: 452
  • Retired
Re: Online Package Manager
« Reply #357 on: November 30, 2016, 02:34:53 am »
Thanks. Play with it, but please keep it working for High DPI screens when you change it! (http://wiki.lazarus.freepascal.org/High_DPI)
No worries - just tweaking captions and adding data checks to make it more idiot-proof for folk like me :)

@GetMem - Is the DownloadURL  fully formed? i.e. does the DownloadURL field contain the full zipfile name?
« Last Edit: November 30, 2016, 02:37:45 am by minesadorada »
GPL Apps: Health MonitorRetro Ski Run
OnlinePackageManager Components: LazAutoUpdate, LongTimer, PoweredBy, ScrollText, PlaySound, CryptINI

lainz

  • Hero Member
  • *****
  • Posts: 4460
    • https://lainz.github.io/
Re: Online Package Manager
« Reply #358 on: November 30, 2016, 02:58:57 am »
Also change the title to something like OPM External JSON Editor, the name I set is really bad.

minesadorada

  • Sr. Member
  • ****
  • Posts: 452
  • Retired
Re: Online Package Manager
« Reply #359 on: November 30, 2016, 03:25:49 am »
Also change the title to something like OPM External JSON Editor, the name I set is really bad.
I've just been tidying up the window-dressing, project options and other stuff.  I've attached what I have so far, but I'll have another go tomorrow (it's late here)
« Last Edit: November 30, 2016, 03:42:16 am by minesadorada »
GPL Apps: Health MonitorRetro Ski Run
OnlinePackageManager Components: LazAutoUpdate, LongTimer, PoweredBy, ScrollText, PlaySound, CryptINI

 

TinyPortal © 2005-2018