* * *

Author Topic: Online Package Manager + ct4laz repository  (Read 17954 times)

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3515
  • I like bugs.
Online Package Manager + ct4laz repository
« on: January 02, 2018, 02:27:52 pm »
Since I have now updated ct2laz to be able to convert latest CT 6.30+ projects and packages, I would like to discuss first what would be the ideal way to go for ct4laz. The general idea would be to use ct4laz from OPM so I write in this thread, but if you feel more like starting a new thread to avoid polution, please say so. Here is the general concept of ct4laz that I would like to discuss:
Avra, I move this to a new thread here. This is specific to your ct4laz repository and not about OPM in general. The replies would be buried to the general comments.

Quote
8 )   Automating conversion process is so far successful with ct2laz. However there might be a need in the future to apply patches to some components. I see that you do it with some OPM packages. Do you have it automated in some way?
This last point raises my alarms a little bit.
The main goal of OPM is to provide released packages. Fixing bugs and applying patches is the duty of package maintainers. The people who want to use a development version of a package can still do so by downloading from the master revision control repo.
Ok, sometimes this rule is fuzzy. There are projects without releases and OPM must use the development version then. Yes, Synapse may also be an exception to the rule. I don't know.

However if you create a process for OPM to patch the packages then you practically fork those projects and force GetMem to become a maintainer of those forked projects. No, it is not sustainable in the long run.
If a project needs a bug fix, then send a patch to the original project.
If the patch is ignored, then put pressure on its maintainer. Explain how important it is for OPM and how it is used by thousands of people (or something).
If the project's maintainer is busy, then offer yourself as a co-maintainer with commit access.
If things are still in halt, then fork the project and become its maintainer. This way the process is not tied to OPM which is meant for delivering packages, not maintaining them.
« Last Edit: January 02, 2018, 02:44:51 pm by JuhaManninen »

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3515
  • I like bugs.
Re: Online Package Manager + ct4laz repository
« Reply #1 on: January 02, 2018, 02:42:19 pm »
This was the avra's post in "Online Package Manager" thread :

Hello GetMem,

if you remember we had initial discussion about central repo for CodeTyphon components which I would like to call ct4laz:
http://forum.lazarus.freepascal.org/index.php/topic,38514.msg261943.html#msg261943

Since I have now updated ct2laz to be able to convert latest CT 6.30+ projects and packages, I would like to discuss first what would be the ideal way to go for ct4laz. The general idea would be to use ct4laz from OPM so I write in this thread, but if you feel more like starting a new thread to avoid polution, please say so. Here is the general concept of ct4laz that I would like to discuss:

1 )   I would use ct2laz to convert all CT packages and CodeOcean examples (excluding ORCA ones, because of the potential licensing problem)
2 )   I would host everything from conversion process on new ct4laz bitbucket git repo, similar as ct2laz is hosted now
3 )   I am thinking about using one repo for everything, using current directory structure of CT packages and CodeOcean examples. However, I am not yet familiar with registering packages to OPM, so I have to ask if this is a good approach? Once repo is established, one by one CT packages would be added to OPM.
4 )   I would like to host everything from the conversion process, although probably only pl_* packages and related CodeOcean examples are interesting for OPM. They should probably be downloaded together in one shot. Would that be a problem for OPM?
5 )   For clear distinction and giving credits to PilotLogic I would like to keep their pl_ prefix for each package originating from CT. OPM can show some other real name to the user, but I would like to keep their "original" package names. The reason for this are users who, for one reason or another (usually bug fixes or cross platform improvements), need to use original pl_* packages even when Lazarus and OPM provide native ones (ct2laz allows this possibility during the conversion process).
6 )   You already have several CT packages in OPM. I think that they should go into ct4laz and have "original" pl_ prefix in their package name, but I clearly would like to know your opinion on this matter. I would give you write access to ct4laz repo if you want, of course.
7 )   Using Synapse in OPM gave me a headache because of 2 versions. Some packages are using stable and some are using trunk. Trouble comes when I need several packages using different Synapse packages. It gets solved after manual fixing everything to use trunk Synapse, but that would be too much for an average OPM user. It also gives trouble when converting CT packages which use pl_synapse, because I have to tell ct2laz to convert all to trunk OPM Synapse package (stable is too old and not compatible). If I tell ct2laz not to convert pl_synapse then I have to install pl_synapse and that again conflicts with any OPM Synapse already installed. You see the problem. Ideally there would be only one Synapse package in OPM and that would be using Synapse trunk. What do you think about this?
8 )   Automating conversion process is so far successful with ct2laz. However there might be a need in the future to apply patches to some components. I see that you do it with some OPM packages. Do you have it automated in some way?

Any idea or improvement to this imaginary concept is most welcome.

wp

  • Hero Member
  • *****
  • Posts: 4974
Re: Online Package Manager + ct4laz repository
« Reply #2 on: January 02, 2018, 04:13:35 pm »
I fear a central repository of all CT packages for usage in Lazarus is a very bad idea and will end up in a terrible mess. I am not talking of these packages which are unique to CT, but of that vast majority which are just copies of Laz packages. Units and packages are renamed, but the classes are named like their Laz counterparts. There is nothing to prevent a user from installing a CT package and the corresponding Laz package simultaneously: now each class of these packages will be duplicate and, therefore, the package will not be functional any more, not sure - but maybe even Lazarus will stop working.

Moreover, these CT packages which duplicate Laz packages are less maintained than the Laz packages. Looking into the CT forum I see almost no activity, compared to forum and bugtracker here, and bug reports are usually commented as "we will look into this issue", and then, when Laz has fixed the issue, the fix will appear in the next CT version.

ct2laz is a great tool to make those CT packages available which don't have a Laz counterpart. But this is a tool for somebody who knows what he is doing. A repository of ready-to-use CT package will cause more trouble than benefits. Instead of this I'd pick those CT-only packages, convert them to Laz and distribute them in OPM. An example is the package "astronomy" which is now in OPM, but originally was a CT-only package.

But - this must be clear: All these packages will not be maintained, and I see the future that OPM will end up like CCR: initially lots of enthusiasm, but a few years later, when the excitment has gone, it will just accumulate dead code. What if FPC introduces another code-breaking feature? Or Laz moves functions to other units again? Or code marked as deprecated is finally removed? These things usually are easily fixed, but somebody must do it. Who? Who will take care of the more than hundred packages in OPM when this happens? With CT in there it will be even worse! (well, I am very pessimistic today...)
Lazarus trunk / fpc 3.0.4 / all 32-bit on Win-10

lainz

  • Hero Member
  • *****
  • Posts: 2828
    • Home
Re: Online Package Manager + ct4laz repository
« Reply #3 on: January 02, 2018, 04:21:37 pm »
I've seen BGRAControls in CT, they have units that I've removed and replaced with new one (components).

So they don't have it up to date, also some days ago I've added new components, and these just as wp said are a copy on CT nothing new or fixes of nothing.

So copy these that are unique and open source somewhere and if someone mantains it or not it's a problem of course.

GetMem

  • Hero Member
  • *****
  • Posts: 3195
Re: Online Package Manager + ct4laz repository
« Reply #4 on: January 02, 2018, 04:38:57 pm »
Hi all,

In the mean time other people also replied. Just to make clear, we are talking about packages that are unique to CT. Packages that was developed by Pilot Logic or converted from old Delphi sources and maintained only by them. It make no sense to re-add packages already available in Lazarus.

Quote
1 )   I would use ct2laz to convert all CT packages and CodeOcean examples (excluding ORCA ones, because of the potential licensing problem)
That would be great. Thanks. But we should add to OPM only the packages not available anywhere else. In fact this is what I did until now.
Quote
2 )   I would host everything from conversion process on new ct4laz bitbucket git repo, similar as ct2laz is hosted now
Bitbucket is perfrectly fine or any other version control system for that matter. 
Quote
3 )   I am thinking about using one repo for everything, using current directory structure of CT packages and CodeOcean examples. However, I am not yet familiar with registering packages to OPM, so I have to ask if this is a good approach? Once repo is established, one by one CT packages would be added to OPM.
After the conversion is complete, every package must be added to OPM manually, I will do this myself, luckily the process is automated, it takes no longer then 1 min. in average for each package.
Quote
4 )   I would like to host everything from the conversion process, although probably only pl_* packages and related CodeOcean examples are interesting for OPM. They should probably be downloaded together in one shot. Would that be a problem for OPM?
No problem at all. The release cycle of CT packages are slow, when new version of a particular package is avaliable, after a quick conversion it can be added again to OPM.
Quote
5 )   For clear distinction and giving credits to PilotLogic I would like to keep their pl_ prefix for each package originating from CT. OPM can show some other real name to the user, but I would like to keep their "original" package names. The reason for this are users who, for one reason or another (usually bug fixes or cross platform improvements), need to use original pl_* packages even when Lazarus and OPM provide native ones (ct2laz allows this possibility during the conversion process).
I don't mind adding pl_ prefix to packages, hower we should also add the original developers to the "Author" field(lpk file). I did this for every CT package available in OPM + I also added PilotLogic Software House to give cretids for the conversion, maintanance. I spent a lot of time seraching for original authors, although in most cases a hint is available somewhere in a txt file.
Quote
6 )   You already have several CT packages in OPM. I think that they should go into ct4laz and have "original" pl_ prefix in their package name, but I clearly would like to know your opinion on this matter. I would give you write access to ct4laz repo if you want, of course.
Yes I converted the packages manually, it was really painful and slow. With CT4Laz the whole process should take much less. Again I have nothing against pl_prefix in the package name. 
Quote
7 )   Using Synapse in OPM gave me a headache because of 2 versions. Some packages are using stable and some are using trunk. Trouble comes when I need several packages using different Synapse packages. It gets solved after manual fixing everything to use trunk Synapse, but that would be too much for an average OPM user. It also gives trouble when converting CT packages which use pl_synapse, because I have to tell ct2laz to convert all to trunk OPM Synapse package (stable is too old and not compatible). If I tell ct2laz not to convert pl_synapse then I have to install pl_synapse and that again conflicts with any OPM Synapse already installed. You see the problem. Ideally there would be only one Synapse package in OPM and that would be using Synapse trunk. What do you think about this?
As Juha mentioned above, the main goal of OPM is to provide released packages. However in the case of synapse I think we can make an exception. The release cycle is so slow that almost six years passed since the latest stable version. I'm gonna fix this issue soon.
Quote
8 )   Automating conversion process is so far successful with ct2laz. However there might be a need in the future to apply patches to some components. I see that you do it with some OPM packages. Do you have it automated in some way?
This is usually done by the "update feature"(http://wiki.freepascal.org/Online_Package_Manager#Update_a_package ). When a package is not maintained or the release cycle is slow I do it myself.

« Last Edit: January 02, 2018, 04:44:34 pm by GetMem »

lainz

  • Hero Member
  • *****
  • Posts: 2828
    • Home
Re: Online Package Manager + ct4laz repository
« Reply #5 on: January 02, 2018, 05:38:26 pm »
So avra or you will keep up to date all these packages, right?

I say because I think CT people will not do that?

Edit: non sense question =)

Of course you already mantain the packages uploading them, and if avra does a tool that automatically put these on bitbucket, it will be easy to do.
« Last Edit: January 02, 2018, 05:49:11 pm by lainz »

GetMem

  • Hero Member
  • *****
  • Posts: 3195
Re: Online Package Manager + ct4laz repository
« Reply #6 on: January 02, 2018, 05:49:56 pm »
@lainz
Quote
So avra or you will keep up to date all these packages, right?
After every major CT release, avra will convert the packages with ct2Laz, I will update them in OPM. The CT release cycle is slow, so it should be no problem. More over not every package is updated with each release.

Quote
I say because I think CT people will not do that?
This project has nothing to do with CT people. The important thing is to import back the good things from CT or at least this is how I see it.

lainz

  • Hero Member
  • *****
  • Posts: 2828
    • Home
Re: Online Package Manager + ct4laz repository
« Reply #7 on: January 02, 2018, 05:51:34 pm »
Good idea, you're getting a lot of help to continue improving! Keep going!

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3515
  • I like bugs.
Re: Online Package Manager + ct4laz repository
« Reply #8 on: January 02, 2018, 05:52:14 pm »
ct2laz is a great tool to make those CT packages available which don't have a Laz counterpart. But this is a tool for somebody who knows what he is doing. A repository of ready-to-use CT package will cause more trouble than benefits. Instead of this I'd pick those CT-only packages, convert them to Laz and distribute them in OPM. An example is the package "astronomy" which is now in OPM, but originally was a CT-only package.
A repository is needed if converted packages need maintenence. Then we are talking about forking instead of only converting. Avra will naturally be the maintainer of that fork. I like the idea, that is how FOSS development works. Copying and modifying code back and forth.

Quote
But - this must be clear: All these packages will not be maintained, and I see the future that OPM will end up like CCR: initially lots of enthusiasm, but a few years later, when the excitment has gone, it will just accumulate dead code. What if FPC introduces another code-breaking feature? Or Laz moves functions to other units again? Or code marked as deprecated is finally removed? These things usually are easily fixed, but somebody must do it. Who? Who will take care of the more than hundred packages in OPM when this happens? With CT in there it will be even worse! (well, I am very pessimistic today...)
OPM and CCR have a big difference.

CCR is a code hosting project in SourceForge. Yes, many sub-project maintainers fleed but it is not the fault of SourceForge or SVN really. Other hosting sites like GitHub are now more fashionable and maybe CCR is just out of fashion in some peoples eyes.

OPM is for delivering packages only. All the fixes and code maintenance must be done in the projects. This way OPM's maintenance will be sustainable also in the future. Ideally the OPM's maintainer should complain about a broken or old package to its maintainer, putting pressure. In a user's eyes a package found in OPM is already "semi-official" and thus gets more attention and users than some other package.
Forking should be used mercilessly for poorly maintained packages, assuming there is a new maintainer up to the task. CCR projects don't even need forking, we just give commit access for a new person.
Any volunteers for the currently neglected CCR packages?
« Last Edit: January 02, 2018, 05:55:48 pm by JuhaManninen »

avra

  • Hero Member
  • *****
  • Posts: 1366
    • Additional info
Re: Online Package Manager + ct4laz repository
« Reply #9 on: January 02, 2018, 08:05:50 pm »
Avra, I move this to a new thread here. This is specific to your ct4laz repository and not about OPM in general. The replies would be buried to the general comments.
Agreed, thanks!

Quote from: JuhaManninen link=topic=39508.msg271353#msg271353
Quote from: avra
8 )   Automating conversion process is so far successful with ct2laz. However there might be a need in the future to apply patches to some components. I see that you do it with some OPM packages. Do you have it automated in some way?
This last point raises my alarms a little bit. The main goal of OPM is to provide released packages. Fixing bugs and applying patches is the duty of package maintainers.
You don't have to worry. My intention is not forking and maintaining forks. I already use ct2laz to convert CT packages, so I would just put that in a repo to allow general use through OPM for packages that are interesting to others. It would ease access to CT components without using ct2laz. The only 2 packages so far that I see need patching are indy (fix for intentional indy leak, which is already applied in CT version package) and wst (CT version is compilable, unlike author's version although I gave him patches long time ago). I don't know if GetMem will decide to use CT versions, but that would give him maintenance free benefit in this case. I think that in the future CT will move further from Lazarus, and that is the area where I sea automated patching should be applied. I don't find fun to apply same patches all over again by hand, so there is my interest in the topic.

I fear a central repository of all CT packages for usage in Lazarus is a very bad idea and will end up in a terrible mess. I am not talking of these packages which are unique to CT, but of that vast majority which are just copies of Laz packages.
I think that GetMem will probably use such repo just for packages not available otherwise, maybe with 1-2 exceptions that will not bring any problems you fear. I myself do not have such problems even after heavy usage. Anyway, I will probably not publish everything in repo simply because currently it would take 1.2GB.

All these packages will not be maintained, and I see the future that OPM will end up like CCR: initially lots of enthusiasm, but a few years later, when the excitment has gone, it will just accumulate dead code.
I do not agree. Code will be maintained as much as PilotLogic maintains it. Through all these years PilotLogic managed to do it well. I have nothing to do with it. However if you want a guarantee for the future, there is none - as with many thing in life.

What if FPC introduces another code-breaking feature? Or Laz moves functions to other units again? Or code marked as deprecated is finally removed?
This catch-up game exists even without CT involved, so I do not see a problem. There will always be packages that work only with stable version, and some that will work only with trunk. With CT using trunk by default and looking at the history of their updates I think that they adapt much faster then you fear. As for OPM, in worse case it will simply not be able to install package in one of Lazarus versions. Nothing new, and we see it anyway without CT involved at all.
« Last Edit: January 02, 2018, 08:08:31 pm by avra »
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

avra

  • Hero Member
  • *****
  • Posts: 1366
    • Additional info
Re: Online Package Manager + ct4laz repository
« Reply #10 on: January 02, 2018, 08:39:47 pm »
...in the case of synapse I think we can make an exception. The release cycle is so slow that almost six years passed since the latest stable version. I'm gonna fix this issue soon.
Thank you very much. I am very happy.

I've seen BGRAControls in CT, they have units that I've removed and replaced with new one (components).
Don't worry. OPM will include just CT specific packages.

A repository is needed if converted packages need maintenence. Then we are talking about forking instead of only converting. Avra will naturally be the maintainer of that fork.
Well, that will not be a fork in a classic sense. I do not intend to make any significant changes to CT code. I will only make ct4laz because PilotLogic does not provide public repo, and because I want those components to be available to Lazarus users. Maintaining will also not be in a classic sense, since I will just convert CT packages and CodeOcean examples to enable their use in Lazarus. OPM is then a natural choice for their delivery to Lazarus users.
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3515
  • I like bugs.
Re: Online Package Manager + ct4laz repository
« Reply #11 on: January 02, 2018, 09:45:12 pm »
You don't have to worry. My intention is not forking and maintaining forks.
Maybe I expressed my meaning poorly. Forking is OK. If you change the code only a little, it is still a fork. My point was that maintaining of the forks must not be done as part of the OPM delivery process. It must be done in a repository somewhere and preferably not by the OPM's maintainer.
I know GetMem is involved now in package maintenence and now it works because OPM is new and everybody is enthusiastic. In future, say 2025, situation may be different. The OPM maintenance must be designed to be light.

Quote
... and wst (CT version is compilable, unlike author's version although I gave him patches long time ago).
Yes, that should be solved by adding more maintainers for WST. Luckily the sources are in CCR. Everybody who has commit access there can also update WST. Do you have commit access for CCR? You could apply the patches, then we inform Inoussa. He is clearly busy and does not check this forum or even his mailing list often.
I think we should make committing to CCR more flexible. Maybe it would become more fashionable again. I don't see why somebody should not fix a bug in a component even if he is not the maintainer. Commit rights must be based on some credit of course, they cannot be given to everybody.

GetMem

  • Hero Member
  • *****
  • Posts: 3195
Re: Online Package Manager + ct4laz repository
« Reply #12 on: January 02, 2018, 10:03:03 pm »
@avra
I fixed the synapse issue. I added the latest trunk(ver. 40.1), also fixed synapse dependencies for other packages, those who pointed to the old version(40.0). Please take a look.

@JuhaManninen
Quote
The OPM maintenance must be designed to be light.
It is light. Everyone can learn in 5 minutes how to add/delete a package. The problem is only I have access to the central repository and this should be changed in the future, so other people can upload packages.

Quote
I think we should make committing to CCR more flexible. Maybe it would become more fashionable again. I don't see why somebody should not fix a bug in a component even if he is not the maintainer. Commit rights must be based on some credit of course, they cannot be given to everybody.
This is a good idea. CCR worked well in the past, it still works well, but a lot of packages are not maintained.

avra

  • Hero Member
  • *****
  • Posts: 1366
    • Additional info
Re: Online Package Manager + ct4laz repository
« Reply #13 on: January 03, 2018, 12:59:46 am »
I fixed the synapse issue.
I will check and let you know. Thanks!

Quote
... and wst (CT version is compilable, unlike author's version although I gave him patches long time ago).
Yes, that should be solved by adding more maintainers for WST. Luckily the sources are in CCR. Everybody who has commit access there can also update WST. Do you have commit access for CCR? You could apply the patches, then we inform Inoussa. He is clearly busy and does not check this forum or even his mailing list often.
If you remember I have notified Inoussa (both by forum and mail) and gave him patches. After a while they were only partially applied and WST could not be compiled in some cases (not checked recently). Patches were not a product of my deeper WST understanding, but a result of comparing original to CT version, which worked well in every Lazarus version I tried. Therefore it is not realistic for me to become a WST maintainer. I prefer to just use CT version of WST (I haven't checked OPM version lately, but I will). Talking about removing maintenance burden from GetMem, it would be more logical to me if CT version will be used for OPM. However, that will be GetMem's decision and for one reason or another he might prefer to continue to maintain his own patched version.

Here are the WST related messages as a reminder:
https://forum.lazarus.freepascal.org/index.php/topic,24519.msg250764.html#msg250764
https://forum.lazarus.freepascal.org/index.php/topic,34297.msg248818.html#msg248818
http://forum.lazarus.freepascal.org/index.php/topic,37285.msg
« Last Edit: January 03, 2018, 01:02:40 am by avra »
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3515
  • I like bugs.
Re: Online Package Manager + ct4laz repository
« Reply #14 on: January 05, 2018, 11:02:36 pm »
If you remember I have notified Inoussa (both by forum and mail) and gave him patches. After a while they were only partially applied and WST could not be compiled in some cases (not checked recently). Patches were not a product of my deeper WST understanding, but a result of comparing original to CT version, which worked well in every Lazarus version I tried. Therefore it is not realistic for me to become a WST maintainer.
Ok, "maintainer" was too strong a word. I meant you can apply your changes which make the code compile again.
For example people who have commit rights for Lazarus SVN repository, fix bugs all around the code base even if they are not authors or main developers of that particular code. It leads to a flexible, light and efficient development process.
With CCR we must be able to do the same. Otherwise this is too bureaucratic. I don't see any reason why you shoud not apply your valid fixes now.
Do you have the commit rights for CCR?

 

Recent

Get Lazarus at SourceForge.net. Fast, secure and Free Open Source software downloads Open Hub project report for Lazarus