Lazarus

Free Pascal => FPC development => Topic started by: dbannon on January 18, 2022, 10:23:29 am

Title: A (Debian) Quality Proposal
Post by: dbannon on January 18, 2022, 10:23:29 am
A very open ended question !

Debian, as part of their QA process, flag  problems in source packages. The recently packaged FPC 3.2.2 has quite few of them (so do many apps). Now, superficially you may see these as mostly quite theoretical or nit-picking but it is a quality question, so, do we do quality for its own sake ?

Or because we'd like to keep in Debian's good books or even to help the long suffering packaging team ?  And most of these things are a problem even if a very minor one.

What are they ?  https://lintian.debian.org/sources/fpc

Things like test files not being UTF8 (or ascii) seem to be about one third of the total -

Quote
A file is not valid UTF-8.
Debian has used UTF-8 for many years. Support for national encodings is being phased out. This file probably appears to users in mangled characters (also called mojibake).  Packaging control files must be encoded in valid UTF-8.

Spelling also figures highly, they have a list of common misspelling and scan against it.

There are also a host of things we probably would argue is the way it should be.

So, my question, would the FPC developers accepts fixes to some of these issues ? Its unlikely it will improve the product significantly but any change carries a risk of breaking something. My approach would be, perhaps -

Identify the ones that can easily be fixed, examine each one in context (absolutely no automatic changes proposed here folks!) and fix it. Get, maybe ten such fixes in a batch, build the source do some superficial test hopefully focused on the possible areas affected.  Submit that batch of fixes as a pull request.  (or patches, or whatever you prefer).

It would be a slow process, done as a spare time activity not a key project.

So, if it worthwhile or not ?

Davo

Title: Re: A (Debain) Quality Proposal
Post by: Thaddy on January 18, 2022, 10:34:39 am
Most if not all have to do that FPC supports so many platforms.
It would be very hard to make an exception for Debian Linux in the sourcecode of the compiler/rtl.
The Debian team seems to recognize that, otherwise fpc would not be included in the standard distribution.
Title: Re: A (Debain) Quality Proposal
Post by: MarkMLl on January 18, 2022, 11:06:29 am
I'd have thought that fixing the highlighted spelling errors would be in everybody's interest.

Codepage issues should take into account (a) the target platform and (b) the provenance of the file in which the offending cavaliers appear: it would be reasonable to treat a target which predated UTF-8 and where an API definition file had a non-Unicode markup convention as a special case.

MarkMLl
Title: Re: A (Debain) Quality Proposal
Post by: dbannon on January 18, 2022, 11:25:58 am
I don't think too many (and I have only looked at six or seven) are deliberate internationalization More historical accidents IMHO

Quote
errln('error setting exception n�'+hexstr(i,2));.

That line is enough for that file to be declared an ISO-8859.  I found a couple that had a mystery symbol where I'd expect the copyright symbol to be used.  Would it be fair to say internationalization belongs in the PO system ?

Similarly, spelling, Debian's list of "frequently misspelt words" (sic) would have been built up over time based on its own very broad coverage.

There are heaps of other things too, some judgement would need to be applied, thus my comment about no automatic changes ....

Davo
Title: Re: A (Debain) Quality Proposal
Post by: MarkMLl on January 18, 2022, 11:36:30 am
Quote
errln('error setting exception n�'+hexstr(i,2));.

If that's intended to be something like nr, nm or no then it definitely needs fixing IMO since it can be done by converting to conventional ASCII.

MarkMLl
Title: Re: A (Debain) Quality Proposal
Post by: munair on January 18, 2022, 12:17:53 pm
Most if not all have to do that FPC supports so many platforms.
It would be very hard to make an exception for Debian Linux in the sourcecode of the compiler/rtl.

If Debian complains about files not being UTF8 then the problem would apply to all platforms natively using UTF8 (probably all OSs with a Linux kernel). It is just that Debian has high quality control checking, which would not be the right reason to make an exception for Debian Linux.
Title: Re: A (Debain) Quality Proposal
Post by: PascalDragon on January 18, 2022, 01:56:58 pm
So, my question, would the FPC developers accepts fixes to some of these issues ?

In principle, yes, though in the end it will be a case-by-case basis, so "batching" as you suggested is not necessarily the correct way (though it depends on the issue in question).

I don't think too many (and I have only looked at six or seven) are deliberate internationalization More historical accidents IMHO

Quote
errln('error setting exception n�'+hexstr(i,2));.

That line is enough for that file to be declared an ISO-8859.  I found a couple that had a mystery symbol where I'd expect the copyright symbol to be used.

And this is one were the context is important (would have been nice if you mentioned the filename, I had to search it myself... ::) ): this is a file for go32v2. That platform has no knowledge about UTF-8 and uses legacy codepages. In this specific case it's a "ø" (and my Lazarus on Windows displays it correctly).

Would it be fair to say internationalization belongs in the PO system ?

The code that's provided by FPC itself does not care about po files. The only applications that provide internationalization support are the compiler and the textmode IDE and both use their own mechanisms.

By the way: please fix the "Debain" in the title :P
Title: Re: A (Debain) Quality Proposal
Post by: marcov on January 18, 2022, 03:26:11 pm
Some might also be fixed meanwhile (most notably ocurred occurred). But it of course can also be more of the same.

Also check trunk to be sure.
Title: Re: A (Debain) Quality Proposal
Post by: marcov on January 18, 2022, 05:30:27 pm
If Debian complains about files not being UTF8 then the problem would apply to all platforms natively using UTF8 (probably all OSs with a Linux kernel).

Not if they convert on startup. Keep in mind that Dos has no native codepage conversion routines. Mostly they are used for older parts of the system like the textmode IDE. On the modern utf8 targets, they mostly use Lazarus

Quote
It is just that Debian has high quality control checking, which would not be the right reason to make an exception for Debian Linux.

It generates a lot, but quality can be debated.  Many things it lists that are not spelling mistakes are deliberate though. (like the static linking, the binary in a versioned dir in  lib etc)

Title: Re: A (Debain) Quality Proposal
Post by: dbannon on January 19, 2022, 02:44:01 am
In principle, yes, though in the end it will be a case-by-case basis, so "batching" as you suggested is not necessarily the correct way (though it depends on the issue in question).
Could make a lot of work for the devs but I now understand it can be more subtle than I thought.


Quote
errln('error setting exception n�'+hexstr(i,2));.
.....this is a file for go32v2. That platform has no knowledge about UTF-8 and uses legacy codepages. In this specific case it's a "ø" (and my Lazarus on Windows displays it correctly).
Right, that does put a different slant on it. I did not realise we could have an OS that does not, itself, support UTF8.  But thinking about it, there must be lots.  :(



Quote
By the way: please fix the "Debain" in the title :P

Damm, that was a deliberate mistake I was going to refer to in the fist post and forgot. Sigh ....

It does sound like far more than I would have expected are un-fixable. Maybe I need to take a bigger sample. Its possible within the debian system to add exceptions that suppress particular lintian warnings (but practicable?). Further research is indicated.

I would like to fix at least some of them, just to prove we are trying to be helpful.

Davo
Title: Re: A (Debian) Quality Proposal
Post by: dbannon on January 19, 2022, 02:49:28 am
Further on the OS that don't support UTF8 issue, it seems to me that it would be possible to keep such OS specific files to just ascii in at least some cases. That might, for example mean replacing a ISO-8859 copyright symbol with (c) ?

Would that be an acceptable solution perhaps ?

Again, obviously, on a case by case basis. Where a symbol appears in compilable code would be a different matter ....

Davo
Title: Re: A (Debain) Quality Proposal
Post by: PascalDragon on January 19, 2022, 09:51:07 am
Quote
errln('error setting exception n�'+hexstr(i,2));.
.....this is a file for go32v2. That platform has no knowledge about UTF-8 and uses legacy codepages. In this specific case it's a "ø" (and my Lazarus on Windows displays it correctly).
Right, that does put a different slant on it. I did not realise we could have an OS that does not, itself, support UTF8.  But thinking about it, there must be lots.  :(

Essentially all the legacy systems that FPC supports.

Quote
By the way: please fix the "Debain" in the title :P

Damm, that was a deliberate mistake I was going to refer to in the fist post and forgot. Sigh ....

Even if it is a deliberate mistake it makes the topic harder to find by only title.

Further on the OS that don't support UTF8 issue, it seems to me that it would be possible to keep such OS specific files to just ascii in at least some cases. That might, for example mean replacing a ISO-8859 copyright symbol with (c) ?

Would that be an acceptable solution perhaps ?

No. Changing encoding especially for files that are intended for legacy systems anyway just to keep Debian's linting happy is not a solution. Not to mention that e.g. box drawing characters used on DOS etc. are not ASCII either.

For example the case with the "nø" might not have a real use, cause it's simply error output, but changing the "ø" to e.g. "o" just for the linting is not a solution either (essentially it's a change for the sake of change).

If there are spelling mistakes or encoding errors in files that are supposed to be UTF-8 then that's a different topic.
Title: Re: A (Debian) Quality Proposal
Post by: dbannon on January 19, 2022, 11:21:36 am

Thanks PascalDragon, one of the really cool things about FPC is the breadth of platforms covered, I would not suggest anything that compromises that.

This is not just to pacify Debian, we are talking about errors in a lot of cases. Sometimes unreadable text. On my Linux system most of the characters Lintian objects to are shown as the blackbox with question mark. The 'file' command identifies the file as one of the 16 possible variations of ISO-8859, we cannot know for sure what the character in question really looks like even on a system that tries to map ISO-8859.

I fully understand your concern about converting a file from ISO-8859-xx to ascii but I would only do that if the affected characters were only in comments. I also found one case where accented characters in a comment were important to understanding the comment, again, don't touch.

Alternatively, converting to UTF8 will, perhaps preserve some information and there is already quite a lot (78) UTF8 files in there, so apparently the non UTF8 Operating Systems can cope, something I don't know for sure ??

I have now looked at about 30 items a lot closer. I reckon we can divide up the Lintian warnings up into three categories.

I. The easy ones. Spelling mistakes, hardening (thats for Debian to do),  using (c) for copyright symbol, and a small part of the National_Encoding ones, ones where the problem is in comments.

2. Ones that we can do nothing about so I/we need to provide a lintian override file. Easy.

3. The hard ones. Might require a policy decision from the developers. For example, where a contributor's name contains "accented characters" or a header is written in non-English. Two possible approaches -

3a.   A translation to UTF8 will probably make it readable by more of today's systems and not break compilability I THINK ??.

3b. Conversely, translating to ascii is safer but might just offend someone (quite reasonably). For example, ä becomes a. Wrong but safe.

Lets be clear, I am absolutely NOT suggesting changing anything in a file that has its issue in compilable code or in a comment where eg the accented character is critical to the sense of the comment. Looks ugly on some systems but play safe !

Sorry I am pushing so hard on this but i think 'we' can actually improve things here if we get it right.

Davo




Title: Re: A (Debian) Quality Proposal
Post by: MarkMLl on January 19, 2022, 12:38:18 pm
Sorry I am pushing so hard on this but i think 'we' can actually improve things here if we get it right.

I for one think your suggestions are reasonable. Comments- as I said earlier- can be a problem where particular character usage echoes the platform's conventions for library definitions etc.

Allowing that Pascal is an English-based programming language and that English has long been this project's working language, I see absolutely no reason why exotic characters should be tolerated in error messages and license texts: /except/ where they are required to render a contributor's name or affiliation correctly (which I consider to be a fundamental respect). I'd also refer to previous discussion where non-ASCII character sets such as EBCDIC were considered, where the core developers made it quite clear that even if this were supported by a cross-compiler FPC itself was- and would remain- ASCII based: there is quite simply no justification for the continued use of e.g. nø in error messages unless these are fully-localised.

MarkMLl
Title: Re: A (Debian) quality proposal
Post by: Kays on January 21, 2022, 12:30:58 am
[…] So, my question, would the FPC developers accepts fixes to some of these issues ? […]
No, there’s a certain moment of inertia: “If it ain’t broke don’t fix it.” I once suggested to replace German-language identifiers (https://gitlab.com/freepascal.org/fpc/source/-/issues/37579) but there was no favorable majority. If the Lintian hints aren’t part of a bug report (i. e. something does not work), they don’t get fixed. Certainly developers can consider the hints, but ultimately the project’s goal is to write a working compiler. Resolving those linting hints is, in a manner of speaking, not within the project scope.
Title: Re: A (Debian) quality proposal
Post by: MarkMLl on January 21, 2022, 09:38:41 am
If the Lintian hints aren’t part of a bug report (i. e. something does not work), they don’t get fixed.

In which case it would be reasonable for Debian to eventually exclude FPC as non-compliant with their standards and leave it to derivatives to bundle it if they wanted.

Which in practice might not be a bad thing, since the version in Debian-stable is invariably a couple of stitches behind the "bleeding edge" that the community is happy supporting.

MarkMLl
Title: Re: A (Debian) Quality Proposal
Post by: dbannon on January 21, 2022, 10:47:43 am
I dare to differ.

If we don't have a fpc in Debian (and elsewhere) then we won't have applications that need FPC to build. And if people looking for a build environment see its not welcome in the distros they will decide its not a good choice.

Davo
Title: Re: A (Debian) Quality Proposal
Post by: Thaddy on January 21, 2022, 10:49:54 am
Note for Debian there is always backports.
Note that the packagers are free to report bugs against FPC on the freepascal bugtracker if they see any. In fact I expect them to do so.
Note that the Debian eco-system contains a great many packages that would be removed because they do not fully adhere to the Debian standards.... A great many...including some that are considered essentals.
Title: Re: A (Debian) Quality Proposal
Post by: dbannon on January 21, 2022, 11:09:09 am
Note for Debian there is always backports.
Yes, and the relevance is ?

Quote
Note that the packagers are free to report bugs against FPC on the freepascal bugtracker if they see any.
Indeed, and thats why I asked the iniial question, is there any point ?

Quote
Note that the Debian eco-system contains a great many packages that would be removed because they do not fully adhere to the Debian standards.... A great many...including some that are considered essentals.
Are you arguing that someone else did something wrong, therefor I should be allowed to too ? Its a widely used argument but one I would have thought beneath you Thaddy.

Again, I say it, this is not a Debian specific issue, Debian is just the messenger, please listen to the message !

Davo
Title: Re: A (Debian) Quality Proposal
Post by: af0815 on January 21, 2022, 11:27:51 am
Quote
A file is not valid UTF-8.
Debian has used UTF-8 for many years. Support for national encodings is being phased out. This file probably appears to users in mangled characters (also called mojibake).  Packaging control files must be encoded in valid UTF-8.
Sorry for my question, but i see here the sentence 'Packaging control files must be encoded in valid UTF-8' this is for me understandable, that this type of files must be UTF-8, but are internal files in the packages ( eg. the sources of FPC/Lazarus) also infected of this rule.
Title: Re: A (Debian) Quality Proposal
Post by: MarkMLl on January 21, 2022, 11:49:36 am
If we don't have a fpc in Debian (and elsewhere) then we won't have applications that need FPC to build. And if people looking for a build environment see its not welcome in the distros they will decide its not a good choice.

But Debian's significance is far more as a "meta-distro" upon which things like Ubuntu and Raspbian are based than something intended for direct consumption.

Focusing on interaction with the derivative distreaux might actually be better from FPC/Lazarus's POV, since it would cut out the plodding politicisation and philosophising which characterise Debian.

MarkMLl
Title: Re: A (Debian) Quality Proposal
Post by: dbannon on January 21, 2022, 11:57:50 am
Hmm, who do you think packages FPC for Ubuntu and Raspbian ?

Ubuntu do create some packages of their own but the vast majority come from Debian, indeed, Debians charts on where a package exists have a column for Ubuntu.

Raspbian is pure Debian, take a Debian source package and recompile.

But, I say it again, I am not suggesting we need to fix these things because of Debian, we need to fix them because they are faults. When someone gets an error message from FPC containing a spelling mistake, they s! just a bit. When they look at the source file and see unknown characters, they raise their eyebrows just a bit. It all adds up ....

Davo
Title: Re: A (Debian) Quality Proposal
Post by: marcov on January 21, 2022, 12:01:00 pm
I've committed a fix to trunk for the first three errors in the demoes (flist* and cgibmp)

These:
Quote
W hardening-no-pie [usr/bin/chmcmd-3.2.2]

W hardening-no-pie [usr/bin/fpdoc-3.2.2]

W hardening-no-pie [usr/bin/fppkg-3.2.2]

W hardening-no-pie [usr/bin/rstconv-3.2.2]

I hardening-no-fortify-functions [usr/bin/chmcmd-3.2.2]

I hardening-no-fortify-functions [usr/bin/fpdoc-3.2.2]

I hardening-no-fortify-functions [usr/bin/rstconv-3.2.2]

probably need to be added to the ignore list. They are all based on the assumption you use GCC.
Title: Re: A (Debian) Quality Proposal
Post by: MarkMLl on January 21, 2022, 12:04:16 pm
Hmm, who do you think packages FPC for Ubuntu and Raspbian ?

Debian, with a systemic delay of months or years.

Quote
Raspbian is pure Debian, take a Debian source package and recompile.

Mostly. I ran Debian on RPis for an extended period before switching to Raspbian (or whatever it's called these days) and there /are/ differences.

Quote
But, I say it again, I am not suggesting we need to fix these things because of Debian, we need to fix them because they are faults. When someone gets an error message from FPC containing a spelling mistake, they s! just a bit. When they look at the source file and see unknown characters, they raise their eyebrows just a bit. It all adds up ....

I agree. Or for that matter when somebody's "AI" mines Debian quality reports and includes things that are basically cosmetic in their reliability rating.

However, I did think that the alternative was worth highlighting.

MarkMLl
Title: Re: A (Debian) Quality Proposal
Post by: marcov on January 21, 2022, 12:08:41 pm
Hmm, who do you think packages FPC for Ubuntu and Raspbian ?

Peter Green aka plugwash, one of the Raspbian starters, was a decade long FPC user and contributor. One of the reasons that at least in the early days Raspbian had a much more up to date FPC than Debian :-)

Quote
W source-contains-prebuilt-windows-binary [fpcsrc/packages/libndsfpc/examples/graphics/3D/Paletted_Cube/data/texture1_RGB16_pal.bin]

Probably a false positive in the "file" command. The file is very short, and the COM format is very minimal, so that detection is not 100%. Another case for lintian ignores.
Title: Re: A (Debian) Quality Proposal
Post by: Fred vS on January 21, 2022, 12:10:52 pm
Hmm, who do you think packages FPC for Ubuntu and Raspbian ?

Ubuntu do create some packages of their own but the vast majority come from Debian, indeed, Debians charts on where a package exists have a column for Ubuntu.

Raspbian is pure Debian, take a Debian source package and recompile.

But, I say it again, I am not suggesting we need to fix these things because of Debian, we need to fix them because they are faults. When someone gets an error message from FPC containing a spelling mistake, they s! just a bit. When they look at the source file and see unknown characters, they raise their eyebrows just a bit. It all adds up ....

Davo

Hello Davo.

Many thanks for your post and I dont understand why some people spit on the messenger.

The Debian report is highly precious and fpc team should be grateful for it rather than trying to find fleas.

Sure that many things could be fixed and it should be a priority, humility is not weakness .

And sorry, but a fpc Debian package is a must for the existence of fpc.

Voila, c'est dit, I let you in peace.

Fre;D

PS: I would be very happy if I could find somebody or something that will check my programs like Debian does for the canditate packages.
Title: Re: A (Debian) Quality Proposal
Post by: marcov on January 21, 2022, 12:33:45 pm
I also fixed the superfluous file commands that relate to the compiler and IDE dir. The only remaining is

Quote
W superfluous-file-pattern fpcsrc/compiler/compinnr.inc [debian/copyright:125]

Since I didn't know what that file was about.

Note that the IDE dir change is first quarter 2015 or so. Sad to see that the debian port files are not updated for that.

All very long line warnings: why does Debian detect lines in binary files? Is this due we don't somehow set an binary attribute on them, or is this impossible, and should we simply add them all to the ignores?

Similarly,

Quote
I spelling-error-in-binary ACount Account [usr/lib/x86_64-linux-gnu/fpc/3.2.2/units/x86_64-linux/fpindexer/fpindexer.o]

is a slightly different case of the same. It probably finds a parameter "A Count" as per Delphi parameter style guide, and thinks this is a text and should be account.

It shouldn't be scanning binary files for spelling, but how to remedy this?
Title: Re: A (Debian) Quality Proposal
Post by: marcov on January 21, 2022, 12:55:32 pm
I also looked at the declartory and orignal cases, and most seem to have already been fixed in trunk. I fixed one more orignal case (which was atypical and might have been missed by some script).

The one remaining declartory is in an MPL license text from an external package. I'm not sure if we are legally allowed to fix that (and thus mutate the license)
Title: Re: A (Debian) Quality Proposal
Post by: dbannon on January 21, 2022, 02:20:20 pm
Marcov, you are a gentleman and a scholar !  I thank you for the positive action.

I also looked at the declartory and orignal cases, and most seem to have already been fixed in trunk.
Yeah, some were fixed a few months ago because they showed up in my binary.

Quote
The one remaining declartory is in an MPL license text from an external package. I'm not sure if we are legally allowed to fix that (and thus mutate the license)
Oh, thats an interesting one.

Quote
    W superfluous-file-pattern fpcsrc/compiler/compinnr.inc [debian/copyright:125]
Since I didn't know what that file was about.
I suspect that is in the Debian license file they create during packaging. Its been fixed at their end anyway.

From whats been going on here, I have to assume the National Code Page v UTF8 issue is a "will not fix" but I suppose its reasonable for us to say that the ISO files are pretty stable, unlikely to be edited or have more added ?

Quote
Note that the IDE dir change is first quarter 2015 or so....
I believe that they are working with FPC3.2.2 so that would seem strange indeed.

Quote
All very long line warnings: why does Debian detect lines in binary files?
I have no idea. That's a new list that only appeared a few days ago, after this thread started. IHMO it makes no sense.

Quote
I spelling-error-in-binary ACount Account..... It shouldn't be scanning binary files for spelling,
Yes, I had a smile about that one too. Fact is they do scan binary files and most of the hits they get are valid. But its not fool proof   ;D

Quote
Peter Green aka plugwash,.... Raspbian
I stand corrected.

I think I know the limits now of what its worthwhile worrying the FPC devs about. If more fixable ones are found, how would you prefer them reported ?  Just a bug report or a pull request ?

Thanks again !

Davo







 
Title: Re: A (Debian) Quality Proposal
Post by: PascalDragon on January 21, 2022, 03:42:46 pm
I've committed a fix to trunk for the first three errors in the demoes (flist* and cgibmp)

These:
Quote
W hardening-no-pie [usr/bin/chmcmd-3.2.2]

W hardening-no-pie [usr/bin/fpdoc-3.2.2]

W hardening-no-pie [usr/bin/fppkg-3.2.2]

W hardening-no-pie [usr/bin/rstconv-3.2.2]

I hardening-no-fortify-functions [usr/bin/chmcmd-3.2.2]

I hardening-no-fortify-functions [usr/bin/fpdoc-3.2.2]

I hardening-no-fortify-functions [usr/bin/rstconv-3.2.2]

probably need to be added to the ignore list. They are all based on the assumption you use GCC.

This has been a topic on the bug tracker (https://gitlab.com/freepascal.org/fpc/source/-/issues/39451) as the same also results in problems on PPC64le. So these shouldn't be ignored, but will instead be fixed by themselves some time in the future once FPC generates correct static PIE binaries.
Title: Re: A (Debian) Quality Proposal
Post by: marcov on January 21, 2022, 03:49:43 pm
This has been a topic on the bug tracker (https://gitlab.com/freepascal.org/fpc/source/-/issues/39451) as the same also results in problems on PPC64le. So these shouldn't be ignored, but will instead be fixed by themselves some time in the future once FPC generates correct static PIE binaries.

The explanation of that feature seems to indicate that GCC implemented features (more dynamic allocation of the register used for PIE (GOT pointer?)) to combat the performance degradation of PIE. that was what I was hinting at.

CHMCMD and fpdoc are quite performance sensitive. (though afaik they should be linked to libc, as they can use threading?)

added: I now realize that ppcx64 is in the log, so it is a 64-bit build where it probably hurts less. So probably this is about optimizations we don't yet do.
Title: Re: A (Debian) Quality Proposal
Post by: PascalDragon on January 21, 2022, 04:02:42 pm
added: I now realize that ppcx64 is in the log, so it is a 64-bit build where it probably hurts less. So probably this is about optimizations we don't yet do.

It's not about optimizations, it's simply that you currently can't generate a working statically linked PIE binary with FPC. Thus FPC is build without PIE and thus hardening-no-pie is triggered by Lintian.
Title: Re: A (Debian) Quality Proposal
Post by: marcov on January 21, 2022, 04:15:21 pm
It's not about optimizations, it's simply that you currently can't generate a working statically linked PIE binary with FPC. Thus FPC is build without PIE and thus hardening-no-pie is triggered by Lintian.

Yeah, but my point was that  if enabling PIE when it has a significant performance degradation might be a good reason to leave it disabled and flag it is as not hardening compatible. But I mostly had 32-bit in mind, which is of course worse than 64-bit from a performance impact (10% and more IIRC)

I now think those comments referred to dynamically allocating a 64-bit register for PIE supporting pointers using liveliness analysis, rather than reserving one outright.
Title: Re: A (Debian) Quality Proposal
Post by: dbannon on January 22, 2022, 01:19:34 pm
I am still unsure how you want these things reported, bug report or but report and pull request. But I do know that they should not be reported here, in the forum. But thats what I am doing it until I know better.  :D

install/man/man5/fpc.cfg.5 contains developpers - spelling mistake

WRT the hardening issue, we can fix the immediate problem (if we wish) just by forcing the compiler to build a dynamically linked binary where it would other wise build a static one. But is it necessary ?  My thoughts are that if Debian need all binaries hardened then they can patch the build themselves ?

Hardening definitely make sense in a server or multiuser system, I am not so excited about it on a single use machine which probably makes up most of our compiler users. (Not end users, the compiler users.).

Apart from things in this particular post, I believe we have answers to original question, thanks !

Davo



Title: Re: A (Debian) Quality Proposal
Post by: marcov on January 22, 2022, 01:27:22 pm
I am still unsure how you want these things reported, bug report or but report and pull request. But I do know that they should not be reported here, in the forum. But thats what I am doing it until I know better.  :D

Separate report and patch per "kind" of issues. IOW you can bundle if they are e.g. spelling errors in similar files.

You can also discuss and post patches or remarks here. Since many are mostly one line fixes, there is not much use for formality.

Quote
install/man/man5/fpc.cfg.5 contains developpers - spelling mistake

Fixed.

Quote
WRT the hardening issue, we can fix the immediate problem (if we wish) just by forcing the compiler to build a dynamically linked binary where it would other wise build a static one. But is it necessary ?  My thoughts are that if Debian need all binaries hardened then they can patch the build themselves ?

Well, Pascaldragon wants to keep it visible, though IMHO an override (so that it is still visible but no longer counted) is better.

Quote
Hardening definitely make sense in a server or multiuser system, I am not so excited about it on a single use machine which probably makes up most of our compiler users. (Not end users, the compiler users.).

Well, that is one of the problems of Linux, everything is in constant flux, and what was ok yesterday will be not today depending on the whimsy of a developer. There are no hard contracts of any kind, and when broken, usually the easiest way (only support the new) is taken.

It is one of the things that puts me off Linux.

Title: Re: A (Debian) Quality Proposal
Post by: MarkMLl on January 22, 2022, 02:46:28 pm
Well, that is one of the problems of Linux, everything is in constant flux, and what was ok yesterday will be not today depending on the whimsy of a developer. There are no hard contracts of any kind, and when broken, usually the easiest way (only support the new) is taken.

Hence in part some of my rude comments about Debian yesterday: they rely on contributed packages which can change at the maintainer's whim, and once put in their repository they are most resistant to fixing stuff in the stable release... although some developers/maintainers are more successful than most.

MarkMLl
Title: Re: A (Debian) Quality Proposal
Post by: dbannon on January 23, 2022, 12:23:55 am
Hence in part some of my rude comments about Debian yesterday: they rely on contributed packages which can change at the maintainer's whim, and once put in their repository they are most resistant to fixing stuff in the stable release... although some developers/maintainers are more successful than most.

We want a stable, predictable platform that does not change but does get absolutely every new feature that comes along.   ;)

Seriously, I think they get it right most of the time. The problem with Lazarus in Bullseye is a bad example but there are very many good examples. Their model of new packages into Sid, then testing and then after an indomitable wait, stable, is really the best you can hope for. Lazarus comes out twice a year, at any one time Testing will have a near current version but sooner or later they freeze it all and call it stable, then, that Lazarus is in Stable for two years. Two years is a long time in Lazarus speak. Only security or extreme functionally problems after that.

Ubuntu's long term release is also two years, same version of Lazarus in the first week of a LTR as the last week.

Otherwise its a rolling release model, massive changes every day, sigh, thats too scary for me !

Davo



Title: Re: A (Debian) Quality Proposal
Post by: dbannon on January 23, 2022, 12:43:51 am
Separate report and patch per "kind" of issues. IOW you can bundle if they are e.g. spelling errors in similar files.
You can also discuss and post patches or remarks here. Since many are mostly one line fixes, there is not much use for formality.
OK, understand. I think we have now covered everything we can reasonably expect from this end.

Quote
install/man/man5/fpc.cfg.5 contains developpers - spelling mistake - Fixed.
Thank you very much.

Quote
> WRT the hardening issue, ....
Well, Pascaldragon wants to keep it visible, though IMHO an override (so that it is still visible but no longer counted) is better.
As a problem for FPC, Hardening is more of an issue on PPC64el IMHO, being a server platform hardening is important. On Intel few user apps will be statically linked and even if they are, they can be forced not to be. No such option with PPC64el.
For my app in Debian I have just turned Hardening off for PPC64el and they accepted that. But no one is likely to be using it on PPC64el anyway !

Davo
Title: Re: A (Debian) Quality Proposal
Post by: MarkMLl on January 23, 2022, 09:54:01 am
Seriously, I think they get it right most of the time. The problem with Lazarus in Bullseye is a bad example but there are very many good examples. Their model of new packages into Sid, then testing and then after an indomitable wait, stable, is really the best you can hope for.

In practice, that's not what appears to happen. Some developers- and I'd highlight Carsten Schoenert who among other things maintains Thunderbird and KiCAD- get a surprising number of changes committed to "stable"... changes with far more effects and far fewer benefits than the current Lazarus version issue.

This for Thunderbird:

2022  4
2021 41
2020 55
2019 28
2018 21
2017 22


And while a few are labelled as CVEs, most are just upstream changes which he's managing to get propagated immediately.

MarkMLl
Title: Re: A (Debian) Quality Proposal
Post by: PascalDragon on January 23, 2022, 02:02:40 pm
Quote
> WRT the hardening issue, ....
Well, Pascaldragon wants to keep it visible, though IMHO an override (so that it is still visible but no longer counted) is better.
As a problem for FPC, Hardening is more of an issue on PPC64el IMHO, being a server platform hardening is important. On Intel few user apps will be statically linked and even if they are, they can be forced not to be. No such option with PPC64el.
For my app in Debian I have just turned Hardening off for PPC64el and they accepted that. But no one is likely to be using it on PPC64el anyway !

x86_64 and Aarch64 are valid server platforms as well (especially the former) and right now static PIE binaries by FPC have problems on all three platforms.
Title: Re: A (Debian) Quality Proposal
Post by: dbannon on January 24, 2022, 01:08:48 am
...
On Intel few user apps will be statically linked and even if they are, they can be forced not to be. No such option with PPC64el.

x86_64 and Aarch64 are valid server platforms as well (especially the former) and right now static PIE binaries by FPC have problems on all three platforms.
[/quote]

You may have missed my point there PascalDragon, on x86_64 and Aarch64 it is possible to to force what would otherwise be a static binary to become a dynamic one and therefore can be hardened. On PPC64el that is not the case.

I'm not saying its not worthwhile fixing the whole issue, and every other issue for that matter. But in the real world, limited resources require us to make priority decisions. I am suggesting that maybe the PPC64el is more of a priority. But if you disagree, thats fine, I respect your decision.

A disclaimer, I do not own any PPC64el hardware, and while once a big user of it, I have not had access to any PPC64el hardware for several years. (and if any past employer wants to offer me such access, No Thanks). So I have no personal interest in PPC64el, my only interest is, as throughout this thread, the quality of FPC.

Davo

TinyPortal © 2005-2018