Exploring FPC/Lazarus units and packages

[silvercoder70]

I wanted to reply to this since the weekend ...

My thought it that units to look at would include

- classes
- sysutils
- units with string helpers and char helpers
- units containing TStreams
- units that contains basic file operations such as FileExists() etc.

This give you a good overview (?) of much of what you might encounter in everyday life. Not saying that you need to memorise the contents of those units. If nothing else, hopefully, you will also get to view what good (subjectively speaking) good code looks like. And if there are things that you don't recognise, it gives you an opportunity to find out what it means. More learning!

[marcov]

Note: I don't say you are wrong per se (except maybe the specific issue of fileexist), just trying to add some nuance.

However from the POV of somebody who's done a fair amount of technical writing, and also from having gone through (some of) the Python material thoroughly, I think we can highlight two things as major problems:

That's for sure. But Python is extremely beginner centric though, while most beginners of FPC came from Delphi, and one could argue that shows in the documentation. Only FPC 3.2 significantly goes beyond the  D6/D7 baseline, so that makes that old practical workaround (just get a Mastering Delphi from some 2nd bookshop) more painful. Also because Delphi beginner documentation is also often more incremental (describing what is new in each version) rather than a complete beginner introduction.

This will become stronger with FPC 3.2.4, as it comes with even newer Delphi units (from package vcl-compat), and then again with whatever the next major version will be.

a) "How do I..."

b) "What changed, and when..."

Some of (b) is in the release manifestos in the wiki, but those descriptions are not beginner level.

For (b), at some point on unix FileExists() changed to /not/ match directories... but one has to look through multiple release notes to find this rather than its being documented with the function description.

The Fileexists/DirectoryExists distinction is age old and documented. The issue is due toexploitation of target specific bugs, and was never valid documented behaviour to begin with.  So I don't agree with that being a defining issue. Even though it might be in the release manifests, it is next level detail.  I suggest to avoid such trial and error programming.

But at the moment, FPC- or more specifically the FPC RTL/FCL- is in precisely the same position as the oft-criticised C++ STL: programming is based on the assimilation of a vast tract describing the standard libraries, with very little help from decent indexing etc.

I'm not sure that is the problem. Yes, grouping all string related routines in some index would be good, but IMHO this is not a dimensional problem of the documentation.
 
The actual problem is simply the reference docs being quite terse, and missing of all encompassing beginner documentation. That is a known issue.

[MarkMLl]

Note: I don't say you are wrong per se (except maybe the specific issue of fileexist), just trying to add some nuance.

However from the POV of somebody who's done a fair amount of technical writing, and also from having gone through (some of) the Python material thoroughly, I think we can highlight two things as major problems:

That's for sure. But Python is extremely beginner centric though, while most beginners of FPC came from Delphi, and one could argue that shows in the documentation.

Actually, I was thinking of the detailed function descriptions to which one refers when embedding Python in some other language (such as, in my case, Object Pascal).

Hence, it applies to both stuff targeted at novices, and at those with substantial experience: which I think equates to being better across the board.

The actual problem is simply the reference docs being quite terse, and missing of all encompassing beginner documentation. That is a known issue.

Disagree. The actual problem is that the RTL/FCL indexing is crap.

I've been trying to get stuff done which required good weather over the last couple of days, but that is expected to change over the next few minutes. I'll see if I can find the stuff I did before, since if nothing else it might be a useful focus for discussion.

And as I said: I believe (from memory) that I was doing things "the wrong way" i.e. by post-processing finished documentation. I would expect that doing things "the right way" would result in something at least as good.

MarkMLl

[marcov]

Actually, I was thinking of the detailed function descriptions to which one refers when embedding Python in some other language (such as, in my case, Object Pascal).

That's mostly 3rd party, so outside of the project documentation scope.

Disagree. The actual problem is that the RTL/FCL indexing is crap.

Good. I'll wait for the examples then. Please try to use the CHMs
 

[MarkMLl]

I'm pretty sure I don't have the stuff I was looking for any more, so all I can do is say where (from memory) I was able to get to. After all, I'm the one complaining and somewhat brashly...

Basically, I had the full set of FPC versions from the earliest (published) v2, and a substantial number of (PDF?) RTL etc. manuals that accompanied them. Working from those I was able to generate ASCII files in a consistent format (ps2ascii?) and extract a brief summary: I vaguely recall that I had to look at a couple of possible fields.

From the summary lines, I generated a permuted index (ptx?) such that if I started of with say

Code: [Select]

Eof Check for end of file
Filepos Position in file
Seek Set file position
SeekEof Set file position to end of file

I got an old-school UNIX-style list

Code: [Select]

Eof:                                 | Check for end of file
FilePos:                             | Position in file
Seek:                                | Set file position
SeekEof:                             | Set file position to end of file
Eof:                       Check for | end of file
SeekEof:        Set file position to | end of file
Eof:                Check for end of | file
FilePos:                 Position in | file
Seek:                            Set | file position
SeekEof                          Set | file position to end of file
SeekEof: Set file position to end of | file
Seek:                       Set file | position
SeekEof:                    Set file | position to end of file

This does obviously need a lot of "stop words" ** : I've had to tweak that a lot manually in lieu of what I did years ago. But if you focus on the index position marked by | you will see that it groups all file operations together, all position operations together and so on.

I might have stopped with a textfile, or I might have generated an output PDF... I really don't remember.

I went on from there trying to generate a "this was added in version..." document, but that might have been when I got bogged down in file formats and the requirement of matching files and utilities... again, this was some years ago and I don't remember.

Disagree. The actual problem is that the RTL/FCL indexing is crap.

Specific points: A large number of fields, constants etc. are highly relevant but not indexed. This also applies in spades to helpers: a few days ago somebody was insisting that /of/ /course/ you'd use the string .Contains helper for something BUT unless you already know that TStringHelper is in SysUtils it's quite simply not indexed.

Those should be fixable.

More difficult, is the number of places that a function has a parameter that is passed as an integer (i.e. unindexable), when it should be an indexable enumeration. For example, fpRecv() at https://www.freepascal.org/docs-html/current/rtl/sockets/fprecv.html has a flags field described as an integer when it should be a set, or at the very least a type that can be indexed with links to its constituent values. There's better examples where only a small number of values are acceptable but the field is still described as an integer, but I feel I've made my point.

I accept that that probably can't be fixed by indexing without a substantial amount of work in the RTL/FCL sources, either to change the types or to add annotations.

My apologies to all for the rant, but I really feel it needed to be said.

MarkMLl

Updated with footnote:

** This is actually an area where an "AI" trained in the computing domain if not the specific language (Pascal in our case) could be useful: recognising that "End of file" is a specific term of art and indexing appropriately.

[MarkMLl]

Actually, I was thinking of the detailed function descriptions to which one refers when embedding Python in some other language (such as, in my case, Object Pascal).

That's mostly 3rd party, so outside of the project documentation scope.

Rubbish. I'm referring to what the python developers have written for their peers' use, in what way is that 3rd party?

MarkMLl

[Curt Carpenter]

...

Specific points: A large number of fields, constants etc. are highly relevant but not indexed. This also applies in spades to helpers: a few days ago somebody was insisting that /of/ /course/ you'd use the string .Contains helper for something BUT unless you already know that TStringHelper is in SysUtils it's quite simply not indexed.

...

I think that captures the essence of the problem from my point of view -- and boy would the sort of index MarkMLI shows here be helpful to someone like me (especially if it included links to source).  I learned about helpers by watching the forum here and asking some questions about using them.  Which is a shame because they add a lot, IMHO, to readability if not functionality.

The "helpers" experience just makes me wonder how many other really useful capabilities there are in the fpc/lazarus system I'm just not aware of.

For all that -- creating the sort of index MarkMLI shows here would obviously require a lot of real and skilled work. Would a go fund me sort of thing attract any interest?

[MarkMLl]

For all that -- creating the sort of index MarkMLI shows here would obviously require a lot of real and skilled work. Would a go fund me sort of thing attract any interest?

To be brutally frank, I don't think there's enough remaining interest in the language to make it viable. Of course, there's people who devote their lives to writing tomes on dead languages: most of whom have tenured positions in an Antiquities department at Oxford or Cambridge.

I'm pretty sure that I used Perl for the utilities which is fairly well suited for this sort of thing: I might be wrong but I don't think I wrote any Pascal for it. /If/ somebody could abstract a one-liner from each function etc. in the RTL I might /possibly/ be persuaded to see how far I could get knocking together something as a foundation but...

Code: [Select]

TStringHelper.Contains: Check is the string contains another
Pos: Search for substring in a string.
ContainsStr: Check whether one text contains another (case sensitive)

even there there are problems with terminology ("contains" vs "in", "text" vs "string") and I don't think there's enough "see also" information to build anything comprehensive.

Add to that that I've got my own workload, and my own commercial opportunities that I'm trying to exploit.

MarkMLl

[Curt Carpenter]

...
To be brutally frank, I don't think there's enough remaining interest in the language to make it viable. Of course, there's people who devote their lives to writing tomes on dead languages: most of whom have tenured positions in an Antiquities department at Oxford or Cambridge.
...

Too bad, but as a Japanese friend reminded me once, nothing is forever.

[marcov]

I went on from there trying to generate a "this was added in version..." document, but that might have been when I got bogged down in file formats and the requirement of matching files and utilities... again, this was some years ago and I don't remember.

(it does assume everything was documented in the version it first appeared).

Specific points: A large number of fields, constants etc. are highly relevant but not indexed.

One of the problems is that the target that the document is generated (and the source is parsed with) influences the docs. So no careful annotation "this is only for windows" or so. If it is not visible on Linux, then it is not there. Period.

This also applies in spades to helpers: a few days ago somebody was insisting that /of/ /course/ you'd use the string .Contains helper for something BUT unless you already know that TStringHelper is in SysUtils it's quite simply not indexed.

But is that reference guide indexing, or lack of introductionary text? I think the latter.

[Mario]

I think a lot could be won if there was a wiki page with these 'essential units', as well as a short introduction to the problems with the documentation as discussed here, because that's very helpful in devising further (re)search strategies.

What could also help is to find a mechanism by which examples, comments and warnings can make if from this forum into the reference documentation. The entry for TByteArray (which started this discussion) could, for example, come with a warning that this particular type is not meant to be used directly, and that there is a duplicate in a different unit. This type of manual annotation could help mitigate the problems little by little and then suddenly.

I find this forum and the community great BTW.

[MarkMLl]

I went on from there trying to generate a "this was added in version..." document, but that might have been when I got bogged down in file formats and the requirement of matching files and utilities... again, this was some years ago and I don't remember.

(it does assume everything was documented in the version it first appeared).

Accepted, but while I don't remember the detail this was something along the lines of having to use FPC 2.4 utilities to process FPC 2.4 documentation sources, and FPC 2.6 utilities to process FPC 2.6 documentation sources.

Specific points: A large number of fields, constants etc. are highly relevant but not indexed.

One of the problems is that the target that the document is generated (and the source is parsed with) influences the docs. So no careful annotation "this is only for windows" or so. If it is not visible on Linux, then it is not there. Period.

Indeed. But things like the Berkeley Sockets API are cross-platform, which is why I used that as an example.

This also applies in spades to helpers: a few days ago somebody was insisting that /of/ /course/ you'd use the string .Contains helper for something BUT unless you already know that TStringHelper is in SysUtils it's quite simply not indexed.

But is that reference guide indexing, or lack of introductionary text? I think the latter.

/Definitely/ not. https://www.freepascal.org/docs.html hence the RTL index at https://www.freepascal.org/docs-html/current/rtl/index.html and its index at https://www.freepascal.org/docs-html/current/rtl/index-8.html: where is .Contains from TStringHelper?

I could give plenty more examples, but basically that is utterly useless for a newcomer to Pascal or for somebody whose experience is "classical" TP but has some old code he'd like to revisit. To be greeted by something like

There I believe you are mistaken. If you spend time looking for it then it is there, see here. I give you that you need to know where to look.

is profoundly unhelpful: at present the only people who can find stuff in the documentation are "old hands" who've watched the compiler and RTL/FCL mature over ten years or so.

MarkMLl

[TRon]

There I believe you are mistaken. If you spend time looking for it then it is there, see here. I give you that you need to know where to look.

is profoundly unhelpful: at present the only people who can find stuff in the documentation are "old hands" who've watched the compiler and RTL/FCL mature over ten years or so.

Ofc. but not reading documentation et all is just as profoundly unhelpful (not pointing to someone in particular).

Even thought it is reference material I do read the documentation from A to Z. Yes, also profoundly unhelpful but the answers to about 80% of the questions asked on the forums can be found there by simply exploring the documentation of an entire unit (or units).

I am more upset (upset, not blame) by the fact that many methods that are listed do not have a description whatsoever so that the only way to know for sure what something actually does is to either guesstimate, test, read the source-code of the implementation, stumble upon an example (wherever it is located) or otherwise (all wasting time though you do get additional respect for those that implemented everything and those who wrote the examples and documentation that does exist). This especially applies to the compiler internals for which no documentation seem to exist at all.

For every release I read the wiki-pages of the changes and I also keep an eye on the Mailinglist regarding announcements of new features and in case possible try them and play with it in order to familiarize them. Although tedious you can pick that up from any release. If I see something new (for me) on the forums or ML then I try to get myself acquainted. This is especially true for weird bugs and situations that some people seem to be running into.

Or in short: I personally do not make a big deal out of the current status of the documentation. And yes on occasion I also completely miss/overlooked a certain feature as well and sometimes make stupidly wrong assumptions.

I truly wish I had a good answer or plan to improve things but as stated earlier I have no idea how things could be improved in a feasible way. In that regards I understand your mentioned issue about the indices.

[dsiders]

/Definitely/ not. https://www.freepascal.org/docs.html hence the RTL index at https://www.freepascal.org/docs-html/current/rtl/index.html and its index at https://www.freepascal.org/docs-html/current/rtl/index-8.html: where is .Contains from TStringHelper?

https://www.freepascal.org/daily/doc/rtl/sysutils/tstringhelper.html

So there is docs out there, when you get a release that actually includes them.

[TRon]

@dsiders: no need for daily docs, see https://www.freepascal.org/docs-html/rtl/sysutils/typehelpers.html

But you are unable to get there if you do not read the main index page of unit sysutils, do a websearch for freepascal + typehelpers and/or are unaware of the wiki entries and ML announcement.