Exploring FPC/Lazarus units and packages

[Curt Carpenter]

(Excerpted from an earlier thread)

I wasn't aware of the "types" unit discussed in a recent thread (https://forum.lazarus.freepascal.org/index.php/topic,67876.0.html), and spent some time with the source (/usr/share/fpcsrc/3.2.2/rtl/objpas/types.pp on my system) this morning.  It raised a general question for me:  can anyone provide a list of the six or seven units that even a casual programmer like me should study (docs and source), at least to be aware of the capabilities they offer? 

I know that I am probably using only a small percentage of the capabilities available in the many units and packages in the fpc/lazarus system, mostly because I'm just not aware of them.  Some suggestions of where to start exploring these would be welcome.  Otherwise, the only way I've found to discover new (to me) units containing useful functionality is to follow the forum here, as in this case.  Trying to explore the fpc source, which I do from time to time, is  just too overwhelming for me, and the docs are great as long as you know what you're looking for, but problematic without a starting point.

(Hope that makes sense...)

[MarkMLl]

(Moved response from earlier thread.)

I feel that would merit a fresh question rather than being piggy-backed onto this one.

The problem is that the organisation of the source is primarily /functional/ rather than being focused on what an end user needs to know. As an example based on the current case, I don't recall ever importing the Types unit myself: everything that's needed is normally brought in implicitly from System.

Add to that that much of what the local gurus think is essential to e.g. strings handling is no longer in System or even Strutils, but is instead embedded in Sysutils as a type helper.

Add to that that the official documentation is very much reference material, rather than coherent user guides. And that indexing of the RTL/FCL material is incomplete (there's no "how do I do this" cross-reference) and gives no indication of when something was introduced or changed.

You need to be aware of the FPC documentation based at https://www.freepascal.org/docs.html, and the FPC+Lazarus documentation at https://lazarus-ccr.sourceforge.io/docs/ which might differ depending on the precise compiler etc. versions being considered. After that you need to consider what sort of things you're doing, and then use the index to track down the appropriate units... with the expectation that the code you really want might actually be in an include file.

In my case if I had to name a handful: System, Sysutils, Strutils, Sockets, BaseUnix, Linux, Serial. Your mileage, as they say, may vary.

MarkMLl

[MarkMLl]

(Maybe something "AI" would help?  Way beyond my abilities, but an approach perhaps?

I've tried to tackle it to some little extent in the past, but was working from "outside in" starting with PDF files rather than hooking into the document preparation programs... I still don't know where document preparation is documented.

What I was trying to get to was a "permuted index", so that if you wanted to know about e.g. file operations you could find everything relevant in a single area. Of course, a few years later you'd simply use a search engine, while these days you'd undoubtedly use "AI" which IMO is by no means the panacea it's cracked out to be (if I'm looking for information on seeking backwards in a file, the last thing I want is a video on somebody abusing metalworking tools).

I was also trying to compare and annotate different versions of the documentation but IIRC fell foul of needing different versions of various utilities to handle each FPC documentation version. At that point something or other more urgent intervened.

The fact that I didn't raise my head too far about the parapet is possibly fortunate, since I have already been accused in a legal hearing of being too interested in obsolete languages and hardware (it didn't stick).

MarkMLl

[cdbc]

Hi
It does make sense and good on you, for making this thread. 
I've sometimes found myself in that 'How do I solve this problem' predicament and found the docs etc. a bit unsurmountable, so yes it would be nice to have xreferenced tasks/areas/solutions, at hand.
Regards Benny

[MarkMLl]

OK, here's where I'm at.

Before anything else, I don't want anybody to think that I'm attempting to disparage MvC's excellent work with the reference manuals. I've done a fair amount of technical writing, and know how difficult it can be.

However from the POV of somebody who's done a fair amount of technical writing, and also from having gone through (some of) the Python material thoroughly, I think we can highlight two things as major problems:

a) "How do I..."

b) "What changed, and when..."

For (a), I'd suggest as an example that a few weeks ago there was somebody in the forum that was trying to invoke an OS-level command (file rename?) and getting into enormous problems with name quoting, when in fact the RTL had something which did the job. Or alternatively, finding documentation on String.Contains() is impossible unless one knows that SysUtils contains TStringHelper.

For (b), at some point on unix FileExists() changed to /not/ match directories... but one has to look through multiple release notes to find this rather than its being documented with the function description.

I'm not, by any means, saying that what I did fixed this. Neither am I saying that I did it the right way, or for that matter that I stand any chance of finding an example of my output.

But at the moment, FPC- or more specifically the FPC RTL/FCL- is in precisely the same position as the oft-criticised C++ STL: programming is based on the assimilation of a vast tract describing the standard libraries, with very little help from decent indexing etc.

Edited: decapitalise "unix". Also: I'll try to find anything I've managed to do on this, but don't make any promises.

MarkMLl

[Curt Carpenter]

OK, here's where I'm at.

Before anything else, I don't want anybody to think that I'm attempting to disparage MvC's excellent work with the reference manuals. I've done a fair amount of technical writing, and know how difficult it can be.

However from the POV of somebody who's done a fair amount of technical writing, and also from having gone through (some of) the Python material thoroughly, I think we can highlight two things as major problems:

a) "How do I..."
10233 in /fpcsrc/packages
b) "What changed, and when..."

Oh absolutely yes.  Some of the most painful times of my career were attempts at technical writing.  It's an exceptional and rare talent.  And one of the wonders of Turbo Pascal and Delphi in the early days was that they clearly had some great tech writers working on them.

One thing I think would be helpful would be a way to just create a specialized list of the available files applicable to a given operating system.  There are 2008 items in my  /fpcsrc/rtl folder and 10233 in /fpcsrc/packages.  Most of them are of no help to someone that is trying to learn fpc/lazarus just for a particular OS.  If there was a way to say "list all of the files and folders in fpcsrc that are applicable to WidnowsCE" for example, it would at least narrow the path up the learning curve.  But to do that, someone would have to go through all those items I think -- real work.

[TRon]

Just some random remarks.


Most if not all of the choices for placing certain type declarations in a particular unit is hinted by Delphi. As the recent practical example the declaration of tbytedynarray (Delphi's system.types)

As already mentioned by MarkMLI the existing documentation is mostly written to be reference material even though some documentation comes with examples to try make things more clear. imho Borland did a much better job when they published their compiler and IDE as their documentation was exhaustive and explained about every (new) feature into details and with accompanied practical examples and explanations to show the ropes.

You can definitely see the difference between a commercial entity and an open source project (which as Mark already stipulated is in no way meant as a comment to MvC and/or other contributors).

Reference documentation simply isn't and never will be exhaustive, full of examples and/or try guide new users to get themselves acquainted. imho its goal is to inform the user about the workings/constructs and that's more or less it. If you want more in depth material you can either opt for 3th party documentation (books comes to mind) or get yourself familiar with the existing examples and the wiki. Having a search on the forums is also not a bad idea in case you get stuck.

That is all far from optimal but do-able.

If you ask what are the main units you would have to get yourself acquainted with then I would answer that with that it depends on your goal(s). For generic use and newcomers: system, sysutils and in the past also unit dos and graphics.

MarkMLI also mentioned the *nix related units and that is of course applicable in case that is your target. That list could be completely different when writing code for another target. And from there on it pretty much depends on the specific topic.

Because also then the real questions begin to pop-up. Do you want to use OOP ? Generics ? Operators ? Helpers ? Do you stick to the default shipped units or prefer 3th party units. Do you want to write GUI applications ?


Till this day (and for over 2 decades) I still haven't read all available reference material shipped with FPC/Laz. When there is need to use a certain class or component that I am unfamiliar with then I start reading, try out examples examine the wiki, forum and try to obtain all relevant information. Sometimes I can get away with just looking at an example other times I am stunned at the complexity and usage of some implementations such as f.e. charts and spreadsheets simply because they offer so many functionality that I barely scratch the surface when in using them.

I agree that things could be much better and afaik the wiki was meant to supplement the reference documentation but quite frankly (and despite some serious efforts) to me it all looks like an enourmous pile of snippets linked together (and yes there are definitely some very good and well written wiki pages but they do not represent the majority).

I believe the discussion has been brought up multiple times in the past that there seem to be something missing between reference documentation and wiki articles and quite frankly I do not seem capable to put my finger on what exactly the missing part is or could be (trees, forest and clueless).


edit:
My take on using AI  to let yourself guide.

Short answer: please don't.

Long answer: AI is
- full of presenting buggy code and explanations (it basically lies straight to your face and when confronting the lie then lies about lying as well).
- based on generic consensus e.g. what the majority deems fit (which you might think is a good idea but is actually bad)
- If you do not already know what you are doing (e.g. ask the exact correct question and basically know the correct answer) you can be deceived.
- AI will eventually lead to (even more) enshittification.
- for any other topic in your life f.e. buy a car or sell your house would you accept a roll of the dice ? (because in basics that is exactly what AI does).
And probably tons of other reasons that do not directly pop into mind.

[MarkMLl]

Oh absolutely yes.  Some of the most painful times of my career were attempts at technical writing.  It's an exceptional and rare talent.  And one of the wonders of Turbo Pascal and Delphi in the early days was that they clearly had some great tech writers working on them.

It's difficult enough when one is working with developers who have to- as a matter of company policy- help. Once subcontractors and loosely-affiliated associates are involved it's virtually impossible to have anything coherent, since each is likely to have coding and markup policies incompatible with anybody else.

The early Borland manuals were gems, as were the JPI/Topspeed manuals... I don't know whether they inherited the writers when the companies split. However I remember something about Borland loosing a large amount of documentation on the Delphi-2 era, which is why later versions were more and more skimpy.

One thing I think would be helpful would be a way to just create a specialized list of the available files applicable to a given operating system.  There are 2008 items in my  /fpcsrc/rtl folder and 10233 in /fpcsrc/packages.  Most of them are of no help to someone that is trying to learn fpc/lazarus just for a particular OS.  If there was a way to say "list all of the files and folders in fpcsrc that are applicable to WidnowsCE" for example, it would at least narrow the path up the learning curve.  But to do that, someone would have to go through all those items I think -- real work.

If you build your own compiler you might find https://github.com/MarkMLl/fpclaz_build_scripts useful. It uses the compiler's -vt flag hence

Code: [Select]

= /usr/local/src.fpc/fpcbuild-3.2.2/fpcsrc/system.pp
+  /lib64/ld-linux-x86-64.so.2
+   /usr/local/src.fpc/fpcbuild-3.2.2/fpcsrc/osdefs.inc
+     /usr/local/src.fpc/fpcbuild-3.2.2/fpcsrc/sysosh.inc
+      /usr/local/src.fpc/fpcbuild-3.2.2/fpcsrc/pmutext.inc
+    /usr/local/src.fpc/fpcbuild-3.2.2/fpcsrc/rtldefs.inc
+   /usr/local/src.fpc/fpcbuild-3.2.2/fpcsrc/sysos.inc
+    /usr/local/src.fpc/fpcbuild-3.2.2/fpcsrc/errno.inc
+    /usr/local/src.fpc/fpcbuild-3.2.2/fpcsrc/ostypes.inc
+     /usr/local/src.fpc/fpcbuild-3.2.2/fpcsrc/ptypes.inc
+      /usr/local/src.fpc/fpcbuild-3.2.2/fpcsrc/pmutext.inc
+    /usr/local/src.fpc/fpcbuild-3.2.2/fpcsrc/./x86_64/stat.inc
+    /usr/local/src.fpc/fpcbuild-3.2.2/fpcsrc/signal.inc
+     /usr/local/src.fpc/fpcbuild-3.2.2/fpcsrc/./x86_64/sighndh.inc
+   /usr/local/src.fpc/fpcbuild-3.2.2/fpcsrc/./x86_64/syscallh.inc
+   /usr/local/src.fpc/fpcbuild-3.2.2/fpcsrc/./x86_64/syscall.inc
+   /usr/local/src.fpc/fpcbuild-3.2.2/fpcsrc/./x86_64/sysnr.inc
+   /usr/local/src.fpc/fpcbuild-3.2.2/fpcsrc/ossysc.inc
+   /usr/local/src.fpc/fpcbuild-3.2.2/fpcsrc/osmacro.inc
+ /usr/local/src.fpc/fpcbuild-3.2.2/fpcsrc/system.pp
+  /usr/local/src.fpc/fpcbuild-3.2.2/fpcsrc/./x86_64/sighnd.inc
+  /usr/local/src.fpc/fpcbuild-3.2.2/fpcsrc/system.pp

and so on. Should be OK on Linux or SunOS/Solaris, untested on Mac, would need to be completely redone on Windows.

MarkMLl

[MarkMLl]

My take on using AI  to let yourself guide.

Short answer: please don't.

It has its place as a way to manage information based on the "flavour" of the material it's processing.

There might possibly be enough in the language/user/programmer reference manuals (which I think we all agree are pretty good) and wiki to train it, after which- once it's decided how to group and categorise concepts- it might possibly be usable as an index generator for the RTL/FCL (which is where the problems are, IMO). But I don't know how that would be tackled in practice, or what resources it would take.

Can anybody comment: if all versions of the relevant manuals were available online, could the relevant LLM training be distributed over the user community?

MarkMLl

[TRon]

It has its place as a way to manage information based on the "flavour" of the material it's processing.

I have no idea if a LLM trained specifically for one programming language/purpose would perform better (you might be inclined to think so).

Can anybody comment: if all versions of the relevant manuals were available online, could the relevant LLM training be distributed over the user community?

You could even do that off-line on a local running LLM. There are engines available that let you do that (I posted something about using whisper on someones question but the same concept also works for other models). The generated data can be distributed as a normal file, f.e. see huggingface.

It takes a lot of horsepower and time though. I do not know if there already exist a distributed way of training a model.

[Curt Carpenter]

Building my own compiler is pretty much beyond my skill level, but that's my problem of course.  I am close to being the the fpc/lazarus newbie that is just trying to find my way around all of the resources the system makes available to be a little more proficient.

Here's an example of a problem for someone like that.

The /fpcsrc/rtl on my system contains 2080 items.  One of these is named "wii."  A little exploration will reveal that this folder contains software specific to something called the "nintendo wii."  For some, this is going to be an important collection of source code for study and certainly warrants a place in the fpc system.  But it is pretty specialized, and not of much use or interest to anyone using a different more common system like Windows, Unix or Linux.  Still, one needs to open the folder and poke around to determine what it is and whether it's going to be important to learn about.  And there are lots of others like this.  Most of us will be able to guess what the folder "Sparc64" contains for example, but exploring it may just be a source of frustration for a younger person trying to engage with fpc/lazarus for the first time.  "Readme" files explaining what a folder contains are relatively rare.  You have to explore the folder "gba" to learn that it contains code specialized to "game boy" -- and so on.

Some sort of database that indicates the purpose of the thousands of folders in fpcsrc/rtl and and fpc/packages would ease the way to learning for a lot of people I think.  The sort of thing we find in the wonderful Online Package Manager, for example, that provides some insight into what a package does when you hover over it. 

You could say "why don't you build it then?" And the answer is that such a thing is beyond my skill level and at 80 attempting it would probably outlast me.  I would be willing to subscribe to such an effort though, in the interest of making fpc/lazarus more accessible to new users.

[TRon]

You could say "why don't you build it then?" And the answer is that such a thing is beyond my skill level and at 80 attempting it would probably outlast me.  I would be willing to subscribe to such an effort though, in the interest of making fpc/lazarus more accessible to new users.

Aside the reasons mentioned for not building, it is the way to go in that case.

The build-system determines what files/directories are relevant for a specific target (the generated makefiles contains all relevant files to be compiled).

Building the compiler itself is not too difficult to accomplish (it is much easier than you might perhaps think though requires a bit of knowledge on the build-system) so if you want help with that then please feel free to let us know. The resulting compiler (or actually the RTL/FCL units) give a fair good indication of what files/directories are relevant. A 100% definite answer comes from the fpmake files (which in itself is actual Pascal source-code).

[Curt Carpenter]

Well I'm certainly game to learn so will have a look at fpmake files for my linux system.  I'm afraid I have no idea where to start in acquiring that bit of knowledge about the build system.  Thanks for the input. 

[MarkMLl]

Well I'm certainly game to learn so will have a look at fpmake files for my linux system.  I'm afraid I have no idea where to start in acquiring that bit of knowledge about the build system.  Thanks for the input.

No, /don't/ do that. Those are what the developers use to define how everything hangs together, and few but they understand them.

(Assuming Linux) if you were building the compiler you'd use something like

Code: Text  [Select][+][-]

1. fpcbuild-3.2.2/fpcsrc$ make all
2.  

I usually add a few extra options relevant to debugging etc. but that't the basic stuff, and is all you need if your option is to duplicate the build rather than actually install a tailored copy

Now from the link I gave you earlier add the -vt option and a postprocessing script:

Code: Text  [Select][+][-]

1. fpcbuild-3.2.2/fpcsrc$ make OPT='-vt' all |/usr/local/bin/fpc-filter-vt
2.  

(noting where it's actually installed), that adds a couple of extre options relevant to debugging etc. but the important thing is the output file which contains a tree of everything the compiler finds relevant. So basically, the compiler is building the list for you.

Most of the "smarts" in that script file relates to working out naming relationships so that it can log the command that was used to invoke the build. Unchanged for years, currently running on Debian 12.

FYI, the full command that I normally run currently stands at

Code: Text  [Select][+][-]

1. fpcbuild-3.2.2/fpcsrc$ make NO_GDB=1 OVERRIDEVERSIONCHECK=1 OPT='-V3.2.2 -O- -gl -gw2 -godwarfsets -Xs- -vt' all |/usr/local/bin/fpc-filter-vt
2.  

That avoids various niggles, and builds an installable compiler with various debugging stuff.

MarkMLl

[TRon]

No, /don't/ do that. Those are what the developers use to define how everything hangs together, and few but they understand them.

The solution you presented is indeed a better idea MarkMLI.

I keep forgetting that my suggestions are not always beginner friendly