Separating definition and implementation parts

[MarkMLl]

I would prefer not to get into an argument over this: I am not trying to enforce a particular point of view or request that anything be changed, but am asking for technical information.

The Object Pascal language as implemented by FPC and (I believe) Delphi puts both the definition and the implementation and the implementation of a unit in the same file.

Various jurisdictions are slowly sorting out their handling of software copyright and licensing, and appear to be making the common-sense distinction that an interface is a statement of fact (hence might not be copyrightable) while an implementation includes the detailed description of an algorithm or "method" which can definitely be copyrighted and possibly patented.

There is a long tradition of a copyright or license applying to the totality of a published work. I don't think there is a tested tradition of different sections or chapters of a published work being covered by different legal protection.

I am concerned that a copyright message in a source file might be jeopardised if the file is examined and a significant portion of it is discovered to be a statement of fact, hence uncopyrightable, hence there is no legitimate owner to enforce a license.

An obvious example of this is C header files, which typically contain a factual description of an API but might also contain macros which are neither descriptive nor obvious since they compute (provide a method for) bit manipulation where the bit significance is an internal implementation detail:

Code: C  [Select][+][-]

1. struct hidraw_devinfo {
2.         __u32 bustype;
3.         __s16 vendor;
4.         __s16 product;
5. };
6.  
7. /* ioctl interface */
8. #define HIDIOCGRDESCSIZE        _IOR('H', 0x01, int)
9. #define HIDIOCGRDESC            _IOR('H', 0x02, struct hidraw_report_descriptor)
10. #define HIDIOCGRAWINFO          _IOR('H', 0x03, struct hidraw_devinfo)
11. #define HIDIOCGRAWNAME(len)     _IOC(_IOC_READ, 'H', 0x04, len)
12. #define HIDIOCGRAWPHYS(len)     _IOC(_IOC_READ, 'H', 0x05, len)
13. /* The first byte of SFEATURE and GFEATURE is the report number */
14. #define HIDIOCSFEATURE(len)    _IOC(_IOC_WRITE|_IOC_READ, 'H', 0x06, len)
15. #define HIDIOCGFEATURE(len)    _IOC(_IOC_WRITE|_IOC_READ, 'H', 0x07, len)
16.  
17. #define HIDRAW_FIRST_MINOR 0
18. #define HIDRAW_MAX_DEVICES 64
19.  

In the distant past, I've borrowed an idea from the old TopSpeed Pascal compiler and put interface specifications into .itf files which I've then included at the top of the corresponding implementation unit.

What are the technical implications of doing this, from the POV of FPC and Lazarus? Will it reduce FPC's compilation efficiency (to an extent greater than implied by the increased number of files) or invariably result in a layout where Lazarus has debugging issues?

MarkMLl

[SymbolicFrank]

I cannot remember where, but some RTTI files include the same declarations for many different types. That's a short .inc file (~10 lines) that's included many times in the same unit. I wouldn't bother about details like that and just split it up exactly how you want it.

For compilation efficiency, an SSD is the main improvement. Next is probably the amount of cores and then RAM. Everything else is pretty much irrelevant, I think.

[PascalDragon]

What are the technical implications of doing this, from the POV of FPC and Lazarus? Will it reduce FPC's compilation efficiency (to an extent greater than implied by the increased number of files) or invariably result in a layout where Lazarus has debugging issues?

There might be problems with the compiler detecting whether a unit needs to be recompiled, because if I remember correctly it only checks the timestamp of the unit file itself against that of the PPU and does not check any of the referenced include files (or include files that are included by include files).

I cannot remember where, but some RTTI files include the same declarations for many different types. That's a short .inc file (~10 lines) that's included many times in the same unit. I wouldn't bother about details like that and just split it up exactly how you want it.

The RTL is however prebuilt from the PoV of the user.

[Martin_fr]

Just some ideas that came to my mind while reading your post.

1)
Afaik, if you put some "source" into an include file and use {$I...} then the compiler will not see if content of the include file is modified (as it compares the date of the ppu and the pas/pp file.
So you may get errors when re-compiling.

2)
In some sense the "definition" (i.e. content of the "interface" section) is repeated in the implementation section. Each function/method... starts with the same header....
(I am aware that not all is function headers, and not all may be repeated / then again, non repeated definitions might be easy to move to a diff unit.. / or not / not the point of the below anyway)

That probably means "definition" needs to be more precisely. I.e. maybe "the public interface" ?

Then a layer might need to clarify, if any repeat of such info will lead into the same potential issue.
Or if a repeat is then just using the already otherwise licensed information, and usage of info that has a license of it's own, is then not affected by the license of the file in which it is repeated?

Depending upon which, the "definition" could be in a separate file:
- as documentation, with/without the code snippets
- as abstract or "type foo = interface" source code.
In either case, the repeat of those headers (in the actual unit that also will have the implementation) is then no different to the repeat of the headers between "interface" and "implementation" section, or is it?

I hope, the 2nd part will not start any discussions. Unless a person with legal background wants to opine, there should be little to add. (Well, if someone has info where such an knowledge based information is to be had, a link might be added)

It's just a sort of rhetorical question, that needs no answer. It is well understood that answers would cover the entire range from "nonsense" to "interesting") with all kind of potentially uninformed reasons given.

[marcov]

What are the technical implications of doing this, from the POV of FPC and Lazarus? Will it reduce FPC's compilation efficiency (to an extent greater than implied by the increased number of files) or invariably result in a layout where Lazarus has debugging issues?

The number of files read rises, and the file searching also becomes more complicated. (multiple extensions). I think both are manageable, but while file reading performance on spinning rust will suffer a bit of course (more files is more chance that a seek is needed), it might not matter that much on a SSD.

For file enumeration, there afaik are already search caches (to avoid to frequent enumerating files in search directories), so file searching just requires extending that.

Probably on e will need some extra crcs and timestamps  (what if def is not changed, but implementation isn't, that case doesn't exist now?)

However the biggest problem is that the unit handling and the automated recompiling isn't the easiest part of the compiler. Import cycles, cross-unit inline, generics, many things complicate it. The usual academic compiler with a clean dialect vs a production compiler with a established dialect and feature set.

I don't really see the copyright being that important, the few big corporates that fight out their legal  wars (like Oracle-Google over Java) will have complete departments on it, and can handle specifying separate clauses for interface and implementation even in one file.

So why bother?

[SymbolicFrank]

Yes, I agree. If it comes to a dispute, it only matters who has the most money. Being in the right is not all that important.

[MarkMLl]

Thanks everybody for the thoughts so far. Sven and others commented on the situation where a changed include (i.e. interface) file doesn't force recompilation of the implementation: yes, I think that's a showstopper. I know that I'm not the only one whose seen problems in this overall area: a directive change which doesn't force recompilation... I've just not seen it in an easily-reproducible form.

The number of files read rises, and the file searching also becomes more complicated. (multiple extensions). I think both are manageable, but while file reading performance on spinning rust will suffer a bit of course (more files is more chance that a seek is needed), it might not matter that much on a SSD.

...on the other hand, I did emphasise that I wasn't asking for anything to be changed so I don't think that extensions and search paths really come into it.

For file enumeration, there afaik are already search caches (to avoid to frequent enumerating files in search directories), so file searching just requires extending that.

Probably on e will need some extra crcs and timestamps  (what if def is not changed, but implementation isn't, that case doesn't exist now?)

However the biggest problem is that the unit handling and the automated recompiling isn't the easiest part of the compiler. Import cycles, cross-unit inline, generics, many things complicate it. The usual academic compiler with a clean dialect vs a production compiler with a established dialect and feature set.

It would be nice if the include-rebuild problem could be pinned down and fixed. I suspect that one thing that would help would be putting the uses clause in the unit rather than the include file since it has an impact on both... but at this point things would be getting decidedly messy since the interface file could no longer be read in isolation :-/

I don't really see the copyright being that important, the few big corporates that fight out their legal  wars (like Oracle-Google over Java) will have complete departments on it, and can handle specifying separate clauses for interface and implementation even in one file.

So why bother?

Well, when various people at Bell Labs thrashed out C (hence the potential for a mix of declarations and implementation in header files) in the early 70s they probably didn't appreciate just how big business software would end up being. and they definitely didn't anticipate Oracle vs Google in the Supreme Court.

I think the fact that that case /did/ merit the consideration of the USA's senior judicial body... which I think is taken seriously by just about everybody else, even if they prefer not to admit it... should be taken as an indication that in another 50 years decisions we are making now might end up getting similar scrutiny. Hypothetically we might, for example, have a situation at some point in the future where somebody (SCO Redux?) challenges the legality of the GPL or some variant of the Creative Commons license since it is routinely applied to files which cannot be licensed since they contain nothing copyrightable: /I/ think it's bad legal reasoning, but some court might decide that the GPL etc. is /always/ invalid because it is /sometimes/ used inappropriately.

MarkMLl

[MarkMLl]

Yes, I agree. If it comes to a dispute, it only matters who has the most money. Being in the right is not all that important.

Actually, that's not always the case:

https://www.nytimes.com/2010/02/09/us/09bar.html

I must say that that's a sufficiently convincing story of somebody who- eventually- started getting things right that it brings tears to my eyes.

MarkMLl

[marcov]

Yeah, sorry, I reacted to a general interface-implementation difference, not on the include trick specifically.

Note that contrary to Delphi,  FPC iirc does have some .inc checking (one level only), which you can see during .ppu loading with -va.

Code: [Select]

[0.030] (FPINTRES) PPU Source: fpintres.pp not available
[0.030] (FPINTRES) PPU Source: winres.inc not available

But as Sven says it is not recursive, so if you move the general interface into an include, it won't do that check on incs in the interface (OTOH, delphi never checks any includes afaik, so if delphi programmers can, you might be able to do without it too).

While I think it is a good FPC feature to check, it is only a first order thingy.

[MarkMLl]

Yeah, sorry, I reacted to a general interface-implementation difference, not on the include trick specifically.

Don't worry, I was expecting reactions which is why I phrased my original question very cautiously :-)

I'm also very much in mind of this from a couple of weeks ago:

In the C world C files can be build in parallel, because they don't depend on each other (declarations and such are in separate header files that are simply included by each). This is not the case for FPC, because units have dependencies between each other and the compiler simply can't continue compiling a unit until all dependencies of that unit are solved.

However thinking about your

Note that contrary to Delphi,  FPC iirc does have some .inc checking (one level only), which you can see during .ppu loading with -va.

Code: [Select]

[0.030] (FPINTRES) PPU Source: fpintres.pp not available
[0.030] (FPINTRES) PPU Source: winres.inc not available

But as Sven says it is not recursive, so if you move the general interface into an include, it won't do that check on incs in the interface (OTOH, delphi never checks any includes afaik, so if delphi programmers can, you might be able to do without it too).

While I think it is a good FPC feature to check, it is only a first order thingy.

OK, that's interesting... if we assumed that a .itf describing an interface and included in a unit file had a uses clause but did not have any $includes, should that be reliable? Would the "only one level deep" prevent units appearing in the uses clause from checking their .itf files, i.e. does the depth get reset for each unit or is it entirely relative to the main program (i.e. top-level sourcefile)?

I'm definitely not asking for help with this but the earlier problem I've come across is illustrated at https://forum.lazarus.freepascal.org/index.php/topic,52849.msg392854.html#msg392854 : it appears there's a $unitpath involved but I saw enough to make me suspect that the conditional was masking the version/age checks. I need to put more time into that and if I can get to a "reliable problem" state I'll raise it formally.

MarkMLl

[PascalDragon]

OK, that's interesting... if we assumed that a .itf describing an interface and included in a unit file had a uses clause but did not have any $includes, should that be reliable? Would the "only one level deep" prevent units appearing in the uses clause from checking their .itf files, i.e. does the depth get reset for each unit or is it entirely relative to the main program (i.e. top-level sourcefile)?

It's completely irrelevant where the uses clause is located. When checking whether something changed the compiler goes purely by date: it first opens the PPU and tries to find the source file for it. If it's found it checks whether the source file is newer than what is stored in the PPU, if so the unit is recompiled. Then it does the same for the include files (with varying degrees of working correctly). If the compiler does not recompile the unit it will take the used units from the PPU otherwise it's recompiling the unit (together with its includes) anyway (it's either an all or nothing).

So if you have the uses-clause in an include the only difference would be whether the change detection would pick it up or not (ideally it should, but currently it might not), but the same applies to any change, cause at that point the compiler doesn't care about content.

[MarkMLl]

It's completely irrelevant where the uses clause is located. When checking whether something changed the compiler goes purely by date: it first opens the PPU and tries to find the source file for it. If it's found it checks whether the source file is newer than what is stored in the PPU, if so the unit is recompiled. Then it does the same for the include files (with varying degrees of working correctly). If the compiler does not recompile the unit it will take the used units from the PPU otherwise it's recompiling the unit (together with its includes) anyway (it's either an all or nothing).

So if you have the uses-clause in an include the only difference would be whether the change detection would pick it up or not (ideally it should, but currently it might not), but the same applies to any change, cause at that point the compiler doesn't care about content.

I think you're actually saying that in principle it /should/ be completely irrelevant where the uses clause is located relative to a $include, but in practice it /might/ not be (which would be a bug, rather than implementation-defined behaviour etc. :-)

OK, so if I'm reading you correctly the timestamp on included files one level deep will be checked correctly, but that /might/ not work at deeper levels and there's a /risk/ that a uses clause in an included file (one level deep) won't be recursed through (with that risk increasing as the depth increases).

So an included .itf should be fairly reliable if a convention is used that it doesn't itself include anything. The residual risk is that a changed unit in the .itf's uses clause won't be picked up, which will obviously apply more to in-project imports rather than references to the RTL etc.

As I've said, I'm not out to argue the merits either way of this approach. But it does appear that as well as- I believe- helping the copyright/license situation it should allow .itf files (I'm trying to avoid the word "interface" here) to be protected from casual modification.

Comparing it with the Modula-2 family's approach, the significant difference is that an import (via a uses clause) in a .itf automatically applies to the implementation part as well, and it will be an error if repeated even if the versioning matches.

MarkMLl

[PascalDragon]

I think you're actually saying that in principle it /should/ be completely irrelevant where the uses clause is located relative to a $include, but in practice it /might/ not be (which would be a bug, rather than implementation-defined behaviour etc. :-)

Yes, or more like a shortcoming than a bug, though that doesn't change the result.

OK, so if I'm reading you correctly the timestamp on included files one level deep will be checked correctly, but that /might/ not work at deeper levels and there's a /risk/ that a uses clause in an included file (one level deep) won't be recursed through (with that risk increasing as the depth increases).

Yes and no. Yes regarding the timestamps, but the uses-clause is irrelevant here. If the compiler decides to recompile the unit it will always read all the include files of that unit and all indirectly used ones (and if one is missing: error). If the compiler does decide not to recompile the unit the used units will always be retrieved from the PPU. So if the compiler doesn't correctly handle checks for some include file it does not matter what change you make (be it a uses-clause or a white space change) it will simply not trigger the recompilation.

[MarkMLl]

Yes, or more like a shortcoming than a bug, though that doesn't change the result.

I can assure you that I'm not criticising, just trying to work things out :-)

Yes and no. Yes regarding the timestamps, but the uses-clause is irrelevant here. If the compiler decides to recompile the unit it will always read all the include files of that unit and all indirectly used ones (and if one is missing: error). If the compiler does decide not to recompile the unit the used units will always be retrieved from the PPU. So if the compiler doesn't correctly handle checks for some include file it does not matter what change you make (be it a uses-clause or a white space change) it will simply not trigger the recompilation.

The key words there are "if the compiler decides".

Reading things through a dozen times to make sure I'm not living dangerously, I think the key scenario is this:

* Program X uses unit Y which includes file Z, and file Z uses unit T.

* A change to (only) Z will not cause Y or X to be recompiled, since the names and attributes of included files are not stored in a .ppu file.

* A change to T will always cause Y and X to be recompiled (hence Z to be reread), since the names and attributes of used files are stored in a .ppu file.

I'm obviously not criticising, and am very much aware that this sort of thing makes demands of the compiler well beyond what gcc etc. would be able to handle without an external build system.

Hence the risk is that if we have this:

* Program A.pas uses unit B.

* Unit B.pas includes its interface definition from B.itf

* A change to B.itf will always be read if unit B is recompiled.

* But if B.itf, included in B.pas, is changed without B.pas being changed, then unit B will not automatically be recompiled.

Hence my suggestion of included .itf files is not safe, if the contained unit interface definition is changed without change to the unit implementation.

I'd be very happy for you to argue that my suggestion is more robust than I've concluded :-)

MarkMLl

[PascalDragon]

The key words there are "if the compiler decides".

Yes, that's the important part.

* Program X uses unit Y which includes file Z, and file Z uses unit T.

Nitpicking: it's always unit Y that uses unit T, but it's file Z that contains the uses-clause (this is an important point, because the compiler keeps track of used units and symbols and such based on units, not based on files)

* A change to (only) Z will not cause Y or X to be recompiled, since the names and attributes of included files are not stored in a .ppu file.

As long as the system works correctly a change to Z will trigger a recompilation of Y (sidenote: the main program file, or to be precise the file provided on the command line - after all it could be a unit as well - is always compiled), because the compiler does store the included files and their timestamps in the PPU.

Take the following example:

Code: Pascal  [Select][+][-]

1. // trecompile.pp
2. program trecompile;
3.  
4. uses
5.   urecompile;
6.  
7. begin
8.  
9. end.
10.  
11. // urecompile.pp
12. unit urecompile;
13.  
14. {$mode objfpc}{$H+}
15.  
16. interface
17.  
18. {$I irecompile.inc}
19.  
20. implementation
21.  
22. end.
23.  
24. // irecompile.inc
25. uses
26.   {$I irecompile2.inc}
27.   Classes;
28.  
29. // irecompile2.inc
30. {SysUtils,}

If you now compile trecompile.pp you will see the following output:

Code: [Select]

PS D:\fpc\git> fpc -FEtestoutput .\fpctests\trecompile.pp
Free Pascal Compiler version 3.2.2 [2022/01/02] for i386
Copyright (c) 1993-2021 by Florian Klaempfl and others
Target OS: Win32 for i386
Compiling .\fpctests\trecompile.pp
Compiling .\fpctests\urecompile.pp
Linking testoutput\trecompile.exe
24 lines compiled, 0.1 sec, 148000 bytes code, 6548 bytes data
If you compile it again without any changes the output will be as following:

Code: [Select]

PS D:\fpc\git> fpc -FEtestoutput .\fpctests\trecompile.pp
Free Pascal Compiler version 3.2.2 [2022/01/02] for i386
Copyright (c) 1993-2021 by Florian Klaempfl and others
Target OS: Win32 for i386
Compiling .\fpctests\trecompile.pp
Linking testoutput\trecompile.exe
8 lines compiled, 0.1 sec, 148000 bytes code, 6548 bytes data
As you can see the line Compiling .\fpctests\urecompile.pp is missing, because the compiler determined that urecompile.pp is up to date.

You can also use ppudump urecompile.ppu and you will see the following (among other information):

Code: [Select]

Interface section
------------------
Module Name: urecompile

Source file 1 : urecompile.pp 2022/06/02 08:58:06
Source file 2 : irecompile.inc 2022/06/02 08:59:52
Source file 3 : irecompile2.inc 2022/06/02 09:00:20
Uses unit: System (Crc: D86E0595, IntfcCrc: 01C77D16, IndCrc: BDD566AD)
Uses unit: objpas (Crc: 869382B2, IntfcCrc: 28EE26C5, IndCrc: AB9908A2)
Uses unit: Classes (Crc: 5FF4CBD7, IntfcCrc: DBC85DDC, IndCrc: 2CAFF89A)
DerefMapsize: 1
DerefMap[0] = SYSTEM
Derefdata length: 27
If you now change irecompile2.inc (by removing the swirly brackets) you will see this:

Code: [Select]

PS D:\fpc\git> fpc -FEtestoutput .\fpctests\trecompile.pp
Free Pascal Compiler version 3.2.2 [2022/01/02] for i386
Copyright (c) 1993-2021 by Florian Klaempfl and others
Target OS: Win32 for i386
Compiling .\fpctests\trecompile.pp
Compiling .\fpctests\urecompile.pp
Linking testoutput\trecompile.exe
24 lines compiled, 0.1 sec, 148000 bytes code, 6548 bytes data
So here the compiler decided to recompile urecompile.pp, because it successfully detected the changed file.

So in this specific example everything is as it should be.

* A change to T will always cause Y and X to be recompiled (hence Z to be reread), since the names and attributes of used files are stored in a .ppu file.

Here you didn't pay attention to Pascal 101: a unit is only recompiled if something significant of a used unit changed. Mainly this means information from the unit's interface, but sometimes this is also the case if the code of some function declared with inline changed (this is still the source of quite some headaches in the compiler and ideally the compiler should also trigger this if a generic implementation is changed...).

So if you change something in the implementation of T then Y likely won't be recompiled (and X will always be as mentioned above), but if you change something in the interface section of T it is very likely that Y will be recompiled. I say likely, because if you only change a typo in a comment and that change doesn't lead to a change of the file position for the remainder of the code (meaning for example that you didn't insert or remove a line) then the relevant content of the unit (from the compiler's PoV) is still considered the same and thus the compiler doesn't see the need to recompile Y.

To come back to your problem: as you see it should work correctly. However we know from past experience that something there is fishy when multiple include files are used, but we've not yet been able to pinpoint the problem. Or maybe it was indeed solved already. That's the problem: we don't have a reproducible case and thus don't know whether the problem has been solved or not.