Proposal: Record Composition

[Warfley]

I would like to make a proposal for a small language feature, which could be really useful. I named this feature "Record Composition", but it already crossed my mind that because of the composition pattern in OOP, this may not be the most fitting name.

As I do not know what the offical way to submit such ideas/contributions is (there seems to be no information in the gitlab or the website), so I will just post it here, as I know that many FPC developers are watching this Forum regularly.

The basic idea is simple, just allow access to the subfields of a record element directly through the parent record. Example:

Code: Pascal  [Select][+][-]

1. {$ModeSwitch RecordComposition}
2.  
3. type
4.   TChildRec = record
5.     A: Integer;
6.   end;
7.  
8.   TComposed = record
9.     uses child: TChildRec;
10.     B: Double;
11.   end;
12.  
13. var
14.   c: TComposed;
15. begin
16.   c.A := 42;
17.   c.B := 3.14;
18.   WriteLn(c.child.A); // Writes 42
19. end.
20.  

I've already implemented it in the FPC sources, to see how feasable such an implementation would be: https://gitlab.com/freepascal.org/fpc/source/-/merge_requests/498
You can look at the commited test cases to see some more examples.

Why I think this is useful/required
There are basically three main reasons for it:

1. Since C11 C allows so called anonymous structures and anonymous unions within their structs. This means that while records have been mostly equivalent with C structs allowing easy porting of C code to Pascal, anonymous records and anonymous unions are not yet compatible, making it challenging to port C code using it. Record composition would basically resolve this problem using anonymous composition fields (already implemented):

Code: C  [Select][+][-]

1. struct foo {
2.   int x;
3.   struct bar; // anonymous struct inclusion of struct bar
4. }

Can now be represented by:

Code: Pascal  [Select][+][-]

1. TFoo = record
2.   x: Integer;
3.   uses TBar;
4. end;

2. Allowing for non trailing and multiple variant parts. As was discussed previously in https://forum.lazarus.freepascal.org/index.php/topic,63226.0.html there seems to be a need for having the variant part of the record not in the end, or having multiple variant parts within a record. Also this can be solved using record composition:

Code: Pascal  [Select][+][-]

1. TFoo = record
2.   x: Integer;
3.   uses record
4.     case Boolean of
5.     True: (I: Integer);
6.     False: (D: Double);
7.   end;
8.   y: Integer;
9. end;

3. It allows to easier organize record contents throughout multiple definitions. For example if you have platform dependent data, you don't need massively nested {$IfDef}-{$Else}-{$EndIf} directives, you can simply write different types and import them:

Code: Pascal  [Select][+][-]

1. TFoo = record
2.   CommonField: Integer;
3.   {$IfDef Windows}
4.   uses TFooWinInternal;
5.   {$Else}
6.   uses TFooUnixInternal;
7.   {$EndIf}
8. end;

There are also a few minor reasons to do this, especially interesting for advanced records. For example, one of the main issues with advanced records is that they don't support inheritance. While this composition is no replacement for inheritance, as it does not provide virtual methods or similars, the ability to compose a record from some base definition with some additional data and/or functionality is probably fully sufficient in most situations, giving more options without having to resort to a full inheritance model.
Second, one thing I experimented with was to use this to enable records as wrappers for objects and classes, for example one could imagine an "shared pointer" like this:

Code: Pascal  [Select][+][-]

1. TSharedSL = record
2. private
3.   ref: TStringList;
4.   refcount: PInteger;
5. public
6.   using ref;
7.   class operator Initialize(...); // Will create ref
8.   class operator Finalize(...); // will decrease refcount
9.   class operator Copy(...); // Decrease -> copy -> increase refcount
10.   class operator AddRef(...); // Will increase refcount
11. end;
12.  
13. var
14.   sl: TSharedSL;
15. begin
16.   sl.Add('Hello');
17.   sl.Add('World');
18.   WriteLn(sl.Text);
19. end.

But I'm a bit unsure about that one, while I currently have built it such that classes can also be composited, I think that just including large classes will mostly create irritating types with way to many fields and field collisions everywhere. So I don't think if there is really any use-case for this that makes it worth it.

Lastly its a very simple addition to the FPC, the diff on the compiler code is only like 200 lines of code, with some of it being for features like allowing to use classes and visibility scoping (such that you can like above have ref within a different visibility than the fields that are imported), which may not even be worth to include in the first place. At it's core it just adds link symbols which when resolved generate the AST for accessing the composite member, with basically only 2 contact points within the FPC source base, such that it will only have very limited effect on other FPC functionality and maintenance.

[TRon]

proposal introduces ambiguity:

Code: Pascal  [Select][+][-]

1. type
2.   TChildRec1 = record
3.     A: Integer;
4.   end;
5.   TChildRec2 = record
6.     A: Integer;
7.   end;
8.  
9.  
10.   TComposed = record
11.     uses child1: TChildRec1;
12.     uses child2: TChildRec2;
13.     B: Double;
14.   end;
15.  

Becomes even more problematic (to detect) when more nesting of record structures occurs.

edit: forgot to rename the child fieldnames introducing my own ambiquity. Fixed now.

[Warfley]

As I implemented it, it is quite easy, when there is a collision, and there is no member variable to resolve this, it throws an error. See the reccomp_dup_unnamed test cases: https://gitlab.com/Warfley/FPC_Source/-/blob/recordcomposition/tests/test/reccomp_dup_unnamed1_f.pp

Code: Pascal  [Select][+][-]

1. type
2.   TChildRec = record
3.     B: Integer;
4.   end;
5.  
6.   TComposed = record
7.     A: Integer;
8.     B: Integer;
9.     uses TChildRec; // Error duplicate identifier B
10.     C: Integer;
11.   end;
12.  

If the composition field is named, it will only throw a warning, and include the fields in a first come first serve manner, where the fields of the owning record are always prefered, as can be seen in the reccomp_dup_named test cases: https://gitlab.com/Warfley/FPC_Source/-/blob/recordcomposition/tests/test/reccomp_dup_named1_f.pp (note, because of the -Sew flag this test will also fail to compile due to it treating warnings as errors, in a live system you of course can always just ignore this warning)

Code: Pascal  [Select][+][-]

1.   TChildRec = record
2.     C: Integer;
3.   end;
4.  
5.   TComposed = record
6.     A: Integer;
7.     B: Integer;
8.     uses child: TChildRec; // Warning Duplicate Identifier C
9.     C: Integer;
10.   end;
11.  
12. var
13.   c: TComposed;
14. begin
15.   c.C := 42; // Sets C of TComposed Record
16.   c.child.C := 42; // Active disambiguation is required
17. end;
18.  

I think that a warning is of course also a very strong incentive to avoid this, but as active disambiguation is always possible I don't think it is truely an error case.

One thing I am unsure about is, when there is a collision between two composites like in your example, it would always use the first one in top to bottom order (so c.A would be c.child1.A), one thing I also though about would be that in such a case to include neither, so you can't write c.A anyway and always must use c.child1.A or c.child2.A. That would be the "cleaner" option I think.

But all in all, those are details, as you can always disambiguate through the field, it does not pose a problem

[Martin_fr]

The following works. So you can only "include" one parent.

"object" is like record, allocated in the stack. No pointer to heap, no "create" needed.

Code: Pascal  [Select][+][-]

1. type
2.   TChildRec = object
3.     A: Integer;
4.   end;
5.  
6.   TComposed = object(TChildRec)
7.     B: Double;
8.   end;

[Warfley]

The following works. So you can only "include" one parent.

"object" is like record, allocated in the stack. No pointer to heap, no "create" needed.

Code: Pascal  [Select][+][-]

1. type
2.   TChildRec = object
3.     A: Integer;
4.   end;
5.  
6.   TComposed = object(TChildRec)
7.     B: Double;
8.   end;

I know about objects and inheritance, the thing is that this does not serve the purposes outlined in my initial post. Unlike records it is not compatible to C, making it infeasable to serve as an interface to a C library that uses anonymous structs/unions. Also inheritance with a vmt and downcasting is much more complex than what I propose, as the goal of this is not to provide OOP functionality, but is more in the spirit of original records to simply provide data access within a structured record.
It is therefore important that when doing this:

Code: Pascal  [Select][+][-]

1. TComposite = record
2.   A: Integer;
3.   using TChildRecord;
4.   C: Integer;
5. end;

The TChildRecord instance is exactly located between A and C, with predictable (and configurable) alignment.
This should be compatible and usable for a wrapper to a C library that publishes the following datastructure:

Code: C  [Select][+][-]

1. struct composite {
2.   int a;
3.   struct childstruct;
4.   int c;
5. }

It specifically tries to address some of the limitations records have currently, which were also discussed in for example: https://forum.lazarus.freepascal.org/index.php/topic,63226.msg479790.html#msg479790

[Fibonacci]

+1

Its essentially anonymous records many ppl asked for, but with addition of including other named records.

An exmaple of how PEB structure may benefit from your Record Composition. Part of the PEB record contains:

Code: Pascal  [Select][+][-]

1. _PEB = packed record
2.   // (...)
3.   a: packed record
4.   case byte of
5.     0: (KernelCallbackTable: Pointer;);
6.     1: (UserSharedInfoPtr: Pointer;);
7.   end;
8.  

Need to call
PEB.a.KernelCallbackTable

With your modification
   

Code: Pascal  [Select][+][-]

1. _PEB = packed record
2.   // (...)
3.   uses packed record
4.   case byte of
5.     0: (KernelCallbackTable: Pointer;);
6.     1: (UserSharedInfoPtr: Pointer;);
7.   end;

Here it would be just
PEB.KernelCallbackTable

So full PEB record would be visible in IDE after typing PEB<dot>

PEB.BeingDebugged
instead of
PEB.byteBools.BeingDebugged

PEB.IsLongPathAwareProcess
instead of
PEB.byteBools.BitField.IsLongPathAwareProcess

[Curt Carpenter]

It appears to me to be writer-friendly, but, like so much from C, not really an improvement to reader friendliness.  And wasn't the later part of the basic philosophy that led to pascal?  (I realize that tools can make complex structures more transparent to the reader.)

[Warfley]

I would actually argue it's the other way around, it's easier to read then to write. Because when you use a record that has some composite fields, you can use all those fields without needing to know how the record is made up internally. On the other hand, now when writing the record it get's harder because you need to know whenever you can change a record you must keep in mind that it may create field collisions with other records that are composited together.

To give you an example where this may be useful, one of my goto projects for trying out a bit more low level programming (especially when learning a new low level language) is writing an emulator, usually for the original Nintendo Gameboy. This system has a few 8 bit registers, A, B, C, D, E, F, H ,L as well as the 16 bit registers AF, BC, DE, HL. As these names imply these registers are just overlays over the 8 bit registers.
But when you read the manual of the gameboy, for all the opcodes and operations they are described as if they are just registers that happen to share the memory with the 8 bit registers.

So when you look at the manual, and it states that the operation for the opcode 0x02 is LD A -> (BC), writing:

Code: Pascal  [Select][+][-]

1. RAMState[CPURegs.BC] := CPURegs.A;

is much more concise and easier to understand than having to go over a seperate hop:

Code: Pascal  [Select][+][-]

1. MemState[CPURegs.BC.Reg16] := CPURegs.AF.Reg8[0];

It allows you to much closer follow the manual and having the code being easy to map onto these constructs.

Of course this is kinda a low level feature, and you probably won't see it being used very commonly in most programs, but when writing such things as an emulator, a network protocol or an interface to some C library, this can make your code much cleaner, as your code can cleanly follow the high level descriptions of the reference material, rather than having to follow your internal representation of the data.

I also wouldn't consider it a typical C feature, as it was added to C only very recently (2011), and  it's kindof a generalization of the variant record feature of Pascal. So it is arguably a very pascallian feature, that C took from pascal (well I think D was the first C ancestor to adopt it and C adopted it then from D), they just lifted the restrictions of variant parts being only at the end of a record and generalized it to any struct or Union at any place.

[Lulu]

a quick comment:
from my Pascal-loving amateur point of view, who doesn't see the possible problems this could cause, at first glance I understood what Warfley was proposing, and, I must say, I can even visualize how I'll integrate it into my future codes! 

[Warfley]

A minor update, just fixed it to also work with generics, now the following is possible:

Code: Pascal  [Select][+][-]

1.   generic TComposed<T> = record
2.     uses child: T;
3.     B: Double;
4.   end;

[Fibonacci]

Bug report

Internal error 200601272 if using it with nested case-of's

Code: Pascal  [Select][+][-]

1. {$modeswitch recordcomposition}
2.  
3. //ASLR on, dynamic imagebase
4. {$SetPEOptFlags $20}
5. {$SetPEOptFlags $40}
6.  
7. uses Windows;
8.  
9. type
10.   _PEB = packed record
11.     uses packed record
12.     case byte of
13.     0: (_dummy: ULONG);
14.     1: (
15.       InheritedAddressSpace: Boolean;
16.       ReadImageFileExecOptions: Boolean;
17.       BeingDebugged: Boolean;
18.       BitField: bitpacked record //OK
19.       //uses bitpacked record //Error: Internal error 200601272
20.         case byte of
21.         0: (_dummy: Byte);
22.         1: (
23.           ImageUsesLargePages: Boolean;
24.           IsProtectedProcess: Boolean;
25.           IsImageDynamicallyRelocated: Boolean;
26.           SkipPatchingUser32Forwarders: Boolean;
27.           IsPackagedProcess: Boolean;
28.           IsAppContainer: Boolean;
29.           IsProtectedProcessLight: Boolean;
30.           IsLongPathAwareProcess: Boolean;
31.         );
32.         end;
33.       );
34.     end;
35.   end;
36.   TPEB = _PEB;
37.   PPEB = ^TPEB;
38.  
39. var
40.   PEB: PPEB;
41.  
42. begin
43.   {$asmmode intel}
44.   asm
45.   {$ifdef CPU64}
46.   mov rax, gs:[$60]
47.   mov PEB, rax
48.   {$else}
49.   mov eax, fs:[$30]
50.   mov PEB, eax
51.   {$endif}
52.   end;
53.  
54.   writeln('debugged  = ', PEB^.BeingDebugged);
55.   writeln('relocated = ', PEB^.BitField.IsImageDynamicallyRelocated);
56.  
57.   readln;
58. end.

[Warfley]

Bug report

Internal error 200601272 if using it with nested case-of's

Fixed and tests added

[TRon]

One thing I am unsure about is, when there is a collision between two composites like in your example, it would always use the first one in top to bottom order (so c.A would be c.child1.A), one thing I also though about would be that in such a case to include neither, so you can't write c.A anyway and always must use c.child1.A or c.child2.A. That would be the "cleaner" option I think.

The latter would imho also be the better option e.g. avoid all ambiguity and interrupt further compilation until the developer corrects.

On another note: have you considered the field and record alignment options that the compiler offers ? (I would like to test myself but unfortunately my agenda is a bit crowded atm). (*)

fwiw I especially like the anonymous embedding as that is really a lacking feature.

(*) By that initially I meant the actual layout in memory. The language construct themselves can be added if not (properly) supported right now.

[Fibonacci]

Fixed, works pretty well, cant spot any problem anymore. This must be merged into FPC main branch Great job.

How about IDE? It needs to be fixed too because it doesnt show hints for records.
And worse: Ctrl+Space doesnt work ANYWHERE. No hints at all in the IDE.

Or am I doing something wrong? Replaced FPC with your fork & copied original fpc.cfg

Code: Pascal  [Select][+][-]

1. {$modeswitch recordcomposition}
2.  
3. //ASLR on, dynamic imagebase
4. {$SetPEOptFlags $20}
5. {$SetPEOptFlags $40}
6.  
7. uses Windows;
8.  
9. type
10.   _UNICODE_STRING = packed record
11.     Length: UINT16;
12.     MaximumLength: UINT16;
13.     {$ifdef CPU64}
14.     Padding: ULONG;
15.     {$endif}
16.     Buffer: PWideChar;
17.   end;
18.  
19.   _CURDIR = packed record
20.     DosPath: _UNICODE_STRING;
21.     Handle: Pointer;
22.   end;
23.  
24.   _STRING = packed record
25.     Length: UINT16;
26.     MaximumLength: UINT16;
27.     {$ifdef CPU64}
28.     Padding: ULONG;
29.     {$endif}
30.     Buffer: PChar;
31.   end;
32.  
33.   _RTL_DRIVE_LETTER_CURDIR = packed record
34.     Flags: UINT16;
35.     Length: UINT16;
36.     TimeStamp: ULONG;
37.     DosPath: _STRING;
38.   end;
39.  
40.   _RTL_USER_PROCESS_PARAMETERS = packed record
41.     MaximumLength: ULONG;
42.     Length: ULONG;
43.     Flags: ULONG;
44.     DebugFlags: ULONG;
45.     ConsoleHandle: Pointer;
46.     ConsoleFlags: ULONG;
47.     {$ifdef CPU64}
48.     Padding1: ULONG;
49.     {$endif}
50.     StandardInput: Pointer;
51.     StandardOutput: Pointer;
52.     StandardError: Pointer;
53.     CurrentDirectory: _CURDIR;
54.     DllPath: _UNICODE_STRING;
55.     ImagePathName: _UNICODE_STRING;
56.     CommandLine: _UNICODE_STRING;
57.     //...
58.   end;
59.   TRTL_USER_PROCESS_PARAMETERS = _RTL_USER_PROCESS_PARAMETERS;
60.   PRTL_USER_PROCESS_PARAMETERS = ^TRTL_USER_PROCESS_PARAMETERS;
61.  
62.   _PEB = packed record
63.     uses packed record
64.     case byte of
65.     0: (_dummy: ULONG);
66.     1: (
67.       InheritedAddressSpace: Boolean;
68.       ReadImageFileExecOptions: Boolean;
69.       BeingDebugged: Boolean;
70.       uses bitpacked record
71.         case byte of
72.         0: (_dummy: Byte);
73.         1: (
74.           ImageUsesLargePages: Boolean;
75.           IsProtectedProcess: Boolean;
76.           IsImageDynamicallyRelocated: Boolean;
77.           SkipPatchingUser32Forwarders: Boolean;
78.           IsPackagedProcess: Boolean;
79.           IsAppContainer: Boolean;
80.           IsProtectedProcessLight: Boolean;
81.           IsLongPathAwareProcess: Boolean;
82.         );
83.         end;
84.       );
85.     end;
86.     {$ifdef CPU64}
87.     Padding0: DWORD;
88.     {$endif}
89.     Mutant: Pointer;
90.     ImageBaseAddress: Pointer;
91.     Ldr: Pointer; //PPEB_LDR_DATA
92.     ProcessParameters: PRTL_USER_PROCESS_PARAMETERS;
93.     SubSystemData: Pointer;
94.     ProcessHeap: Pointer;
95.     FastPebLock: Pointer; //PRTL_CRITICAL_SECTION
96.     AtlThunkSListPtr: Pointer;
97.     IFEOKey: Pointer;
98.     uses bitpacked record
99.     case byte of
100.       0: (_dummy: ULONG);
101.       1: (
102.         ProcessInJob: Boolean;
103.         ProcessInitializing: Boolean;
104.         ProcessUsingVEH: Boolean;
105.         ProcessUsingVCH: Boolean;
106.         ProcessUsingFTH: Boolean;
107.       );
108.     end;
109.     {$ifdef CPU64}
110.     Padding1: DWORD;
111.     {$endif}
112.     uses packed record
113.     case byte of
114.       0: (KernelCallbackTable: Pointer;);
115.       1: (UserSharedInfoPtr: Pointer;);
116.     end;
117.     SystemReserved: ULONG;
118.     AtlThunkSListPtr32: ULONG;
119.     ApiSetMap: Pointer;
120.     TlsExpansionCounter: ULONG;
121.     {$ifdef CPU64}
122.     Padding2: ULONG;
123.     {$endif}
124.     TlsBitmap: Pointer;
125.     TlsBitmapBits: array[0..1] of ULONG;
126.     ReadOnlySharedMemoryBase: Pointer;
127.     SharedData: Pointer;
128.     ReadOnlyStaticServerData: Pointer;
129.     AnsiCodePageData: Pointer;
130.     OemCodePageData: Pointer;
131.     UnicodeCaseTableData: Pointer;
132.     NumberOfProcessors: ULONG;
133.     NtGlobalFlag: ULONG;
134.     CriticalSectionTimeout: _LARGE_INTEGER; //unit Windows
135.     HeapSegmentReserve: Pointer;
136.     HeapSegmentCommit: Pointer;
137.     HeapDeCommitTotalFreeThreshold: Pointer;
138.     HeapDeCommitFreeBlockThreshold: Pointer;
139.     NumberOfHeaps: ULONG;
140.     MaximumNumberOfHeaps: ULONG;
141.     ProcessHeaps: Pointer;
142.     GdiSharedHandleTable: Pointer;
143.     ProcessStarterHelper: Pointer;
144.     GdiDCAttributeList: ULONG;
145.     Padding3: array[0..4-1] of UINT8;
146.     LoaderLock: Pointer; //PRTL_CRITICAL_SECTION
147.     OSMajorVersion: ULONG;
148.     OSMinorVersion: ULONG;
149.     OSBuildNumber: UINT16;
150.     OSCSDVersion: UINT16;
151.     OSPlatformId: ULONG;
152.     //...
153.   end;
154.   TPEB = _PEB;
155.   PPEB = ^TPEB;
156.  
157. var
158.   PEB: PPEB;
159.  
160. begin
161.   {$asmmode intel}
162.   asm
163.   {$ifdef CPU64}
164.   mov rax, gs:[$60]
165.   mov PEB, rax
166.   {$else}
167.   mov eax, fs:[$30]
168.   mov PEB, eax
169.   {$endif}
170.   end;
171.  
172.   writeln('debugged   = ', PEB^.BeingDebugged);
173.   writeln('relocated  = ', PEB^.IsImageDynamicallyRelocated);
174.   writeln('OS version = ', PEB^.OSMajorVersion, '.', PEB^.OSMinorVersion, ' build ', PEB^.OSBuildNumber);
175.   writeln('CPU cores  = ', PEB^.NumberOfProcessors);
176.   writeln;
177.  
178.   writeln('image path = ', WideString(PWideChar(PEB^.ProcessParameters^.ImagePathName.Buffer)));  
179.   writeln('workdir    = ', WideString(PWideChar(PEB^.ProcessParameters^.CurrentDirectory.DosPath.Buffer)));
180.   writeln('cmdline    = ', WideString(PWideChar(PEB^.ProcessParameters^.CommandLine.Buffer)));
181.  
182.   readln;
183. end.

[Warfley]

Fixed, works pretty well, cant spot any problem anymore. This must be merged into FPC main branch Great job.

How about IDE? It needs to be fixed too because it doesnt show hints for records.
And worse: Ctrl+Space doesnt work ANYWHERE. No hints at all in the IDE.

Or am I doing something wrong? Replaced FPC with your fork & copied original fpc.cfg

This is to be expected, Lazarus has it's own parser for the code-tools (autocomplete, jump to implementation and so on), which would need to get it's own additions to support this