structured types structure fpc/kylix problem

[Чебурашка]

Hello,
I have a .pas containing a record definition which is

1. used in fpc to define variables from this record definition
2. used in kylix to define variables from this record definition
3. there is a variable from this record definition created in a program compiled with kylix, which is streamed in a file and later read by a program compiled using fpc.

I saw that there are some differences in the way kylix and fpc compiler arrange the internal record structure (for example, if there is a enum, kylix used one byte, while fpc uses 4 as default).

I managed to make the structures generated by the two compilers exactely coincident and everithing works (I substituted the enums with integer, and later I did that needed casts).

I am now concerned with that fact that in future several things could happen:

1. I could unwittingly modify the record definition in a improper way
2. the fpc compiler might change the way records are structured (kylix compiler I believe will not change as borland is dead)

I would like to see if I can put something in the record structure that helps me to understand if the two generated structures are still aligned or not, so that I can put appropriate checks in code that help me understand at runtime if I broke smtg.

I was thinking to put some variable at the end of the record with fixed values that I can read before processing the received stream that guarantees with good reliance that record is aligned. Something like:

Code: Pascal  [Select][+][-]

1. type
2.   TSharedRecord = packed record
3.       // Many variables...
4.       //
5.       CheckVariable: string[8];
6.    end;
7.  
8. const
9.   CheckVariableValue = 'ILOVEFPC';
10.  

sending side:

Code: Pascal  [Select][+][-]

1. var
2.   x: TSharedRecord;
3. begin
4.   // x. set all values
5.   x.CheckVariable := ControlVariableValue;
6.   Send(x);
7.  

receiving side:

Code: Pascal  [Select][+][-]

1. var
2.   x: TSharedRecord;
3. begin
4.   Receive(x);
5.   if x.CheckVariable <> ControlVariableValue then // tell big problem
6.   else // x. read all values
7.  

Any ideas/suggestions/critics?

[JuhaManninen]

Any ideas/suggestions/critics?

Just dump Kylix and use FPC.

[Чебурашка]

Just dump Kylix and use FPC.

Thanks for the suggestion.

Others that do not foresee the dismission of kylix, as it is used to compile the software that operates the machines that my company sells worldwide since many years, and for which there is no plan for dismission in the near future (despite the long term goal is to switch to fpc)?

[440bx]

Any ideas/suggestions/critics?

Given that FPC likely does everything you want Kylix to do, JuhaManninen's suggestion is likely the best course of action.

If, for some reason, that is not an option for you then you can add code to calculate the offset of every field in the record definition and compare those offsets with the values you expect them to be.  A bit tedious to maintain but, it guarantees the offsets match.

Also, just checking the offset of the last field isn't sufficient because one alignment difference of one field could be compensated by another one for some other field.

[Чебурашка]

If, for some reason, that is not an option for you then you can add code to calculate the offset of every field in the record definition and compare those offsets with the values you expect them to be.  A bit tedious to maintain but, it guarantees the offsets match.

If it was an option I would not have asked.

Also, just checking the offset of the last field isn't sufficient because one alignment difference of one field could be compensated by another one for some other field.

I did not consider the mutual compensation, thanks for the suggestion.

I start thinking that another possibility could be to make a md5 or sha1 of the raw memory area used by the record and transmit it separately? Could it work?

[JuhaManninen]

I saw that there are some differences in the way kylix and fpc compiler arrange the internal record structure (for example, if there is a enum, kylix used one byte, while fpc uses 4 as default).

It can be adjusted. See:
 https://wiki.freepascal.org/Enum_Type#Size

[Чебурашка]

I saw that there are some differences in the way kylix and fpc compiler arrange the internal record structure (for example, if there is a enum, kylix used one byte, while fpc uses 4 as default).

It can be adjusted. See:
 https://wiki.freepascal.org/Enum_Type#Size

I saw, thanks.

Initially I was thinking to use this compiler feature. The reason why I didn't is that I could do mistakes (I will, indeed ) because some months from now maybe I won't remember anymore that I have to set this in the new fields I will add.  Basically the concern no 1. Maybe I am worrying too much, but it comes from past experiences (if something is fragile, either you or someone else will broke sooner or later).

[440bx]

I start thinking that another possibility could be to make a md5 or sha1 of the raw memory area used by the record and transmit it separately? Could it work?

It probably would but, that's probably more work than needed to ensure the alignments are as they should be.  Plus, if the method indicates that the offsets don't match, you won't have any indication of which field is misaligned.  Doesn't sound like a desirable method.

Here is something I do to ensure Pascal records are aligned the same as translated C structures: for both structures, write short programs (C and Pascal in this example) that outputs the offset of each field then compare the outputs, they should be identical.

The only difference in your case is that instead of C and Pascal, you'd have Kylix Pascal and FPC but the process/method is the same.  It is very simple and reliable too.

Also, I don't know if this applies in your case but, when I translate hundreds of C structs to Pascal, I write an AWK script to generate the programs that output the offsets (along with the struct and field name) then compare the outputs using a file diff. 

For a single record, manually creating the code to get the offsets in both cases should probably take just a few minutes.

[Чебурашка]

I start thinking that another possibility could be to make a md5 or sha1 of the raw memory area used by the record and transmit it separately? Could it work?

It probably would but, that's probably more work than needed to ensure the alignments are as they should be.  Plus, if the method indicates that the offsets don't match, you won't have any indication of which field is misaligned.  Doesn't sound like a desirable method.

True, having something that also tells me (or others) where fixing is needed would be very good.

Here is something I do to ensure Pascal records are aligned the same as translated C structures: for both structures, write short programs (C and Pascal in this example) that outputs the offset of each field then compare the outputs, they should be identical.

The only difference in your case is that instead of C and Pascal, you'd have Kylix Pascal and FPC but the process/method is the same.  It is very simple and reliable too.

Also, I don't know if this applies in your case but, when I translate hundreds of C structs to Pascal, I write an AWK script to generate the programs that output the offsets (along with the struct and field name) then compare the outputs using a file diff. 

For a single record, manually creating the code to get the offsets in both cases should probably take just a few minutes.

Thanks I will do this way.

[TRon]

@Чебурашка
I have no idea how it is called/named but I seem to remember there is a utility and/or build-script in the Free Pascal source tree that is able to generate both c and pascal source-code, compile it and create some output that seem to do what you ask for: compare the generated structures against each other. Perhaps you can use that as a start to create your own utility/tool/script for Kylix vs FPC ?

Other than that it is no different then converting c-structures to Pascal: you never know if the alignment is the same unless trying/experiencing. There are a lot of option for the FPC compiler to try make the alignment as compatible as possible.

edit: found it ! It is named "h2paschk" and can be found in the utils/h2pas directory

[marcov]

I saw that there are some differences in the way kylix and fpc compiler arrange the internal record structure (for example, if there is a enum, kylix used one byte, while fpc uses 4 as default).

Kylix, like delphi allows to set the enum size in the project. IOW only a source file that always declares records as packed or with alignment differences 100% defines a structure.

In other cases it is always a mix of project and source settings.

[440bx]

Thanks I will do this way.

Just in case it might be useful to you, here is an example program and its output.

Program to verify offsets in the Process Environment Block:

Code: Pascal  [Select][+][-]

1. {$APPTYPE       CONSOLE}
2.  
3. {$TYPEDADDRESS  ON}
4.  
5. {$LONGSTRINGS   OFF}
6.  
7. { --------------------------------------------------------------------------- }
8.  
9. program _PebOffsets;
10.   { program to verify the offsets of the PEB structure                        }
11.  
12. uses
13.   _8905_Utils,
14.  
15.   _9000_000_ntdll
16.   ;
17.  
18.  
19. function sprintf(
20.                  { _out_ } OutAsciizBuffer : pchar;
21.                  { _in_  } InAsciizFormat  : pchar;
22.                  { _in_  } InFieldName     : pchar
23.                 )
24.          : integer; cdecl; external ntdll;
25.   { used to left justify the field name in a field of spaces                  }
26.  
27.  
28.  
29. procedure WriteOffset(FieldName : pchar; Offset : ptruint);
30.   { output the field name and its offset in hex and decimal                   }
31. const
32.   WIDTH = 5;
33.  
34. var
35.   Name    : packed array[0..255] of char;
36.   WorkBuf : packed array[0..255] of char;
37.  
38. begin
39.   sprintf(Name, '  %-40s : ', FieldName);
40.  
41.   writeln(Name, DToHexD(Offset, 0, WorkBuf):WIDTH, '  ', Offset:WIDTH);
42. end;
43.  
44.  
45. var
46.   WorkBuf : packed array[0..255] of char;
47.  
48. begin
49.   writeln;
50.   writeln;
51.  
52.   case sizeof(pointer) of
53.     4 : write('  32bit PEB');
54.     8 : write('  64bit PEB');
55.   end;
56.   writeln(' - sizeof(PEB) = ', DToHexD(sizeof(TPEB), 0, WorkBuf), '  ', sizeof(TPEB), 'd');
57.   writeln;
58.  
59.   WriteOffset('InheritedAddressSpace                 ', ptruint(@TPEB(nil^).InheritedAddressSpace                 ));
60.   WriteOffset('ReadImageFileExecOptions              ', ptruint(@TPEB(nil^).ReadImageFileExecOptions              ));
61.   WriteOffset('BeingDebugged                         ', ptruint(@TPEB(nil^).BeingDebugged                         ));
62.   writeln;
63.   WriteOffset('UnionA                                ', ptruint(@TPEB(nil^).UnionA                                ));
64.   WriteOffset('UnionA.SpareBool                      ', ptruint(@TPEB(nil^).UnionA.SpareBool                      ));
65.   WriteOffset('UnionA.Bitfields                      ', ptruint(@TPEB(nil^).UnionA.Bitfields                      ));
66.  
67.   {$ifdef FPC}
68.   WriteOffset('UnionA.BitFlags                       ', ptruint(@TPEB(nil^).UnionA.BitFlags                       ));
69.   {$endif}
70.  
71.   writeln;
72.   WriteOffset('Mutant                                ', ptruint(@TPEB(nil^).Mutant                                ));
73.  
74.   < additional fields removed to make acceptable to forum software >
75.  
76.   { ------------------------------------------------------------------------- }
77.   { fields appended for NT 4.0 and above                                      }
78.  
79.   writeln;
80.   WriteOffset('ProcessStarterHelper                  ', ptruint(@TPEB(nil^).ProcessStarterHelper                  ));
81.   WriteOffset('GdiDCAttributeList                    ', ptruint(@TPEB(nil^).GdiDCAttributeList                    ));
82.  
83.   writeln;
84.   WriteOffset('UnionI                                ', ptruint(@TPEB(nil^).UnionI                                ));
85.   WriteOffset('UnionI.LoaderLockPointer              ', ptruint(@TPEB(nil^).UnionI.LoaderLockPointer              ));
86.   WriteOffset('UnionI.LoaderLock                     ', ptruint(@TPEB(nil^).UnionI.LoaderLock                     ));
87.  
88.   writeln;
89.   WriteOffset('OSMajorVersion                        ', ptruint(@TPEB(nil^).OSMajorVersion                        ));
90.   WriteOffset('OSMinorVersion                        ', ptruint(@TPEB(nil^).OSMinorVersion                        ));
91.   WriteOffset('OSBuildNumber                         ', ptruint(@TPEB(nil^).OSBuildNumber                         ));
92.   WriteOffset('OSCSDVersion                          ', ptruint(@TPEB(nil^).OSCSDVersion                          ));
93.   WriteOffset('OSPlatformId                          ', ptruint(@TPEB(nil^).OSPlatformId                          ));
94.   WriteOffset('ImageSubsystem                        ', ptruint(@TPEB(nil^).ImageSubsystem                        ));
95.   WriteOffset('ImageSubsystemMajorVersion            ', ptruint(@TPEB(nil^).ImageSubsystemMajorVersion            ));
96.   WriteOffset('ImageSubsystemMinorVersion            ', ptruint(@TPEB(nil^).ImageSubsystemMinorVersion            ));
97.  
98.   writeln;
99.   WriteOffset('UnionJ                                ', ptruint(@TPEB(nil^).UnionJ                                ));
100.   WriteOffset('UnionJ.ImageProcessAffinityMask       ', ptruint(@TPEB(nil^).UnionJ.ImageProcessAffinityMask       ));
101.   WriteOffset('UnionJ.ActiveProcessAffinityMask      ', ptruint(@TPEB(nil^).UnionJ.ActiveProcessAffinityMask      ));
102.  
103.   writeln;
104.   {$ifdef WIN32}
105.   WriteOffset('GdiHandleBuffer                       ', ptruint(@TPEB(nil^).GdiHandleBuffer                       ));
106.   {$endif}
107.  
108.   {$ifdef WIN64}
109.   WriteOffset('GdiHandleBuffer                       ', ptruint(@TPEB(nil^).GdiHandleBuffer                       ));
110.   {$endif}
111.   writeln;
112.   WriteOffset('PostProcessInitRoutine                ', ptruint(@TPEB(nil^).PostProcessInitRoutine                ));
113.  
114.   { ------------------------------------------------------------------------- }
115.   { fields appended for Window 2000 and above                                 }
116.  
117.   writeln;
118.   WriteOffset('TlsExpansionBitmap                    ', ptruint(@TPEB(nil^).TlsExpansionBitmap                    ));
119.   WriteOffset('TlsExpansionBitmapBits                ', ptruint(@TPEB(nil^).TlsExpansionBitmapBits                ));
120.   writeln;
121.   WriteOffset('SessionId                             ', ptruint(@TPEB(nil^).SessionId                             ));
122.   writeln;
123.   WriteOffset('AppCompatFlags                        ', ptruint(@TPEB(nil^).AppCompatFlags                        ));
124.   WriteOffset('AppCompatFlagsUser                    ', ptruint(@TPEB(nil^).AppCompatFlagsUser                    ));
125.   WriteOffset('pShimData                             ', ptruint(@TPEB(nil^).pShimData                             ));
126.   WriteOffset('AppCompatInfo                         ', ptruint(@TPEB(nil^).AppCompatInfo                         ));
127.   WriteOffset('CSDVersion                            ', ptruint(@TPEB(nil^).CSDVersion                            ));
128.  
129.  
130.   { ------------------------------------------------------------------------- }
131.   { fields appended for Window XP and above                                   }
132.  
133.   writeln;
134.   WriteOffset('ActivationContextData                 ', ptruint(@TPEB(nil^).ActivationContextData                 ));
135.   WriteOffset('ProcessAssemblyStorageMap             ', ptruint(@TPEB(nil^).ProcessAssemblyStorageMap             ));
136.   WriteOffset('SystemDefaultActivationContextData    ', ptruint(@TPEB(nil^).SystemDefaultActivationContextData    ));
137.   WriteOffset('SystemAssemblyStorageMap              ', ptruint(@TPEB(nil^).SystemAssemblyStorageMap              ));
138.   writeln;
139.   WriteOffset('MinimumStackCommit                    ', ptruint(@TPEB(nil^).MinimumStackCommit                    ));
140.  
141.  
142.   { ------------------------------------------------------------------------- }
143.   { fields appended for Windows server 2003 and above                         }
144.  
145.   writeln;
146.   WriteOffset('UnionK                                ', ptruint(@TPEB(nil^).UnionK                                ));
147.   WriteOffset('UnionK.FlsCallback                    ', ptruint(@TPEB(nil^).UnionK.FlsCallback                    ));
148.   WriteOffset('UnionK.FlsListHead                    ', ptruint(@TPEB(nil^).UnionK.FlsListHead                    ));
149.   WriteOffset('UnionK.FlsBitmap                      ', ptruint(@TPEB(nil^).UnionK.FlsBitmap                      ));
150.   WriteOffset('UnionK.FlsBitmapBits                  ', ptruint(@TPEB(nil^).UnionK.FlsBitmapBits                  ));
151.   WriteOffset('UnionK.FlsHighIndex                   ', ptruint(@TPEB(nil^).UnionK.FlsHighIndex                   ));
152.   writeln;
153.   WriteOffset('UnionK.SparePointers                  ', ptruint(@TPEB(nil^).UnionK.SparePointers                  ));
154.   WriteOffset('UnionK.SpareUlongs                    ', ptruint(@TPEB(nil^).UnionK.SpareUlongs                    ));
155.  
156.  
157.   { ------------------------------------------------------------------------- }
158.   { fields appended for Windows Vista and above                               }
159.  
160.   writeln;
161.   WriteOffset('WerRegistrationData                   ', ptruint(@TPEB(nil^).WerRegistrationData                   ));
162.   WriteOffset('WerShipAssertPtr                      ', ptruint(@TPEB(nil^).WerShipAssertPtr                      ));
163.  
164.  
165.   { ------------------------------------------------------------------------- }
166.   { fields appended for Windows 7 beta and up                                 }
167.  
168.   writeln;
169.   WriteOffset('UnionL                                ', ptruint(@TPEB(nil^).UnionL                                ));
170.   WriteOffset('UnionL.pContextData                   ', ptruint(@TPEB(nil^).UnionL.pContextData                   ));
171.   WriteOffset('UnionL.pUnused                        ', ptruint(@TPEB(nil^).UnionL.pUnused                        ));
172.   writeln;
173.   WriteOffset('pImageHeaderHash                      ', ptruint(@TPEB(nil^).pImageHeaderHash                      ));
174.  
175.  
176.   { ------------------------------------------------------------------------- }
177.   { fields appended for Windows 7 RTM and above                               }
178.  
179.   writeln;
180.   WriteOffset('TracingFlagsUnion                     ', ptruint(@TPEB(nil^).TracingFlagsUnion                     ));
181.   WriteOffset('TracingFlagsUnion.TracingFlags        ', ptruint(@TPEB(nil^).TracingFlagsUnion.TracingFlags        ));
182.   {$ifdef FPC}
183.   WriteOffset('TracingFlagsUnion.BitFlags            ', ptruint(@TPEB(nil^).TracingFlagsUnion.BitFlags            ));
184.   {$endif}
185.   writeln;
186.   WriteOffset('AlignmentFiller                       ', ptruint(@TPEB(nil^).AlignmentFiller                       ));
187.  
188.  
189.   { ------------------------------------------------------------------------- }
190.   { fields appended for Windows 8 and above                                   }
191.  
192.   writeln;
193.   WriteOffset('CsrServerReadOnlySharedMemoryBase     ', ptruint(@TPEB(nil^).CsrServerReadOnlySharedMemoryBase     ));
194.  
195.  
196.   { ------------------------------------------------------------------------- }
197.   { fields appended for Windows 10 and above                                  }
198.  
199.   writeln;
200.   WriteOffset('TppWorkerpListLock                    ', ptruint(@TPEB(nil^).TppWorkerpListLock                    ));
201.  
202.   writeln;
203.   WriteOffset('UnionM                                ', ptruint(@TPEB(nil^).UnionM                                ));
204.   WriteOffset('UnionM.TppWorkerpList                 ', ptruint(@TPEB(nil^).UnionM.TppWorkerpList                 ));
205.   WriteOffset('UnionM.dwSystemCallMode               ', ptruint(@TPEB(nil^).UnionM.dwSystemCallMode               ));
206.   writeln;
207.   WriteOffset('WaitOnAddressHashTable                ', ptruint(@TPEB(nil^).WaitOnAddressHashTable                ));
208.   writeln;
209.   WriteOffset('TelemetryCoverageHeader               ', ptruint(@TPEB(nil^).TelemetryCoverageHeader               ));
210.   WriteOffset('CloudFileFlags                        ', ptruint(@TPEB(nil^).CloudFileFlags                        ));
211.   writeln;
212.   WriteOffset('CloudFileDiagFlags                    ', ptruint(@TPEB(nil^).CloudFileDiagFlags                    ));
213.   WriteOffset('PlaceholderCompatibilityMode          ', ptruint(@TPEB(nil^).PlaceholderCompatibilityMode          ));
214.   WriteOffset('PlaceholderCompatibilityModeReserved  ', ptruint(@TPEB(nil^).PlaceholderCompatibilityModeReserved  ));
215.   writeln;
216.   WriteOffset('LeapSecondData                        ', ptruint(@TPEB(nil^).LeapSecondData                        ));
217.   writeln;
218.   WriteOffset('UnionN                                ', ptruint(@TPEB(nil^).UnionN                                ));
219.   WriteOffset('UnionN.LeapSecondFlags                ', ptruint(@TPEB(nil^).UnionN.LeapSecondFlags                ));
220.   {$ifdef FPC}
221.   WriteOffset('UnionN.BitFields                      ', ptruint(@TPEB(nil^).UnionN.BitFields                      ));
222.   {$endif}
223.   writeln;
224.   WriteOffset('NtGlobalFlag2                         ', ptruint(@TPEB(nil^).NtGlobalFlag2                         ));
225.  
226.   writeln;
227.   writeln;
228.   writeln('press ENTER/RETURN to end this program');
229.   readln;
230. end.

Output of above program for 32bit and 64bit:

Code: Text  [Select][+][-]

1.  
2.   32bit PEB - sizeof(PEB) = $480  1152d
3.  
4.   InheritedAddressSpace                    :    $0      0
5.   ReadImageFileExecOptions                 :    $1      1
6.   BeingDebugged                            :    $2      2
7.  
8.   UnionA                                   :    $3      3
9.   UnionA.SpareBool                         :    $3      3
10.   UnionA.Bitfields                         :    $3      3
11.   UnionA.BitFlags                          :    $3      3
12.  
13.   Mutant                                   :    $4      4
14.  
15.   < lots of additional fields >
16.  
17.   CloudFileDiagFlags                       :  $464   1124
18.   PlaceholderCompatibilityMode             :  $468   1128
19.   PlaceholderCompatibilityModeReserved     :  $469   1129
20.  
21.   LeapSecondData                           :  $470   1136
22.  
23.   UnionN                                   :  $474   1140
24.   UnionN.LeapSecondFlags                   :  $474   1140
25.   UnionN.BitFields                         :  $474   1140
26.  
27.   NtGlobalFlag2                            :  $478   1144
28.  
29.  
30. press ENTER/RETURN to end this program
31.  

Code: Text  [Select][+][-]

1.  
2.   64bit PEB - sizeof(PEB) = $7C8  1992d
3.  
4.   InheritedAddressSpace                    :    $0      0
5.   ReadImageFileExecOptions                 :    $1      1
6.   BeingDebugged                            :    $2      2
7.  
8.   UnionA                                   :    $3      3
9.   UnionA.SpareBool                         :    $3      3
10.   UnionA.Bitfields                         :    $3      3
11.   UnionA.BitFlags                          :    $3      3
12.  
13.   Mutant                                   :    $8      8
14.  
15.   < lots of additional fields >
16.  
17.   CloudFileDiagFlags                       :  $7AC   1964
18.   PlaceholderCompatibilityMode             :  $7B0   1968
19.   PlaceholderCompatibilityModeReserved     :  $7B1   1969
20.  
21.   LeapSecondData                           :  $7B8   1976
22.  
23.   UnionN                                   :  $7C0   1984
24.   UnionN.LeapSecondFlags                   :  $7C0   1984
25.   UnionN.BitFields                         :  $7C0   1984
26.  
27.   NtGlobalFlag2                            :  $7C4   1988
28.  
29.  
30. press ENTER/RETURN to end this program
31.  

I have an equivalent program written in C that produces the same output, that way ensuring the offsets are correct is a simple matter of comparing the corresponding files.

Also, the structures are verified to be compatible with FPC and Delphi 2.0 (which makes the verification requirements similar to yours using Kylix.)

NOTE: had to trim the program and its output to make it be less than 20,000 characters which is the limit the forum software imposes.

HTH.

[Чебурашка]

Thank you all guys!