C-specific record functionality in FPC

[ALLIGATOR]

Moderator Note: Split from Feature announcement: Function References and Anonymous Functions based on this post by 440bx:

Most people won't even notice any bugs.

That alone is an indication of a significant problem.

I cannot even port the Windows API definitions without running into FPC bugs and deficiencies.    IOW, it doesn't even take writing code to run into problems, just declarations/definitions does it.

How about that for a "new feature" ?... being able to faithfully and correctly port the Windows API from C to FPC.  Personally, I think that would be a great "new feature" (and a rather useful one too.)

without running into FPC bugs

Could you give a small example, please?
Perhaps there are workarounds

[440bx]

I just want to note that FPC 3.2.2 and 3.2.4 also have bugs, just like FPC[main]

And it's still a big question where there are more of them

Yes, my post above is about v3.2.2. 

I have little to no idea what the situation is with v3.2.4 and FPC main. I'd like to see (really, just hoping) some of those bugs addressed in v3.2.4.

without running into FPC bugs

Could you give a small example, please?
Perhaps there are workarounds

Sure.

The first example is when defining bit fields in a record.

In 32 bit, if the field has more than 32 bits then it simply cannot be defined.

In 64 bit, there are a number of cases where the definition ends up being wrong.  In some cases, if the number of bits add up to 64, the compiler will not accept the definition.  In other cases, it will accept the definition but, either the offset of the next field (following the bit field) is incorrect and/or the total size of the record is incorrect.  This requires a number of acrobatics to get the definition to be somewhat usable, i.e, the correct size and correct field offsets.

I recently reported this bug here in the forums and in the bug tracker.

Another case, also involving bit fields is when the C definition uses arithmetic expression to specify the number of bits a particular field uses.  Since FPC uses ranges to specify the number of bits, using expressions is not possible.  In this case, it takes a manual conversion to set the sizes but, the definition, unlike the expression, isn't scaleable.   Fortunately, there are very few definition that use bit field expressions.

Another problem and that one is common enough to be a real problem is when the record/structure needs to be aligned on a boundary greater than 8.  The same problem if any one of the fields in the record/structure needs to be aligned on a boundary greater than 8.  The field alignment requires manually calculating the padding and declaring the padding fields.  Must not forget that the alignment of those fields also affects the total size of the structure, IOW, there is a "trailing" alignment field required and that one is far from obvious.  Aligning the record at a boundary greater than 8 is simply beyond FPC's abilities.  Therefore that one requires the code to _create_ blocks of memory that are aligned as required.

Those are just some examples I've recently ran into.  They are far from the only problems I've run into when porting the Windows API from C to FPC.

[LeP]

@440bx
I'm not convinced that Pascal should try to replicate everything C, C++, C#, Rust, GO, etc. do at the language level -> therefore compiler.
This is a serious mistake, in my opinion. C, like other languages, goes its own way with its own statements, and the Pascal compiler shouldn't be the one to resolve any inconsistencies between languages.
I've always viewed the implementation of C features in Pascal as a bad thing (actually, very bad), and they've never really been completed.
I think such situations should be resolved with standard procedures (even in ASM) or, at the very least, by creating external interfaces with DLLs.
In Delphi, there's the HPPEMIT directive (and others) that help build any "interfaces" in C or C++ (for those with Rad Studio) directly from Delphi. The beauty is that if something changes in Delphi, and the C DLL project is included in the group, the DLL will also be built automatically, ensuring that it is always maintained in line with the project.
In FPC, this can be done manually if necessary (without the automatic process), with the use of external tool like gcc or mingw but I believe this should be done.
Pascal cannot bind to C, or any other language at the compiler level.

[marcov]

In Delphi, there's the HPPEMIT directive (and others) that help build any "interfaces" in C or C++ (for those with Rad Studio) directly from Delphi. The beauty is that if something changes in Delphi, and the C DLL project is included in the group, the DLL will also be built automatically, ensuring that it is always maintained in line with the project.

Afaik a lot of that only works with BCB and/or windows (many compilers have similar object layout on Windows due to COM requirements)

[LeP]

In Delphi, there's the HPPEMIT directive (and others) that help build any "interfaces" in C or C++ (for those with Rad Studio) directly from Delphi. The beauty is that if something changes in Delphi, and the C DLL project is included in the group, the DLL will also be built automatically, ensuring that it is always maintained in line with the project.

Afaik a lot of that only works with BCB and/or windows (many compilers have similar object layout on Windows due to COM requirements)

This is true, but 'cause Delphi is multiplatform and BCB not.

OT
COM is not involved in this and the objects between Delphi and BCB can be shared without any COM intervention.

[marcov]

COM is not involved in this and the objects between Delphi and BCB can be shared without any COM intervention.

My point is that to be able to implement COM on top of them, you need a certain layout. I know some people (Rudi Velthuis) used visual studio classes from delphi with careful hacks based on similarities of object models.

But the point is that those are chosen pairings, simply because of $HPEMIT you can't pair random Pascal and C++ compilers.

[LeP]

COM is not involved in this and the objects between Delphi and BCB can be shared without any COM intervention.

My point is that to be able to implement COM on top of them, you need a certain layout. I know some people (Rudi Velthuis) used visual studio classes from delphi with careful hacks based on similarities of object models.
But the point is that those are chosen pairings, simply because of $HPEMIT you can't pair random Pascal and C++ compilers.

I wrote HELP to .... of course a simple directive cannot do all the works. But if I had somthing that i Pascal cannot be done, simply I do it with C and export some functions, or others things. In C one can read for example the result of an API (like in bit, nibble, or ....) and export the correct structure in Pascal without any effort in change the compiler.
I Delphi this is relative simple (of course under Windows) 'cause the integration between Delhp and BCB, but the same can be obtained in FPC with a little more work and some external tools.

[440bx]

@440bx
I'm not convinced that Pascal should try to replicate everything C, C++, C#, Rust, GO, etc. do at the language level -> therefore compiler.

When I state that FPC should have every feature C has, I don't mean they should be implemented the same way.  As far as implementation goes, C's features are often very poorly implemented even though the feature itself is good.

For instance, in C it is _trivial_ to align a structure and fields within a structure.   In FPC, if the alignment isn't to an 8 byte or less boundary, it is a nightmare and, it it is greater than 8, you are out of luck (as far as the structure's alignment itself.  the fields you need to manually align and the structure size also manually adjusted.  It's simply baroque.)

Another example is declaring and specifying bit fields.  In FPC, it's unbelievably poorly implemented and on top of that, it's buggy.   It should be done the obvious way, by numerically specifying the number of bits (or by using a constant expression) and certainly not using a range.

C has "inline" variable declarations which Delphi has poorly "shoe-horned" into Pascal.  The feature is extremely useful but, it is very poorly implemented in C (no surprise there) and even more poorly in Delphi's implementation.

C has a very good, very well thought out feature set.  Unfortunately, that feature set is probably as poorly implemented as humanly possible.  I sometimes joke that the missing link is still missing because it doesn't want to be associated with the C language's atrociously poor syntactic design.

I hope the above sheds some light that when I say C's features, I really mean the features not their implementation in any way (on the contrary, the C implementation is usually poor, even though the bit field one isn't as bad as it could be and certainly much better than FPC's but, that's an exception.)

[LeP]

@440bx
I think everything you've reported has a single cause: replicating C functionality in Pascal (as you mentioned... the functionality, not its exact implementation).
Why do you need to know how the compiler performs alignments, obviously taking into account that the compiler always executes them the same way... even if in a "strange" way...?
If the compiler guarantees access to the correct data, what problems are there?
Problems arise when you need to replicate C functionality, perhaps with direct access to data via pointers and pointer math (which in my opinion should be limited).
Honestly, I've never had these problems, because if I need to use the functionality typical of other languages... I simply use other languages.
But it's right that if the FPC/Lazarus teams have decided to implement this, it should be done.
And here comes the problem that has been reported in numerous other topics: who is responsible for this? And in how much time? And why would anyone do it in a day, a month, or a year, or a decade?

[440bx]

If the compiler guarantees access to the correct data, what problems are there?

That's the problem, it does not.

When alignment is on a boundary greater than 8, the FPC guarantees nothing! (actually, it almost always guarantees the definition is wrong.)  The programmer has to _manually_ calculate all the padding between fields, padding for structure size and write code to ensure the structure is properly aligned.

I keep referring to alignment because that's the most recent problem I've run into but, it is far from the only one.

Also, it isn't about other languages.    The fact is that C has become the "lingua franca" of programming.  Because of this, any language, not just FPC, that does not have all the features present in C is going to be a source of headaches simply because C's features are found in most code/definitions found out there.

[LeP]

Also, it isn't about other languages.    The fact is that C has become the "lingua franca" of programming.  Because of this, any language, not just FPC, that does not have all the features present in C is going to be a source of headaches simply because C's features are found in most code/definitions found out there.

We cannot reinvent the wheel: if C has the wheel and Pascal not, simply we must use the wheel of C. Reinvent the wheel in Pascal, in Rust, in C# in .... it's a no sense.

If it's possible to replicate to wheel of C in Pascal in simple way or may be in complex way too (without involving the compiler) we can do of course. But if this it is a possible generator of bugs ... I'm still with my idea to use the original wheel.

And if the compiler doesn't make the correct alignements ... this is a serious problem. I think this is the first problem in absolute. Any issues of the compiler should have the priority.

[440bx]

We cannot reinvent the wheel:

It's not about re-inventing the wheel, it's about making a _better_ wheel.

Any issues of the compiler should have the priority.

I can agree with that but, that does not mean making Pascal a better wheel than C shouldn't be a goal, particularly considering that C is now the standard wheel in most low level programming and, there is plenty of possible improvements in that wheel.