Recent

Author Topic: Zero Sized Array at end of Records, can this test code be verified ?  (Read 16070 times)

PascalDragon

  • Hero Member
  • *****
  • Posts: 5444
  • Compiler Developer
Re: Zero Sized Array at end of Records, can this test code be verified ?
« Reply #15 on: December 03, 2021, 04:16:54 pm »
This all works fine with delphi, it's an old project.
Code: Pascal  [Select][+][-]
  1. procedure TForm1.Button1Click(Sender: TObject);
  2. Var
  3.   T:PMyIntBaseRecord;
  4. begin
  5.   T := GetMem(SizeOf(T)+SizeOf(integer)); {Create an item plus one integer at end}
  6.   T^.IntCount := $FFFFFFFF;
  7.   T^.Ints[0]:= 1;
  8.   Caption := T^.IntCount.ToHexString+','+T^.ints[0].Tostring;
  9.   FreeMem(T);
  10. end;
  11.  

I assume you mean SizeOf(TMyIntBaseRecord) + SizeOf(Integer) instead of SizeOf(P) as otherwise you might get a nasty surprise on 32-bit systems ;)

It is a quite neat trick if it is legal. (that self of a nested record points to the nested record, not the encompassing one)

 I would hope it would be legal, I've doing such tricks for some time now and the SELF. It would be broken if not pointing to the immediate record when referenced which btw is also part of the parent record that over hangs at the end. That is the intent here.

It is legal. And rather elegant I have to say compared to some other “hacks”.

I've found another solution that does not throw warnings and requires less code:

If one writes the record once as a generic one can easily reuse it (and maybe we should even add this to the RTL... :-X ):

Code: Pascal  [Select][+][-]
  1. program trecarr;
  2.  
  3. {$mode objfpc}
  4. {$modeswitch advancedrecords}
  5.  
  6. uses
  7.   SysUtils;
  8.  
  9. type
  10.   generic TEndOfRecArray<T> = packed record
  11.   private type
  12.     PT = ^T;
  13.   private
  14.     function GetItem(aIndex: Integer): T; inline;
  15.     procedure SetItem(aIndex: Integer; aValue: T); inline;
  16.   public
  17.     property Items[Index: Integer]: T read GetItem write SetItem; default;
  18.   end;
  19.  
  20. function TEndOfRecArray.GetItem(aIndex: Integer): T;
  21. begin
  22.   Result := PT(@Self)[aIndex];
  23. end;
  24.  
  25. procedure TEndOfRecArray.SetItem(aIndex: Integer; aValue: T);
  26. begin
  27.   PT(@Self)[aIndex] := aValue;
  28. end;
  29.  
  30. type
  31.   TMyIntBaseRecord = packed record
  32.     IntCount: LongWord;
  33.     Ints: specialize TEndOfRecArray<LongInt>;
  34.   end;
  35.   PMyIntBaseRecord = ^TMyIntBaseRecord;
  36.  
  37. var
  38.   p: PMyIntBaseRecord;
  39. begin
  40.   p := GetMem(SizeOf(TMyIntBaseRecord) + SizeOf(LongInt));
  41.   p^.IntCount := $FFFFFFFF;
  42.   p^.Ints[0] := 1;
  43.   Writeln(p^.IntCount.ToHexString, ', ', p^.Ints[0].ToString);
  44.   FreeMem(p);
  45. end.

But compiling this let's the FPC enter an infinite loop with 100% CPU usage so. This is probably a bug in the FPC

Would you please report that? :)

I have some doubts about the sizeof(self)  (always pointer size =4/8 or nested record size=0?)
This is actually quite a good question. The pointer size does not really matter but the record size does. I think that with packed it should always be 0 but I am not sure if this is the case and/or is documented somewhere.

It should be 0, cause when using SizeOf from the outside returns the same, but for some reason it's 1... %)

One exception I could think of is, when more advanced RTTI comes, that records might get hidden RTTI fields added if enabled. But TBH I don't know how the plans are for RTTI in advanced records

The RTTI is per type, not per instance of that type, thus there is nothing related to RTTI stored in the record itself.

jamie

  • Hero Member
  • *****
  • Posts: 6077
Re: Zero Sized Array at end of Records, can this test code be verified ?
« Reply #16 on: December 03, 2021, 04:57:27 pm »
Quote
I assume you mean SizeOf(TMyIntBaseRecord) + SizeOf(Integer) instead of SizeOf(P) as otherwise you might get a nasty surprise on 32-bit systems ;)


Yes that is what I meant, That was a slipup in the Test demo because I started from a non pointer type reccord but you see what the intent is.

Good catch.   :)
The only true wisdom is knowing you know nothing

jamie

  • Hero Member
  • *****
  • Posts: 6077
Re: Zero Sized Array at end of Records, can this test code be verified ?
« Reply #17 on: December 03, 2021, 05:40:41 pm »
You guys with your GENERICS !  >:D
The only true wisdom is knowing you know nothing

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 1313
Re: Zero Sized Array at end of Records, can this test code be verified ?
« Reply #18 on: December 03, 2021, 05:57:24 pm »
I don't get it. Isn't that how lists and such work? At least in Delphi, they were declared like this:

Code: Pascal  [Select][+][-]
  1. type
  2.   TSomeList: array[0..MaxInt - 1] of SomeType;    

Or just a pointer to the first element. And code to manage the size.

As far as I get it, you want to do the same, but manage it yourself? Or is this to have the memory footprint of a C-struct:

Code: C  [Select][+][-]
  1. struct SomeArray {
  2.     int Length;
  3.     int* Elements;
  4. };

Like Pascal strings/arrays?

jamie

  • Hero Member
  • *****
  • Posts: 6077
Re: Zero Sized Array at end of Records, can this test code be verified ?
« Reply #19 on: December 03, 2021, 06:29:31 pm »
That would generate a large record field that isn't empty..

The idea is to have a define without substance so the record can be used as a header to a chuck of data that follows without actually spanning into that region without memory.

 Your example would cause some issues even if you were use a Getmem call and subtracting that field size.

 This is to reproduce what other languages do with having zero sized arrays at the end of the struct or at any place for that matter.

  this generates a place holder of a memory address and in this case a hang over.

 Using your method you would need to subract that field when dynamically creating the record and if using it as a static record or one on the heap will just flood the pool.
The only true wisdom is knowing you know nothing

Warfley

  • Hero Member
  • *****
  • Posts: 1499
Re: Zero Sized Array at end of Records, can this test code be verified ?
« Reply #20 on: December 03, 2021, 06:35:43 pm »
The difference between this and a pointer to a memory location is that the list is directly tied to the control structure in the same memory region.
For example, in the DNS protocoll, you have a header followed by the DNS records in a memory list. This could be directly implemented with such a datastructure, without requiring any additional control data and code, which can be extremly helpful if you need to process such information at a high performance (e.g. when implementing your own DNS server)

What can be done alternatively is the following:
Code: Pascal  [Select][+][-]
  1. PMyRecord = ^TMyRecord;
  2. TMyRecord = record
  3.   // Header...
  4.   Items: Array[0..SizeInt.MaxValue] of Integer;
  5. end;
  6.  
  7. p: PMyRecord;
  8. begin
  9.   p := GetMem(HeaderSize + Itemsize);
  10.   ...
  11. end;
But this is kinda inconvinient, as you need to compute the Headersize by yourself, while with the datastructures shown here, you can simply use sizeof and let the compiler do the work. Especially if the header might be changed in the future, sizeof will always return the true size, while in the other case you need to update every instance where you compute the size
« Last Edit: December 03, 2021, 06:38:08 pm by Warfley »

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 1313
Re: Zero Sized Array at end of Records, can this test code be verified ?
« Reply #21 on: December 03, 2021, 06:46:44 pm »
Yes, that's what you put inside a class, to hold the actual data.

Code: Pascal  [Select][+][-]
  1. type
  2.   TItem = Integer;
  3.   PItems = ^TItems;
  4.   TItems = Array[0..MaxInt - 1] of TItem;
  5.  
  6.   { TItemList }
  7.  
  8.   TItemList = class
  9.   private
  10.     FItems: TItems;
  11.     ItemCount: Integer;
  12.     ItemsSize: Integer;
  13.     function GetItem(Index: Integer): TItem;
  14.     procedure SetItem(Index: Integer; AValue: TItem);
  15.     procedure SetItems(AValue: TItems);
  16.   public
  17.     constructor Create;
  18.     destructor Destroy;
  19.     function Add(TItem): Integer;
  20.     property Count: Integer read ItemCount;
  21.     property Items: TItems read FItems write SetItems;
  22.     property Item[Index: Integer]: TItem read GetItem write SetItem; default;
  23.   end;

Better example (x3).
« Last Edit: December 03, 2021, 07:28:03 pm by SymbolicFrank »

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 1313
Re: Zero Sized Array at end of Records, can this test code be verified ?
« Reply #22 on: December 03, 2021, 07:22:53 pm »
Ok, now the example is better. The property Items is only included because it can be done, not that you should do that (it defeats the purpose of the class).

Warfley

  • Hero Member
  • *****
  • Posts: 1499
Re: Zero Sized Array at end of Records, can this test code be verified ?
« Reply #23 on: December 03, 2021, 08:22:46 pm »
It should be noted that this can run into problems with Initialize.

So whenever raw memory is allocated through GetMem (or malloc if the c memory manager is used), before using it it must be initialized. The reason for this is to perform all the neccessary management operations on managed types, which since the introduction of management operators, could be any record now.

Consider the following (using string as an example here as managed type):
Code: Pascal  [Select][+][-]
  1. type
  2.   PTestRec = ^TTestRec;
  3.   TTestRec = record
  4.     Name: String;
  5.     Items: array[0..MaxInt] of String;
  6.   end;
  7.  
  8. var
  9.   p: PTestRec;
  10. begin
  11.   p := GetMem(1024);
  12.   Initialize(p^);
  13. end.
The initialize will look at the type TTestRec and see it is a normal record, so it will initialize every single element of that record. When considering Items it sees MaxInt many Strings, so it tries to initialize them. But the allocated memory only covers 1024/sizeof(String) strings, meaning it will cause the initialize to overflow on the allocated memory, and in this example most likely cause a segfault.
But initialize is defenetly needed, as otherwise both the name and the items are not usable

When considering a type with a function or [] property, it can be done like this very easiely and safe:
Code: Pascal  [Select][+][-]
  1. type
  2.   PTestRec = ^TTestRec;
  3.  
  4.   TTestRec = record
  5.     Name: String;
  6.     function Items: PString;
  7.     procedure Destroy;
  8.  
  9.     class function Alloc(Count: SizeInt): PTestRec; static;
  10.   end;
  11.  
  12. function TTestRec.Items: PString;
  13. begin
  14.   Result := PString(Pointer(@self) + SizeOf(Self));
  15. end;
  16.  
  17. procedure TTestRec.Destroy(Count: SizeInt);
  18. var
  19.   i: SizeInt;
  20. begin
  21.   for i:=0 to Count-1 do
  22.     Finalize(Items[i]);
  23.   Finalize(Self);
  24. end;
  25.  
  26. class function TTestRec.Alloc(Count: SizeInt): PTestRec;
  27. var
  28.   i: SizeInt;
  29. begin
  30.   Result := GetMem(SizeOf(TTestRec) + Count * SizeOf(String));
  31.   Initialize(Result^);
  32.   for i:=0 to Count-1 do
  33.     Initialize(Result^.Items[i]);
  34. end;
  35.  
  36. var
  37.   p: PTestRec;
  38. begin
  39.   p := TTestRec.Alloc(1024);
  40.   try
  41.  
  42.   finally
  43.     p^.Destroy(1024);
  44.     Freemem(p);
  45.   end;
  46. end.
« Last Edit: December 03, 2021, 08:25:21 pm by Warfley »

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 1313
Re: Zero Sized Array at end of Records, can this test code be verified ?
« Reply #24 on: December 03, 2021, 09:15:31 pm »
Well, yes. But that's why you normally double the size of the array when it gets full and simply copy everything to the new one. And why you're not using pointers, but the array. You're not going to initialize memory for each element. If you want that, use a linked list instead.

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11351
  • FPC developer.
Re: Zero Sized Array at end of Records, can this test code be verified ?
« Reply #25 on: December 03, 2021, 09:36:52 pm »
If one writes the record once as a generic one can easily reuse it (and maybe we should even add this to the RTL... :-X ):

That was what I thinking too, not in the RTL, but in my own root record. (Records because they are opengl vertex buffers)

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 1313
Re: Zero Sized Array at end of Records, can this test code be verified ?
« Reply #26 on: December 04, 2021, 01:11:53 am »
If one writes the record once as a generic one can easily reuse it (and maybe we should even add this to the RTL... :-X ):

That was what I thinking too, not in the RTL, but in my own root record. (Records because they are opengl vertex buffers)
Why? Could you explain?

jamie

  • Hero Member
  • *****
  • Posts: 6077
Re: Zero Sized Array at end of Records, can this test code be verified ?
« Reply #27 on: December 04, 2021, 01:35:50 am »
Playing with the Generic's sample that Pascal posted it compiles etc but it does not INLINE.

do it without generics and it does inline.

I did this on 3.0.4 fpc so I don't know if higher levels have fixed this ?

For now I will stick with non generics because I don't want the calls for the setter and getter...
The only true wisdom is knowing you know nothing

jamie

  • Hero Member
  • *****
  • Posts: 6077
Re: Zero Sized Array at end of Records, can this test code be verified ?
« Reply #28 on: December 04, 2021, 02:44:52 am »
It seems that if I am using a GUI app and do that code within a Button Click event which is a procedure of OBJECT the generics version does not inline but not using GUI or maybe even not doing Method of Classes/Record calls embedded does work either way.

 Something about the compiler not happy with inlining when type is declared within an Object body.

 Maybe the compiler is not dealing with the background code that implies the SELF pointer etc.

 its strange this would work outside of method bodies though?


 Anyways, I know the non-generic version works both ways and will stick with it.

 Thanks.
The only true wisdom is knowing you know nothing

Warfley

  • Hero Member
  • *****
  • Posts: 1499
Re: Zero Sized Array at end of Records, can this test code be verified ?
« Reply #29 on: December 04, 2021, 03:51:32 am »
You're not going to initialize memory for each element. If you want that, use a linked list instead.
Initialize is always called on each element, just normally it is hidden by compiler magic. TList from Generics.Collections for example uses "array of T" and allocates the memory with Setlength, which will internally call Initialize for each element.

You *always* need to initialize the memory for each element. The question is just if it is done automatically, either through a constructor, New or SetLength, or if you need to do it manually which you need to do if you use raw allocation e.g. via getmem, stack allocation or, in case of microcontrollers and you don't have virtual memory, when performing direct memory access.

This can also be easiely tested check the memory consumption of the following two programs:
Code: Pascal  [Select][+][-]
  1. program Project1;
  2.  
  3. {$mode objfpc}{$H+}
  4.  
  5. type
  6.   TLargeArray = Array[1..1024*1024*1024 div SizeOf(String)] of String; // 1 GB memory
  7.   PLargeArray = ^TLargeArray;
  8. var
  9.   p: PLargeArray;
  10. begin
  11.   p := GetMem(SizeOf(TLargeArray));
  12.   ReadLn;
  13.   FreeMem(p);
  14. end.
  15.  
and
Code: Pascal  [Select][+][-]
  1. program Project1;
  2.  
  3. {$mode objfpc}{$H+}
  4.  
  5. type
  6.   TLargeArray = Array[1..1024*1024*1024 div SizeOf(String)] of String; // 1 GB memory
  7.   PLargeArray = ^TLargeArray;
  8. var
  9.   p: PLargeArray;
  10. begin
  11.   New(p);
  12.   ReadLn;
  13.   Dispose(p);
  14. end.
  15.  
The first program takes takes up no physical memory at all (only 0.6 MB when launched with debugger symbols), while the second one takes 1GB of memory.
The reason for that is that allocating virtual memory does by itself not use any physical memory, only once a page is touched it is loaded into actual memory. What happens in the second example that implicetly (as part of calling New), the code will iterate over every single element of the array and call initialize on it.
But the first program is not complete yet, you can't use any array element until you called Initialize on it.

As a simple rule of thumb: If you use GetMem, always call Initialize and if you use FreeMem always use Finalize.

PS: You can use this as an alternative to geometric growth, to allocate large chunks of virtual memory in advance but only touch the elements (and call Initialize on them) when you need them (for performance reasons this can also be done in a geometric fashion). If you know the maximum size of the list beforehand, and also have enough virtual memory available (2^48 bytes on most modern x86_64 CPUs) this can improve performance by avoiding multiple calls to the memory managers and also copy operations.

 

TinyPortal © 2005-2018