Array of structure -> structure of arrays

[LemonParty]

Is there a code that allow to do pointed transformation automatically? I mean some tricky generic code.

[Zvoni]

Is there a code that allow to do pointed transformation automatically? I mean some tricky generic code.

Not sure if i understood it correctly.

You have an Array of Structures/Records.
This could be read as
"The record represents the Columns, the Array the Rows"

So you're basically looking for "Transposing" (Column To Rows and vice versa)?

Don't think this exists, because it implies there being something like a "Record-Factory"

[Thaddy]

Or do you mean something like the Windows World transform API? Yes, that is also available on Linux, but I have to think deep for an example.
(WWT is what you use when writing CAD applications on Windows and is standard since win 3.0)

If my guess is correct I will post some code for Windows.

[LemonParty]

Transformation should work like this:

Code: Pascal  [Select][+][-]

1. type
2.   TRec = record
3.     D: DWord;
4.     B: Byte;
5.   end;
6.  
7.   TRecArrays = record
8.     PD: PDWord;
9.     PB: PByte;
10.   end;
11. var
12.   i: Integer;
13.   Arr: array of TRec;
14.   RArr: TRecArrays;
15. begin
16.   RArr.PD:= GetMem(Length(Arr) * SizeOf(DWord));
17.   RArr.PB:= GetMem(Length(Arr) * SizeOf(Byte));
18.   for i:= 0 to High(Arr) do begin
19.     RArr.PD[i]:= Arr[i].D;
20.     RArr.PB[i]:= Arr[i].B;
21.   end;
22. end.

As you can see in this code I done it by hand, but I am interesting if there is an automatic way to do that.

[jcmontherock]

Find an extract of an appli I used with StringGrid:

Code: Pascal  [Select][+][-]

1. type
2.   TRecord = Record
3.     drInteger:    Integer;
4.     drFloat:      Double;
5.     drDateTime:   TDateTime;
6.     drString:     String[255];
7.   end;
8.  
9.   TRecordsArray = Array of TRecord;
10. ........
11.   DataRecord:   TRecord;
12.   RecordsArray: TRecordsArray;
13. .......
14.   RecordsArray := TRecordsArray.Create;
15.   SetLength(RecordsArray, 0);
16. ........
17.     for iRow := 0 to RowCount - 1 do begin
18.       DataRecord.drInteger    := iRow;
19.       DataRecord.drFloat      := Random(100000) / 25;
20.       DataRecord.drDateTime   := DTCur;
21.       DataRecord.drString     := 'We don''t use ' + IntToStr(iRow) + ' Embarcadééro';
22.       System.Insert(DataRecord, RecordsArray, Length(RecordsArray));
23.     end;
24.     SaveAll('MyRecords.dat');
25. ...
26. procedure TForm1.SaveAll(const aFilename: String);
27. var
28.   fs: TfileStream;
29. begin
30.   try
31.     fs := TfileStream.Create(aFilename, fmCreate);
32.     fs.WriteBuffer(Pointer(RecordsArray)^, SizeOf(TRecord) * (Length(RecordsArray)));
33.   finally
34.     fs.Free;
35.   end;
36.   Memo1.Append('');
37.   Memo1.Append('File  "' + aFilename + '"  has been successfully saved !...');
38.   Memo1.Append('-----------------------------------------------------------');
39. end;
40. ......................
41. procedure TForm1.ReadAll(const aFilename: String; Var Records: TRecordsArray);
42. var
43.   fs: TfileStream;
44. begin
45.   try
46.     fs := TfileStream.Create(aFilename, fmopenRead);
47.     SetLength(Records, Round(fs.Size / SizeOf(Trecord)));
48.     fs.Read(Pointer(Records)^, fs.Size);
49.   finally
50.     fs.Free;
51.   end;
52.   Memo1.Append('');
53.   Memo1.Append((Length(Records)).ToString + ' entries from file  "' + aFilename + '"  has been successfully restored !...');
54.   Memo1.Append('-----------------------------------------------------------------------------');
55. end;
56.  

[Martin_fr]

I don't really know the answer...

There may be with the extended RTTI. But then you need to add the RTTI info. And it might be slow (mind that any simple transformation will run into cache misses too, if you have huge data. That would need to be addressed separate).

Without  RTTI, you can maybe write some pointer based helpers.

If the source and dest don't have any additional pointers in the data.... (e.g. no fields of class instance, were you would first have to follow the pointer to the instance data of each class)

Code: Pascal  [Select][+][-]

1. procedure DoMap(ADest, ASource: Pointer; ACount, ADestSize, ASourceSize: integer);
2. begin
3.   for i := 0 to Acount do begin
4.     move(ASource, ADest, ADestSize);
5.     inc(ASource, ASourceSize):
6.     inc(ADest, ADestSize);
7.   end;
8. end;
9.  
10. //////////////// for each field...
11. DoMap(@ADest[0], ASrc[0].FieldFoo, length(ASrc), sizeof(ADest[0] {must be same as FieldFoo}, sizeof(ASrc[0]);
12.  

[MathMan]

I don't really know the answer...

There may be with the extended RTTI. But then you need to add the RTTI info. And it might be slow (mind that any simple transformation will run into cache misses too, if you have huge data. That would need to be addressed separate).

Without  RTTI, you can maybe write some pointer based helpers.

If the source and dest don't have any additional pointers in the data.... (e.g. no fields of class instance, were you would first have to follow the pointer to the instance data of each class)

Code: Pascal  [Select][+][-]

1. procedure DoMap(ADest, ASource: Pointer; ACount, ADestSize, ASourceSize: integer);
2. begin
3.   for i := 0 to Acount do begin
4.     move(ASource, ADest, ADestSize);
5.     inc(ASource, ASourceSize):
6.     inc(ADest, ADestSize);
7.   end;
8. end;
9.  
10. //////////////// for each field...
11. DoMap(@ADest[0], ASrc[0].FieldFoo, length(ASrc), sizeof(ADest[0] {must be same as FieldFoo}, sizeof(ASrc[0]);
12.  

Hm - I would rather break this down to individual handling per structure. Agreed that this may end as pointer chasing on the read side, if the structures are randomly distributed over the heap. But on the write side you'd have n (number of elements in the structure) linear writes - which is much more cache friendly. In addition there will only be one traversal through the array of structure. And the intermediate total memory requirement stays 2*k*SizeOf( structure ) anyway.

Edit:
I missed the array of structure aspect - so the pointer chasing in the above is moot. Regarding RTTI - one can easily do without I think. From the offset of each field in structure one can derive the size of each field. As the majority will be in 1, 2, 4 or 8 the transfer can be done without Move speeding things up again.

[Jorg3000]

Hi!
There's also something special in Pascal: >> array of const << what is a flexible array of variant records ( TVarRec ).

Code: Pascal  [Select][+][-]

1. procedure DumpArgs(const Args: array of const);
2. var
3.   I: Integer;
4. begin
5.   for I := Low(Args) to High(Args) do
6.   begin
7.     case Args[I].VType of
8.       vtInteger: Writeln('Integer: ', Args[I].VInteger);
9.  
10.       vtBoolean: Writeln('Boolean: ', BoolToStr(Args[I].VBoolean, True));
11.  
12.       vtChar: Writeln('Char: ', Args[I].VChar);
13.  
14.       vtExtended: Writeln('Float: ', Args[I].VExtended^);
15.  
16.       vtString: Writeln('ShortString: ', Args[I].VString^);
17.  
18.       vtAnsiString:  Writeln('AnsiString: ', AnsiString(Args[I].VAnsiString));
19.  
20.       vtUnicodeString:  Writeln('UnicodeString: ', UnicodeString(Args[I].VUnicodeString));
21.  
22.       vtPChar:  Writeln('PChar: ', Args[I].VPChar);
23.  
24.       vtObject: Writeln('Object: ', Args[I].VObject.ClassName);
25.  
26.     else  Writeln('Unknown type: ', Args[I].VType);
27.     end;
28.   end;
29. end;
30.  
31. begin
32.   DumpArgs(['Hello', 42, True, 3.14]);
33. end.
34.  

[jamie]

Trying to comprehend the request and this is what I get out of it.

It looks like the request is to shorten the syntax of transferring the values in a single := which would mean an Operator := (.....) can be applied in the records (Advanced) records so that the background assignments can be handled and all you need is a simple := from one type of record to another.

So I think maybe Operator := overloading is what's being looked for here.


But then again, I was born in HickVill so who am I

Jamie

[Warfley]

If I understand this correctly, the question is basically how to build a zip and unzip function as exists for example in Haskell.

The short answer is: you can't. This level of meta programming is not possible in pascal. There are no tuples or anonymous unpacking of records. Ive actually made a merge request with which this would probably be possible using generics. But without something like that you can't, unless under very specific circumstances like specific names for the record fields

[runewalsh]

@Warfley, you in fact can. Record RTTI describes all fields, including unmanaged ones, and it can be used to do what the OP wants. Not that it *should* be used, nor that the problem is remotely plausible.

[Warfley]

Well you can do so in theory at runtime, but you dont have any type checks or other compiler support for what you are doing.

But I don't agree that the problem is not plausible, in functional languages like Haskell, Ocaml, etc. Zip and unzip are two of the most basic functions. It may not make as much sense with arrays due to the overhead of creating new arrays and copying everything over, so in non functional languages like Java you usually see such functionality be implemented on streams or iterators not arrays directly. But the problem itself is very plausible and such functions can be really useful. Imagine you get information from two sources, e.g. two files and you have a function that needs to combine both records at the same time. You could write one monster function that parses two files simultaneously, or you could write two small functions one for each file and zip their results together

[runewalsh]

I'm not saying the zip() function is useless — I'm saying AoS ↔ SoA conversions are useless. SoA is a fundamental choice about designing your data structure for efficiency, and conversions will eat up that efficiency.

The author may rather need something like:

Code: Pascal  [Select][+][-]

1. {$modeswitch advancedrecords}
2. type
3.         PSoA = ^TSoA;
4.  
5.         TRecRef = record
6.         private
7.                 soa: PSoA;
8.                 index: SizeUint;
9.                 function GetV32: uint32;
10.                 procedure SetV32(value: uint32);
11.                 function GetVB: byte;
12.                 procedure SetVB(value: byte);
13.         public
14.                 property v32: uint32 read GetV32 write SetV32;
15.                 property vb: byte read GetVB write SetVB;
16.         end;
17.  
18.         TSoA = record
19.         private
20.                 function GetRecRef(index: SizeUint): TRecRef;
21.         public
22.                 v32: array[0 .. 99] of uint32;
23.                 vb: array[0 .. 99] of byte;
24.                 property Rec[index: SizeUint]: TRecRef read GetRecRef; default;
25.         end;
26.  
27.         function TRecRef.GetV32: uint32;
28.         begin
29.                 result := soa^.v32[index];
30.         end;
31.  
32.         procedure TRecRef.SetV32(value: uint32);
33.         begin
34.                 soa^.v32[index] := value;
35.         end;
36.  
37.         function TRecRef.GetVB: byte;
38.         begin
39.                 result := soa^.vb[index];
40.         end;
41.  
42.         procedure TRecRef.SetVB(value: byte);
43.         begin
44.                 soa^.vb[index] := value;
45.         end;
46.  
47.         function TSoA.GetRecRef(index: SizeUint): TRecRef;
48.         begin
49.                 result.soa := @self;
50.                 result.index := index;
51.         end;
52.  
53. var
54.         soa: TSoA;
55.         i: SizeInt;
56.  
57. begin
58.         soa[3].v32 := 12345;
59.         soa[3].vb := 99;
60.         soa[4].v32 := 54321;
61.         soa[4].vb := 100;
62.         write('soa.v32: (');
63.         for i := 0 to 9 do write(soa.v32[i], ', ');
64.         writeln('...)');
65.         write('soa.vb: (');
66.         for i := 0 to 9 do write(soa.vb[i], ', ');
67.         writeln('...)');
68. end.

soa.v32: (0, 0, 0, 12345, 54321, 0, 0, 0, 0, 0, ...)
soa.vb: (0, 0, 0, 99, 100, 0, 0, 0, 0, 0, ...)

[LemonParty]

runewalsh, no, I actually want a SoA <-> AoS convertation.
Thank you for answers.

[Thaddy]

I think after all that what you mean is a pivot table structure. (Like Excel)
There is some code for that around for Delphi, old code, so translatable.