Recent

Author Topic: Arrays as parameters  (Read 7030 times)

jollytall

  • Sr. Member
  • ****
  • Posts: 306
Arrays as parameters
« on: February 17, 2020, 01:25:13 pm »
Apologies for the long Post. It sounds like a tutorial, but there are some issues, questions or potential bugs reported in it.

If I want to hand over a bunch of e.g. integers to a procedure, I have many ways. If I know in advance how many I will have, then I can list them (I1,I2:integer), or use a record (R:TRecord) or a static array (SA:TStaticArray). If I do not know the number of integers in advance, I am left with Dynamic arrays (https://www.freepascal.org/docs-html/ref/refsu14.html, https://wiki.freepascal.org/Dynamic_array) or Open arrays. So I can have
Code: Pascal  [Select][+][-]
  1. type
  2.   TDynamicArray=array of integer;
  3. procedure ProcDynamic(V:TDynamicArray);
  4. procedure ProcOpen(V:array of integer);
  5.  

The first question already comes up here. Why do we have the last option? (Before answering, please read on, I will also answer it later and challenge the answer as well). For all other structured types, there is no such an option to declare the type in the procedure header. How would it look to write:
Code: Pascal  [Select][+][-]
  1. procedure ProcWrong1(V:array[1..5] of integer);
  2. procedure ProcWrong2(V:record I1,I2:integer; end);
  3.  
Ok, in my 30+ years of Pascal it never came up as a need, but it is somewhat similar to an open array parameter. Actually, I do not mind if we have more options (both Open and Dynamic array parameters), but it has some anomalies.

To make it more precise I can call a procedure with three different parameter calling methods, two by pointer (const and var) and one by value. So I have 6 options:
Code: Pascal  [Select][+][-]
  1. procedure ProcDynamicConst( const V: TDynamicArray);
  2. procedure ProcDynamicVar(   var   V: TDynamicArray);
  3. procedure ProcDynamicValue(       V: TDynamicArray);
  4. procedure ProcOpenConst(    const V: array of integer);
  5. procedure ProcOpenVar(      var   V: array of integer);
  6. procedure ProcOpenValue(          V: array of integer);
  7.  

When I call the above procedures, I can do it two ways. Either call it with a Variable or with values (e.g. [5,10,15,20,25,30]);
From the 12 possible combinations, I understand that calling the Var versions is not possible with Values (ProcOpenVar([1,2]) is wrong). It is logical, that if it cannot hand back the value to a variable, you cannot call it. It is also consistent with e.g. trying to call Proc(var I:integer); with Proc(5) would fail as well.
I do not understand however, why I cannot call the other two of the Dynamic versions? (Small remark: technically the compiler it seems first checks this restriction and only after that the Var, so even for the ProcDynamicVar([5,10,15,20,25,30]) I get the compiler error "Error: Call by var for arg no. 1 has to match exactly: Got "{Array Of Const/Constant Open} Array of ShortInt" expected "TDynamicArray").
My issue (Question 1) with this: If calling ProcOpen versions with a Variable can be "converted" into an open array, then theoretically it could and should also be done the other way. One could say, that calling with values is different, because it is not a proper array, just a bunch of numbers, but it seems that under the hood the compiler makes some sort of array object from it, e.g. Length, High work just like for a "proper" Dynamic Array. The fact that Dynamic cannot be called with values, would explain why we needed the Open version, but as in the question, I do not see why it is needed to be like this. Furthermore, if we have two versions, those should work well, what is not always the case, as shown below:

My confusions starts when I look at the memory addresses.

In ProcDynamicVar everything works the way it should. Both V and V[0] points the place where MyArray and MyArray[0] points respectively. If only an element is changed, it is overwritten in the original memory area and so the Caller will see it. If the array structure is changed (Length increased) then it updates where V[0] is and again it is reflected in MyArray as well.
ProcDynamicConst is already somewhat wrong. When called, V gets a new pointer but V[0] points to MyArray[0]. Question 2: Why does const V get a new pointer? It seems wrong, as it means that the object has to be physically copied, even if the real data (the heavy memory load) is not. Then the compiler is careful enough not to allow the change of V being read-only (but again, then why a new pointer). But then the compiler is not careful enough, so the value of the array can be changed and since V[0] points the same place as MyArray[0], it actually overwrites a read-only parameter. Question 3: Is it a bug or a functionality. Somehow I understand that MyObject or V are objects and the compiler protects them, but not the memory area where they point to. Nonetheless I think it is a bug, especially in light of ProcDynamicValue.
The real problem is with ProcDynamicValue. When the method is called, then just like in case of ProcDynamicConst V gets a new pointer (a new object) pointing the old memory area. If now we change an element of the array it is still changed in the old memory area, and so the Caller gets it back. If then we change the Length of the array (here it is allowed), then a new physical memory is blocked for it, and V now points to that. If we change the value of an element of V again, it is changed in the copy and when the control is given back to the Caller, then it does not see it. I think this is totally wrong. It cannot be like that, that the Caller sees some of the changes, but not all. This also indicates why even in case of ProcDynamicConst I call it wrong. Question 4: Do you agree, it is a bug?

Things get even more strange when I look at the Open versions.
Unlike in Dynamic versions here the address of the array @V is always the same as the first element @V[0]. If V only points to the first element, how does it know the length, high, etc. It seems that internally there IS an object that points the payload to the first element, but then @V should give back the address of that (technical, temporary) object and not the address of V[0]. This causes problems later as well. Question 5: Is it a bug to not see the object as a whole? How does the compiler know e.g. Length(V) in this case, if there is no proper object behind?
Unlike the ProcDynamicConst version, the ProcOpenConst version does not allow changing even the value of an element of the array. It is good news, but raises the question, (Question 6) that if the compiler can check it for Open arrays, then why it could not check for Dynamic arrays.
On the other hand in ProcOpenVar and ProcOpenValue where the array is changeable (should be a local copy), it is not allowed to change the length of the array ("Type mismatch"), but it is allowed to change the value. I think it is a bug again. An array either can be changed (both length and values) or not. It should not be, that values can be changed, but not the length. Also, I do not understand the technical reason behind. When ProcOpenValue is called, it creates a sort-of copy of the original object, so why can't it create a new object if I want to change the size? Question 7: So why cannot change the length of the array in ProcOpenVar and ProcOpenValue?

Due to the @V=@V[0] in the Open versions, there is another consequence (this is where the problem originally started for me). I have a set-up like this (much more complex of course):
Code: Pascal  [Select][+][-]
  1. type TArr=array of integer;
  2. procedure A(p:pointer); // I need an untyped pointer, because different B-s, call it with integer array, real array, etc. and pass as a second parameter the type of it
  3. // or it could be A(const p:pointer) or A(var p:pointer) I would not mind
  4.   begin
  5.   writeln(length(TArr(p^)));
  6.   end;
  7. procedure B(V:TArr); // The parameter of the different B-s can be different, but inside they call the same A
  8. // or it could be B(const V:TArr) or B(var V:TArr)
  9.   begin
  10.   A(@V);
  11.   end;
  12. procedure C;
  13.   begin
  14.   B([1,2,3]); // I want to call it with actual values
  15.   end;
  16.  

This does not work, because B cannot be called by C.
To fix it I have to construct a TArr variable in C from the values and call B with it. This is quite some overhead.
Alternatively I can declare B as B(V:array of integer), but then the p pointer is wrong for A (not the address of an object but the address of its first element and so Length crashes), so I would need to create a proper Dynamic array inside B, and give its pointer to A. The same burden again.
Question 8
. Any better idea?

Can someone help with the above many questions? Thanks in advance.

The full code, to test this:
Code: Pascal  [Select][+][-]
  1. program arraytest;
  2.  
  3. uses
  4.   SysUtils;
  5.  
  6. const
  7.   Fmt='[0x%p]';
  8.  
  9. type
  10.   TDynamicArray=array of integer;
  11.   TStaticArray=array[10..15] of integer;
  12.  
  13. procedure ProcAddress(P:pointer);
  14.   begin
  15.   writeln(Format(Fmt,[P]));
  16.   end;
  17.  
  18. procedure ProcStatiConst(const V:TStaticArray);
  19.   begin
  20. //  V[13]:=0; // Compile error - OK
  21.   end;
  22. procedure ProcDynamicConst(const V:TDynamicArray);
  23.   begin
  24.   writeln('After the call: Array address ',Format(Fmt,[@V]),' Array[0] address ',Format(Fmt,[@(V[0])]),' - Why has V got a new address');
  25.   writeln('Length ',Length(V));
  26.   writeln('High ',High(V));
  27.   writeln('V[0] ',V[0]);
  28. //  SetLength(V,3); // Compile error - OK
  29.   V[0]:=1111; // Not an error - strange, but somewhat understandable
  30.   writeln('Before return: Array address ',Format(Fmt,[@V]),' Array[0] address ',Format(Fmt,[@(V[0])]),' - Value changed in V[0] pointing to the Caller''s area');
  31.   end;
  32. procedure ProcDynamicVar(var V:TDynamicArray);
  33.   begin
  34.   writeln('After the call: Array address ',Format(Fmt,[@V]),' Array[0] address ',Format(Fmt,[@(V[0])]),' - This is OK');
  35.   writeln('Length ',Length(V));
  36.   writeln('High ',High(V));
  37.   writeln('V[1] ',V[1]);
  38.   V[1]:=2222;
  39.   writeln('After changing V[1], but not the length: Array address ',Format(Fmt,[@V]),' Array[0] address ',Format(Fmt,[@(V[0])]),' - No change in the pointers when only a value changed, OK');
  40.   SetLength(V,16);
  41.   V[1]:=2345;
  42.   writeln('New length ',Length(V));
  43.   writeln('Before return (also the length is changed): Array address ',Format(Fmt,[@V]),' Array[0] address ',Format(Fmt,[@(V[0])]),' - New pointer of V[0] as the length also changed, OK');
  44.   end;
  45. procedure ProcDynamicValue(V:TDynamicArray);
  46.   begin
  47.   writeln('After the call: Array address ',Format(Fmt,[@V]),' Array[0] address ',Format(Fmt,[@(V[0])]),' - V has a new address, but V[0] is still the same');
  48.   writeln('Length ',Length(V));
  49.   writeln('High ',High(V));
  50.   writeln('V[2] ',V[2]);
  51.   V[2]:=3333; // This is updated in the original memory area
  52.   writeln('After changing V[2], but not the length: Array address ',Format(Fmt,[@V]),' Array[0] address ',Format(Fmt,[@(V[0])]),' - Just like for Var, no change in the pointers, so again it wrote to the Caller''s area');
  53.   SetLength(V,26);
  54.   V[2]:=3456; // This is updated in the new - local use only - memory area
  55.   writeln('New length ',Length(V));
  56.   writeln('Before return (also the Length is changed): Array address ',Format(Fmt,[@V]),' Array[0] address ',Format(Fmt,[@(V[0])]),' - Now the payload area is updated, but that is not handed back to Caller');
  57.   end;
  58. procedure ProcOpenConst(const V:array of integer);
  59.   begin
  60.   writeln('After the call: Array address ',Format(Fmt,[@V]),' Array[0] address ',Format(Fmt,[@(V[0])]),' - Now even V points to the same as V[0], not an "object"');
  61.   writeln('Length ',Length(V));
  62.   writeln('High ',High(V));
  63.   writeln('V[3] ',V[3]);
  64. //  SetLength(V,36); // Compile error - OK, with the expected error message
  65. //  V[3]:=4444; // Compile error - Here it notices that it is a const parameter, so why not above
  66.   writeln('Before return: Array address ',Format(Fmt,[@V]),' Array[0] address ',Format(Fmt,[@(V[0])]));
  67.   end;
  68. procedure ProcOpenVar(var V:array of integer);
  69.   begin
  70.   writeln('After the call: Array address ',Format(Fmt,[@V]),' Array[0] address ',Format(Fmt,[@(V[0])]),' - Again @V=@V[0]');
  71.   writeln('Length ',Length(V));
  72.   writeln('High ',High(V));
  73.   writeln('V[4] ',V[4]);
  74. //  SetLength(V,46); // Compile error with "Type mismatch"?
  75. //  writeln('New length ',Length(V));
  76.   V[4]:=5555;
  77.   writeln('Before return: Array address ',Format(Fmt,[@V]),' Array[0] address ',Format(Fmt,[@(V[0])]));
  78.   end;
  79. procedure ProcOpenValue(V:array of integer);
  80.   begin
  81.   writeln('After the call: Array address ',Format(Fmt,[@V]),' Array[0] address ',Format(Fmt,[@(V[0])]),' - Here even if called with a variable, a new area is allocated - OK');
  82.   writeln('Length ',Length(V));
  83.   writeln('High ',High(V));
  84.   writeln('V[5] ',V[5]);
  85. //  SetLength(V,56); // Compile error with "Type mismatch"?
  86. //  writeln('New length ',Length(V));
  87.   V[5]:=6666;
  88.   writeln('Before return: Array address ',Format(Fmt,[@V]),' Array[0] address ',Format(Fmt,[@(V[0])]));
  89.   end;
  90.  
  91. procedure Caller;
  92.   var
  93.     MyArray:TDynamicArray;
  94.     i:integer;
  95.   begin
  96.   SetLength(MyArray,6);
  97.   for i:=0 to 5 do
  98.     MyArray[i]:=i+11;
  99.  
  100.   writeln('Versions to call the 6 procedures with a Variable.');
  101.   writeln;
  102.   writeln('Calling ProcDynamicConst with MyArray address ',Format(Fmt,[@MyArray]),' MyArray[0] address ',Format(Fmt,[@MyArray[0]]));
  103.   ProcDynamicConst(MyArray);
  104.   writeln('The value is changed. It should not be allowed. ',MyArray[0]);
  105.   writeln;
  106.   writeln('Calling ProcDynamicVar with MyArray address ',Format(Fmt,[@MyArray]),' MyArray[0] address ',Format(Fmt,[@MyArray[0]]));
  107.   ProcDynamicVar(MyArray);
  108.   writeln('The value is changed. OK. ',MyArray[1]);
  109.   writeln;
  110.   writeln('Calling ProcDynamicValue with MyArray address ',Format(Fmt,[@MyArray]),' MyArray[0] address ',Format(Fmt,[@MyArray[0]]));
  111.   ProcDynamicValue(MyArray);
  112.   writeln('The value is changed as per the first change, but not as per the second. This is clearly wrong! ',MyArray[2]);
  113.   writeln;
  114.   writeln('Calling ProcOpenConst with MyArray address ',Format(Fmt,[@MyArray]),' MyArray[0] address ',Format(Fmt,[@MyArray[0]]));
  115.   ProcOpenConst(MyArray);
  116.   writeln('The value is not changed. Already prevented by the Compiler. OK.');
  117.   writeln;
  118.   writeln('Calling ProcOpenVar with MyArray address ',Format(Fmt,[@MyArray]),' MyArray[0] address ',Format(Fmt,[@MyArray[0]]));
  119.   ProcOpenVar(MyArray);
  120.   writeln('The value is changed, despite the strange pointer within ProcOpenVar. OK. ',MyArray[4]);
  121.   writeln;
  122.   writeln('Calling ProcOpenValue with MyArray address ',Format(Fmt,[@MyArray]),' MyArray[0] address ',Format(Fmt,[@MyArray[0]]));
  123.   ProcOpenValue(MyArray);
  124.   writeln('The value is not changed. OK. ',MyArray[5]);
  125.  
  126.   writeln;
  127.   writeln('Versions to call the 6 (actually only 2 of them allowed) procedures with Values');
  128. //  ProcDynamicConst([5,10,15,20,25,30]); // Compile error, not necessarily OK
  129. //  ProcDynamicVar([5,10,15,20,25,30]); // Compile error, would be OK with "Variable identifier expected", but here the same error is given as previously
  130. //  ProcDynamicValue([5,10,15,20,25,30]); // Compile error, not necessarily OK
  131.   writeln;
  132.   writeln('Calling ProcOpenConst');
  133.   ProcOpenConst([5,10,15,20,25,30]);
  134. //  ProcOpenVar([5,10,15,20,25,30]); // Compile error with the expected error message ("Variable identifier expected")
  135.   writeln;
  136.   writeln('Calling ProcOpenValue');
  137.   ProcOpenValue([5,10,15,20,25,30]);
  138.  
  139.   end;
  140.  
  141. begin
  142. Caller;
  143. end.
  144.  

lucamar

  • Hero Member
  • *****
  • Posts: 4219
Re: Arrays as parameters
« Reply #1 on: February 17, 2020, 03:16:24 pm »
I may be wrong but I think most differences arise from the fact that a dynamic array is a new type while an open array is considered kind of a a "calling convention" allowing (almost) any array to be passed to a procedure/function.

The corollary is that using variables declared of a dynamic array   type must obey the relevant type-checking rules which are somewhat different from those for open-arrays parameters.

I'm sure, though, that someone else can provide you with a more detailed explanation :-[
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus/FPC 2.0.8/3.0.4 & 2.0.12/3.2.0 - 32/64 bits on:
(K|L|X)Ubuntu 12..18, Windows XP, 7, 10 and various DOSes.

Thaddy

  • Hero Member
  • *****
  • Posts: 14202
  • Probably until I exterminate Putin.
Re: Arrays as parameters
« Reply #2 on: February 17, 2020, 03:59:31 pm »
Note dynamic arrays and open array parameters start always at index zero. Only static arrays can have a different index from zero.
Code: Pascal  [Select][+][-]
  1. // NEEDS FPC 3.2.0
  2. {$mode delphi}{$H+}
  3. type
  4.   Ta1 = array[15..19] of integer;
  5.   Ta2 = array of integer;
  6.  
  7.   procedure OpenArray(const a:array of integer);
  8.   var i:integer;
  9.   begin
  10.     for i := Low(a) to High(a) do
  11.       writeln(i:3,a[i]:4);
  12.   end;
  13.  
  14.   procedure DynArray(const a:Ta2);
  15.   var i:integer;
  16.   begin
  17.     for i := Low(a) to High(a) do
  18.       writeln(i:3,a[i]:4);
  19.   end;
  20.  
  21.   procedure StaticArray(const a:Ta1);
  22.   var i:integer;
  23.   begin
  24.     for i := Low(a) to High(a) do
  25.       writeln(i:3,a[i]:4);
  26.   end;
  27. var
  28.   arr1:Ta1 = (10,20,30,40,50);
  29.   arr2:Ta2 = [1,2,3,4,5];  
  30. begin
  31.  { Allowed, but note the index values count from zero, not 15 }
  32.  OpenArray(arr1);  
  33.  
  34.  { Allowed }
  35.  OpenArray(Arr2);
  36.  
  37.  { Not allowed, type must match exactly }
  38.  // DynArray(arr1); // not allowed
  39.  { Allowed, types match }
  40.  DynArray(arr2);
  41.  
  42.  { Not allowed even if the array has the proper amount of elements. Types must match }
  43.  // StaticArray(arr2); // not allowed
  44.  
  45.  { Allowed, types match exactly and the defined index is used too. }
  46.  StaticArray(Arr1);
  47. end.
This is all documented behavior.
https://freepascal.org/docs-html/ref/refsu14.html
https://freepascal.org/docs-html/ref/refsu68.html

So remember that if you pass a static array as an open array parameter its index changes internally in the procedure if the index does not start at zero.
The rest is pretty straightforward.

Does this explain it?
« Last Edit: February 17, 2020, 04:15:43 pm by Thaddy »
Specialize a type, not a var.

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11383
  • FPC developer.
Re: Arrays as parameters
« Reply #3 on: February 17, 2020, 04:30:34 pm »
I may be wrong but I think most differences arise from the fact that a dynamic array is a new type while an open array is considered kind of a a "calling convention" allowing (almost) any array to be passed to a procedure/function.

Yes, const open array is basically passing a descriptor record and a reference to first element of the data. If you pass something to a dynamic array, it must first be converted to the dynamic array type.

 

Thaddy

  • Hero Member
  • *****
  • Posts: 14202
  • Probably until I exterminate Putin.
Re: Arrays as parameters
« Reply #4 on: February 17, 2020, 04:55:26 pm »
I may be wrong but I think most differences arise from the fact that a dynamic array is a new type while an open array is considered kind of a a "calling convention" allowing (almost) any array to be passed to a procedure/function.

Yes, const open array is basically passing a descriptor record and a reference to first element of the data. If you pass something to a dynamic array, it must first be converted to the dynamic array type.
Actually, they are assignment compatible as this function shows. Note TIntegerArray is a predefined type
Code: Pascal  [Select][+][-]
  1.  
  2.   function Convert(const a: {open} array of Integer):{dynamic}TintegerArray;
  3.   begin
  4.     Result := a;
  5.   end;
This is also implicitly demonstrated in my previous example.
Another thing that follows is that you can convert a static array passed as open array to a dynamic array.
Just mind your index: it will be zero based, not range based.
« Last Edit: February 17, 2020, 06:22:01 pm by Thaddy »
Specialize a type, not a var.

jollytall

  • Sr. Member
  • ****
  • Posts: 306
Re: Arrays as parameters
« Reply #5 on: February 17, 2020, 06:31:00 pm »
Thanks for the answers, but I did not get much closer.

Does this explain it?
I know the fact that Open arrays and Dynamic arrays are zero based, while Static arrays can be indexed any way. I had no question related to this. Sorry if I was misunderstandable somewhere.

Yes, const open array is basically passing a descriptor record and a reference to first element of the data. If you pass something to a dynamic array, it must first be converted to the dynamic array type.
Somehow it is very similar to a Dynamic array object, as that also has some descriptive part and a reference to the first element of the data. So I still do not understand why not create a fully operational Dynamic array when an Open array is called.
Also Open array does a bit more than what you write. If it is an open array by value (as in my example ProcOpenValue), then the whole data part of the array is copied and the descriptor points to the new data area (so again, some sort of construction of an array).
So, while I understand that in the current set-up a Dynamic array parameter (even if not var) cannot be called by values, I still do not understand the deeper reason why it is designed like this.

Actually, they are assignment compatible as this function shows. Note TIntegerArray is a predefined type
I could not make your code run. It seems to me that the TIntegerArray is defined as a Static array (IntegerArray  = array[0..$effffff] of Integer;) and my program runtime crashes immediately (at the first begin) when such a variable is declared. (Maybe too big to have a 1GB array?)
I tried to make it replacing TIntegerArray with TDynamicArray from my example, but it fails at Compile time, with Incompatible types.

Anyway, all these above were related to my suggestion (Q1) how to make Open array and Dynamic arrays more alike. This still leaves all other questions, potential bugs still open.
Q2. Why do I see a new object in case of (const V:TDynamicArray) called with a variable. Even if the payload (the array itself) is not copied, why is the "container" object? I would expect that a const parameter is handing over the pointer just like var. Otherwise does it mean that var is faster than const.
Q3-Q6. To me the fact that in (const V:TDynamicArray) the value of V[n] can be changed, is not right (even if Length(V) cannot). For static arrays (const S:TStaticArray) and Open arrays (const O:array of integer) S[n]:=1 and O[n]:=1 both fail, then why is it allowed for dynamic arrays?
Q4. Similarly, in (V:TDynamicArray) if only V[n] is changed it goes back to the Caller, but if Length(V) is changed or any other V[m] is changed after the length is changed, it does not go back. This is almost unpredictable and so I strongly think it is a bug.
Q5. Would it be possible to get the pointer of the descriptor record mentioned by marcov when reading @V, instead of getting the same as @V[0]?
Q6. Another non-consistent use of the Open array, when inside the procedure called with (V:array of integer) or (var V:array of integer), the value of V[n] can be changed and - as it should - either it goes back to the Caller (var version) or remains local (value version), but the length of V cannot be changed. In my opinion both should be allowed (normal array once we are inside the procedure), or changing should not be allowed at all (then const, var or value would all be the same).

Thaddy

  • Hero Member
  • *****
  • Posts: 14202
  • Probably until I exterminate Putin.
Re: Arrays as parameters
« Reply #6 on: February 17, 2020, 06:54:21 pm »
Yes, I forgot about the TIntegerArray declaration being static. (strange)
The rest is OK.
Note I test against 3.2.0 and trunk, not 3.0.4.
Specialize a type, not a var.

avk

  • Hero Member
  • *****
  • Posts: 752
Re: Arrays as parameters
« Reply #7 on: February 17, 2020, 07:02:45 pm »
Q2.Why do I see a new object in case of (const V:TDynamicArray) called with a variable.

@V is the address of the local variable V.

Technically, a dynamic array is a pointer(it can be compared with nil, right?), which heavily involved compiler magic.
Accordingly, when you pass it as a const parameter, the compiler will not mind until you try to change it(but you can change the data it points to).
What does this pointer indicate and where are the remaining data of the dynamic array (length and reference count) stored?
Code: Pascal  [Select][+][-]
  1. program arrays;
  2.  
  3. {$mode objfpc}
  4.  
  5. uses
  6.   SysUtils;
  7.  
  8. type
  9.   TIntArray = array of Integer;
  10.  
  11. var
  12.   a: TIntArray = nil;
  13.  
  14. procedure Test1(const a: TIntArray);
  15. begin
  16.   a[1] := 5;
  17.   //SetLength(a, 10); // <- compile time error
  18. end;
  19.  
  20. procedure Test2;
  21. var
  22.   a, b: TIntArray;
  23.   c: Byte absolute a;
  24. begin
  25.   SetLength(a, 64);
  26.   WriteLn('address of a: ' + Format('0x%p', [@a]));
  27.   WriteLn('address of c: ' + Format('0x%p', [@c]));
  28.   WriteLn('Pointer(a): ' + Format('0x%p', [Pointer(a)]));
  29.   WriteLn('address of a[0]: ' + Format('0x%p', [@a[0]]));
  30.   WriteLn('data at address PSizeInt(a) - 1: ' + Format('%d', [(PSizeInt(a) - 1)^]));
  31.   WriteLn('data at address PSizeInt(a) - 2: ' + Format('%d', [(PSizeInt(a) - 2)^]));
  32.   b := a;
  33.   WriteLn('after b assignment data at address PSizeInt(a) - 2: ' + Format('%d', [(PSizeInt(a) - 2)^]));
  34. end;
  35.  
  36. begin
  37.   Test2;
  38. end.
  39.  
« Last Edit: February 17, 2020, 07:31:18 pm by avk »

jollytall

  • Sr. Member
  • ****
  • Posts: 306
Re: Arrays as parameters
« Reply #8 on: February 17, 2020, 08:32:53 pm »
Thanks avk, this is very useful.
If I understand you correctly, the system stores the reference count and the length always immediately before the data (strange though that data is both before and after the address where V points to, but as you say it is magic). So, if the data is resized (upwards) and thus need to be moved, then the two parameters move with it as well. So far I thought - I don't know why, - that the variable is the object (i.e. its address is the same as the first parameter in the record) and then after the parameters there is a last pointer showing where in the heap the data (V[0]) is. This is how I would create an object. I should have read the source code.

I also understand (this I guessed too) what you say related to Q3, why the data in the array can be changed even for a const parameter, but I still think it is wrong.
It is wrong design-wise: a Dynamic array should "copy" the functionality of a Static array as much as it is possible, and for static arrays you cannot change the value of an element in the array, just like you cannot do it either for an Open array.
It is wrong conceptually: an array either can be changed or cannot be changed. Allowing it to change a value, but not allowing it to change the shape is inconsistent behaviour.
But the biggest problem with allowing the called method (not called with "var") to write back into a memory area managed by the calling method  is in the bug I write in Q4. I cannot imagine a good reasoning (not explanation, because it is clear), why it can happen that lines
Code: Pascal  [Select][+][-]
  1. procedure Proc(V:TDynamicArray);
  2.   begin
  3.   V[0]:=200;
  4.   SetLength(V,Length(V+1));
  5.   V[0]:=300;
  6.   end.
  7. procedure A;
  8.   begin
  9.   U[0]:=100;
  10.   Proc(U);
  11.   writeln(U[0]);
  12.   end;
prints 200.

The pointer topic (Q2 and Q5) leaves me still with two points to understand:
Why does it make a new local variable (V) pointing to the V[0], if for a var type call V is the same absolute memory address as the calling variable? Couldn't it be the same for const as well? Why do we need a local variable?
What data is stored for an Open array parameter and where is that stored? Is it also only reference count and size? Are they right before V[0] just like the Dynamic array?

And last Q6: Open array parameters still show an inconsistent behavior when the value can be changed for var and value calls, but the shape can never be changed. Again, an array is either changeable or it is not. Here I think both should be allowed, but surely it should not be mixed.

ASerge

  • Hero Member
  • *****
  • Posts: 2222
Re: Arrays as parameters
« Reply #9 on: February 17, 2020, 09:14:01 pm »
Let's look at a small example:
Code: Pascal  [Select][+][-]
  1. {$APPTYPE CONSOLE}
  2. {$MODE OBJFPC}
  3.  
  4. procedure OpenArrProc(X: array of Integer);
  5. begin
  6.   X[0] := 0;
  7.   Writeln('OpenArrProc: ', X[0]);
  8. end;
  9.  
  10. procedure DynArrayProc(const X: TBoundArray);
  11. begin
  12.   X[0] := 0;
  13.   Writeln('DynArrayProc: ', X[0]);
  14. end;
  15.  
  16. var
  17.   A: TBoundArray;
  18.   B: array[1..5] of Integer = (1, 2, 3, 4, 5);
  19. begin
  20.   OpenArrProc([1, 2, 3, 4, 5]);
  21.   A := TBoundArray.Create(1, 2, 3, 4, 5);
  22.   OpenArrProc(A);
  23.   Writeln('A after OpenArrProc: ', A[0]);
  24.   OpenArrProc(B);
  25.   Writeln('B after OpenArrProc: ', B[1]);
  26.   DynArrayProc(A);
  27.   Writeln('A after DynArrayProc: ', A[0]);
  28.   Readln;
  29. end.
1. The OpenArrProc can accept both dynamic and static arrays, so the array length parameter is always passed separately in call.
2. If we declare X in the OpenArrProc with the const modifier, compiler will throw an error, because const cannot be changed.
3. Since the parameter is without the const modifier, a copy of the array is created inside the OpenArrProc, so although we change the value of the array element, the original array is not changed.
4. On the first call (line 20), the compiler creates an array to call on the fly, usually on the stack.
5. A dynamic array behaves like objects - it's just a pointer. And just like objects, you can change the internal content - this is normal, and the const modifier has to do with the object itself, not its content. Therefore, after calling in line 26, the array contents will be changed.
If I remember correctly, the open array parameter appeared first, and they were done classically, with const in mind, and copying non-const. And dynamic arrays came later, they already show the nature of objects and dynamic strings.

avk

  • Hero Member
  • *****
  • Posts: 752
Re: Arrays as parameters
« Reply #10 on: February 18, 2020, 07:17:22 am »
So, if the data is resized (upwards) and thus need to be moved, then the two parameters move with it as well.

No, why do you think so? Only the record fields will change.

Open array parameters still show an inconsistent behavior when the value can be changed for var and value calls, but the shape can never be changed. Again, an array is either changeable or it is not. Here I think both should be allowed, but surely it should not be mixed.
I believe the main reason is Delphi compatibility. But why it was implemented in Delphi in this way, I don't know who to ask.
Nevertheless, as @ASerge already mentioned, if you imagine a dynamic array as an object, then it is quite possible to live with it.
IMO the most universal way to pass an array as a parameter is an open array. It is also possible to pass only part of the array(AFAIK Delphi incompatible). 
Small example:
Code: Pascal  [Select][+][-]
  1. program arrays;
  2.  
  3. {$mode objfpc}
  4.  
  5. uses
  6.   SysUtils;
  7.  
  8. type
  9.   TIntArray = array of Integer;
  10.  
  11. procedure TestOpenArray(a: array of Integer);
  12. begin
  13.   WriteLn(a[0]);
  14.   WriteLn(Length(a));
  15. end;
  16.  
  17. var
  18.   a: TIntArray = (1, 2, 3, 4, 5, 6, 7, 8);
  19.  
  20. begin
  21.   TestOpenArray(a);
  22.   TestOpenArray(a[2..High(a)-1]);
  23. end.
  24.  
and implementation fragment:
Code: ASM  [Select][+][-]
  1. PASCALMAIN:
  2. ; [20] begin
  3.                 call    fpc_initializeunits
  4. ; [21] TestOpenArray(a);
  5.                 mov     eax,dword ptr [TC_$P$ARRAYS_$$_A]
  6.                 sub     eax,1
  7.                 jc      @@j11
  8.                 mov     eax,dword ptr [eax-3]
  9. @@j11:
  10.                 mov     edx,eax
  11.                 mov     eax,dword ptr [TC_$P$ARRAYS_$$_A]
  12.                 call    P$ARRAYS_$$_TESTOPENARRAY$array_of_LONGINT
  13. ; [22] TestOpenArray(a[2..High(a)-1]);
  14.                 mov     edx,dword ptr [TC_$P$ARRAYS_$$_A]
  15.                 sub     edx,1
  16.                 jc      @@j12
  17.                 mov     edx,dword ptr [edx-3]
  18. @@j12:
  19.                 sub     edx,3
  20.                 mov     eax,dword ptr [TC_$P$ARRAYS_$$_A]
  21. ; Peephole Optimization: Lea2Add done
  22.                 add     eax,8
  23.                 call    P$ARRAYS_$$_TESTOPENARRAY$array_of_LONGINT
  24. ; [23] end.
  25.                 call    fpc_do_exit
  26.                 ret
  27. _CODE           ENDS
  28.  

It seems to contain the answer to your question:
What data is stored for an Open array parameter and where is that stored?

Thaddy

  • Hero Member
  • *****
  • Posts: 14202
  • Probably until I exterminate Putin.
Re: Arrays as parameters
« Reply #11 on: February 18, 2020, 08:21:22 am »
IMO the most universal way to pass an array as a parameter is an open array. It is also possible to pass only part of the array(AFAIK Delphi incompatible). 
Yes, it is, I agree, but be aware of the change of index when passing a static array that is non-zero based, like array[15..19] of integer. This becomes in effect an array [0..4] of integer when passed to an open array as I demonstrated before.
Specialize a type, not a var.

jollytall

  • Sr. Member
  • ****
  • Posts: 306
Re: Arrays as parameters
« Reply #12 on: February 18, 2020, 09:28:48 am »
5. A dynamic array behaves like objects - it's just a pointer. And just like objects, you can change the internal content - this is normal, and the const modifier has to do with the object itself, not its content. Therefore, after calling in line 26, the array contents will be changed.
ASerge, Everything you write is true. I also understand why it works like this. What I questioned is why it is designed like this. It is against the logic that a Dynamic array should be like a variable length "normal" (i.e. static) array.
You do not cover though the case Proc(V:TDynamicArray), what is not only against this principle, but also, almost unpredictable. See my relevant small example in my last reply. I still strongly think that such a behavior should never happen.

No, why do you think so? Only the record fields will change.
I understood it from your reply (apparently wrong?). You wrote that the reference count and the length are on memory addresses directly before the 0th element of the array. So, the memory looks like [ref_count][length][V0][V1]etc. and that V being a pointer, points to the memory where V0 is. If I change the length of the array upwards, and the memory after V[last] is not available, the program moves the whole array to a new location. Then V will point to the new V0 and since ref_count and length is immediately before it, they will have to move as well.

IMO the most universal way to pass an array as a parameter is an open array.
In general I agree, but in my special case that does not work.
Imagine in your example, you want to call a new procedure Proc(P:pointer) from TestOpenArray. You would do Proc(@A),and in Proc you want to check the length of A. So far I cannot solve it, unless I pass a length parameter separately, what kills the standard interface of Proc (complicated enough in my real example).
If instead of your TestOpenArray I make something called TestDynamicArray, then I can safely call from it Proc(@A) and in Proc I can safely do writeln(Length(TDynamicArray(P^));
The only problem with this one, that TestDynamicArray cannot be called like TestDynamicArray([1,2,3]), so either I use your TestOpenArray and inside it I create a "real" Dynamic array (D), copy all data from A to D and then call Proc(@D), OR I use my TestDynamicArray and before calling it, my main program has to make an array and call TestDynamicArray with that.
AFAIK, in C++ you can easily solve it, as their - what we call - Open array and Dynamic array are the same thing.

It seems to contain the answer to your question:
I am sure it does, but honestly I am not good in Asm, so, I will have to study it a bit.

PascalDragon

  • Hero Member
  • *****
  • Posts: 5446
  • Compiler Developer
Re: Arrays as parameters
« Reply #13 on: February 18, 2020, 10:13:37 am »
My issue (Question 1) with this: If calling ProcOpen versions with a Variable can be "converted" into an open array, then theoretically it could and should also be done the other way. One could say, that calling with values is different, because it is not a proper array, just a bunch of numbers, but it seems that under the hood the compiler makes some sort of array object from it, e.g. Length, High work just like for a "proper" Dynamic Array. The fact that Dynamic cannot be called with values, would explain why we needed the Open version, but as in the question, I do not see why it is needed to be like this.

Pre-3.2.0 FPC does not support the following either, so why should it support it for parameters?

Code: Pascal  [Select][+][-]
  1. var
  2.   arr: array of LongInt;
  3. begin
  4.   arr := [1, 2, 3, 4];
  5. end.

With 3.2.0 array constructors for dynamic arrays are supported however and they also work for dynamic arrays.

In ProcDynamicVar everything works the way it should. Both V and V[0] points the place where MyArray and MyArray[0] points respectively. If only an element is changed, it is overwritten in the original memory area and so the Caller will see it. If the array structure is changed (Length increased) then it updates where V[0] is and again it is reflected in MyArray as well.
ProcDynamicConst is already somewhat wrong. When called, V gets a new pointer but V[0] points to MyArray[0]. Question 2: Why does const V get a new pointer? It seems wrong, as it means that the object has to be physically copied, even if the real data (the heavy memory load) is not.

You forget an important point about const: it's up to the compiler to decide how to pass it. With var, out or constref the compiler will always pass the pointer to the variable you passed, but with const there is no such guarantee. E.g. on i386 it will pass a pointer on the stack to the original array data. On other platforms that might be different again.

Then the compiler is careful enough not to allow the change of V being read-only (but again, then why a new pointer). But then the compiler is not careful enough, so the value of the array can be changed and since V[0] points the same place as MyArray[0], it actually overwrites a read-only parameter. Question 3: Is it a bug or a functionality. Somehow I understand that MyObject or V are objects and the compiler protects them, but not the memory area where they point to. Nonetheless I think it is a bug, especially in light of ProcDynamicValue.

This is simply a quirk of dynamic arrays and is retained for Delphi compatibility.

The real problem is with ProcDynamicValue. When the method is called, then just like in case of ProcDynamicConst V gets a new pointer (a new object) pointing the old memory area. If now we change an element of the array it is still changed in the old memory area, and so the Caller gets it back. If then we change the Length of the array (here it is allowed), then a new physical memory is blocked for it, and V now points to that. If we change the value of an element of V again, it is changed in the copy and when the control is given back to the Caller, then it does not see it. I think this is totally wrong. It cannot be like that, that the Caller sees some of the changes, but not all. This also indicates why even in case of ProcDynamicConst I call it wrong. Question 4: Do you agree, it is a bug?

This is simply how dynamic arrays behave. You'd see the same if you'd do this with two dynamic array variables in the same function:

Code: Pascal  [Select][+][-]
  1. var
  2.   a, b: array of LongInt;
  3. begin
  4.   a := [1, 2, 3, 4];
  5.   b := a;
  6.   b[2] := 42;
  7.   Writeln(a[2]);
  8.   SetLength(b, Length(b) + 1);
  9.   b[3] := 10;
  10.   Writeln(a[3]);
  11. end.

This will write

Code: [Select]
42
4

Unlike the managed string types (AnsiString and UnicodeString) dynamic arrays are not copy-on-write.

Things get even more strange when I look at the Open versions.
Unlike in Dynamic versions here the address of the array @V is always the same as the first element @V[0]. If V only points to the first element, how does it know the length, high, etc. It seems that internally there IS an object that points the payload to the first element, but then @V should give back the address of that (technical, temporary) object and not the address of V[0]. This causes problems later as well. Question 5: Is it a bug to not see the object as a whole? How does the compiler know e.g. Length(V) in this case, if there is no proper object behind?

The important difference from dynamic arrays to open arrays is that open arrays always point to the first element of the array and a High parameter is passed as a second parameter. Unlike with dynamic arrays you can also pass in static arrays or single elements for open array parameters (this should answer some of your further questions already).

Unlike the ProcDynamicConst version, the ProcOpenConst version does not allow changing even the value of an element of the array. It is good news, but raises the question, (Question 6) that if the compiler can check it for Open arrays, then why it could not check for Dynamic arrays.

As said above, Delphi compatibility.

On the other hand in ProcOpenVar and ProcOpenValue where the array is changeable (should be a local copy), it is not allowed to change the length of the array ("Type mismatch"), but it is allowed to change the value. I think it is a bug again. An array either can be changed (both length and values) or not. It should not be, that values can be changed, but not the length. Also, I do not understand the technical reason behind. When ProcOpenValue is called, it creates a sort-of copy of the original object, so why can't it create a new object if I want to change the size? Question 7: So why cannot change the length of the array in ProcOpenVar and ProcOpenValue?

Because open arrays can also take static arrays or single elements as values. The called function does not care where it stems from, it just gets a pointer to the first element and a high value. Thus it can change the contents (because it knows the location of the elements and the size), but never the size.

Due to the @V=@V[0] in the Open versions, there is another consequence (this is where the problem originally started for me). I have a set-up like this (much more complex of course):
Code: Pascal  [Select][+][-]
  1. type TArr=array of integer;
  2. procedure A(p:pointer); // I need an untyped pointer, because different B-s, call it with integer array, real array, etc. and pass as a second parameter the type of it
  3. // or it could be A(const p:pointer) or A(var p:pointer) I would not mind
  4.   begin
  5.   writeln(length(TArr(p^)));
  6.   end;
  7. procedure B(V:TArr); // The parameter of the different B-s can be different, but inside they call the same A
  8. // or it could be B(const V:TArr) or B(var V:TArr)
  9.   begin
  10.   A(@V);
  11.   end;
  12. procedure C;
  13.   begin
  14.   B([1,2,3]); // I want to call it with actual values
  15.   end;
  16.  

This does not work, because B cannot be called by C.
To fix it I have to construct a TArr variable in C from the values and call B with it. This is quite some overhead.

For 3.0.x you can use a Dynamic Array Constructor:

Code: Pascal  [Select][+][-]
  1. procedure C;
  2. begin
  3.   B(TArr.Create(1, 2, 3));
  4. end;

For 3.2.0 and newer you can use the normal array syntax (which also works for assignments):

Code: Pascal  [Select][+][-]
  1. procedure C;
  2. begin
  3.   B([1, 2, 3]);
  4. end;

Alternatively I can declare B as B(V:array of integer), but then the p pointer is wrong for A (not the address of an object but the address of its first element and so Length crashes), so I would need to create a proper Dynamic array inside B, and give its pointer to A. The same burden again.
Question 8
. Any better idea?

Dynamic and open arrays are simply different types, you can not handle them the same when you go down to the pointer level.

AFAIK, in C++ you can easily solve it, as their - what we call - Open array and Dynamic array are the same thing.

You are in so far right that in C++ there is no difference, because when passing an array you only pass a pointer. There is no dynamic array type that contains the size and you can't do any Length like operation on a passed array, because it is only a simple, stupid pointer. Thus when passing an array you always need to pass a size separately or have an array with a NULL element at the end.

avk

  • Hero Member
  • *****
  • Posts: 752
Re: Arrays as parameters
« Reply #14 on: February 18, 2020, 10:29:53 am »
I completely forgot to mention: as it seems, using modern FPC features, it is quite possible to create a type of dynamic array that will behave exactly as you need.

 

TinyPortal © 2005-2018