Apologies for the long Post. It sounds like a tutorial, but there are some issues, questions or potential bugs reported in it.
If I want to hand over a bunch of e.g. integers to a procedure, I have many ways. If I know in advance how many I will have, then I can list them (I1,I2:integer), or use a record (R:TRecord) or a static array (SA:TStaticArray). If I do not know the number of integers in advance, I am left with Dynamic arrays (
https://www.freepascal.org/docs-html/ref/refsu14.html,
https://wiki.freepascal.org/Dynamic_array) or Open arrays. So I can have
type
TDynamicArray=array of integer;
procedure ProcDynamic(V:TDynamicArray);
procedure ProcOpen(V:array of integer);
The first question already comes up here. Why do we have the last option? (Before answering, please read on, I will also answer it later and challenge the answer as well). For all other structured types, there is no such an option to declare the type in the procedure header. How would it look to write:
procedure ProcWrong1(V:array[1..5] of integer);
procedure ProcWrong2(V:record I1,I2:integer; end);
Ok, in my 30+ years of Pascal it never came up as a need, but it is somewhat similar to an open array parameter. Actually, I do not mind if we have more options (both Open and Dynamic array parameters), but it has some anomalies.
To make it more precise I can call a procedure with three different parameter calling methods, two by pointer (const and var) and one by value. So I have 6 options:
procedure ProcDynamicConst( const V: TDynamicArray);
procedure ProcDynamicVar( var V: TDynamicArray);
procedure ProcDynamicValue( V: TDynamicArray);
procedure ProcOpenConst( const V: array of integer);
procedure ProcOpenVar( var V: array of integer);
procedure ProcOpenValue( V: array of integer);
When I call the above procedures, I can do it two ways. Either call it with a Variable or with values (e.g. [5,10,15,20,25,30]);
From the 12 possible combinations, I understand that calling the Var versions is not possible with Values (ProcOpenVar([1,2]) is wrong). It is logical, that if it cannot hand back the value to a variable, you cannot call it. It is also consistent with e.g. trying to call Proc(var I:integer); with Proc(5) would fail as well.
I do not understand however, why I cannot call the other two of the Dynamic versions? (Small remark: technically the compiler it seems first checks this restriction and only after that the Var, so even for the ProcDynamicVar([5,10,15,20,25,30]) I get the compiler error "Error: Call by var for arg no. 1 has to match exactly: Got "{Array Of Const/Constant Open} Array of ShortInt" expected "TDynamicArray").
My issue (
Question 1) with this: If calling ProcOpen versions with a Variable can be "converted" into an open array, then theoretically it could and should also be done the other way. One could say, that calling with values is different, because it is not a proper array, just a bunch of numbers, but it seems that under the hood the compiler makes some sort of array object from it, e.g. Length, High work just like for a "proper" Dynamic Array. The fact that Dynamic cannot be called with values, would explain why we needed the Open version, but as in the question, I do not see why it is needed to be like this. Furthermore, if we have two versions, those should work well, what is not always the case, as shown below:
My confusions starts when I look at the memory addresses.
In ProcDynamicVar everything works the way it should. Both V and V[0] points the place where MyArray and MyArray[0] points respectively. If only an element is changed, it is overwritten in the original memory area and so the Caller will see it. If the array structure is changed (Length increased) then it updates where V[0] is and again it is reflected in MyArray as well.
ProcDynamicConst is already somewhat wrong. When called, V gets a new pointer but V[0] points to MyArray[0].
Question 2: Why does const V get a new pointer? It seems wrong, as it means that the object has to be physically copied, even if the real data (the heavy memory load) is not. Then the compiler is careful enough not to allow the change of V being read-only (but again, then why a new pointer). But then the compiler is not careful enough, so the value of the array can be changed and since V[0] points the same place as MyArray[0], it actually overwrites a read-only parameter.
Question 3: Is it a bug or a functionality. Somehow I understand that MyObject or V are objects and the compiler protects them, but not the memory area where they point to. Nonetheless I think it is a bug, especially in light of ProcDynamicValue.
The real problem is with ProcDynamicValue. When the method is called, then just like in case of ProcDynamicConst V gets a new pointer (a new object) pointing the old memory area. If now we change an element of the array it is still changed in the old memory area, and so the Caller gets it back. If then we change the Length of the array (here it is allowed), then a new physical memory is blocked for it, and V now points to that. If we change the value of an element of V again, it is changed in the copy and when the control is given back to the Caller, then it does not see it. I think this is totally wrong. It cannot be like that, that the Caller sees some of the changes, but not all. This also indicates why even in case of ProcDynamicConst I call it wrong.
Question 4: Do you agree, it is a bug?
Things get even more strange when I look at the Open versions.
Unlike in Dynamic versions here the address of the array @V is always the same as the first element @V[0]. If V only points to the first element, how does it know the length, high, etc. It seems that internally there IS an object that points the payload to the first element, but then @V should give back the address of that (technical, temporary) object and not the address of V[0]. This causes problems later as well.
Question 5: Is it a bug to not see the object as a whole? How does the compiler know e.g. Length(V) in this case, if there is no proper object behind?
Unlike the ProcDynamicConst version, the ProcOpenConst version does not allow changing even the value of an element of the array. It is good news, but raises the question, (
Question 6) that if the compiler can check it for Open arrays, then why it could not check for Dynamic arrays.
On the other hand in ProcOpenVar and ProcOpenValue where the array is changeable (should be a local copy), it is not allowed to change the length of the array ("Type mismatch"), but it is allowed to change the value. I think it is a bug again. An array either can be changed (both length and values) or not. It should not be, that values can be changed, but not the length. Also, I do not understand the technical reason behind. When ProcOpenValue is called, it creates a sort-of copy of the original object, so why can't it create a new object if I want to change the size?
Question 7: So why cannot change the length of the array in ProcOpenVar and ProcOpenValue?
Due to the @V=@V[0] in the Open versions, there is another consequence (this is where the problem originally started for me). I have a set-up like this (much more complex of course):
type TArr=array of integer;
procedure A(p:pointer); // I need an untyped pointer, because different B-s, call it with integer array, real array, etc. and pass as a second parameter the type of it
// or it could be A(const p:pointer) or A(var p:pointer) I would not mind
begin
writeln(length(TArr(p^)));
end;
procedure B(V:TArr); // The parameter of the different B-s can be different, but inside they call the same A
// or it could be B(const V:TArr) or B(var V:TArr)
begin
A(@V);
end;
procedure C;
begin
B([1,2,3]); // I want to call it with actual values
end;
This does not work, because B cannot be called by C.
To fix it I have to construct a TArr variable in C from the values and call B with it. This is quite some overhead.
Alternatively I can declare B as B(V:array of integer), but then the p pointer is wrong for A (not the address of an object but the address of its first element and so Length crashes), so I would need to create a proper Dynamic array inside B, and give its pointer to A. The same burden again.
Question 8. Any better idea?
Can someone help with the above many questions? Thanks in advance.
The full code, to test this:
program arraytest;
uses
SysUtils;
const
Fmt='[0x%p]';
type
TDynamicArray=array of integer;
TStaticArray=array[10..15] of integer;
procedure ProcAddress(P:pointer);
begin
writeln(Format(Fmt,[P]));
end;
procedure ProcStatiConst(const V:TStaticArray);
begin
// V[13]:=0; // Compile error - OK
end;
procedure ProcDynamicConst(const V:TDynamicArray);
begin
writeln('After the call: Array address ',Format(Fmt,[@V]),' Array[0] address ',Format(Fmt,[@(V[0])]),' - Why has V got a new address');
writeln('Length ',Length(V));
writeln('High ',High(V));
writeln('V[0] ',V[0]);
// SetLength(V,3); // Compile error - OK
V[0]:=1111; // Not an error - strange, but somewhat understandable
writeln('Before return: Array address ',Format(Fmt,[@V]),' Array[0] address ',Format(Fmt,[@(V[0])]),' - Value changed in V[0] pointing to the Caller''s area');
end;
procedure ProcDynamicVar(var V:TDynamicArray);
begin
writeln('After the call: Array address ',Format(Fmt,[@V]),' Array[0] address ',Format(Fmt,[@(V[0])]),' - This is OK');
writeln('Length ',Length(V));
writeln('High ',High(V));
writeln('V[1] ',V[1]);
V[1]:=2222;
writeln('After changing V[1], but not the length: Array address ',Format(Fmt,[@V]),' Array[0] address ',Format(Fmt,[@(V[0])]),' - No change in the pointers when only a value changed, OK');
SetLength(V,16);
V[1]:=2345;
writeln('New length ',Length(V));
writeln('Before return (also the length is changed): Array address ',Format(Fmt,[@V]),' Array[0] address ',Format(Fmt,[@(V[0])]),' - New pointer of V[0] as the length also changed, OK');
end;
procedure ProcDynamicValue(V:TDynamicArray);
begin
writeln('After the call: Array address ',Format(Fmt,[@V]),' Array[0] address ',Format(Fmt,[@(V[0])]),' - V has a new address, but V[0] is still the same');
writeln('Length ',Length(V));
writeln('High ',High(V));
writeln('V[2] ',V[2]);
V[2]:=3333; // This is updated in the original memory area
writeln('After changing V[2], but not the length: Array address ',Format(Fmt,[@V]),' Array[0] address ',Format(Fmt,[@(V[0])]),' - Just like for Var, no change in the pointers, so again it wrote to the Caller''s area');
SetLength(V,26);
V[2]:=3456; // This is updated in the new - local use only - memory area
writeln('New length ',Length(V));
writeln('Before return (also the Length is changed): Array address ',Format(Fmt,[@V]),' Array[0] address ',Format(Fmt,[@(V[0])]),' - Now the payload area is updated, but that is not handed back to Caller');
end;
procedure ProcOpenConst(const V:array of integer);
begin
writeln('After the call: Array address ',Format(Fmt,[@V]),' Array[0] address ',Format(Fmt,[@(V[0])]),' - Now even V points to the same as V[0], not an "object"');
writeln('Length ',Length(V));
writeln('High ',High(V));
writeln('V[3] ',V[3]);
// SetLength(V,36); // Compile error - OK, with the expected error message
// V[3]:=4444; // Compile error - Here it notices that it is a const parameter, so why not above
writeln('Before return: Array address ',Format(Fmt,[@V]),' Array[0] address ',Format(Fmt,[@(V[0])]));
end;
procedure ProcOpenVar(var V:array of integer);
begin
writeln('After the call: Array address ',Format(Fmt,[@V]),' Array[0] address ',Format(Fmt,[@(V[0])]),' - Again @V=@V[0]');
writeln('Length ',Length(V));
writeln('High ',High(V));
writeln('V[4] ',V[4]);
// SetLength(V,46); // Compile error with "Type mismatch"?
// writeln('New length ',Length(V));
V[4]:=5555;
writeln('Before return: Array address ',Format(Fmt,[@V]),' Array[0] address ',Format(Fmt,[@(V[0])]));
end;
procedure ProcOpenValue(V:array of integer);
begin
writeln('After the call: Array address ',Format(Fmt,[@V]),' Array[0] address ',Format(Fmt,[@(V[0])]),' - Here even if called with a variable, a new area is allocated - OK');
writeln('Length ',Length(V));
writeln('High ',High(V));
writeln('V[5] ',V[5]);
// SetLength(V,56); // Compile error with "Type mismatch"?
// writeln('New length ',Length(V));
V[5]:=6666;
writeln('Before return: Array address ',Format(Fmt,[@V]),' Array[0] address ',Format(Fmt,[@(V[0])]));
end;
procedure Caller;
var
MyArray:TDynamicArray;
i:integer;
begin
SetLength(MyArray,6);
for i:=0 to 5 do
MyArray[i]:=i+11;
writeln('Versions to call the 6 procedures with a Variable.');
writeln;
writeln('Calling ProcDynamicConst with MyArray address ',Format(Fmt,[@MyArray]),' MyArray[0] address ',Format(Fmt,[@MyArray[0]]));
ProcDynamicConst(MyArray);
writeln('The value is changed. It should not be allowed. ',MyArray[0]);
writeln;
writeln('Calling ProcDynamicVar with MyArray address ',Format(Fmt,[@MyArray]),' MyArray[0] address ',Format(Fmt,[@MyArray[0]]));
ProcDynamicVar(MyArray);
writeln('The value is changed. OK. ',MyArray[1]);
writeln;
writeln('Calling ProcDynamicValue with MyArray address ',Format(Fmt,[@MyArray]),' MyArray[0] address ',Format(Fmt,[@MyArray[0]]));
ProcDynamicValue(MyArray);
writeln('The value is changed as per the first change, but not as per the second. This is clearly wrong! ',MyArray[2]);
writeln;
writeln('Calling ProcOpenConst with MyArray address ',Format(Fmt,[@MyArray]),' MyArray[0] address ',Format(Fmt,[@MyArray[0]]));
ProcOpenConst(MyArray);
writeln('The value is not changed. Already prevented by the Compiler. OK.');
writeln;
writeln('Calling ProcOpenVar with MyArray address ',Format(Fmt,[@MyArray]),' MyArray[0] address ',Format(Fmt,[@MyArray[0]]));
ProcOpenVar(MyArray);
writeln('The value is changed, despite the strange pointer within ProcOpenVar. OK. ',MyArray[4]);
writeln;
writeln('Calling ProcOpenValue with MyArray address ',Format(Fmt,[@MyArray]),' MyArray[0] address ',Format(Fmt,[@MyArray[0]]));
ProcOpenValue(MyArray);
writeln('The value is not changed. OK. ',MyArray[5]);
writeln;
writeln('Versions to call the 6 (actually only 2 of them allowed) procedures with Values');
// ProcDynamicConst([5,10,15,20,25,30]); // Compile error, not necessarily OK
// ProcDynamicVar([5,10,15,20,25,30]); // Compile error, would be OK with "Variable identifier expected", but here the same error is given as previously
// ProcDynamicValue([5,10,15,20,25,30]); // Compile error, not necessarily OK
writeln;
writeln('Calling ProcOpenConst');
ProcOpenConst([5,10,15,20,25,30]);
// ProcOpenVar([5,10,15,20,25,30]); // Compile error with the expected error message ("Variable identifier expected")
writeln;
writeln('Calling ProcOpenValue');
ProcOpenValue([5,10,15,20,25,30]);
end;
begin
Caller;
end.