Brand new iMac i5 27" retina OS10.10.6 cannot create array

[Thaddy]

Whether int64 and qword are taken as ordinal seems to depend on 32 bit / 64 bit.

No it is not. The code compiles even on 32 bit arm. As Jonas pointed out this is an OS limitation?

[Noodly]

I think the TS's main intention is not focus on the use case, he/she just wanted to push FPC to its limits to see how far it can handle large memory or at least to make sure it can handle large memory.

There is an earlier related topic from the same user stating that they had problems running this on an older Mac and they were going to buy a new computer. I guess this is the follow up topic after the new computer arrived. It's not just a memory testing exercise, or at least it's a very expensive one if it is.

[summerleas]

First, my delay in replying is because I live in Australia and the time difference causes problems.

Jonas wrote "OS X does not support more than 4GB of statically allocated data, even on 64 bit platforms (probably for efficiency reasons)."

But this contradicts the Apple developer document. Also, it seems pointless to have 64 bit addressing if it cannot be used.

Handoko wrote "he/she just wanted to push FPC to its limits to see how far it can handle large memory or at least to make sure it can handle large memory." I have a real problem, but I am not going to write the program unless it will execute. So I gave a fragment to illustrate the problem. Noodly is correct, it is not just a memory test. I was and am not sure where this thread should be posted so I shifted it.

I have been trying something simpler to explain the problem. But the problem vanished!!!

Below is a test program that creates and writes a big array. Assuming fpc worked OK  this should create an array of about 9.6 GB; my iMac has 8 GB of RAM.

It compiled, ran and finished in about 7 seconds. I don't know how to check if it actually worked, so I have set all values to 0 and written the last value.

This test appears to show that 64-bit addressing is working. If so, why don't the declarations in the first post work?



Program TestArray;
uses ExecutionTimeUnit, sysutils;
var
   BigArray : array[1..1200000000] of int64; {about 9.6GB}
   i : int64;
Begin
StartClock;
for i :=1 to 1200000000 do
   BigArray := 0;
   BigArray := i;
writeln('Last number is ', BigArray[1200000000]);
StopClock;
end.

fpc -Px86_64 -FeDesktop/Senate-preferences/TestArrayDebug Desktop/Senate-preferences/TestArray.pas

Desktop/Senate-preferences/TestArray

Terminal session:

Richards-iMac:~ richardwatkins$ fpc -Px86_64 -FeDesktop/Senate-preferences/TestArrayDebug Desktop/Senate-preferences/TestArray.pas
Richards-iMac:~ richardwatkins$ Desktop/Senate-preferences/TestArray
Last number is 1200000000
Start time: 14 hours  17 min  19 sec  528 ms
End time: 14 hours  17 min  25 sec  290 ms
Execution time: 0 hours  0 min  6 sec  762 ms


PS: I tried the above with
   BigArray : array[1..2200000000] of int64;
It worked; took 14 seconds.

[Thaddy]

Ahum:

Code: Pascal  [Select][+][-]

1. Program TestArray;
2. uses ExecutionTimeUnit, sysutils;
3. var
4.    BigArray : array[1..1200000000] of int64; {about 9.6GB}
5.    i : int64;
6. Begin
7. StartClock;
8.    for i :=1 to 1200000000 do
9.        BigArray := 0; // this will loop, but is meaningless: always same location in memory
10.    BigArray := i;// this won't....
11. writeln('Last number is ', BigArray[1200000000]);
12. StopClock;
13. end.

Do you mean:

Code: Pascal  [Select][+][-]

1. Program TestArray;
2. uses ExecutionTimeUnit, sysutils;
3. var
4.    BigArray : array[1..1200000000] of int64; {about 9.6GB}
5.    i : int64;
6. Begin
7. StartClock;
8.  for i :=1 to 120000000 do // fill the array...
9.    BigArray[i] := i;
10. writeln('Last number is ', BigArray[1200000000]);
11. StopClock;
12. end.

That will bomb out with EOutOfMemory in your case.

[mischi]

No it is not. The code compiles even on 32 bit arm. As Jonas pointed out this is an OS limitation?

Interesting. My result with the crosssompiler for arm-linux is:

[Hal:~/Desktop] michael% ppcarm -Tlinux inttest.pas
Free Pascal Compiler version 3.0.0 [2015/11/15] for arm
Copyright (c) 1993-2015 by Florian Klaempfl and others
Target OS: Linux for ARMEL
Compiling inttest.pas
inttest.pas(4,7) Error: Ordinal expression expected
inttest.pas(6,4) Fatal: There were 1 errors compiling module, stopping
Fatal: Compilation aborted
[Hal:~/Desktop] michael%

[Jonas Maebe]

First, my delay in replying is because I live in Australia and the time difference causes problems.

Jonas wrote "OS X does not support more than 4GB of statically allocated data, even on 64 bit platforms (probably for efficiency reasons)."

But this contradicts the Apple developer document.

No, it does not.

Also, it seems pointless to have 64 bit addressing if it cannot be used.

It can perfectly be used, just not for statically allocated data. You can dynamically allocate as much as you want (up to the limits mentioned in the Apple developer documentation). In FPC you can do this, a.o., using getmem/freemem/new/dispose, dynamic arrays and classes.

This test appears to show that 64-bit addressing is working. If so, why don't the declarations in the first post work?

It's because the limit is related to addresses that are encoded directly in an instruction.  As long as such an address is within 4GB of the instruction that accesses it, things will work. The address that is encoded in your test program is the begin address of BigArray, and that address is well with 4GB of your main program. If you would try to directly access BigArray[high(BigArray)], then again you would get an error because that address is located at 9.6GB from your main program. The reason it works if you use an index, is that then again the start address of BigArray is directly encoded in the instruction, and afterwards the index is loaded, it is multiplied by 8, and then added to this first address (all using —semantically— separate operations). So part of the address that is encoded in the instruction will still be within the 4GB limit.

Technically: the relocation used on OS X to address symbols in a position-independent way is limited to a 4GB offset (32 bit RIP-relative offset). While it is possible to use operations that allows for a full 64 bit offset to be statically encoded, this is slower and increases the code size (you need an extra, and large, instruction per access). C compilers use the slow version by default and the fast version if you compile with -fno-common. FPC always uses the fast version. It would be possible to add a command line option to force the use of the slow version, but that would be work for very little gain (and we would have to compile all of our RTL code also with that switch for it to be useful, because if your main program contains an 8GB array and data from the RTL is placed after it, we can no longer use the fast instruction variant either).

[summerleas]

Sorry about the stupid slip. I tried the following:

var
   BigArray : array[1..2200000000] of int64;
...
StartClock;
for i :=1 to 2200000000 do
   BigArray := 0;
for i :=1 to 2200000000 do
   BigArray := i;
   
writeln('BigArray[1] = ', BigArray[1]);
writeln('BigArray[675118] = ', BigArray[675118]);
writeln('BigArray[70516] = ', BigArray[70516]);

randomize;
For i := 1 to 20 do
   begin
   j := random(2200000000);
   writeln('BigArray[', j, '] = ', BigArray[j]);
   end;
writeln('Last number is ', BigArray[2200000000]);
StopClock;


The Terminal session was:

Richards-iMac:~ richardwatkins$ fpc -Px86_64 -FeDesktop/Senate-preferences/TestArrayDebug Desktop/Senate-preferences/TestArrayRandom.pas
Richards-iMac:~ richardwatkins$ Desktop/Senate-preferences/TestArrayRandom
BigArray[1] = 1
BigArray[675118] = 675118
BigArray[70516] = 70516
BigArray[405057250] = 405057250
BigArray[1681106892] = 1681106892
BigArray[99467957] = 99467957
BigArray[1232969132] = 1232969132
BigArray[415403036] = 415403036
BigArray[1219148714] = 1219148714
BigArray[87783797] = 87783797
BigArray[1509630515] = 1509630515
BigArray[2170999049] = 2170999049
BigArray[141933739] = 141933739
BigArray[1704486524] = 1704486524
BigArray[149338612] = 149338612
BigArray[1530232675] = 1530232675
BigArray[736534994] = 736534994
BigArray[724727909] = 724727909
BigArray[1857829501] = 1857829501
BigArray[762102694] = 762102694
BigArray[1690900447] = 1690900447
BigArray[2103242906] = 2103242906
BigArray[877522511] = 877522511
Last number is 2200000000
Start time: 11 hours  23 min  10 sec  868 ms
End time: 11 hours  23 min  43 sec  455 ms
Execution time: 0 hours  0 min  33 sec  587 ms

Why doesn't it bomb out with EOutOfMemory?

When I get time I will try dynamic arrays.

[summerleas]

Actually my code was OK. The problem was that the forum software converts "bracket i bracket" to italic text. Using "j" instead:

var
   BigArray : array[1..2200000000] of int64; {about 9.6GB}
   i, j : int64;
Begin
StartClock;
for j :=1 to 2200000000 do
   BigArray[j] := 0;
for i :=1 to 2200000000 do
   BigArray[j] := j;
   
writeln('BigArray[1] = ', BigArray[1]);
writeln('BigArray[675118] = ', BigArray[675118]);
writeln('BigArray[70516] = ', BigArray[70516]);

randomize;
For i := 1 to 20 do
   begin
   j := random(2200000000);
   writeln('BigArray[', j, '] = ', BigArray[j]);
   end;
writeln('Last number is ', BigArray[2200000000]);
StopClock;

Adding -gh produces:
Heap dump by heaptrc unit
6 memory blocks allocated : 851/864
6 memory blocks freed     : 851/864
0 unfreed memory blocks : 0
True heap size : 622592 (64 used in System startup)
True free heap : 622528

How big is a memory block?

[Phil]

Why doesn't it bomb out with EOutOfMemory?

You have monitored this thing in Activity Monitor, right? I would guess that the memory is getting swapped out to disk.

Why are you running Yosemite? New Macs come with Sierra.

You still haven't said anything that changes my assessment that you're going about this problem the wrong way by using very large arrays.

[summerleas]

It is getting swapped out. it creates a 16+ GB array.

It is El Capitan OS10.11.6. Don't ask me why - it was what was sold to me. So far my opinion is that it is **** compared to 10.6.8.

Assume there are 4.5 million records in a file of 2.4 GB. I need to access a random subset of 80,000 records. Then I need to access a different random subset. And then ... Repeat the process about 1,300 times. I don't know how long it would take to read the file 1,300 times, but I suspect it would be considerable.

I have not yet investigated dynamic arrays or random accessing a file. The reason is simple:

It SHOULD be possible to load the file into RAM with the OS virtual memory handling it. It SHOULD be possible for free Pascal to compile such a program. So it SHOULD be EASY for me to write the program. But although fpc will correctly compile the example above, it does not compile correctly the program with the array consisting of records. But if it can in one case it should in the other case.