Forum > Other OS

Brand new iMac i5 27" retina OS10.10.6 cannot create array

<< < (5/5)

Jonas Maebe:

--- Quote from: summerleas on January 28, 2017, 04:47:55 am ---First, my delay in replying is because I live in Australia and the time difference causes problems.

Jonas wrote "OS X does not support more than 4GB of statically allocated data, even on 64 bit platforms (probably for efficiency reasons)."

But this contradicts the Apple developer document.

--- End quote ---
No, it does not.


--- Quote from: summerleas ---Also, it seems pointless to have 64 bit addressing if it cannot be used.

--- End quote ---
It can perfectly be used, just not for statically allocated data. You can dynamically allocate as much as you want (up to the limits mentioned in the Apple developer documentation). In FPC you can do this, a.o., using getmem/freemem/new/dispose, dynamic arrays and classes.


--- Quote from: summerleas ---This test appears to show that 64-bit addressing is working. If so, why don't the declarations in the first post work?

--- End quote ---
It's because the limit is related to addresses that are encoded directly in an instruction.  As long as such an address is within 4GB of the instruction that accesses it, things will work. The address that is encoded in your test program is the begin address of BigArray, and that address is well with 4GB of your main program. If you would try to directly access BigArray[high(BigArray)], then again you would get an error because that address is located at 9.6GB from your main program. The reason it works if you use an index, is that then again the start address of BigArray is directly encoded in the instruction, and afterwards the index is loaded, it is multiplied by 8, and then added to this first address (all using —semantically— separate operations). So part of the address that is encoded in the instruction will still be within the 4GB limit.

Technically: the relocation used on OS X to address symbols in a position-independent way is limited to a 4GB offset (32 bit RIP-relative offset). While it is possible to use operations that allows for a full 64 bit offset to be statically encoded, this is slower and increases the code size (you need an extra, and large, instruction per access). C compilers use the slow version by default and the fast version if you compile with -fno-common. FPC always uses the fast version. It would be possible to add a command line option to force the use of the slow version, but that would be work for very little gain (and we would have to compile all of our RTL code also with that switch for it to be useful, because if your main program contains an 8GB array and data from the RTL is placed after it, we can no longer use the fast instruction variant either).

summerleas:
Sorry about the stupid slip. I tried the following:

var
   BigArray : array[1..2200000000] of int64;
...
StartClock;
for i :=1 to 2200000000 do
   BigArray := 0;
for i :=1 to 2200000000 do
   BigArray := i;
   
writeln('BigArray[1] = ', BigArray[1]);
writeln('BigArray[675118] = ', BigArray[675118]);
writeln('BigArray[70516] = ', BigArray[70516]);

randomize;
For i := 1 to 20 do
   begin
   j := random(2200000000);
   writeln('BigArray[', j, '] = ', BigArray[j]);
   end;
writeln('Last number is ', BigArray[2200000000]);
StopClock;


The Terminal session was:

Richards-iMac:~ richardwatkins$ fpc -Px86_64 -FeDesktop/Senate-preferences/TestArrayDebug Desktop/Senate-preferences/TestArrayRandom.pas
Richards-iMac:~ richardwatkins$ Desktop/Senate-preferences/TestArrayRandom
BigArray[1] = 1
BigArray[675118] = 675118
BigArray[70516] = 70516
BigArray[405057250] = 405057250
BigArray[1681106892] = 1681106892
BigArray[99467957] = 99467957
BigArray[1232969132] = 1232969132
BigArray[415403036] = 415403036
BigArray[1219148714] = 1219148714
BigArray[87783797] = 87783797
BigArray[1509630515] = 1509630515
BigArray[2170999049] = 2170999049
BigArray[141933739] = 141933739
BigArray[1704486524] = 1704486524
BigArray[149338612] = 149338612
BigArray[1530232675] = 1530232675
BigArray[736534994] = 736534994
BigArray[724727909] = 724727909
BigArray[1857829501] = 1857829501
BigArray[762102694] = 762102694
BigArray[1690900447] = 1690900447
BigArray[2103242906] = 2103242906
BigArray[877522511] = 877522511
Last number is 2200000000
Start time: 11 hours  23 min  10 sec  868 ms
End time: 11 hours  23 min  43 sec  455 ms
Execution time: 0 hours  0 min  33 sec  587 ms

Why doesn't it bomb out with EOutOfMemory?

When I get time I will try dynamic arrays.

summerleas:
Actually my code was OK. The problem was that the forum software converts "bracket i bracket" to italic text. Using "j" instead:

var
   BigArray : array[1..2200000000] of int64; {about 9.6GB}
   i, j : int64;
Begin
StartClock;
for j :=1 to 2200000000 do
   BigArray[j] := 0;
for i :=1 to 2200000000 do
   BigArray[j] := j;
   
writeln('BigArray[1] = ', BigArray[1]);
writeln('BigArray[675118] = ', BigArray[675118]);
writeln('BigArray[70516] = ', BigArray[70516]);

randomize;
For i := 1 to 20 do
   begin
   j := random(2200000000);
   writeln('BigArray[', j, '] = ', BigArray[j]);
   end;
writeln('Last number is ', BigArray[2200000000]);
StopClock;

Adding -gh produces:
Heap dump by heaptrc unit
6 memory blocks allocated : 851/864
6 memory blocks freed     : 851/864
0 unfreed memory blocks : 0
True heap size : 622592 (64 used in System startup)
True free heap : 622528

How big is a memory block?

Phil:

--- Quote from: summerleas on January 29, 2017, 01:30:18 am ---Why doesn't it bomb out with EOutOfMemory?

--- End quote ---

You have monitored this thing in Activity Monitor, right? I would guess that the memory is getting swapped out to disk.

Why are you running Yosemite? New Macs come with Sierra.

You still haven't said anything that changes my assessment that you're going about this problem the wrong way by using very large arrays.

summerleas:
It is getting swapped out. it creates a 16+ GB array.

It is El Capitan OS10.11.6. Don't ask me why - it was what was sold to me. So far my opinion is that it is **** compared to 10.6.8.

Assume there are 4.5 million records in a file of 2.4 GB. I need to access a random subset of 80,000 records. Then I need to access a different random subset. And then ... Repeat the process about 1,300 times. I don't know how long it would take to read the file 1,300 times, but I suspect it would be considerable.

I have not yet investigated dynamic arrays or random accessing a file. The reason is simple:

It SHOULD be possible to load the file into RAM with the OS virtual memory handling it. It SHOULD be possible for free Pascal to compile such a program. So it SHOULD be EASY for me to write the program. But although fpc will correctly compile the example above, it does not compile correctly the program with the array consisting of records. But if it can in one case it should in the other case.

Navigation

[0] Message Index

[*] Previous page

Go to full version