Lazarus

Free Pascal => Other OS => Topic started by: summerleas on January 27, 2017, 01:36:43 am

Title: Brand new iMac i5 27" retina OS10.10.6 cannot create array
Post by: summerleas on January 27, 2017, 01:36:43 am
Have I bought a useless computer?

   BallotData = record
      BallotNumber : longint;
      NumberOfPreferences : smallint;
      Preferences : array[1..256] of byte;
      CurrentPreference : byte;
      CurrentValue : double;
      end;

var
   BallotsGroup1 : array[1 .. 1000000] of BallotData;
   BallotsGroup2 : array[1 .. 1000000] of BallotData;
   BallotsGroup3 : array[1 .. 1000000] of BallotData;
   BallotsGroup4 : array[1 .. 1000000] of BallotData;
   BallotsGroup5 : array[1 .. 1000000] of BallotData;
   BallotsGroup6 : array[1 .. 1000000] of BallotData;


I compiled this with:

fpc -Px86_64 -FeDesktop/Senate-preferences/SenateDebug Desktop/Senate-preferences/SenateTestExhaustedVotes.pas

The SenateDebug file reported:

Target OS: Darwin for x86_64
Compiling Desktop/Senate-preferences/SenateTestExhaustedVotes.pas
Assembling (pipe) Desktop/Senate-preferences/SenateTestExhaustedVotes.s
Linking Desktop/Senate-preferences/SenateTestExhaustedVotes
SenateTestExhaustedVotes.pas(132,49) Error: Error while linking
SenateTestExhaustedVotes.pas(132,49) Fatal: There were 1 errors compiling module, stopping
Fatal: Compilation aborted

Terminal reported:

    __TEXT/__text addr=0x1000015F0, size=0x00045FCE, fileOffset=0x000015F0, type=1
    __TEXT/__stubs addr=0x1000475BE, size=0x0000018C, fileOffset=0x000475BE, type=28
    __TEXT/__stub_helper addr=0x10004774C, size=0x00000318, fileOffset=0x0004774C, type=32
    __TEXT/.fpc addr=0x100047A68, size=0x0000002A, fileOffset=0x00047A68, type=0
    __TEXT/__const addr=0x100047AA0, size=0x00005513, fileOffset=0x00047AA0, type=0
    __TEXT/__unwind_info addr=0x10004CFB4, size=0x00000048, fileOffset=0x0004CFB4, type=22
    __DATA/__dyld addr=0x10004D000, size=0x00000038, fileOffset=0x0004D000, type=30
    __DATA/__la_symbol_ptr addr=0x10004D038, size=0x00000210, fileOffset=0x0004D038, type=27
    __DATA/__const addr=0x10004D250, size=0x0001158E, fileOffset=0x0004D250, type=0
    __DATA/__data addr=0x10005E7E0, size=0x00005468, fileOffset=0x0005E7E0, type=0
    __DATA/__common addr=0x100063C48, size=0x00001414, fileOffset=0x00000000, type=25
    __DATA/__bss addr=0x100065060, size=0x00001C44, fileOffset=0x00000000, type=25
    __DATA/__huge addr=0x100066CB0, size=0xBCE41084, fileOffset=0x00000000, type=25
ld: 32-bit RIP relative reference out of range (2640414934 max is +/-4GB): from _P$COUNTEXHAUSTEDBALLOTS_$$_PUTBALLOT$LONGINT$BALLOTDATA (0x100001650) to _U_$P$COUNTEXHAUSTEDBALLOTS_$$_BALLOTSGROUP6@0x9D5B3C40 (0x19D61A0B0) in '_P$COUNTEXHAUSTEDBALLOTS_$$_PUTBALLOT$LONGINT$BALLOTDATA' from Desktop/Senate-preferences/SenateTestExhaustedVotes.o for architecture x86_64
An error occurred while linking
Error: /usr/local/bin/ppcx64 returned an error exitcode


Does this mean (as I suspect) the iMac is useless because it can only access 32 bit memory blocks?

Or have I missed something?

Title: Re: Brand new iMac i5 27" retina OS10.10.6 cannot create array
Post by: Phil on January 27, 2017, 02:45:49 am
Have I bought a useless computer?

More likely useless code. Whenever I see someone using very large arrays, the "bad idea" alarm goes off in my head.

Can you explain why you can't write whatever it is you're trying to accomplish so you don't need such large arrays?

Title: Re: Brand new iMac i5 27" retina OS10.10.6 cannot create array
Post by: molly on January 27, 2017, 02:52:41 am
rinse and repeat (http://forum.lazarus.freepascal.org/index.php/topic,35467.0.html)
Title: Re: Brand new iMac i5 27" retina OS10.10.6 cannot create array
Post by: summerleas on January 27, 2017, 03:20:19 am
The data must be accessed repeatedly out of order. The alternative is to repeatedly read the data file maybe 2000 times. There is nothing wrong with large arrays per se, although I would be happy to avoid them.

Anyway, the problem appears to be with the iMac.
Title: Re: Brand new iMac i5 27" retina OS10.10.6 cannot create array
Post by: Phil on January 27, 2017, 03:25:05 am
The data must be accessed repeatedly out of order. The alternative is to repeatedly read the data file maybe 2000 times. There is nothing wrong with large arrays per se, although I would be happy to avoid them.

Anyway, the problem appears to be with the iMac.

Put the data into a file in the "out of order" order that you'll need it in.

Or put the data into a binary random-access file. Then your "array" can be as big as your disk.
Title: Re: Brand new iMac i5 27" retina OS10.10.6 cannot create array
Post by: summerleas on January 27, 2017, 03:30:25 am
The former won't work, the latter might.

As a first step I was hoping that someone might know how Macs access RAM.

I am assuming that the problem is not in free Pascal, but maybe it is ...
Title: Re: Brand new iMac i5 27" retina OS10.10.6 cannot create array
Post by: summerleas on January 27, 2017, 05:34:30 am
On another forum I was directed to:

https://developer.apple.com/library/content/documentation/Performance/Conceptual/ManagingMemory/Articles/AboutMemory.html

It states that "OS X provides approximately 18 exabytes of addressable space for 64-bit processes."

In which case, my program should compile and run because it requires only about 4 or 5 GB. So the problem is with free Pascal or the way I am using it.

Any suggestions?
Title: Re: Brand new iMac i5 27" retina OS10.10.6 cannot create array
Post by: summerleas on January 27, 2017, 06:20:44 am
In the Free Pascal Reference guide (page 24)

"With the exception of int64, qword and Real types, all base types are ordinal types."

And on page 39 an array is defined as: array[ordinal type, ...]

So can the index of an array be of type int64?
Title: Re: Brand new iMac i5 27" retina OS10.10.6 cannot create array
Post by: Thaddy on January 27, 2017, 11:49:51 am
Most likely the problem is NOT the Mac.
Can you give us a compilable example to reproduce the error.I dusted off my mac recently, so I can debug things again.

The problem is that if you yourself don't provide a compilable example, we start to "assume" the probably correct things and then it suddenly works.
Better have you give a small example that can be compiled.

BTW: I suspect a documentation bug: With the exception of int64, qword and real types"
By now int64 and qword are native ordinal types on most platforms, afaik.
And even real is an alias for double, which has never been an ordinal type anyway.

Anyway some proof:
Code: Pascal  [Select][+][-]
  1. program untitled;
  2. var
  3.   a:int64 = 0;
  4.   b:qword = 0;
  5.   c:real := 0.0;
  6. begin
  7.   inc(a);
  8.   inc(b);
  9.   a := pred(a);
  10.   b := pred(b);
  11. end.

I filed a bug report against the documentation. Mantis #0031280

Don't let this distract you: we will still need a compilable example that replicates your issue.
Title: Re: Brand new iMac i5 27" retina OS10.10.6 cannot create array
Post by: mischi on January 27, 2017, 01:56:46 pm
Whether int64 and qword are taken as ordinal seems to depend on 32 bit / 64 bit.
The following test program compiles ok with 64 bit but gives the error message below with 32 bit.

Code: Pascal  [Select][+][-]
  1. program int4Test;
  2. var index: int64;
  3. begin
  4.   for index := 1 to 10 do
  5.     writeln (index)
  6. end.

Excerpt of the error message on 32 bit (Darwin for i386):
...
Compiling int4Test.pas
int4Test.pas(4,7) Error: Ordinal expression expected
Title: Re: Brand new iMac i5 27" retina OS10.10.6 cannot create array
Post by: Cyrax on January 27, 2017, 04:15:15 pm
Use dynamic arrays, not static ones.

Also make sure that you have enough swap space, too.
Title: Re: Brand new iMac i5 27" retina OS10.10.6 cannot create array
Post by: Jonas Maebe on January 27, 2017, 05:38:07 pm
OS X does not support more than 4GB of statically allocated data, even on 64 bit platforms (probably for efficiency reasons). You have to allocate your data dynamically if you want more than that.
Title: Re: Brand new iMac i5 27" retina OS10.10.6 cannot create array
Post by: Noodly on January 27, 2017, 06:40:38 pm
The Preferences array is taking up most of the memory.

For each BallotData record do you really need 256 preferences, each with 256 potential values? Obviously I don't know what your use cases are, but have you considered if there might be a different way of handling preferences with a smaller data structure?

Title: Re: Brand new iMac i5 27" retina OS10.10.6 cannot create array
Post by: Handoko on January 27, 2017, 07:07:36 pm
I think the TS's main intention is not focus on the use case, he/she just wanted to push FPC to its limits to see how far it can handle large memory or at least to make sure it can handle large memory.

Several 'smarter' solutions have already been suggested, but one interesting question still remain:
Why 64-bit iMac OS 10 with 8 GB RAM can't compile program that uses 4 GB memory. Is it FPC problem or OS issue?
Title: Re: Brand new iMac i5 27" retina OS10.10.6 cannot create array
Post by: Thaddy on January 27, 2017, 07:11:38 pm
Is it FPC problem or OS issue?
Well. Jonas is the expert: OS issue.
Title: Re: Brand new iMac i5 27" retina OS10.10.6 cannot create array
Post by: Thaddy on January 27, 2017, 07:15:05 pm
Whether int64 and qword are taken as ordinal seems to depend on 32 bit / 64 bit.
No it is not. The code compiles even on 32 bit arm. As Jonas pointed out this is an OS limitation?
Title: Re: Brand new iMac i5 27" retina OS10.10.6 cannot create array
Post by: Noodly on January 27, 2017, 09:34:47 pm
I think the TS's main intention is not focus on the use case, he/she just wanted to push FPC to its limits to see how far it can handle large memory or at least to make sure it can handle large memory.

There is an earlier related topic from the same user stating that they had problems running this on an older Mac and they were going to buy a new computer. I guess this is the follow up topic after the new computer arrived. It's not just a memory testing exercise, or at least it's a very expensive one if it is.
Title: Re: Brand new iMac i5 27" retina OS10.10.6 cannot create array
Post by: summerleas on January 28, 2017, 04:47:55 am
First, my delay in replying is because I live in Australia and the time difference causes problems.

Jonas wrote "OS X does not support more than 4GB of statically allocated data, even on 64 bit platforms (probably for efficiency reasons)."

But this contradicts the Apple developer document. Also, it seems pointless to have 64 bit addressing if it cannot be used.

Handoko wrote "he/she just wanted to push FPC to its limits to see how far it can handle large memory or at least to make sure it can handle large memory." I have a real problem, but I am not going to write the program unless it will execute. So I gave a fragment to illustrate the problem. Noodly is correct, it is not just a memory test. I was and am not sure where this thread should be posted so I shifted it.

I have been trying something simpler to explain the problem. But the problem vanished!!!

Below is a test program that creates and writes a big array. Assuming fpc worked OK  this should create an array of about 9.6 GB; my iMac has 8 GB of RAM.

It compiled, ran and finished in about 7 seconds. I don't know how to check if it actually worked, so I have set all values to 0 and written the last value.

This test appears to show that 64-bit addressing is working. If so, why don't the declarations in the first post work?



Program TestArray;
uses ExecutionTimeUnit, sysutils;
var
   BigArray : array[1..1200000000] of int64; {about 9.6GB}
   i : int64;
Begin
StartClock;
for i :=1 to 1200000000 do
   BigArray := 0;
   BigArray := i;
writeln('Last number is ', BigArray[1200000000]);
StopClock;
end.

fpc -Px86_64 -FeDesktop/Senate-preferences/TestArrayDebug Desktop/Senate-preferences/TestArray.pas

Desktop/Senate-preferences/TestArray

Terminal session:

Richards-iMac:~ richardwatkins$ fpc -Px86_64 -FeDesktop/Senate-preferences/TestArrayDebug Desktop/Senate-preferences/TestArray.pas
Richards-iMac:~ richardwatkins$ Desktop/Senate-preferences/TestArray
Last number is 1200000000
Start time: 14 hours  17 min  19 sec  528 ms
End time: 14 hours  17 min  25 sec  290 ms
Execution time: 0 hours  0 min  6 sec  762 ms


PS: I tried the above with
   BigArray : array[1..2200000000] of int64;
It worked; took 14 seconds.
Title: Re: Brand new iMac i5 27" retina OS10.10.6 cannot create array
Post by: Thaddy on January 28, 2017, 09:51:07 am
Ahum:
Code: Pascal  [Select][+][-]
  1. Program TestArray;
  2. uses ExecutionTimeUnit, sysutils;
  3. var
  4.    BigArray : array[1..1200000000] of int64; {about 9.6GB}
  5.    i : int64;
  6. Begin
  7. StartClock;
  8.    for i :=1 to 1200000000 do
  9.        BigArray := 0; // this will loop, but is meaningless: always same location in memory
  10.    BigArray := i;// this won't....
  11. writeln('Last number is ', BigArray[1200000000]);
  12. StopClock;
  13. end.

Do you mean:
Code: Pascal  [Select][+][-]
  1. Program TestArray;
  2. uses ExecutionTimeUnit, sysutils;
  3. var
  4.    BigArray : array[1..1200000000] of int64; {about 9.6GB}
  5.    i : int64;
  6. Begin
  7. StartClock;
  8.  for i :=1 to 120000000 do // fill the array...
  9.    BigArray[i] := i;
  10. writeln('Last number is ', BigArray[1200000000]);
  11. StopClock;
  12. end.

That will bomb out with EOutOfMemory in your case.
Title: Re: Brand new iMac i5 27" retina OS10.10.6 cannot create array
Post by: mischi on January 28, 2017, 11:17:20 am
No it is not. The code compiles even on 32 bit arm. As Jonas pointed out this is an OS limitation?
Interesting. My result with the crosssompiler for arm-linux is:

[Hal:~/Desktop] michael% ppcarm -Tlinux inttest.pas
Free Pascal Compiler version 3.0.0 [2015/11/15] for arm
Copyright (c) 1993-2015 by Florian Klaempfl and others
Target OS: Linux for ARMEL
Compiling inttest.pas
inttest.pas(4,7) Error: Ordinal expression expected
inttest.pas(6,4) Fatal: There were 1 errors compiling module, stopping
Fatal: Compilation aborted
[Hal:~/Desktop] michael%
Title: Re: Brand new iMac i5 27" retina OS10.10.6 cannot create array
Post by: Jonas Maebe on January 28, 2017, 12:50:57 pm
First, my delay in replying is because I live in Australia and the time difference causes problems.

Jonas wrote "OS X does not support more than 4GB of statically allocated data, even on 64 bit platforms (probably for efficiency reasons)."

But this contradicts the Apple developer document.
No, it does not.

Quote from: summerleas
Also, it seems pointless to have 64 bit addressing if it cannot be used.
It can perfectly be used, just not for statically allocated data. You can dynamically allocate as much as you want (up to the limits mentioned in the Apple developer documentation). In FPC you can do this, a.o., using getmem/freemem/new/dispose, dynamic arrays and classes.

Quote from: summerleas
This test appears to show that 64-bit addressing is working. If so, why don't the declarations in the first post work?
It's because the limit is related to addresses that are encoded directly in an instruction.  As long as such an address is within 4GB of the instruction that accesses it, things will work. The address that is encoded in your test program is the begin address of BigArray, and that address is well with 4GB of your main program. If you would try to directly access BigArray[high(BigArray)], then again you would get an error because that address is located at 9.6GB from your main program. The reason it works if you use an index, is that then again the start address of BigArray is directly encoded in the instruction, and afterwards the index is loaded, it is multiplied by 8, and then added to this first address (all using —semantically— separate operations). So part of the address that is encoded in the instruction will still be within the 4GB limit.

Technically: the relocation used on OS X to address symbols in a position-independent way is limited to a 4GB offset (32 bit RIP-relative offset). While it is possible to use operations that allows for a full 64 bit offset to be statically encoded, this is slower and increases the code size (you need an extra, and large, instruction per access). C compilers use the slow version by default and the fast version if you compile with -fno-common. FPC always uses the fast version. It would be possible to add a command line option to force the use of the slow version, but that would be work for very little gain (and we would have to compile all of our RTL code also with that switch for it to be useful, because if your main program contains an 8GB array and data from the RTL is placed after it, we can no longer use the fast instruction variant either).
Title: Re: Brand new iMac i5 27" retina OS10.10.6 cannot create array
Post by: summerleas on January 29, 2017, 01:30:18 am
Sorry about the stupid slip. I tried the following:

var
   BigArray : array[1..2200000000] of int64;
...
StartClock;
for i :=1 to 2200000000 do
   BigArray := 0;
for i :=1 to 2200000000 do
   BigArray := i;
   
writeln('BigArray[1] = ', BigArray[1]);
writeln('BigArray[675118] = ', BigArray[675118]);
writeln('BigArray[70516] = ', BigArray[70516]);

randomize;
For i := 1 to 20 do
   begin
   j := random(2200000000);
   writeln('BigArray[', j, '] = ', BigArray[j]);
   end;
writeln('Last number is ', BigArray[2200000000]);
StopClock;


The Terminal session was:

Richards-iMac:~ richardwatkins$ fpc -Px86_64 -FeDesktop/Senate-preferences/TestArrayDebug Desktop/Senate-preferences/TestArrayRandom.pas
Richards-iMac:~ richardwatkins$ Desktop/Senate-preferences/TestArrayRandom
BigArray[1] = 1
BigArray[675118] = 675118
BigArray[70516] = 70516
BigArray[405057250] = 405057250
BigArray[1681106892] = 1681106892
BigArray[99467957] = 99467957
BigArray[1232969132] = 1232969132
BigArray[415403036] = 415403036
BigArray[1219148714] = 1219148714
BigArray[87783797] = 87783797
BigArray[1509630515] = 1509630515
BigArray[2170999049] = 2170999049
BigArray[141933739] = 141933739
BigArray[1704486524] = 1704486524
BigArray[149338612] = 149338612
BigArray[1530232675] = 1530232675
BigArray[736534994] = 736534994
BigArray[724727909] = 724727909
BigArray[1857829501] = 1857829501
BigArray[762102694] = 762102694
BigArray[1690900447] = 1690900447
BigArray[2103242906] = 2103242906
BigArray[877522511] = 877522511
Last number is 2200000000
Start time: 11 hours  23 min  10 sec  868 ms
End time: 11 hours  23 min  43 sec  455 ms
Execution time: 0 hours  0 min  33 sec  587 ms

Why doesn't it bomb out with EOutOfMemory?

When I get time I will try dynamic arrays.
Title: Re: Brand new iMac i5 27" retina OS10.10.6 cannot create array
Post by: summerleas on January 29, 2017, 02:09:29 am
Actually my code was OK. The problem was that the forum software converts "bracket i bracket" to italic text. Using "j" instead:

var
   BigArray : array[1..2200000000] of int64; {about 9.6GB}
   i, j : int64;
Begin
StartClock;
for j :=1 to 2200000000 do
   BigArray[j] := 0;
for i :=1 to 2200000000 do
   BigArray[j] := j;
   
writeln('BigArray[1] = ', BigArray[1]);
writeln('BigArray[675118] = ', BigArray[675118]);
writeln('BigArray[70516] = ', BigArray[70516]);

randomize;
For i := 1 to 20 do
   begin
   j := random(2200000000);
   writeln('BigArray[', j, '] = ', BigArray[j]);
   end;
writeln('Last number is ', BigArray[2200000000]);
StopClock;

Adding -gh produces:
Heap dump by heaptrc unit
6 memory blocks allocated : 851/864
6 memory blocks freed     : 851/864
0 unfreed memory blocks : 0
True heap size : 622592 (64 used in System startup)
True free heap : 622528

How big is a memory block?
Title: Re: Brand new iMac i5 27" retina OS10.10.6 cannot create array
Post by: Phil on January 29, 2017, 02:18:23 am
Why doesn't it bomb out with EOutOfMemory?

You have monitored this thing in Activity Monitor, right? I would guess that the memory is getting swapped out to disk.

Why are you running Yosemite? New Macs come with Sierra.

You still haven't said anything that changes my assessment that you're going about this problem the wrong way by using very large arrays.
Title: Re: Brand new iMac i5 27" retina OS10.10.6 cannot create array
Post by: summerleas on January 29, 2017, 03:46:31 am
It is getting swapped out. it creates a 16+ GB array.

It is El Capitan OS10.11.6. Don't ask me why - it was what was sold to me. So far my opinion is that it is **** compared to 10.6.8.

Assume there are 4.5 million records in a file of 2.4 GB. I need to access a random subset of 80,000 records. Then I need to access a different random subset. And then ... Repeat the process about 1,300 times. I don't know how long it would take to read the file 1,300 times, but I suspect it would be considerable.

I have not yet investigated dynamic arrays or random accessing a file. The reason is simple:

It SHOULD be possible to load the file into RAM with the OS virtual memory handling it. It SHOULD be possible for free Pascal to compile such a program. So it SHOULD be EASY for me to write the program. But although fpc will correctly compile the example above, it does not compile correctly the program with the array consisting of records. But if it can in one case it should in the other case.
TinyPortal © 2005-2018