Large array problem: The Computer or free Pascal?

[summerleas]

I need to process non-sequential subsets of about 4.5 million records. Allowing a bit extra I have:
type
   BallotData = record
      BallotNumber : longint;
      NumberOfPreferences : smallint;
      Preferences : array[1..256] of smallint;
      CurrentPreference : byte;
      CurrentValue : double;
      end;
The first attempt was to define:
var   Ballots : array[1 .. 6000000] of BallotData;
From memory it compiled but failed in execution.
The second attempt was to spit up the array:
var   Ballots : array[1 .. 3, 1 .. 2000000] of BallotData;
It compiled but generated meaningless output; for example, it referenced Ballots[1376268, 1638422] and a few seconds later I got:
An unhandled exception occurred at $000115B2:
EInOutError: File not open
  $000115B2

I then tried splitting the array:
const
   ArrayBallotNumbers = 800000;
var
   BallotsGroup1 : array[1 .. ArrayBallotNumbers] of BallotData;
   BallotsGroup2 : array[1 .. ArrayBallotNumbers] of BallotData;
   BallotsGroup3 : array[1 .. ArrayBallotNumbers] of BallotData;
   BallotsGroup4 : array[1 .. ArrayBallotNumbers] of BallotData;
   BallotsGroup5 : array[1 .. ArrayBallotNumbers] of BallotData;
   BallotsGroup6 : array[1 .. ArrayBallotNumbers] of BallotData;
This resulted in a compilation error:
ld: rel32 out of range in __start in __start from /usr/lib/crt1.o
An error occurred while linking
Error: /usr/local/bin/ppc386 returned an error exitcode

After some experimenting I found it works with ArrayBallotNumbers = 650000, but this is too small!

Is the problem with Free Pascal?

Or is it my computer, a first generation Mac desktop pro with a Quad-core 64 bit Intel Xeon, but EFI32 and Kernel mode 32 bit suggesting it might have problems addressing large arrays?

[molly]

Did you count those bitty buggers ? If you did and read something similar like this then things might perhaps become a bit more clear ?

[SurferJoe]

Odds are there isn’t enough ram to support an array of this size.
The size could be calculated with some effort.
Large volumes of data are better managed in TSQL, returning only the results.

What original form is the data in, i.e. Text, File, SQL?

G

[lainz]

Code: Pascal  [Select][+][-]

1. ShowMessage(IntToStr(SizeOf(BallotData))); // 528

528 * 4500000 = 2376000000 bytes = 2.376 GB (according google converter, may differ depending on how you count)

So you can on a 64 bit OS and 64 bit application.

Edit:

If I want to compile with 32 bit

Code: Pascal  [Select][+][-]

1. Ballots : array[1 .. 6000000] of BallotData;

The compiler gives me a hint that can't be done, when I switch to 64 bit output it compiles and run.

[summerleas]

I have 10 GB RAM; anyway virtual  memory should solve any problems.

I tried using:

fpc -Px86_64 ... and I got:

reference out of range from _P$COUNTEXHAUSTEDBALLOTS_$$_PUTBALLOT$LONGINT$BALLOTDATA (1000014A0) in Desktop/Senate-preferences/SenateTestExhaustedVotes.o to _U_$P$COUNTEXHAUSTEDBALLOTS_$$_BALLOTSGROUP6 (18DA55010) in Desktop/Senate-preferences/SenateTestExhaustedVotes.o
ld: rel32 out of range in _P$COUNTEXHAUSTEDBALLOTS_$$_PUTBALLOT$LONGINT$BALLOTDATA from Desktop/Senate-preferences/SenateTestExhaustedVotes.o
An error occurred while linking

So this means my OS cannot use 64 bit mode?

[lainz]

At least not for GUI?

http://forum.lazarus.freepascal.org/index.php?topic=12619.0

http://forum.lazarus.freepascal.org/index.php/topic,12589.0.html

Mac OS X:      10.4, LCL only 32bit, non LCL apps can be 64bit

[User137]

Code: Pascal  [Select][+][-]

1. Preferences : array[1..256] of smallint;

So am i right to assume that each preference value can be higher than 255? Otherwise byte or shortint would fit better.

There is also a big difference in memory handling for static

Code: Pascal  [Select][+][-]

1. Ballots : array[1 .. 6000000] of BallotData;

and dynamic arrays

Code: Pascal  [Select][+][-]

1. Ballots : array of BallotData;
2. ...
3.   Setlength(Ballots, 6000000);

Generally i think dynamic arrays are used when you want to use more RAM.

I didn't find older topics other than this: http://forum.lazarus.freepascal.org/index.php?topic=25802.0
So try setting {$setpeflags $20} ? I tried that old app in my post now with 64-bit compiler and it allocated to 6GB of RAM. Didn't go more because i only have 8GB and i fear it could have bad effects for my system

[derek.john.evans]

Try something like this:

Code: Pascal  [Select][+][-]

1. var
2.   BallotDataArray: array of array of array of BallotData;
3. procedure TForm1.FormCreate(Sender: TObject);
4. begin
5.   SetLength(BallotDataArray, 256, 256, 50);
6. end;    
7.  

I was able to allocate 3,276,800 records on my Windows XP 32bit 2G machine.

[summerleas]

I got up to about 4,000,000 before it crashed, but not enough. Looks like I have to buy a new computer.

Anyway, thanks for the help.