"Data element too large"

[eli]

Compling the following program, I receive the message: "Data element too large".

Are there any tricks to overcome the problem, without splitting the arrays/records?

Code: Pascal  [Select][+][-]

1. type
2.  
3.   n: 0..1849
4.  
5.   sequence: array[1..13] of n;
6.  
7. var
8.  
9.   ArrayOfRecords: array[1..8192] of record
10.  
11.                                       sq: sequence;
12.  
13.                                       ar: array[1..8192] of record
14.                                                               s: sequence;
15.                                                               x: integer
16.                                                             end
17.                                     end
18. begin end.

[marcov]

Not really. But usually that means that a structure is over 2GB, and then it would be a good thing to rethink.

[440bx]

Are there any tricks to overcome the problem, without splitting the arrays/records?

The simplest workaround is to define a separate record for the inner record and have the ArrayOfRecords simply be an array of pointers to those records. 

The only additional work that causes is, the program has to allocate memory for the inner records (in Windows, using VirtualAlloc(Ex) would be the preferable method though, even HeapAlloc could do it.)

There is one gotcha though, that data structure is large enough that there may not be enough contiguous virtual space available in a 32 bit process to accommodate it but, it wouldn't be a problem for a 64 bit program.

HTH.

[eli]

The simplest workaround is to define a separate record for the inner record and have the ArrayOfRecords simply be an array of pointers to those records.

Unfortunately, your advice doesn't help when I try to use the NEW operation for starting those pointers. The result is "out of memory" (exitcode 217).
The same happens if I also repalce the other array of records by an array of pointers to records (but with another exitcode, 255, and without the message "out of memory").

However I noticed that the NEW operation does run well, if I turn off the {$mode objfpc}. But I need this mode, because my program uses the unit gmp.

So it seems I have to give up... 

[marcov]

ArrayOfRecords is  +/- 2 billion bytes (8192*8192 * (4+13*2)). That is simply too much for 32-bit apps for all practical purposes.

If it is only this one array, allocate it immediately after startup.

[eli]

Not really. But usually that means that a structure is over 2GB, and then it would be a good thing to rethink.

I noticed, that if I replace the arrays of records by arrays of pointers to records, the program is compiled well.

If I also turn off the {$mode objfpc} then also the NEW operation runs well. But I need this mode, because my program uses the unit gmp.

So it seems I have to give up... 

[wp]

If you cannot work around this huge data structure you must go to a 64-bit system: The following code works flawlessly on my 64-bit installation of Laz on Windows:

Code: Pascal  [Select][+][-]

1. program Project1;
2.  
3. type
4.   n = 0..1849;
5.   sequence = array[1..13] of n;
6.  
7. var
8.   ArrayOfRecords: array[1..8192] of record
9.     sq: sequence;
10.     ar: array[1..8192] of record
11.       s: sequence;
12.       x: integer
13.     end
14.   end;
15. begin
16.   WriteLn(SizeOf(ArrayOfRecords) div (1024*1024),'MB');
17.   ReadLn;
18. end.

[440bx]

The result is "out of memory" (exitcode 217).
The same happens if I also repalce the other array of records by an array of pointers to records (but with another exitcode, 255, and without the message "out of memory").

It sounds like you're compiling the program for 32bit.  That data structure is too large for a 32bit program running in a 2GB address space.  It should be fine as a 64bit program (provided the O/S is configured properly.)  A 32bit program _may_ be ok under some specific circumstances but, not the usual ones.

However I noticed that the NEW operation does run well, if I turn off the {$mode objfpc}. But I need this mode, because my program uses the unit gmp.

Personally, for a structure that large, I'd use VirtualAlloc/Ex.  I'd stay away from "sugar coated" memory blocks.


ETA:

pretty much what @marcov and @wp stated while I was typing my reply.

[eli]

It sounds like you're compiling the program for 32bit.  That data structure is too large for a 32bit program running in a 2GB address space.  It should be fine as a 64bit program (provided the O/S is configured properly.)  A 32bit program _may_ be ok under some specific circumstances but, not the usual ones.

Actually, the system (Windows 7) in my (old) laptop is of 64-bit, but its RAM memory is 3G only...

Personally, for a structure that large, I'd use VirtualAlloc/Ex.  I'd stay away from "sugar coated" memory blocks.

Interesting. I was not aware of that. Do I have to write anything in my program (written in Free Pascal) for executing VirtualAlloc/Ex?

[winni]

Actually, the system in my (old) laptop is of 64-bit, but its RAM memory is 3G only...

But - depending on your OS - you got a swapfile or a swappartition which ist larger that that.

Winni

[MarkMLl]

I'm surprised that GMP or anything else /demands/ that you use an array structured like that: "computer science types" have eschewed flat arrays for decades, preferring trees etc. where each element is comparatively small. Or a (sparse) array of pointers.

However looking at your code with one minor change:

Code: Pascal  [Select][+][-]

1. type
2.  
3.   n: word; // 0..1849 NOTE: SIZE EXPLICITLY FORCED TO MINIMUM HERE
4.  
5.   sequence: array[1..13] of n; // 14 * 2 = 28 bytes
6.  
7. var
8.  
9.   ArrayOfRecords: array[1..8192] of record
10.  
11.                                       sq: sequence; // 28 bytes
12.  
13.                                       ar: array[1..8192] of record
14.                                                               s: sequence; //28 bytes
15.                                                               x: integer // 4 bytes
16.                                                             end // i.e. 32 bytes total
17.                                     end
18. begin end.

Hence the entire array is going to be ((28 + 4) * 8192 + 28) * 8192 = 2048.2 Mb i.e. greater than the 2Gb space available.

If you absolutely have to have a flat array, I'd suggest that two things worth trying are (a) moving the s field into an array of its own to keep the inner element <32 bytes and (b) packing the sequence type since you only need 11 bits per value... that's 154 bits i.e. ~20 bytes rather than 28, but you'll lose efficiency due to having to bit-twiddle.

Far easier to install a 64-bit OS.

MarkMLl

[eli]

Or a (sparse) array of pointers.

Ok, that's what I finally did: Every array of records was replaced by an array of pointers (to records). But I still need the array (of pointers) itself, because I need the indices (for some computation on these indices).

I'd suggest that two things worth trying are (a) moving the s field into an array of its own to keep the inner element <32 bytes.

Unfortunately, every record must have its own s field, so the s field must be in the record.

and (b) packing the sequence type since you only need 11 bits per value... that's 154 bits i.e. ~20 bytes rather than 28, but you'll lose efficiency due to having to bit-twiddle.

To me, the efficiency is a marginal issue. The most important thing is to let the program run. I've adopted your idea, of packing the arrays, but still, the program exits before its task is completed. The program does complete the task with smaller arrays, though. The task is simple: NEW (pointer), for every pointer in those arrays of pointers (to records).

Far easier to install a 64-bit OS.

Actually my OS is Windows 7 of 64bit. However, the RAM is 3G only.
How about VirtualAlloc/Ex (suggested by 440bx in this thread)? How is it used in Free Pascal?

[440bx]

The mission is simple: NEW (pointer), for every pointer in those arrays of pointers (to records).

You really don't want to do that.  I haven't looked exactly at what "new" does internally but, I'm almost certain that it allocates the memory blocks on a heap (likely the system heap.)  If that is correct and, it quite likely is, there is additional overhead with every single allocation.  Given that addressable space is already tight, that additional overhead gets in the way.

Actually my OS is Windows 7 of 64bit. However, the RAM is 3G only.
How about VirtualAlloc/ex? How may I use it in Free pascal?

The fact that the system only has 3GB is a concern only for performance.  The system may be forced to swap things out to accommodate the structures but as long as there is enough space in the swapfile, the allocations should succeed.

As far how to use VirtualAlloc, I attached a very simple sample program to this post that uses VirtualAlloc to allocate a 4K block of memory. Note that it could have allocated 40,000KB exactly the same way just by changing the number of bytes in the call.

The other thing that sample shows is, you can have more than 2GB available in a 32bit program if it is running on a 64bit system.  The "trick" is to mark the executable as "large address aware" (I don't know if FPC makes that option available.  If it doesn't, get the free CFF explorer from Mark Pistelli (use Google) and mark your executable as "Large address aware".  That's done by opening the exe with CFF explorer, click on "File Header" (on the left) then double click on the "Click here" part of the "Characteristics", check the option "App can handle > 2GB address space" then go to "File->save".  After that your program will have about 3.7GB of memory but, NOT contiguous.  You will still need to break your structure into chunks but, with VirtualAlloc, you can break it into 2 chunks and then calculate the value of the pointers to each individual record, you save those pointers into your array of pointers and you're done.  You'll have about 1GB of address space to spare.

The sample program is already marked as being able to handle addresses greater than 2GB.  If you look at its process space (Process Hacker would allow you to do that), you'll see that it has a full 4GB of address space.  There is one large free chunk of 1.7GB and another of about 2GB.  That should be plenty enough for you.

HTH.

[PascalDragon]

If I also turn off the {$mode objfpc} then also the NEW operation runs well. But I need this mode, because my program uses the unit gmp.

It works without {$mode objfpc}, because in the default mode Integer is a 16-bit signed integer instead of a 32-bit signed integer.

So it seems I have to give up...

Maybe you tell us what exactly you're trying to achieve? Maybe we can point you to a better way then...

[eli]

I haven't looked exactly at what "new" does internally but, I'm almost certain that it allocates the memory blocks on a heap (likely the system heap.)

Correct. NEW is a reserved word in Pascal, intended to allocate memery for every pointer before it's used in the program.

you can have more than 2GB available in a 32bit program if it is running on a 64bit system.

How can I know if my program is a 64bit program or a 32bit program? All I know about it is, that it only contains the text indicated in the beginning of this thread, and that my OS is Windows Seven 64bit.

The fact that the system only has 3GB is a concern only for performance.  The system may be forced to swap things out to accommodate the structures but as long as there is enough space in the swapfile, the allocations should succeed.

As far how to use VirtualAlloc, I attached a very simple sample program to this post that uses VirtualAlloc to allocate a 4K block of memory. Note that it could have allocated 40,000KB exactly the same way just by changing the number of bytes in the call.

...The "trick" is to mark the executable as "large address aware" (I don't know if FPC makes that option available.  If it doesn't, get the free CFF explorer from Mark Pistelli (use Google) and mark your executable as "Large address aware".  That's done by opening the exe with CFF explorer, click on "File Header" (on the left) then double click on the "Click here" part of the "Characteristics", check the option "App can handle > 2GB address space" then go to "File->save".  After that your program will have about 3.7GB of memory but, NOT contiguous.  You will still need to break your structure into chunks but, with VirtualAlloc, you can break it into 2 chunks and then calculate the value of the pointers to each individual record, you save those pointers into your array of pointers and you're done.  You'll have about 1GB of address space to spare.

The sample program is already marked as being able to handle addresses greater than 2GB.  If you look at its process space (Process Hacker would allow you to do that), you'll see that it has a full 4GB of address space.  There is one large free chunk of 1.7GB and another of about 2GB.  That should be plenty enough for you.

HTH.

Thank you. I've downloaded your sample program, for the future. I haven't used it yet, though, because I have just added some (physical) memory. Now the RAM is six G, with the OS of Windows Seven, 64 bits.
Unfortunately, it still doesn't work. if I use arrays of records (rather than arrays of pointers to records), the compiler still presents the message "data element too large". Indeed, the program is compiled well, if I use arrays of pointers (to records), but it exits before the task is completed, with the exitcode 5 (sometimes 255), and I wonder why. By "task" I mean, even the simple task of using the reserved word NEW for all of the pointers.