Forum > Database

"Data element too large"

<< < (3/6) > >>

MarkMLl:
I'm surprised that GMP or anything else /demands/ that you use an array structured like that: "computer science types" have eschewed flat arrays for decades, preferring trees etc. where each element is comparatively small. Or a (sparse) array of pointers.

However looking at your code with one minor change:


--- Code: Pascal  [+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i < x.length; i++) { x[i].style.maxHeight='none'; x[i].style.height = Math.min(x[i].clientHeight+15,306)+'px'; x[i].style.resize = "vertical";}};} ---type   n: word; // 0..1849 NOTE: SIZE EXPLICITLY FORCED TO MINIMUM HERE   sequence: array[1..13] of n; // 14 * 2 = 28 bytes var   ArrayOfRecords: array[1..8192] of record                                       sq: sequence; // 28 bytes                                       ar: array[1..8192] of record                                                              s: sequence; //28 bytes                                                              x: integer // 4 bytes                                                            end // i.e. 32 bytes total                                    endbegin end.
Hence the entire array is going to be ((28 + 4) * 8192 + 28) * 8192 = 2048.2 Mb i.e. greater than the 2Gb space available.

If you absolutely have to have a flat array, I'd suggest that two things worth trying are (a) moving the s field into an array of its own to keep the inner element <32 bytes and (b) packing the sequence type since you only need 11 bits per value... that's 154 bits i.e. ~20 bytes rather than 28, but you'll lose efficiency due to having to bit-twiddle.

Far easier to install a 64-bit OS.

MarkMLl

eli:

--- Quote from: MarkMLl on November 15, 2021, 06:54:07 pm ---Or a (sparse) array of pointers.
--- End quote ---
Ok, that's what I finally did: Every array of records was replaced by an array of pointers (to records). But I still need the array (of pointers) itself, because I need the indices (for some computation on these indices).


--- Quote from: MarkMLl on November 15, 2021, 06:54:07 pm ---I'd suggest that two things worth trying are (a) moving the s field into an array of its own to keep the inner element <32 bytes.
--- End quote ---
Unfortunately, every record must have its own s field, so the s field must be in the record.



--- Quote from: MarkMLl on November 15, 2021, 06:54:07 pm ---and (b) packing the sequence type since you only need 11 bits per value... that's 154 bits i.e. ~20 bytes rather than 28, but you'll lose efficiency due to having to bit-twiddle.
--- End quote ---

To me, the efficiency is a marginal issue. The most important thing is to let the program run. I've adopted your idea, of packing the arrays, but still, the program exits before its task is completed. The program does complete the task with smaller arrays, though. The task is simple: NEW (pointer), for every pointer in those arrays of pointers (to records).


--- Quote from: MarkMLl on November 15, 2021, 06:54:07 pm ---Far easier to install a 64-bit OS.
--- End quote ---
Actually my OS is Windows 7 of 64bit. However, the RAM is 3G only.
How about VirtualAlloc/Ex (suggested by 440bx in this thread)? How is it used in Free Pascal?

440bx:

--- Quote from: eli on November 15, 2021, 09:38:04 pm ---The mission is simple: NEW (pointer), for every pointer in those arrays of pointers (to records).

--- End quote ---
You really don't want to do that.  I haven't looked exactly at what "new" does internally but, I'm almost certain that it allocates the memory blocks on a heap (likely the system heap.)  If that is correct and, it quite likely is, there is additional overhead with every single allocation.  Given that addressable space is already tight, that additional overhead gets in the way.


--- Quote from: eli on November 15, 2021, 09:38:04 pm ---Actually my OS is Windows 7 of 64bit. However, the RAM is 3G only.
How about VirtualAlloc/ex? How may I use it in Free pascal?

--- End quote ---
The fact that the system only has 3GB is a concern only for performance.  The system may be forced to swap things out to accommodate the structures but as long as there is enough space in the swapfile, the allocations should succeed.

As far how to use VirtualAlloc, I attached a very simple sample program to this post that uses VirtualAlloc to allocate a 4K block of memory. Note that it could have allocated 40,000KB exactly the same way just by changing the number of bytes in the call.

The other thing that sample shows is, you can have more than 2GB available in a 32bit program if it is running on a 64bit system.  The "trick" is to mark the executable as "large address aware" (I don't know if FPC makes that option available.  If it doesn't, get the free CFF explorer from Mark Pistelli (use Google) and mark your executable as "Large address aware".  That's done by opening the exe with CFF explorer, click on "File Header" (on the left) then double click on the "Click here" part of the "Characteristics", check the option "App can handle > 2GB address space" then go to "File->save".  After that your program will have about 3.7GB of memory but, NOT contiguous.  You will still need to break your structure into chunks but, with VirtualAlloc, you can break it into 2 chunks and then calculate the value of the pointers to each individual record, you save those pointers into your array of pointers and you're done.  You'll have about 1GB of address space to spare.

The sample program is already marked as being able to handle addresses greater than 2GB.  If you look at its process space (Process Hacker would allow you to do that), you'll see that it has a full 4GB of address space.  There is one large free chunk of 1.7GB and another of about 2GB.  That should be plenty enough for you.

HTH.

PascalDragon:

--- Quote from: eli on November 15, 2021, 03:30:28 pm ---If I also turn off the {$mode objfpc} then also the NEW operation runs well. But I need this mode, because my program uses the unit gmp.
--- End quote ---

It works without {$mode objfpc}, because in the default mode Integer is a 16-bit signed integer instead of a 32-bit signed integer.


--- Quote from: eli on November 15, 2021, 03:30:28 pm ---So it seems I have to give up...

--- End quote ---

Maybe you tell us what exactly you're trying to achieve? Maybe we can point you to a better way then...

eli:

--- Quote from: 440bx on November 15, 2021, 10:09:50 pm ---I haven't looked exactly at what "new" does internally but, I'm almost certain that it allocates the memory blocks on a heap (likely the system heap.)
--- End quote ---
Correct. NEW is a reserved word in Pascal, intended to allocate memery for every pointer before it's used in the program.


--- Quote from: 440bx on November 15, 2021, 10:09:50 pm ---you can have more than 2GB available in a 32bit program if it is running on a 64bit system.
--- End quote ---

How can I know if my program is a 64bit program or a 32bit program? All I know about it is, that it only contains the text indicated in the beginning of this thread, and that my OS is Windows Seven 64bit.


--- Quote from: 440bx on November 15, 2021, 10:09:50 pm ---The fact that the system only has 3GB is a concern only for performance.  The system may be forced to swap things out to accommodate the structures but as long as there is enough space in the swapfile, the allocations should succeed.

As far how to use VirtualAlloc, I attached a very simple sample program to this post that uses VirtualAlloc to allocate a 4K block of memory. Note that it could have allocated 40,000KB exactly the same way just by changing the number of bytes in the call.

...The "trick" is to mark the executable as "large address aware" (I don't know if FPC makes that option available.  If it doesn't, get the free CFF explorer from Mark Pistelli (use Google) and mark your executable as "Large address aware".  That's done by opening the exe with CFF explorer, click on "File Header" (on the left) then double click on the "Click here" part of the "Characteristics", check the option "App can handle > 2GB address space" then go to "File->save".  After that your program will have about 3.7GB of memory but, NOT contiguous.  You will still need to break your structure into chunks but, with VirtualAlloc, you can break it into 2 chunks and then calculate the value of the pointers to each individual record, you save those pointers into your array of pointers and you're done.  You'll have about 1GB of address space to spare.

The sample program is already marked as being able to handle addresses greater than 2GB.  If you look at its process space (Process Hacker would allow you to do that), you'll see that it has a full 4GB of address space.  There is one large free chunk of 1.7GB and another of about 2GB.  That should be plenty enough for you.

HTH.

--- End quote ---

Thank you. I've downloaded your sample program, for the future. I haven't used it yet, though, because I have just added some (physical) memory. Now the RAM is six G, with the OS of Windows Seven, 64 bits.
Unfortunately, it still doesn't work. if I use arrays of records (rather than arrays of pointers to records), the compiler still presents the message "data element too large". Indeed, the program is compiled well, if I use arrays of pointers (to records), but it exits before the task is completed, with the exitcode 5 (sometimes 255), and I wonder why. By "task" I mean, even the simple task of using the reserved word NEW for all of the pointers.

Navigation

[0] Message Index

[#] Next page

[*] Previous page

Go to full version