"Data element too large"

[eli]

Maybe you tell us what exactly you're trying to achieve? Maybe we can point you to a better way then...

What I need is my structure, as defined in the beginning of this thread. The structure contains arrays of records with a "huge" size, as defined in the beginning of this thread.

I'm fine with replacing the arrays of records by arrays of pointers (to records), but I need the arrays themselves, because of some arithmetical computation that has to be made with the indices of those arrays.

The probelm is, that FPC doesn't accept my structure: Actually, if I use arrays of records (rather than arrays of pointers to records), the compiler presents the message "data element too large". Indeed, the program is compiled well if I use arrays of pointers (to records), but it exits before the task is completed, with the exitcode 5 (sometimes 255), and I wonder why. By "task" I mean, even the simple task of using the reserved word NEW for all of the pointers.

My OS is Windows Seven, 64 bit. My current RAM is 6 G, as for today (as opposed to yesterday). my FPC version is 3.2.2.win32.and.win 64.exe. Maybe I should download another version, e.g. 3.2.2.i386-win32.cross.x86_64-win64.exe ???

[winni]

Hi!

I compiled your initiale structure with ppcross64.exe on win7/64.

Compiles and runs fine. Shows a sizeof( ArrayOfRecords) of 1,8... GigaByte.

Winni

[eli]

Hi!

I compiled your initiale structure with ppcross64.exe on win7/64.

Compiles and runs fine. Shows a sizeof( ArrayOfRecords) of 1,8... GigaByte.

Winni

Do you have a link to the ppcross64.exe you have suggested?

However, when I click the "Help" button, and then the "About" button, I get the message "Free Pascal IDE for Win32 for i386. Target CPU: i386".

My problem is as follows: When I add the directive {$mode objfpc} in the beginning of my initial program, the compiler presents the mesgae "data element too large".

Additionally, even without adding the directive {$mode objfpc}, when I replace the arrays of records by arrays of pointers to records, and try using the reserved word NEW to allocate memory for all of the pointers, the program exits before it completes its task.

OS: Windows Seven, 64 bit.
RAM: 6 G.

[marcov]

Let's start with the basics. Do you use 64-bit Windows FPC compiler or win32?

(if you go to project options, and then click the "show options" button, what do you get?)

[eli]

Let's start with the basics. Do you use 64-bit Windows FPC compiler or win32?

(if you go to project options, and then click the "show options" button, what do you get?)

As for your question: my free Pascal IDE doesn't show the "show options" button. It does show the "options" button. What should I click after clicking the "options" button?

When I click the "Help" button, and then the "About" button, I get the message "Free Pascal IDE for Win32 for i386. Target CPU: i386".

However, my computer shows the following data:
OS: Windows Seven, 64 bit.
RAM: 6 G.

The problem is as follows:
When I add the directive {$mode objfpc} in the beginning of the my structure (indicated in the beginning of this thread), the compiler presents the messsage "data element too large".

Additionally, even without adding the directive {$mode objfpc}, when I replace the arrays of records by arrays of pointers to records, and try using the reserved word NEW to allocate memory for all of the pointers, the program exits (with exitcode=203) before it completes its task.

[tetrastes]

Your system is 64 bit, but your program is 32 bit. Compile it with "fpc -Px86_64".

[eli]

Your system is 64 bit, but your program is 32 bit. Compile it with "fpc -Px86_64".

Do you mean fpc-3.2.2.i386-win32.cross.x86_64-win64.exe ?
I have just installed it.
But now I don't see any icon on my desktop to execute the editor, So how do I execute it? I even don't know the path where the editor is located, and what its name is.

[tetrastes]

fpc-3.2.2.i386-win32.cross.x86_64-win64.exe is installer, it adds cross compiler for x86_64-win64 to already installed i386 compiler. But if you did full install of 3.2.2.win32.and.win64.exe, you already have cross compiler.
It seems that 32-bit Free Pascal IDE (fp.exe) cannot do cross compilation to 64 bit, so you have to use compiler (fpc.exe) directly from command line.

[PascalDragon]

Maybe you tell us what exactly you're trying to achieve? Maybe we can point you to a better way then...

What I need is my structure, as defined in the beginning of this thread. The structure contains arrays of records with a "huge" size, as defined in the beginning of this thread.

The question is why you need this structure? What are you trying to achieve?

[winni]

Do you have a link to the ppcross64.exe you have suggested?

Hi!

I got it here:

Code: Bash  [Select][+][-]

1. C:\FPC\3.2.0\bin\i386-win32\ppcross64.exe

Download from the fpc/Lazarus page

Winni

[440bx]

However, my computer shows the following data:
OS: Windows Seven, 64 bit.
RAM: 6 G.

The problem is as follows:
When I add the directive {$mode objfpc} in the beginning of the my structure (indicated in the beginning of this thread), the compiler presents the messsage "data element too large".

Additionally, even without adding the directive {$mode objfpc}, when I replace the arrays of records by arrays of pointers to records, and try using the reserved word NEW to allocate memory for all of the pointers, the program exits (with exitcode=203) before it completes its task.

You're computer is fine, no problem there.

No problem with {$mode objfpc} either.

The reserved word "new" only reserves problems for you.  THAT is a problem. 

Here is a 32bit program, with {$mode objfpc} that gives you an array of 8192 pointers to those records you want and, it could handle about 5,000 more if need be.  REMEMBER, that after compiling the program it must be > 2GB address space enabled.  You do that using CFF Explorer or a little utility called 4gb_patch.exe (also written by Daniel Pistelli - disclaimer: I haven't used it, I know nothing about it - I use CFF Explorer).  The program that allocates the records you want is:

Code: Pascal  [Select][+][-]

1. {$MODE OBJFPC}
2.  
3. {$define ALLOCATE_TRUE_SIZE}
4.  
5. program _LargeArray;
6.  
7. uses
8.   sysutils,
9.  
10.   Windows
11.   ;
12.  
13.  
14. type
15.   TELEMENT_RANGE = 1..8192;
16.  
17. const
18.   WIDTH         = 10;
19.  
20.   ELEMENT_COUNT = high(TELEMENT_RANGE);
21.  
22.  
23.   { 2GB - 1MB for safety                                                      }
24.  
25.   BLOCK_HI_SIZE = 2 * (1024 * 1024 * 1023); { one meg short of actual size    }
26.  
27.  
28. type
29.   n              = 0..1849;
30.  
31.   TSEQUENCE      = array[1..13] of n;
32.  
33.   PINNER_RECORD  = ^TINNER_RECORD;
34.   TINNER_RECORD  = record
35.     sq               : TSEQUENCE;
36.     ar               : array[TELEMENT_RANGE] of record
37.                                                  s : TSEQUENCE;
38.                                                  x : longint;
39.                                                end
40.   end;
41.  
42. const
43.   {$ifdef ALLOCATE_TRUE_SIZE}
44.   ALLOC_HI_COUNT = BLOCK_HI_SIZE div sizeof(TINNER_RECORD);
45.   ALLOC_LO_COUNT = ELEMENT_COUNT - ALLOC_HI_COUNT;
46.   {$else}
47.   ALLOC_HI_COUNT = 16;     { counts arbitrarily chosen                        }
48.   ALLOC_LO_COUNT =  8;
49.   {$endif}
50.  
51.  
52. var
53.   OuterRecordStaticSize : qword;
54.  
55.   { the memory blocks that hold the records                                   }
56.  
57.   MemoryBlocks          : packed array[0..1]    of PINNER_RECORD;
58.  
59.   { the array of pointers to inner records                                    }
60.  
61.   RecordsArray          : array[TELEMENT_RANGE] of PINNER_RECORD;
62.  
63.   i                     : TELEMENT_RANGE;
64.  
65.   p                     : PINNER_RECORD;
66.  
67.   SCOPE                 : DWORD;
68.  
69.  
70. begin
71.   writeln;
72.   writeln;
73.  
74.   writeln('  approx size of above 2GB block : ', BLOCK_HI_SIZE:WIDTH);
75.  
76.   writeln('  Inner record size              : ', sizeof(TINNER_RECORD):WIDTH);
77.   writeln;
78.   OuterRecordStaticSize := ELEMENT_COUNT * sizeof(TINNER_RECORD);
79.   writeln('  Outer record size              : ', OuterRecordStaticSize:WIDTH);
80.  
81.   if OuterRecordStaticSize >= 1700000000 then    { about 1.7GB                }
82.   begin
83.     writeln;
84.     writeln('  data structure is too large for a 32bit program');
85.     writeln('  that is not large address [ > 2GB ] space aware');
86.   end;
87.  
88.   writeln;
89.   writeln('  will allocate ', ALLOC_HI_COUNT:4, ' records in the block above 2GB');
90.   writeln('  will allocate ', ALLOC_LO_COUNT:4, ' records in the block below the 2GB line');
91.  
92.  
93.   repeat     { scope delimiter - NOT a loop                                   }
94.  
95.     MemoryBlocks[0] := VirtualAlloc(nil,                   { anywhere is fine }
96.                                     ALLOC_HI_COUNT * sizeof(TINNER_RECORD),
97.                                     MEM_COMMIT or MEM_RESERVE,
98.                                     PAGE_READWRITE);
99.  
100.     MemoryBlocks[1] := VirtualAlloc(nil,                   { anywhere is fine }
101.                                     ALLOC_LO_COUNT * sizeof(TINNER_RECORD),
102.                                     MEM_COMMIT or MEM_RESERVE,
103.                                     PAGE_READWRITE);
104.  
105.     if (MemoryBlocks[0] = nil) or (MemoryBlocks[0] = nil) then
106.     begin
107.       writeln;
108.       writeln;
109.       writeln('  failed to allocate the necessary memory blocks');
110.  
111.       break;
112.     end;
113.  
114.     writeln;
115.     writeln('  block ABOVE the 2GB line is at : ', IntToHex(ptruint(MemoryBlocks[0]), 0):10);
116.     writeln('  block below the 2GB line is at : ', IntToHex(ptruint(MemoryBlocks[1]), 0):10);
117.  
118.     { populate the array of pointers to inner blocks                          }
119.  
120.     p := PINNER_RECORD(MemoryBlocks[0]);
121.     for i := low(TELEMENT_RANGE) to TELEMENT_RANGE(ALLOC_HI_COUNT) do
122.     begin
123.       RecordsArray[i] := p;
124.  
125.       {$ifndef ALLOCATE_TRUE_SIZE}
126.         { mark the records for testing and verification purposes only         }
127.  
128.         FillChar(p^, sizeof(p^), byte(i));
129.       {$endif}
130.  
131.       inc(p);
132.     end;
133.  
134.     p := PINNER_RECORD(MemoryBLocks[1]);
135.     for i := TELEMENT_RANGE(ALLOC_HI_COUNT + 1)
136.           to TELEMENT_RANGE(ALLOC_HI_COUNT + ALLOC_LO_COUNT) do
137.     begin
138.       RecordsArray[i] := p;
139.  
140.       {$ifndef ALLOCATE_TRUE_SIZE}
141.         { mark the records for testing and verification purposes only         }
142.  
143.         FillChar(p^, sizeof(p^), byte(i));
144.       {$endif}
145.  
146.       inc(p);
147.     end;
148.  
149.     { output the pointers in RecordsArray                                     }
150.  
151.     writeln;
152.     for i := low(RecordsArray)
153.           to TELEMENT_RANGE(ALLOC_LO_COUNT + ALLOC_HI_COUNT) do
154.     begin
155.       write('  ', ord(i):4, ':', IntToHex(ptruint(RecordsArray[i]), 0):10);
156.  
157.       if ord(i) mod 4 = 0 then writeln;
158.     end;
159.  
160.   until SCOPE = SCOPE;
161.  
162.  
163.   writeln;
164.   writeln;
165.   writeln('press ENTER/RETURN to end this program');
166.   readln;
167. end.

AGAIN, remember to patch it every time you recompile it or the first call to VirtualAlloc will fail.  The {$define .... } at the top is to allow you to run the program unpatched with a smaller number of records that fit in 2GB, that way you can use the debugger to step through it and see how it all works.  Once the define is enabled then the program MUST be patched for the memory allocations to succeed.

Attached is a version of the program that is patched and allocates the 8192 records you want.

HTH.

PS: use Google to locate that 4gb_patch.exe utility if you'd rather use that instead of CFF Explorer.

[PascalDragon]

Here is a 32bit program, with {$mode objfpc} that gives you an array of 8192 pointers to those records you want and, it could handle about 5,000 more if need be.  REMEMBER, that after compiling the program it must be > 2GB address space enabled.  You do that using CFF Explorer or a little utility called 4gb_patch.exe (also written by Daniel Pistelli - disclaimer: I haven't used it, I know nothing about it - I use CFF Explorer).

You can also do this:

Code: Pascal  [Select][+][-]

1. program ttest;
2.  
3. uses
4.   Windows;
5.  
6. {$SetPEFlags IMAGE_FILE_LARGE_ADDRESS_AWARE}
7.  
8. begin
9. end.

[Thaddy]

Yup. Value is $200. This came up last week already and for the past couple of years.

[440bx]

You can also do this:

Code: Pascal  [Select][+][-]

1. program ttest;
2.  
3. uses
4.   Windows;
5.  
6. {$SetPEFlags IMAGE_FILE_LARGE_ADDRESS_AWARE}
7.  
8. begin
9. end.

Thank you for that @PascalDragon.  I suspected that FPC might support setting that bit but, I use it so rarely that I never spent any time finding out if it could, instead I just patched the exe.  Your suggestion is the right way to do it.


@eli

That's how you set that bit.  Much better than patching the exe.

[PascalDragon]

Yup. Value is $200. This came up last week already and for the past couple of years.

No, it's $20 (which is also how it's declared in the Windows unit). $200 is IMAGE_FILE_DEBUG_STRIPPED.

To quote from the Windows SDK's winnt.h:

Code: C  [Select][+][-]

1. #define IMAGE_FILE_RELOCS_STRIPPED           0x0001  // Relocation info stripped from file.
2. #define IMAGE_FILE_EXECUTABLE_IMAGE          0x0002  // File is executable  (i.e. no unresolved external references).
3. #define IMAGE_FILE_LINE_NUMS_STRIPPED        0x0004  // Line nunbers stripped from file.
4. #define IMAGE_FILE_LOCAL_SYMS_STRIPPED       0x0008  // Local symbols stripped from file.
5. #define IMAGE_FILE_AGGRESIVE_WS_TRIM         0x0010  // Aggressively trim working set
6. #define IMAGE_FILE_LARGE_ADDRESS_AWARE       0x0020  // App can handle >2gb addresses
7. #define IMAGE_FILE_BYTES_REVERSED_LO         0x0080  // Bytes of machine word are reversed.
8. #define IMAGE_FILE_32BIT_MACHINE             0x0100  // 32 bit word machine.
9. #define IMAGE_FILE_DEBUG_STRIPPED            0x0200  // Debugging info stripped from file in .DBG file
10. #define IMAGE_FILE_REMOVABLE_RUN_FROM_SWAP   0x0400  // If Image is on removable media, copy and run from the swap file.
11. #define IMAGE_FILE_NET_RUN_FROM_SWAP         0x0800  // If Image is on Net, copy and run from the swap file.
12. #define IMAGE_FILE_SYSTEM                    0x1000  // System File.
13. #define IMAGE_FILE_DLL                       0x2000  // File is a DLL.
14. #define IMAGE_FILE_UP_SYSTEM_ONLY            0x4000  // File should only be run on a UP machine
15. #define IMAGE_FILE_BYTES_REVERSED_HI         0x8000  // Bytes of machine word are reversed.