Lazarus

Free Pascal => Beginners => Topic started by: kazhar on August 05, 2018, 08:23:17 pm

Title: Can't read COFF Header...
Post by: kazhar on August 05, 2018, 08:23:17 pm
Hello, i am not exactly a beginner, but this feels to me like a real noob-problem...

I have a quite large unit containing a vast array of qWords:

Code: Pascal  [Select][+][-]
  1. unit xyzList; {$mode objfpc}{$H+}
  2. interface
  3. const KnownList: array[0..322244336] of qWord = (
  4. // < --- here go all the numbers --- >
  5. );
  6. implementation
  7. end.

This unit is generated by an export program to be used as part of a quick&dirty testing and integrity check routine. Worked well while the array was smaller, but now the compiler throws  this error:

Projekt kompilieren, Ziel: CheckResFile.exe: Exit code 1, Fehler: 1
Error: Can't read COFF Header while reading C:\Users\<user>\Documents\SourceCode\CheckResFile\lib\x86_64-win64\xyzList.o


xyzList.pas is about 3,7 GB large and has about 10,7 million lines. Is there a way to compile that monster succesfully? It also dont help, that the IDE wont display that file...

thanks!
Title: Re: Can't read COFF Header...
Post by: 440bx on August 05, 2018, 08:34:05 pm
I am not absolutely sure of this but, I believe that a single array in FPC is limited to 2GB and your array exceeds that.

Title: Re: Can't read COFF Header...
Post by: Thaddy on August 05, 2018, 08:46:10 pm
I am not absolutely sure of this but, I believe that a single array in FPC is limited to 2GB and your array exceeds that.
Depends on platform. As in Delphi or C++. The limit you refer to is 32 bit platform (D7 Windows32?) and not even all of them..
And pointers for 32 bit code are unsigned in FPC, so double the amount.

He needs to mention Platform and Bitness, as usual. And the fpc version.
This can be solved in at least two ways: 64 bit: buy more memory. 32 bit: e.g. use a file and a buffered stream.
Title: Re: Can't read COFF Header...
Post by: engkin on August 05, 2018, 08:54:01 pm
Code: Pascal  [Select][+][-]
  1. //ogcoff.pas
  2.     function  TCoffObjInput.ReadObjData(AReader:TObjectreader;out objdata:TObjData):boolean;
  3. ...
  4.            { Read COFF header }
  5.            if not AReader.read(header,sizeof(tcoffheader)) then
  6.              begin
  7.                InputError('Can''t read COFF Header');
  8.                exit;
  9.              end;



Code: Pascal  [Select][+][-]
  1. //owbase.pas
  2. function tobjectreader.read(out b;len:longint):boolean;
  3. begin
  4.   result:=true;
  5.   if bufidx+len>bufmax then
  6.     begin
  7.       result:=false;
  8.       len:=bufmax-bufidx;
  9.     end;
  10.   move(buf[bufidx],b,len);
  11.   inc(bufidx,len);
  12. end;

Code: Pascal  [Select][+][-]
  1. //owbase.pas
  2. function tobjectreader.openfile(const fn:string):boolean;
  3. ...
  4.   bufmax:=f.Size;
  5.  

bufmax and property Size are of type Longint (4 bytes).
Title: Re: Can't read COFF Header...
Post by: kazhar on August 05, 2018, 08:58:11 pm
thanks for your replies!

target platform is win64. the compiler runs on win7 64bit with 16GB or RAM.
Title: Re: Can't read COFF Header...
Post by: 440bx on August 05, 2018, 08:59:15 pm
I am not absolutely sure of this but, I believe that a single array in FPC is limited to 2GB and your array exceeds that.
Depends on platform. As in Delphi or C++. The limit you refer to is 32 bit platform (D7 Windows32?) and not even all of them..
And pointers for 32 bit code are unsigned in FPC, so double the amount.

He needs to mention Platform and Bitness, as usual. And the fpc version.
if you got into the habit of paying attention instead of babbling the first thing that brews in your brain, you would have noticed:
Quote
Error: Can't read COFF Header while reading C:\Users\<user>\Documents\SourceCode\CheckResFile\lib\x86_64-win64\xyzList.o
"C:\users\"... not something you're going to see on many different OSs.

As far as the bitness... close to the end of that line, you might notice "x86_64"... that should answer your question.

To be more specific, I know this for a fact.  A PE file cannot exceed 2GB, this is due to the data type used to store offsets in the PE file.   If a data structure is declared in such a way that it would be stored in the PE file (such as initialized data), that would prevent the linker from successfully building the PE file.

If I knew ELF well enough (which I don't) I'd tell you how it works there but, fortunately for you, I can't help you with that one.

@kazhar:

If you give a reasonably detailed description of what you want to do, I can probably offer a good way of working around that limitation.
Title: Re: Can't read COFF Header...
Post by: engkin on August 05, 2018, 09:03:21 pm
If you check my previous post, and assuming I did not jump into wrong conclusions:

FPC cannot handle object files bigger than 2147483647 bytes.

longint is 4 bytes: $7FFFFFFF = 2147483647
Title: Re: Can't read COFF Header...
Post by: kazhar on August 05, 2018, 09:06:53 pm
bufmax and property Size are of type Longint (4 bytes).

... wrote bullshit.

the object file is 2,5GB large. longint can count to 2GB -> problem found

Title: Re: Can't read COFF Header...
Post by: engkin on August 05, 2018, 09:09:44 pm
bufmax and property Size are of type Longint (4 bytes).

So the object files size(?) has to be smaller than 4GB, if i interprete that correctly?
But that file is "only" about 2,5GB.

I gonna try what happens, if i slightly reduce the array size. Sadly one "run" takes hours :-(
It has to be less than 2GB. Divide your array in half into two arrays in two different units.
Title: Re: Can't read COFF Header...
Post by: rvk on August 05, 2018, 09:11:01 pm
xyzList.pas is about 3,7 GB large and has about 10,7 million lines. Is there a way to compile that monster succesfully? It also dont help, that the IDE wont display that file...
Wouldn't it be much easier to put those numbers in a CSV file (or plain text-file)? Converting the .pas text file with that array to a CSV/text file wouldn't be much work and you would be much more flexible. Ultimately you could/would transfer that CSV/text to a database because I can't see any advantage of that whole array directly in the executable (you already saw that with the time to run it).
Title: Re: Can't read COFF Header...
Post by: kazhar on August 05, 2018, 09:17:46 pm
thanks for your comments!

I think i have to modify that check routine; the old approch wont work any more.
Dividing the array will make things more compilcated than necessary, because i can also read the original, compressed ressource file.

The array-solution had 2 reasons: easy human readable backup and an easy and fast way to access the data.

Title: Re: Can't read COFF Header...
Post by: 440bx on August 05, 2018, 09:22:36 pm
This unit is generated by an export program to be used as part of a quick&dirty testing and integrity check routine. Worked well while the array was smaller, but now the compiler throws  this error:

As rvk above suggested, why not write a quick and dirty program that scans the generated unit and creates a file that only contains the array values, then you can simply load the file and address it as an array.

Quote
The array-solution had 2 reasons: easy human readable backup and an easy and fast way to access the data.
writing a program that strips the unit of the pascal declarations leaving only the data retains both of those advantages.  (at least, it looks that way.)
Title: Re: Can't read COFF Header...
Post by: marcov on August 05, 2018, 09:34:59 pm
Use  -Aas -Xe and see if  GNU's as and ld buy it ?
Title: Re: Can't read COFF Header...
Post by: engkin on August 05, 2018, 10:35:23 pm
Dividing the array will make things more compilcated than necessary, because i can also read the original, compressed ressource file.

I would follow Marco's suggestion, but even if your divide the data into two arrays:
Code: Pascal  [Select][+][-]
  1. {$mode objfpc}{$H+}
  2.  
  3. Interface
  4.  
  5. const
  6.   CKnownIdxMax=10;// or 322244336
  7.   CKnownIdxMid=5; // or 161122168
  8.  
  9. type
  10.   TArrayRange = 0..CKnownIdxMax;
  11.  
  12. Function GetKnowList(AIDX: TArrayRange) : QWord;
  13.  
  14. Property KnownList[AIDX: TArrayRange] : QWord Read GetKnowList;
  15.  
  16. Implementation
  17.  
  18. Uses sysutils;
  19.  
  20. type
  21.   TLowerArrayRange = 0..CKnownIdxMid;
  22.   TUpperArrayRange = CKnownIdxMid+1..CKnownIdxMax;
  23.  
  24. var
  25.   KnownListL: array[TLowerArrayRange] of QWord=(0,1,2,3,4,5); // in a different units uKnownListL
  26.   KnownListH: array[TUpperArrayRange] of QWord=(66,77,88,99,10); // also in a different units uKnownListH
  27.  
  28. Function GetKnowList(AIDX: TArrayRange) : QWord;
  29. begin
  30.   if AIDX<=High(TLowerArrayRange) then
  31.     Result:=KnownListL[AIDX]
  32.   else
  33.     Result:=KnownListH[AIDX];
  34. end;
  35.  
  36. end.
Title: Re: Can't read COFF Header...
Post by: PascalDragon on August 07, 2018, 12:21:26 pm
Even with two arrays it would exceed the maximum length as the file would still be > 2 GB. Better find another way to deal with that data than a static array.
Title: Re: Can't read COFF Header...
Post by: engkin on August 08, 2018, 06:20:37 am
Even with two arrays it would exceed the maximum length as the file would still be > 2 GB. Better find another way to deal with that data than a static array.
Not if each array is in a different unit (if you are referring to the object file)

Divide your array in half into two arrays in two different units.
TinyPortal © 2005-2018