Getting File Size from PE Header Fails with Lazarus

[msintle]

So here's an odd one. Code like this works beautifully in Delphi:

Code: Pascal  [Select][+][-]

1. { call passing HInstance as the Executable parameter, add Windows to uses }
2. function GetExeSize(Executable: THandle): Cardinal;
3. type
4.   PImageDosHeader = ^TImageDosHeader;
5.   TImageDosHeader = packed record
6.     e_magic: Word;
7.     e_ignore: packed array [0..28] of Word;
8.     _lfanew: Longint;
9.   end;
10. var
11.   i, NumSections: Integer;
12.   p: PAnsiChar;
13. begin
14.   Result := 0;
15.   p := Pointer(Executable);
16.   Inc(p, PImageDosHeader(p)^._lfanew + SizeOf(DWORD));
17.   NumSections := PImageFileHeader(p)^.NumberOfSections;
18.   Inc(p, SizeOf(TImageFileHeader) + SizeOf(TImageOptionalHeader));
19.   for i := 1 to NumSections do
20.   begin
21.     with PImageSectionHeader(p)^ do
22.       if PointerToRawData+SizeOfRawData > Result then
23.         Result := PointerToRawData + SizeOfRawData;
24.     Inc(p, SizeOf(TImageSectionHeader));
25.   end;
26. end;

But it fails in Lazarus.

To clarify, it does still compile and run - but returns the wrong file size (always falling short a bit).

Is Lazarus constructing the PE header differently somehow, perhaps?

[Fibonacci]

Works 100% fine if reading 32bit i386.

For 64bit the code will be differet. eg TImageOptionalHeader default is TImageOptionalHeader32, and for 64bit you need to use TImageOptionalHeader64

BTW. Remove debug info, this is non standard and weird method to get "file size"

[msintle]

Works 100% fine if reading 32bit i386.

Not for me. I am building x86 targets. Happy to upload both a console and a GUI test app if people would like. Reproduces instantly.

For 64bit the code will be differet. eg TImageOptionalHeader default is TImageOptionalHeader32, and for 64bit you need to use TImageOptionalHeader64

Thanks for noting that, will definitely come in handy in the future.

BTW. Remove debug info, this is non standard and weird method to get "file size"

How do you mean? In Delphi, it works properly even with debug info. Is the way Lazarus is adding debug information then non-standard and breaking the standard header based file size calculation process?

Regarding your comment that this is a weird method to get file size, it is the only method to get file size when you are building a stub application (such as a self-extractor which would have an archive of variable size appended to its end, for example).

[440bx]

@msintle,

For a general utility your code will _not_ work because sizeof(TImageOptionalHeader) is likely determined at compile time (otherwise the code you showed would not compile.)

if that code has to deal with executables other than itself then it will only be right when the executable it is dealing with happens to be the same bitness as the program.  If the bitness  between them differ then the result will be wrong due to the wrong size of TImageOptionalHeader.

On a different note, why do you calculate the file size that way instead of simply asking the O/S for the file size ?

HTH.

[KodeZwerg]

I do use something similar in my Advanced-Properties app that was compiled with Delphi.
It is very handy to detect anything that in my app I declared as "overlay" (or extract/strip such data)
It is also very reliable used in stubs (no matter what bitness when using correct header translation) to detect the end of stub in a 100% accurate way.
Many oldschool and also newschool archivers that do produce executable files, like WinRAR, are using this way to know where streaming of archive begin.

To OP:
I will try to port my way, pure WinAPI, to Lazarus and send source here.

[Fibonacci]

How do you mean? In Delphi, it works properly even with debug info. Is the way Lazarus is adding debug information then non-standard and breaking the standard header based file size calculation process?

I dont have Delphi so cant compare the outputs, but FPC is adding debug info after EOF, overlay. If you build your exe without debug info, your function will display correct size. In my case debug info adds ~60 KB after PE sections (with only uses Windows).

170496 {your function} + 62931 {overlay size} = 233427 {actual file size on the disk}

Other than overlay debug info, it also creates many actual PE sections.

[440bx]

but FPC is adding debug info after EOF, overlay.

I'm probably misunderstanding something but, I have not seen an instance where FPC adds debug info _after_ the executable's EOF in a PE file. 

Do you have an example of such an occurrence and/or how to produce such a file ?   TIA.

The following is what I've always seen (Note: details for non-debug sections have been removed for brevity):

Code: Text  [Select][+][-]

1.  
2.      40.0178            - IMAGE_SECTION_HEADER(s) - Number of sections : 10
3.  
4.      40.0178              SECTION:  1 of 10
5.  
6.      40.0178     178  [8]                                 Name : .text
7.  
8.  
9.      40.01a0              SECTION:  2 of 10  contains: Thread Local Storage (TLS) directory
10.  
11.      40.01a0     1a0  [8]                                 Name : .data
12.  
13.  
14.      40.01c8              SECTION:  3 of 10
15.  
16.      40.01c8     1c8  [8]                                 Name : .rdata
17.  
18.  
19.      40.01f0              SECTION:  4 of 10
20.  
21.      40.01f0     1f0  [8]                                 Name : .bss
22.  
23.  
24.      40.0218              SECTION:  5 of 10
25.  
26.      40.0218     218  [8]                                 Name : .CRT
27.  
28.  
29.      40.0240              SECTION:  6 of 10  contains: Imports directory
30.      40.0240 +                                         Imports address table directory
31.  
32.      40.0240     240  [8]                                 Name : .idata
33.  
34.  
35.      40.0268              SECTION:  7 of 10
36.  
37.      40.0268     268  [8]                                 Name : /4
38.               1.c3f0  [12]                                       .debug_info
39.  
40.      40.0270     270  [4]                         Virtual size :      9202    (37,378)
41.  
42.      40.0274     274  [4]           (Relative) Virtual address :    1.6000    [Va:     41.6000] [FO:   ea00] [      /4]
43.  
44.      40.0278     278  [4]                     Size of raw data :      9400    (37,888)
45.  
46.      40.027c     27c  [4]      File offset/pointer to raw data :      ea00    [Va:     41.6000] [FO:   ea00] [      /4]
47.      40.0280     280  [4]   File offset/pointer to relocations :         0
48.      40.0284     284  [4]   File offset/pointer to linenumbers :         0
49.  
50.      40.0288     288  [2]                Number of relocations :         0
51.      40.028a     28a  [2]                Number of linenumbers :         0
52.  
53.      40.028c     28c  [4]                      Characteristics : 4200.0040
54.      40.028c +                        SCN_CNT_INITIALIZED_DATA          40
55.      40.028c +                          SCN_ALIGN_NOTSPECIFIED     0
56.      40.028c +                             SCN_MEM_DISCARDABLE    200.0000
57.      40.028c +                                    SCN_MEM_READ   4000.0000
58.  
59.  
60.      40.0290              SECTION:  8 of 10
61.  
62.      40.0290     290  [8]                                 Name : /16
63.               1.c3fc  [14]                                       .debug_abbrev
64.  
65.      40.0298     298  [4]                         Virtual size :       244    (  580)
66.  
67.      40.029c     29c  [4]           (Relative) Virtual address :    2.0000    [Va:     42.0000] [FO: 1.7e00] [     /16]
68.  
69.      40.02a0     2a0  [4]                     Size of raw data :       400    (1,024)
70.  
71.      40.02a4     2a4  [4]      File offset/pointer to raw data :    1.7e00    [Va:     42.0000] [FO: 1.7e00] [     /16]
72.      40.02a8     2a8  [4]   File offset/pointer to relocations :         0
73.      40.02ac     2ac  [4]   File offset/pointer to linenumbers :         0
74.  
75.      40.02b0     2b0  [2]                Number of relocations :         0
76.      40.02b2     2b2  [2]                Number of linenumbers :         0
77.  
78.      40.02b4     2b4  [4]                      Characteristics : 4200.0040
79.      40.02b4 +                        SCN_CNT_INITIALIZED_DATA          40
80.      40.02b4 +                          SCN_ALIGN_NOTSPECIFIED     0
81.      40.02b4 +                             SCN_MEM_DISCARDABLE    200.0000
82.      40.02b4 +                                    SCN_MEM_READ   4000.0000
83.  
84.  
85.      40.02b8              SECTION:  9 of 10
86.  
87.      40.02b8     2b8  [8]                                 Name : /30
88.               1.c40a  [12]                                       .debug_line
89.  
90.      40.02c0     2c0  [4]                         Virtual size :       bd5    (3,029)
91.  
92.      40.02c4     2c4  [4]           (Relative) Virtual address :    2.1000    [Va:     42.1000] [FO: 1.8200] [     /30]
93.  
94.      40.02c8     2c8  [4]                     Size of raw data :       c00    (3,072)
95.  
96.      40.02cc     2cc  [4]      File offset/pointer to raw data :    1.8200    [Va:     42.1000] [FO: 1.8200] [     /30]
97.      40.02d0     2d0  [4]   File offset/pointer to relocations :         0
98.      40.02d4     2d4  [4]   File offset/pointer to linenumbers :         0
99.  
100.      40.02d8     2d8  [2]                Number of relocations :         0
101.      40.02da     2da  [2]                Number of linenumbers :         0
102.  
103.      40.02dc     2dc  [4]                      Characteristics : 4200.0040
104.      40.02dc +                        SCN_CNT_INITIALIZED_DATA          40
105.      40.02dc +                          SCN_ALIGN_NOTSPECIFIED     0
106.      40.02dc +                             SCN_MEM_DISCARDABLE    200.0000
107.      40.02dc +                                    SCN_MEM_READ   4000.0000
108.  
109.  
110.      40.02e0              SECTION: 10 of 10
111.  
112.      40.02e0     2e0  [8]                                 Name : /42
113.               1.c416  [13]                                       .debug_frame
114.  
115.      40.02e8     2e8  [4]                         Virtual size :       29c    (  668)
116.  
117.      40.02ec     2ec  [4]           (Relative) Virtual address :    2.2000    [Va:     42.2000] [FO: 1.8e00] [     /42]
118.  
119.      40.02f0     2f0  [4]                     Size of raw data :       400    (1,024)
120.  
121.      40.02f4     2f4  [4]      File offset/pointer to raw data :    1.8e00    [Va:     42.2000] [FO: 1.8e00] [     /42]
122.      40.02f8     2f8  [4]   File offset/pointer to relocations :         0
123.      40.02fc     2fc  [4]   File offset/pointer to linenumbers :         0
124.  
125.      40.0300     300  [2]                Number of relocations :         0
126.      40.0302     302  [2]                Number of linenumbers :         0
127.  
128.      40.0304     304  [4]                      Characteristics : 4200.0040
129.      40.0304 +                        SCN_CNT_INITIALIZED_DATA          40
130.      40.0304 +                          SCN_ALIGN_NOTSPECIFIED     0
131.      40.0304 +                             SCN_MEM_DISCARDABLE    200.0000
132.      40.0304 +                                    SCN_MEM_READ   4000.0000
133.  
134.  
135.  
136.      40.0308             - IMAGE HEADER - 248 filler bytes -
137.  
138.      40.0308     308  [ 8]   __ __ __ __ __ __ __ __  00 00 00 00 00 00 00 00   ________........
139.      40.0310     310  [16]   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
140.      40.0320     320  [16]   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00   ................
141.  
142.      etc
143.      etc
144.      etc
145.  
146.  

In the above, all file offsets are within the real size of the PE file (IOW, not appended to what "used to be" its real size)

[KodeZwerg]

My app does confirm what Fibonacci say.
I attached an Image that visualize it.

[Fibonacci]

Do you have an example of such an occurrence and/or how to produce such a file ?   TIA.

1. OPs function returns real image size, which is different than file size, ergo "there is something more in the file"
2. Open binary in eg. PE-Bear and see Overlay

[440bx]

@KodeZwerg & @Fibonacci

Which binary ?... I don't see a binary anywhere in this thread ... did I miss an elephant in the room ?

[Fibonacci]

Ok, if you cant/dont want to compile yourself, here you go, binary in the attachment

Full source code

Code: Pascal  [Select][+][-]

1. uses Windows;
2.  
3. { call passing HInstance as the Executable parameter, add Windows to uses }
4. function GetExeSize(Executable: THandle): Cardinal;
5. type
6.   PImageDosHeader = ^TImageDosHeader;
7.   TImageDosHeader = packed record
8.     e_magic: Word;
9.     e_ignore: packed array [0..28] of Word;
10.     _lfanew: Longint;
11.   end;
12. var
13.   i, NumSections: Integer;
14.   p: PAnsiChar;
15. begin
16.   Result := 0;
17.   p := Pointer(Executable);
18.   Inc(p, PImageDosHeader(p)^._lfanew + SizeOf(DWORD));
19.   NumSections := PImageFileHeader(p)^.NumberOfSections;
20.   Inc(p, SizeOf(TImageFileHeader) + SizeOf(TImageOptionalHeader));
21.   for i := 1 to NumSections do
22.   begin
23.     with PImageSectionHeader(p)^ do
24.       if PointerToRawData+SizeOfRawData > Result then
25.         Result := PointerToRawData + SizeOfRawData;
26.     Inc(p, SizeOf(TImageSectionHeader));
27.   end;
28. end;
29.  
30. begin
31.   writeln('image size = ', GetExeSize(GetModuleHandle(nil)));
32.   readln;
33. end.

(app return) image size = 58368

real file size = 98818

40450 bytes overlay data

[KodeZwerg]

@KodeZwerg & @Fibonacci

Which binary ?... I don't see a binary anywhere in this thread ... did I miss an elephant in the room ?

I just opened Lazarus, a new form, pressed run and on screen image i clicked "okay"
Afterwards I run my app that I mentioned earlier over the compiled executable.

[440bx]

Thank you both, Fibonacci and KodeZwerg, for the additional information.  Now I see what you guys are talking about. 

Just for the record, there is no need to calculate the "real executable size" because that is given in the PE header.  Using the executable that Fibonacci posted:

Code: Text  [Select][+][-]

1.      40.0080            - IMAGE_NT_HEADERS -
2.  
3.      40.0080      80  [4]                            Signature : 00004550    (PE00)
4.  
5.      40.0084               - IMAGE_FILE_HEADER -
6.  
7.      40.0084      84  [2]                              Machine :  14c
8.      40.0084 +                         IMAGE_FILE_MACHINE_I386
9.  
10.      40.0086      86  [2]                   Number of sections :    b    ( 11)
11.  
12.      40.0088      88  [4]                      Time Date Stamp :    0
13.  
14.      40.008c      8c  [4]              Pointer to symbol table : e400
15.      40.0090      90  [4]                    Number of symbols :  314    (788)
16.  
17.      40.0094      94  [2]              Size of optional header :   e0    (224)
18.  
19.      40.0096      96  [2]                      Characteristics :  107
20.      40.0096 +                      IMAGE_FILE_RELOCS_STRIPPED      1
21.      40.0096 +                     IMAGE_FILE_EXECUTABLE_IMAGE      2
22.      40.0096 +                   IMAGE_FILE_LINE_NUMS_STRIPPED      4
23.      40.0096 +                        IMAGE_FILE_32BIT_MACHINE    100
24.  

Note that the "Pointer to symbol table" is where what you guys have been calling the "overlay" starts.  Therefore the "executable" size is 0xE400 which is 58,368.

The size of the "overlay" can be calculated by simply subtracting 0x400 0xE400 from the file size reported by the O/S. (note: corrected wrong 0x400 size to 0xE400)

Also, if needed, the symbol table (referred to as "overlay" in this thread) has a header with information that may be useful (depending on what you're looking for.) e.g, in the file posted:

Code: Text  [Select][+][-]

1.  
2.  address n/a             - COFF SYMBOL TABLE -
3.  
4.                              File pointer to symbol table :   e400
5.                                         Number of symbols :    314    (   788)
6.  
7.                              File pointer to string table : 1.1b68
8.                                         String table size :   669a    (26,266)
9.  

Again, thank you both for providing the information I needed to make sense of what was being talked about in this thread.

And just for the record, since the symbol table is NOT in a section, it MUST be after the last section, therefore using the "pointer to symbol" table as the "executable" size is reliable (barring a malformed PE file.)

ETA:

Corrected: 0x400 to 0xE400

[msintle]

To OP:
I will try to port my way, pure WinAPI, to Lazarus and send source here.

Perhaps you can also paste that code here and I will see if I can make any progress with it, if you haven't yet had time to port it. A pure WinAPI approach does sound good indeed.

Excellent discussion and contributions from everyone, thank you very much. To clarify - is it true (or not) that Lazarus adds debug information to the tail end of an executable in non-standard form? Probably asking for this behavior to be changed is moot, but yes, it does break things and would prevent debugging of stubs in the Lazarus IDE (not a happy proposition).

Also: It does not happen on Linux. Meaning, debug data does not produce incorrect file sizes when reading the ELF header directly and computing the file size in that manner.

For completeness, macOS is untested due to its app-bundle structure and how it is innately hostile to data appended to the tail end of a stub with things like code signing, notarization, etc.

[440bx]

To clarify - is it true (or not) that Lazarus adds debug information to the tail end of an executable in non-standard form?

It's important to realize there is more than one type of debug information.

What is now considered debug information is DWARF or stabs.  I'll mention specifics about DWARF because those are fresh in my mind.  In the case of DWARF, every "part" of the DWARF debug info is in a PE section, therefore the debug info is part of the PE file and, as a result, that information will be included in the PE size.

If I remember correctly, stabs, which is another type of debugging information, is also stored in a PE section.  It would be good if someone confirmed this for stabs because it's been a while since I dealt with it and my recollection might not be right.

The other type of debug information, is the COFF symbol table, what has been referred to as "overlay" in this thread.  That information, unlike DWARF, is NOT stored in PE sections therefore it can be considered as not being part of the executable.   In the case of Windows, there is the "Pointer to symbol table" to locate it in the program file. 

As far as standard/non-standard... first, it isn't Lazarus that adds debug information, it is FPC that puts it there (regardless of the type of debug information), second, at least in the case of Windows, the "standard" way to store the COFF symbol table is to append it to what is PE information proper (what is in sections) and update the "pointer to symbol table" to enable locating it.  In Windows, that's pretty much the standard way. 

An alternative would be to create a debug directory entry (which is what Delphi does for its debug information) but, that seems redundant for COFF symbols given that the "pointer to symbol table" is there for that.

HTH.