[Solved] Compiler not finding missing forward declaration

[Wilko500]

Laz/FPC 4.3/3.2.3 on MacOs Sequoia 15.7.1

I have been chasing a strange memory problem on a "working" application where there are no memory leaks shown when compiling but after running for a couple of days it crashes with out of application memory.

I have been doing many, many tests to try and identify the cause and have identified a group of procedures that are the likely culprits.  During the last of the tests I discovered that two of these procedures did not have a forward declaration but the compilation was successful and provided no errors.

The missing declarations were discovered when one of the procedures was called from the main form rather than from with the same unit.  I get that that is to be expected because the declaration is missing from the interface of the unit thus out of scope from the main form.  What I don't get is why the compiler doesn't object to them missing in the unit.  Other procedures used only in the unit will generate compiler errors if their declarations are commented out.

I was my understanding that every procedure had to have a forward declaration.  What am I missing?

[jamie]

Possible you have a unit in the uses that happens to have the same name id.

Rest your mouse on the suspected missing item and see what code tools tells you about where it comes from.

Also, you can enable HeapTrc, run the app a little and abort it to see the list of possible unfree items which if are periodically being created without freeing, will accumulate.

Jamie

[cdbc]

Hi
Does any of that look like this e.g.:

Code: Pascal  [Select][+][-]

1. procedure ObjFreeProc(aPtr: pointer); external name 'BC_OBJFREEPROC';

or like this e.g.:

Code: Pascal  [Select][+][-]

1. { dispose-proc for 'TObject's - convenience publicly exported }
2. procedure ObjFreeProc(aPtr: pointer); public name 'BC_OBJFREEPROC';

Then you've encountered "Compiler-Magic" 
No worries, it's supposed to work like that...
The solution is to /import/ the func/proc with 'external name' xxxx; where you need it.
Regards Benny

[Thaddy]

Variables, typed constants, functions/procedures/classes and records can be intentionally hidden from external code by leaving them out in the interface section. This is a feature.
There are two ways to solve this, provided you have access to the sourcecode:
1. add their declaration to the interface section, the normal option.
2.what cdbc proposed:
example:

Code: Pascal  [Select][+][-]

1. // unit exports from its implementation section
2. unit expfromimpl;
3. {$mode objfpc}
4. interface
5. // left intentionally empty
6. implementation
7. // This unhides the code by exporting it
8. // instead of export you can use public
9. function echo(const s:string):string;export alias:'echo';
10. begin
11.   result := s;
12. end;
13. end.

and then use it like this:

Code: Pascal  [Select][+][-]

1. program echotest;
2. {$ifdef fpc}{$mode objfpc}{$endif}
3. uses expfromimpl;
4.  
5. function echo(const s:string):string; external name 'echo';
6. // main
7. begin
8.   writeln(echo('echo from implementation section unit expfromimpl'));
9. end.

The latter is a bit of an uncommon technique, but it is supported.
In that case note the names are case sensitive and must match the exported name and calling convention exactly.
The exported name does not need to be the name of the function itself, but can be any name.
Usually this is only done for external libraries, but it is perfectly legal to use it like so.

Note that if it is not your code, the author probably means that that code should stay hidden, so be careful: you have to know what you are doing. It can have unintentional side effects, one of which is memory leaks.... An example would be a managed type where the code relies on being manipulated just from that unit and the reference count is broken because you called it from outside the unit.

[Martin_fr]

The missing declarations were discovered when one of the procedures was called from the main form rather than from with the same unit.  I get that that is to be expected because the declaration is missing from the interface of the unit thus out of scope from the main form.  What I don't get is why the compiler doesn't object to them missing in the unit.  Other procedures used only in the unit will generate compiler errors if their declarations are commented out.

I was my understanding that every procedure had to have a forward declaration.  What am I missing?

No, they don't all need a "declaration" (in the interface).

Code: Pascal  [Select][+][-]

1. unit abc;
2. interface
3. implementation
4.  
5. procedure foo;
6. begin
7. end;
8.  
9. procedure bar;
10. begin
11. end;

In the above bar can call foo. But foo can't call bar.

As soon as a procedure has been in the code (either in the interface, or it's full body in the implementation) then any other function (in the same unit) can call it.


On top of that you can do

Code: Pascal  [Select][+][-]

1. unit abc;
2. interface
3. implementation
4.  
5. procedure bar; forward;
6.  
7. procedure foo;
8. begin
9. end;
10.  
11. procedure bar;
12. begin
13. end;

And they can both call each other. Yet neither is in the interface.

[creaothceann]

Laz/FPC 4.3/3.2.3 on MacOs Sequoia 15.7.1
The missing declarations were discovered when one of the procedures was called from the main form rather than from with the same unit.

You could try always using unique names (not shared across units), or always adding the name of the unit to the calls. (You can enforce the latter style by putting the procedures as static methods into a record, essentially treating the record as a namespace.)

[Thaddy]

Not relevant. Martin's response is a welcome addition to explain it.

[d2010]

You  have other solution, but (A OR B) is not very happy solutions.
a) You move all, but all function/procedures declarations before i"implementation"

Code: Pascal  [Select][+][-]

1. Implementation
2. Function if_variant(condtion:wordbool;s1,s2:variant):variant; forward;
3. Function if_cha(accessCheck:boolean;var1,var2:char):char; forward;
4. Function if_int(zeroone:wordbool;var1,var2:integer):integer;forward;
5. Function dos_beeep:wordbool;forward;
6.  

Code: Pascal  [Select][+][-]

1. Function if_variant(condtion:wordbool;s1,s2:variant):variant;
2. Function if_cha(accessCheck:boolean;var1,var2:char):char;
3. Function if_int(zeroone:wordbool;var1,var2:integer):integer
4. Function dos_beeep:wordbool;
5. Implementation
6.  

b) You re-order all functions/procedures from minin-used  to max-used by you.

[Wilko500]

Thank you all for your responses and I do apologise for the delay in getting back to this post.  I am still very distracted by the memory "leak/excessive usage" issue and will post separately soon, I hope.

I was hoping in vain that the missing declarations would point to some coding error that would in turn help me understand the memory issue.  Alas, that was not the case and Martin's explanation is what I was missing.  What had confused me was the order of the declarations and the actual calls which had by chance hidden my coding error in the first instance.  Of course I ought to have known that since it works like that in nested procedures which I know about but have never used. 

The other comments also provided info that I was not aware of so thank you for your input.

I will mark this as solved

[Martin_fr]

I am still very distracted by the memory "leak/excessive usage" issue and will post separately soon, I hope.

Leaks can often be found with HeapTrc. (On Mac you might need 3.3.1 or some trickery, as it may not print line numbers).

Though, they are not always easy to understand.

- Often it is best to start with the leak at the end of the list, because if that is some object (class), then other leaks above may be sub-objects of that class (and they go away if that object is fixed)>

- Managed types (strings) are reported where they were first created. But they may then be hold by some object, that did not create them. So the reported location is (mostly) useless. But then if you work bottom up, you likely solve the object first, and they go away.


If you are on Linux, you may also use "valgrind --tool=memcheck" which is a very good leak and dangling pointer detector.


Both, the output of heaptrc and valgrind can be pasted/loaded in menu : "view" > "Leaks and traces". And then you can doubleclick to go to the source.

[jamie]

@Wilko500
   What add on components are you using, those that are not normally installed on the IDE by default?

Jamie

[Wilko500]

@Jamie,  The additional components I am using are Industrial, LazSerial and Chart.  Those additions made the testing particularly challenging.  I attempted to create a test program to limit the number of variables but that wasn't very successful.  In retrospect I wish that I had been using debug mode early on in the development.
In early testing I think I managed to rule out those packages as the cause.  Industrial and Chart , easy to comment out but LazSAerial not so because I am interacting with live inverter. However I did manage to duplicate the memory usage problem using Synaser instead of LazSerial so it is probable that neither is cause.

@Martin, HeapTrc, I have been using it.  That's the tool that helped me identify the canvas font leak with help from this forum.  In the current problem there is no leak shown by HeapTrc.  I have done some research on using Xcode Instruments to profile my memory usage and how to install Valgrind on MacOs. However I do hope I don't have to go down that route.

I have further tests to run and then I will post again in a new thread. I think I have identified a procedure that can cause the problem but I have doubts about it being that straightforward, I fear that there might a "MacOs" elements lurking somewhere.

[Martin_fr]

In the current problem there is no leak shown by HeapTrc.  I have done some research on using Xcode Instruments to profile my memory usage and how to install Valgrind on MacOs.

You may not have a "classical leak" in the sense that the leaked memory is unreachable.

That is, usually when thinking of a leak, you think of

Code: Pascal  [Select][+][-]

1. a := alloc_some_mem; // could be an object created or anything direct memalloc, or....
2. a := nil; // you did not free it, but you no longer have a variable to reach it, so you can no longer free it

But of course you could have

Code: Pascal  [Select][+][-]

1. var a: array of TFoo;
2. begin
3.   SetLength(a, length(a) + 1);
4.   a[length(a)-1] := TFoo.create;

You keep the pointers forever, you just never free them, until the app exits (so HeapTrc says they all got freed).
If you run that long enough, you run out of mem.

The above is of course simplified. But a linked list would be less obvious.
Or:

Code: Pascal  [Select][+][-]

1. TButton.Create(Form1);

The button is hold by the owner Form1. You may never destroy it yourself, the form will, but only once the form is destroyed, which may only be when the app exits. (Hide/Show does not destroy it).


A complete other scenario to run out of memory is, to not run out of mem, but not to have any large enough block: fragmentation.

If your app runs for a long long time, and does frequent alloc and free. Imagine you memory as continuous list of bytes.
- Alloc 10 from 0 to 9
- Alloc 10 from 10 to 19
- Free the first 10
- Alloc 11 => you can't use the free block of 10 at the start, you need to get fresh mem.

Imagine you got all the memory fragmented, so the largest block is maybe 1000, but you need 1100.

You can get the fpc heapstatus, and query (and writeln for debugging) the amount of really allocated mem. Then you see if that goes up, or not.

But keep in mind, fragmentation is not overly common. It does happen, but its on the lower likelihood of being your problem.
Though, then again, some server apps are written to alloc mem once at start up, and then never alloc or free any after that, and at least in part to avoid fragmentation.

[jamie]

HeapTrc is good when it comes to monitoring the local heap in your application. Items that get created within your application are reported via HeapTrc when exiting if not relinquished back to the local Heap, which is in your application.

 However, it does not report items that are pass to your application via handles and you could get an out of memory error which would not mean it's actually out of memory but maybe out of resources. That could be anything that involves handles from the OS.

 If you have a resource monitor for the Mac you could run maybe you can figure it out.

 I remember years ago I helped someone with a Toolbar app that will crash the system with an out of system resource error if you keep passing the mouse over it. What that problem was an ICON from the system was being fetched each time the mouse passed and never freed. After a while the system ran out of handle space and whatever else.
 
 So, anything that involves handles should always been released.

Jamie

[Wilko500]

Thank you both.  I think that both scenarios are possible  and perhaps likely.  It's time to show you some code of where I think the problem may be.  To that end I'm going to start a new thread because this one is heading off in a different direction from where it started.