mORMot memory manager

[spenov]

I'm using the mORMot memory manager. When handling an exception in the program, the stack is being read (DumpCallStack). While reading the stack, the executable (exe) file is opened (OpenExeFile). When the application shuts down, the memory manager's unit is finalized first, and then the system units that close the previously opened exe file (CloseExeFile). At this point, an exception occurs because the memory manager has already been changed (deinitialized). How can I avoid this conflict?

[Martin_fr]

Add the unit with the mem mgr by using the command line  -FaUnitWithMemMgr

That way it will be added earlier, initialized earlier, and de-initialized later.

[spenov]

This didn't help. The unit initialization order remained the same, and the same error still occurs.

[spenov]

I have attached a test project. On the form, you need to click the button, and an error will occur when closing the application.

[Fred vS]

I'm using the mORMot memory manager.

Hello.

In your attached project, I dont not see how you use the mormot memory manager.

Usually it is done by adding  mormot.core.fpcx64mm (for  x86_64) as first unit in uses section of the main program unit (project1.lpr).

Sure I miss something but what?

[EDIT]
I suppose that you have added this parameter, so, sorry for the noise.
$ fpc -Famormot.core.fpcx64mm project1.lpr

[Fred vS]

Re-hello.

I think the problem comes from sysutils conflicting with mormot manager.
CloseExeFile is used by sysutils after a OpenExeFile at finalization of sysutils.
But if using mormot.core.fpcx64mm as first unit, finalization of this unit will be done after finalization of sysutils (it is the fpc reverse order).


So a workaround is to create a "PreSysUtils" unit, add it before mormot.core.fpcx64mm, then as usual, add the "original" sysutils:

Code: Pascal  [Select][+][-]

1. unit PreSysUtils;
2. interface
3. implementation
4. uses
5.   SysUtils;
6. end.

and in project:

Code: Pascal  [Select][+][-]

1. program project1;
2. uses
3.   PreSysUtils,
4.   mormot.core.fpcx64mm,
5.   SysUtils;
6. begin
7.   // Your program code
8. end.

Compile it without -Famormot.core.fpcx64mm.

That way, finalization of SysUtils will be done before finalization of mormot manager.

(Yes, indeed, Gork helped me a little  )

[Zoran]

Re-hello.

I think the problem comes from sysutils conflicting with mormot manager.
CloseExeFile is used by sysutils after a OpenExeFile at finalization of sysutils.
But if using mormot.core.fpcx64mm as first unit, finalization of this unit will be done before finalization of sysutils (it is the fpc order).


So a workaround is to create a "PreSysUtils" unit, add it before mormot.core.fpcx64mm, then as usual, add the "original" sysutils:

Code: Pascal  [Select][+][-]

1. unit PreSysUtils;
2. implementation
3. uses
4.   SysUtils;
5. end.

and in project:

Code: Pascal  [Select][+][-]

1. program project1;
2. uses
3.   PreSysUtils,
4.   mormot.core.fpcx64mm,
5.   SysUtils;
6. begin
7.   // Your program code
8. end.

Compile it without -Famormot.core.fpcx64mm.

That way, finalization of SysUtils will be done before finalization of mormot manager.

(Yes, indeed, Gork helped me a little  )

That is same as just putting SysUtils before mormot manager unit in main unit uses section. SysUtils is initialized before mormot and finalized after it.
Just unnecessary complicated.

[Zoran]

And, I think the main problem is that the default memory manager is initialized in SysUtils, hence the point is that alternative memory manager must be initialized befoure SysUtils.
So, Fred, your proposal cannot help.

[Fred vS]

That is same as just putting SysUtils before mormot manager unit in main unit uses section. SysUtils is initialized before mormot and finalized after it.
Just unnecessary complicated.

That is exactly the problem: SysUtils is initialized before mormot and finalized after it.

// from Gork:

Why PreSysUtils Is NecessaryControl Over Finalization Order: Simply listing SysUtils before mormot.core.fpcx64mm in the main program doesn’t guarantee that SysUtils finalizes before the memory manager, because the main program’s uses clause directly controls the order of the listed units. SysUtils finalizing last in this case is the problem.

Dependency Chain: PreSysUtils creates an indirect dependency, making SysUtils part of a unit that is finalized after mormot.core.fpcx64mm but whose dependencies (i.e., SysUtils) finalize before it. This subtle shift in the dependency chain is what fixes the issue.
Avoiding Direct Conflict: Without PreSysUtils, there’s no way to force SysUtils to finalize before mormot.core.fpcx64mm using only the main program’s uses clause, as SysUtils would always finalize after the memory manager if listed directly.

Using the PreSysUtils unit is good if you want to rely on unit initialization order to ensure SysUtils finalizes before mormot.core.fpcx64mm, avoiding the need for manual file-closing workarounds.

the default memory manager is initialized in SysUtils

SysUtils uses the one defined by System that is replaced directly by mormot.core.fpcx64mm.
.
To resume:

Without PreSysUtils (with "direct" SysUtils):

Init: SysUtils → mormotMM
Fini: mormotMM → SysUtils (crash risk)

With PreSysUtils:

Init: PreSysUtils(SysUtils) → mormotMM
Fini: mormotMM → PreSysUtils(SysUtils) (safe)

That said, mORMot2 memory manager is a game changer and I dont understand why fpc does not adopt it as default memory manager.

Also note that this workaround was proposed by Elon's AI (and Elon has been rambling a bit lately  ... but his Dragon worked great yesterday  )

[Fred vS]

Ooops, forget all what I gave, I tested it and get a error when using sysutils or PreSysUtils as first unit in uses clause.

Code: Pascal  [Select][+][-]

1. Run error(211) Call to abstract method.

So, I apologize for all the loud noise.

And Grrrrrr for Gork.   

[EDIT] I created a issue on mORMot2. https://github.com/synopse/mORMot2/issues/373

[spenov]

The problem can be solved by modifying the compiler file lnfodwrf.pp. The module finalization procedure needs to be made public, and then called before finalizing the memory manager in your own unit.
But modifying the compiler's source code is not a very convenient or practical solution.

Code: Pascal  [Select][+][-]

1. unit lnfodwrf;
2. ...
3. procedure FinalizeInfodwrf;
4.  
5. implementation
6. ...
7. procedure FinalizeInfodwrf;
8. begin
9.    CloseDwarf;
10.    Abbrev_Offsets:=nil;
11.    Abbrev_Tags:=nil;
12.    Abbrev_Children:=nil;
13.    Abbrev_Attrs:=nil;
14. end;
15. ...
16. finalization
17.   FinalizeInfodwrf;
18.  

Code: Pascal  [Select][+][-]

1. unit Unit1;
2.  
3. uses ... , lnfodwrf;
4. ...
5. finalization
6.    FinalizeInfodwrf;
7.  

[abouchez]

... the memory manager's unit is finalized first ...

I don't understand this statement.
I have never observed this.

Here is how the mormot.core.fpcx64mm unit is setup:

Code: [Select]

initialization
  InitializeMemoryManager;
  GetMemoryManager(OldMM);
  SetMemoryManager(NewMM);

finalization
  SetMemoryManager(OldMM);
  FreeAllMemory;
I don't see anything wrong in this code. It even restores any previous memory manager.

Therefore, two remarks:

1) debug a little more and check exactly the order of units initialization/finalization

2) mORmot has its own Dwarf reader and exception handling for the stack trace - you may consider using it instead of the one from the RTL (it can also shrink a lot the debug information using a compressed format on production)

[spenov]

1) debug a little more and check exactly the order of units initialization/finalization

During call stack determination, memory is allocated, which is released in the module's finalization section. Since this module is finalized after the memory manager module, an error occurs when attempting to free the memory, as it was allocated by the mormot memory manager. I have verified this under the debugger. Therefore, my fix involves releasing the memory before unloading the mormot memory manager.

2) mORmot has its own Dwarf reader and exception handling for the stack trace - you may consider using it instead of the one from the RTL (it can also shrink a lot the debug information using a compressed format on production)

Could you provide a link where I can read more about this?

[abouchez]

this module is finalized after the memory manager module

I simply can not believe this.

[spenov]

I simply can not believe this.

I uploaded the project demonstrating this above.