Finally I realized there is no clean solution for this in Lazarus.
To show a form in a DLL in the context of the main application would need a clean and well defined solution how to link the entire LCL context from the main project (e.g. WidgetSet, Application, Screen etc.) to the DLL project. If both the DLL and main project are written in Lazarus.
I tried to switch the context, i.e. supplant those static instances I mentioned above from main project into the DLL but this produced exceptions. I am sure those three are not enough, there are many more, going deeply into the particular widget set etc.
Finally I encapsulated the DLL call with Screen's DisableForms and EnableForms methods, which at least disables all forms from the main project before the DLL modal form is shown:
function TPlugin.PropConfig(AHandle, AAppHandle: THandle): Boolean;
var
List: TList;
begin
if Assigned(FFuncPropConfig) then
begin
List := Screen.DisableForms(nil);
try
Result := FFuncPropConfig(AHandle, AAppHandle);
finally
Screen.EnableForms(List);
end;
end
else
Result := False;
end;
It produces some problems when switching programs with ALT+TAB but this is the only at least partially clean solution I could find.
Delphi has no explicit solution as well, but somehow supplanting Application.Handle in the DLL works fine.