Dynamic linkage on Linux x86_64 (calling back into main program)

[MarkMLl]

About five years ago I put together a useful little Lazarus/LCL-based app which could collect data from a cheap oscilloscope, display it, apply an FFT and so on. I left it in this state:

Anybody who wants a copy is welcome, but I'm not sticking it onto Github yet. The issue is that the project can use dynamic linkage between the frontend (generic) and backend (device-specific) parts, and while the frontend could call the backend I was having problems with calls from the backend to frontend. The worrying thing is that it worked properly with 32-bit code (on a 64-bit Linux system with multiarch, FPC 3.0.4) but not with 64-bit code, and when I last looked at this I got bemired a long way into the runtimes and standard libraries.

I've just put a solid few days work into it, with a view to writing more instrument-specific backends, which has uncovered something "interesting".

The overall programming and in particular linkage style is something I used for a MIDI-filter program 10+ years ago, which worked sufficiently reliably that it was- as an example- able to pick up a recompiled backend on-the-fly. However at the time everything was 32-bit...

The main part of the app ("frontend") is able to call into the dynamically-linked .so library ("backend") reliably on both x86 (i386) and x86_64 (amd).

On x86, the backend can call into the frontend to recover its API magic number, display a dialog(ue) box and so on.

On x86_64, the backend cannot call into the frontend, even to recover an integer magic number.

I've now got that to a state where I can demonstrate a problem in dynlibs reliably, by asking the frontend to recover its own magic number via a dynamically-linked entry point: it works on x86, but not x86_64.

I think I remember Sven writing something about a fundamental change to the x86_64 calling conventions (relative to x86) a couple of months ago. Can anybody remind me of the detail before I concede defeat and attach a test project?

Currently using FPC 3.2.2, but was probably on 3.0 when I first became aware of the problem.

MarkMLl

[cdbc]

Hi
On linux, I can call back and forth between lib & app with no obstacles.
I use NO declarations, i.e.: *no* 'cdecl' or 'stdcall' etc...
I use 'cmem' in both app & lib
All of this works for me on PCLinuxOS 64bit AMD.
I've also got some using fpc's sharemem, which I forked to linux and made dynamic... It too works.
I can even use strings arrays and com interfaces across library boundries...
Regards Benny

[MarkMLl]

Absolutely definitely not working here. The specific case is exposing an entry point in the main program to allow a magic number to be recovered, which fails even if the main program tries to link to itself dynamically.

I agree about having cmem and broadly agree about the lack of other modifiers. I'm sufficiently confident in the underlying technology to e.g. copy menu structure from a (non-displayed) backend form to the frontend.

MarkMLl

[cdbc]

Hi
Mark take a look at the attachment, it's just quick'n'dirty, but works here...
When you compile the library, just copy it to the host-directory...
Regards Benny

[MarkMLl]

Will do, but apart from that I'm working on knocking redundant stuff out of my own program to make a decent POC.

And bringing up a machine with Debian-11 32- and 64-bit, to eliminate various other known problems: I spent a chunk of Saturday trying to get FPC 32-bit on Debian-12 and ultimately resolved that there were quite simply too many things in the way.

MarkMLl

[cdbc]

Hi
Boy, do I know what you mean... Sometimes, tsk tsk ...sometimes 
Regards Benny

[MarkMLl]

OK, we're not quite talking about the same thing here.

Your test code opens a library dynamically, then makes an indirect call back into the frontend.

My app (before being stripped down for test purposes), opens a library dynamically and that library then opens the main program dynamically.

Before I went on a saga of building a test system... and getting dual boot sorted out etc. which appears to be getting more difficult as Linux tries to appeal to the masses, I added a bit of extra code into my app to make sure that the main program can open itself dynamically: and while that works on a 32-bit system it fails on 64-bits (Debian 11, Lazarus 2.2.6, FPC 3.2.2 with the systems being as similar as possible).

The attached test has chopped virtually everything out of the app (OK, there's still a lot of extra comments and variables etc., but the units are minimised).

If you run under a debugger and go to the end of BackendDynamic, then go up slightly to testTFrontend.MainProgramMagicNumber, you will see a marker of where things fail on a 64-bit system.

testTFrontend is a class encapsulating the dynamically-loadable aspects of the frontend (i.e. main program binary), LoadRoutine() encapsulates GetProcedureAddress() from the standard Dynlibs unit, and it's that that's failing.

I'd add that I'm specifically using Debian 11 to eliminate the compatibility issues that messed up the FPC 3.0 to 3.2 transition.

MarkMLl

[MarkMLl]

Found it I think. The unit which turns a loadable file into a unit was using a 32-bit THandle for a module rather than a 64-bit TLibHandle, which was overwriting a return address.

I've not yet worked out why that wasn't clobbering loading a library. More updates here if I find anything interesting.

MarkMLl

[cdbc]

Hi Mark
Yup, that pestering 'THandle', screwed me over before, when compiling old(er) code on 64bit...
Changing that, makes your program work, at least I get the magic number...
...But why all these /shenanigans/, when you can just serve up your *FrontEndAPI* on a platter, via an interface, like I do?!?
Is it because you're not the implementer of the library? Then I'd understand.
Regards Benny

[MarkMLl]

I've found the model of wrapping a library in a class to be very useful in the past, and being able to switch painlessly between static and dynamic linkage works well particularly when something needs debugging... except obviously in this case.

I've also got a program that will convert a definition of a C library into a similar format, e.g. the MIDI example code I pointed Carl at a few days ago.

So this works both for locally-written modules (e.g. plugins) and for externally-defined libraries, and the magic numbers etc. are an abundance of caution after seeing many problems with mismatched APIs over the years.

Mark

[cdbc]

Hi
Okidoki, that makes sense... 
Regards Benny

[MarkMLl]

Oh, and there's still a great deal of crap in those test files.

Basically, if you take those .inc files as being descriptions of the APIs then I can generate the rest of the module/class structures automatically: BackendProcs.inc becomes Backend.pas (for static linkage) and backend_dynamic.pas containing TBackend instantiated to a variable Backend for dynamic linkage... the syntax at the call points is identical.

The underlying technology goes back to some code that somebody gave me which he'd been using to interface Delphi to the (FORTRAN) NAG libraries.

Updated: This is now tested against a cheap DSO112A 'scope, and appears fairly solid subject to Linux occasionally losing the input device.

It can seamlessly reload the backend, either on receipt of a HUP or if it's seen to have been recompiled. This could potentially check an external repository for an updated version, but I've not implemented that.

There's plenty more which I'd like to add, in particular support for additional cheap measurement devices ** . However I'm going to stick it on Github as is, in the hope that there might be something in there of use or interest to somebody.

** If anybody knows of any sub-£75 devices with a usable PC interface please let me know. The one I'm using downloads the buffer via XModem, but I could potentially use other protocols.

MarkMLl