[Solved] Attach GDB to a TDaemon application

[ArminLinder]

Hi,

this question has been asked a while before (I found threads back until 2009), but IMHO never got a satisfactory answer.

Is it possible to attach GDB and the Lazuarus IDE holding the source code to a running daemon (TDaemon) bullt with the debug configuration/switches, obviously to debug the daemon with the same level of functionality one is used to from "normal" applications?

Thnx, Armin.

[Martin_fr]

I haven't tried....

What happens if you use the attach feature of the IDE? (Run > Attach)
If your daemon process runs as different user, you may have to start the entire IDE with sufficient rights.

Note, if your daemon forks new processes (processes, not threads), only the process to which you attached will be debugged. So obviously at the time of attaching the process you are interested in has to be running already.

[Martin_fr]

Or try gdbserver (search wiki). You may need to install the package LazDebuggerGdbmiServer.

Then you can start the daemon using:
  gdbserver [--host...]  your_daemon

[ArminLinder]

I haven't tried....

What happens if you use the attach feature of the IDE? (Run > Attach)
If your daemon process runs as different user, you may have to start the entire IDE with sufficient rights.

Note, if your daemon forks new processes (processes, not threads), only the process to which you attached will be debugged. So obviously at the time of attaching the process you are interested in has to be running already.

IDE and Service are running under the same account. No forking involved.

Nevertheless, attach to process fails.

[Martin_fr]

What OS ?

In order to comment on the gdb error, I would need a log / or the gdb "debug output":
https://wiki.lazarus.freepascal.org/GDB_Debugger_Tips#Log_info_for_debug_session

Have you tried FpDebug? Or is there a reason it has to be gdb?

[ArminLinder]

Thanks very much for your support!

My OS is Windows 11.

I did some more testing. First of all, I updated Lazarus from 2.2.0RC2 to 2.2.0 Release. The message has changed, I get a readable "Access denied (Code 5)" message now.

Tried:

- User account is a member of the Administrators group
- Lazarus and the affected Daemon are both running with this account
- Started Lazarus with elevated privileges (got UAC confirmation dialog)

Best regards, Armin.

[Martin_fr]

OK, it looks like the logger got broken in 2.2 (The log you returned is empty, in the sense that the debug-log-msgs are not in it. And never mind where the debugging failed, there would be some of them).

Anyway, with the error you get now, I am pretty sure that the log will contain the same error. That is the error comes from gdb, and not from the IDE. The latter could only be, if gdb could not be started). So the log is no longer needed.

Testing on Win10, an elevated Lazarus process, does start gdb as elevated process too.  (Though I did not have a target that was running elevated)

Anyway, it seems to fail due to restrictions from the OS.
So other than googling, I don't have any recommendation.

If you can successfully attach gdb (without the IDE), and know what was needed to do so, I can look if that can be achieved from inside the IDE.

Btw, elevated. Faint memory of mine... If your TDaemon is some driver (hardware driver), then it may run as part of the kernel. Not sure if simple elevated, is enough for that.

[balazsszekely]

It works fine at my side(see attached picture). I did the following:
1. Create a debug profile for my service
2. Run Lazarus elevated as SYSTEM with PSExec from SysInternals, like this(PSExec -sid c:\lazarus\lazarus.exe).
3. Run->Attach to Program..

[Martin_fr]

Great info. We should have a wiki page for stuff like this...

[440bx]

Great info. We should have a wiki page for stuff like this...

I agree, that is really good stuff.  Thank you for sharing that GetMem.

[ArminLinder]

Great info. We should have a wiki page for stuff like this...

I agree, that is really good stuff.  Thank you for sharing that GetMem.

I think I can hijack this info and add a chapter about "daemon debugging" to "https://wiki.freepascal.org/Daemons_and_Services", as soon as I am done with the basics. For Windows I know what you did (I had a similar idea and tried to run the daemon and lazarus under the same Administrator account, which did not work), but I did not try to run Lazarus and the daemon under the "System" account. I'll check on Linux as well and see back to you in case my current Linux skills proove insufficient.

Armin.

[ArminLinder]

Unfortunately, I am stuck again. What worked quite well yesterday, doesn't work any longer today. I suspect I got a deadlock somewhere, and thus my breakpoints are never reached.

What's happening is that I can start my service, get a PID, and attach the debugger using "run - attach to program", and then the debugger breaks in "ntdll!DbgBreakPoint", From there I can use "Run (F9)" to continue. Unfortunately this is where success ends, no further breakpoint is ever hit. I suspect that my daemon may hang somewhere, but I cannot find out where.

At the core of the problem lies, the the PID gets known only after the daemon has been started, and there won't be any way to let it hit a breakpont on start, because I cannot find out the PID and attach the debugger before all the onStart code has already executed.

Two possible soutions come to my mind, but each arises a bunch of further questions I do not yet have answers for:

- there could probably be a construct DebuggerAttached.waitfor() to tell the code to wait, until the debugger is attached. I could put this into the onStart method, maybe. Does such a possibility exist?

- I could try to run the code from normal lazarus in user context, I suspect the problem is not at all related to background operations, and then somehow trigger OnStart just like the OS service manager does. I did always suspect that this could be the functionality behind the -run command line parameter. So I put a breakpoint into onStart, and set the command line param string in Lazarus to -run. Nothing happened, the program started and ended immedately, without doing anything. Is there any way to achieve what I want? What is the functionality behind the -run parameter?

Thnx, Armin.

[Martin_fr]

Not sure if it works....
https://docs.microsoft.com/en-us/windows/win32/api/debugapi/nf-debugapi-isdebuggerpresent

while not IsDebuggerPresent do sleep(10);

If indeed it is hanging, hit pause. And then search through the thread window.

On Windows, pausing a process (as well as attaching, afaik  / but not hitting a breakpoint) injects a thread. So you will be in the injected thread, which is meaningless to you.

Btw, this means, your process must have finished the initialization in the system (and maybe sysutils) unit(s).
Because the setup code and data, that is called/used when new threads are created (even if the thread is "injected", it's a callback, so it happens for every thread).
If you manage to ask the OS to pause an fpc app, before that initialization => then the app will crash.

[Martin_fr]

Also to find the "hang"...

You can try https://docs.microsoft.com/en-us/windows/win32/api/debugapi/nf-debugapi-outputdebugstringw (both gdb and (afaik) fpdebug should show them / in the "debug event" window - if enabled in Tools > Options).

Or open a regular file, and log to it (LazLogger sholud be able to do that).

Then you can see, if certain points of your app are called. And (if you log, at locations where breakpoints will be) you can see if code with breakpoints is run, and therefore if the problem is indeed that they are not called, or if they do not work.

Mind, there are some bugs with breakpoints, if your code contains subroutines that are "dead code". Fpc removes them, but it keeps some debug info (and/or messes up info of the code in front/behind). In rare cases this affects breakpoints set in live code, after a dead subroutine. (dead "if then" blocks are not known to be an issue). https://gitlab.com/freepascal.org/fpc/source/-/issues/39499  // This affects only gdb, not fpdebug  // fpdebug on the other hand does not handle methods of old style "type TFoo = object .... end;"

[ArminLinder]

Not sure if it works....
https://docs.microsoft.com/en-us/windows/win32/api/debugapi/nf-debugapi-isdebuggerpresent

Martin, u'r genius :-)

The line worked like a charm, and now I got a really strong idea where the problem in my code stems from.

I herewith promise to add all those extremely useful tips in the wikis about daemons, and probably general debugging within the enxt 1 or 2 weeks, I need to finish my daemon first, but afterwards I got some free time.

It seems though, that IsDebuggerPresent is part of the Windows unit, so I suspect it won't work with Linux. Moreover I am mainly using Windows, and Linux only occasionally. Right now I'd test on Linux as good as I can, but some cross-checking with someone more familiar with debugging on Linux would be very welcome. Same applies to macOS, but since there is no requirement to run on macOS for my current project I hardly justify spending time on macOS, and I don't own any hardware Mac anyway, only VMs. nevertheless I consider Windows, Linux and macOS the "big three" we should always strive to support, so if anyone is willing to contribute the macOS part he'd be very welcome.

Armin.