Remote debugging (or other powerful debugging possibilities)

[Ahi]

Remote debugging (or other powerful debugging possibilities)

I ported the xine.h to objectpascal.

But it seems that I got something wrong. Lazarus' integrated debugger is of no help here.

gdb seems to have a user interface next to unusable.

Next I tried insight (which I think internally uses gdb) but at least it has a nice GUI.

I was surprised (and disappointed) to learn that although insight  can show disassembly of code it also has source code available, it cannot disassemble code that it has no source!

I don't know whether this is a limitation of insight itself or the underlying gdb.

But something even worse: when debugging, suddenly the machine locks up totally. Even "panic" keys such as Ctrl-Alt-Backspace or Ctrl-Alt-F1 etc do not do anything.

The first time this happened I had no choice but to use the hardware reset button.

Then I installed the ssh server so I can login my linux PC (having KUbuntu 7.10 installed) from a windows PC using putty. Now I can at least kill the offending process to avoid a hardware reset but that still does not help with debugging.

A remote debugger that could show the debugging user interface either on a windows PC or another linux PC would be useful, because then a temporary XWindow system lockup while debugging would be no problem.

What remote debuggers are available to debug linux programs ?
the IDA claims to be able to do it, but does that include the free 4.9 version ?

I guess that while porting the xwindow and gtk libraries to objectpascal, those who did that work may have run into similar problems.
If you did, what debugger did you use to find what is theproblem so you can correct it ?

So all those who have used a remote debugger to debug linux applications, please write about what debugger dis you use.

I also have the Kylix3 professional, but is that of any help here?
(then I would need to install for example debian sarge into a virtual machine to run kylix3, since it doesn't run in more modern linuxes). Also, I have no idea, if xine was useful at the time of selecting a version of xine to be included in debian sarge.


The xine engine is hard to debug, because the video playing needs to lock the XWindow system (as in the following example, which I translated to objectpascal from the "C" language example program shown here:

http://xinehq.de/index.php/hackersguide

)

    XLockDisplay(display);
    got_event :=  XPending(display);
    if LongBool( got_event )
      then XNextEvent(display, @xevent);
    XUnlockDisplay(display);

Without remote debugging, I think what happens is that  after the XLockDisplay -call, the debugger cannot access the XWindow -server, and so that results in a total machine lockup.

This is all theoretical, since the machine locks up in a much earlier place.

The only thing that I have got working so far, is that I started by converting the small example "C" program to a .so dynamic library, and the main() to dll_main() and called that from a freepascal program.

That  works otherwise perfectly, but it produces some strange error message about a mutex when closing the application. Until app closing time, there are no problems at all.

I tried to debug that situation too, but the lack of the possibility to single-step disassembly listing of a code I have no source in Insight left me with nothing to do a succesful debugging.

The only workaround to suppress the error message about the mutex I found was this:

program XineTest;
uses
  MyFirstUnit, {other used units here} ;

...

end.

---------------------------

Unit MyFirstUnit;

interface

implementation

initialization

finalization
  // do a syscall: sys_exit; this is equivalent to Halt; - but halt still produces the error message about the mutex.

  sys_exit(0);

end.

By putting MyFirstUnit as the first unit in the main program, I can guarantee that the finalization sections in all other units used will run before this one - so the only thing bypassed is the system unit finalization section.

[Marc]

choose remote gdb as debugger in lazarus, then a debug session will be started through ssh.
You yourself are resposible of copieng the exe to the remote target.

[Ahi]

choose remote gdb as debugger in lazarus, then a debug session will be started through ssh.

Nice to know, if that really works.

The file docs/RemoteDebugging.txt claims "* Remote debugging is under construction *". Maybe just nobody updated that file, it is dated 2003-07-23.

But, for hard-to-debug bugs I think I need a remote debugger that also can debug at the assembly language level.

It seems that the current lazarus version cannot do it.

But at least when an exception occurs, I remember seeing a dialog "someday a CPU window might popup here".

How about making that CPU window appear rather sooner than later.

I could even develope at least part of such CPU window.

Parts that I think I can do:

1.GUI

2. assembler / disassembler (so far for x86 linux only).
My solution would be to include a ".so" shared library, that provides the assembler / disassembler functions. They are developed in "C" and were originally written by Oleh Yuschuk, and modified to make it a ".so" instead of an independent program by me. License of that code is GPL, either version 2 or any later version, if the FPC project leader so desires.
for more info, google "ollydbg".

My modifications for the code mentioned also make it compile under linux gcc instead of windows Borland C++ builder.


3. CPU register display (possibly modifying them too).

4. visual part of assembly-level Step Into / Step over.

But where I would appreciate some help / more info, is this part:

How to integrate a new CPU window to the existing lazarus debugger ?
Isn't the lazarus debugger based on gdb ?
If so, does GDB have assembly level single step functionality, or should a temporary breakpoint be used instead ( as I believe is done when source-level debugging, how else would GDB understand something about objectpascal source and it's relationship to machine language addresses).

I have now downloaded the Lazarus 0.9.25 beta nightly snapshot, my download is stamped 2008-04-15 23:30.

Can someone with knowledge of lazarus internals please give some information, where should new instances of classes be created / destroyed and how to integrate new parts so that they will work correctly.

Also, since lazarus is multiplatform, how to deal with the fact that such solution for integrated assembly/disassembly will only work for x86, and at least this time, only for linux.

And, the new parts should be integrated in a way that works both for local and remote debugging.

While developing and testing new functionality for lazarus, is there any way to compile lazarus using lazarus ide instead of the make?

Doing that would make possible to debug it using lazarus' integrated debugger.

A filename conflict here is no problem at all: I am currently running the newer 0.9.25 beta renamed as "laz".

[Marc]

choose remote gdb as debugger in lazarus, then a debug session will be started through ssh.

Nice to know, if that really works.

The file docs/RemoteDebugging.txt claims "* Remote debugging is under construction *". Maybe just nobody updated that file, it is dated 2003-07-23.

There is not much changed since that option is added

But, for hard-to-debug bugs I think I need a remote debugger that also can debug at the assembly language level.

Thats not a requirement of the remote debugger, it something for the debugger as a whole

It seems that the current lazarus version cannot do it.

Correct. I still need to add some new class definitions to our debuggerwrapper to make display of it possible

But at least when an exception occurs, I remember seeing a dialog "someday a CPU window might popup here".

How about making that CPU window appear rather sooner than later.

I could even develope at least part of such CPU window.

Everyone can. As said the first thing is to add the generic classes. Something which is on my roadmap since I wrote the "someday a CPU window might popup here" message

Parts that I think I can do:

1.GUI

2. assembler / disassembler (so far for x86 linux only).
My solution would be to include a ".so" shared library, that provides the assembler / disassembler functions. They are developed in "C" and were originally written by Oleh Yuschuk, and modified to make it a ".so" instead of an independent program by me. License of that code is GPL, either version 2 or any later version, if the FPC project leader so desires.
for more info, google "ollydbg".

This is not needed, the disassembly wil be generated by the underlying debugger, not by ourselves. In this case GDB

My modifications for the code mentioned also make it compile under linux gcc instead of windows Borland C++ builder.


3. CPU register display (possibly modifying them too).

When a TDBGRegisters class is designed, it is just adding another debugger view to that class. Similar to the Locals view.

4. visual part of assembly-level Step Into / Step over.

That's not a problem, it's already supported.

But where I would appreciate some help / more info, is this part:

How to integrate a new CPU window to the existing lazarus debugger ?

Look at the localsview, how it is created etc.

Isn't the lazarus debugger based on gdb ?

Yes

If so, does GDB have assembly level single step functionality, or should a temporary breakpoint be used instead ( as I believe is done when source-level debugging, how else would GDB understand something about objectpascal source and it's relationship to machine language addresses).

Is has instruction step (stepi)

I have now downloaded the Lazarus 0.9.25 beta nightly snapshot, my download is stamped 2008-04-15 23:30.

Can someone with knowledge of lazarus internals please give some information, where should new instances of classes be created / destroyed and how to integrate new parts so that they will work correctly.

All generic debugger classess are defined in debugger.pp, the derived classes for gdb in gdbmidebugger.pp. Those debugger classes are managed in the debugmanager.

Also, since lazarus is multiplatform, how to deal with the fact that such solution for integrated assembly/disassembly will only work for x86, and at least this time, only for linux.

Id doesn't matter since it wont be a specific x86 solution

And, the new parts should be integrated in a way that works both for local and remote debugging.

There isn't a big difference between the local and remote debugger

While developing and testing new functionality for lazarus, is there any way to compile lazarus using lazarus ide instead of the make?

see build lazarus menu. However personally I compile lazarus in the IDE through makefiles by defining the make as a precompilation step. (Compiler options, last tab)

Doing that would make possible to debug it using lazarus' integrated debugger.

A filename conflict here is no problem at all: I am currently running the newer 0.9.25 beta renamed as "laz".

For other questions, you can contact me by email or at IRC(giantm) somewhere during european day or evening