CPU-View

[Martin_fr]

Interesting, it works if the app is a GUI app, and has a form open. But not if it is a console app. (Well, may be something else, but that is the main diff I noted)

Thanks, I'll check into why it's not connecting.

My test is just a basic program, main begin/end block. No units. Just a few global vars for testing.

[Okoba]

It seems very cool! I will follow this work.
Thank you!

[Alexander (Rouse_) Bagel]

My test is just a basic program, main begin/end block. No units. Just a few global vars for testing.

Got it, I'll test it.

[Martin_fr]

I owe you an apology. My fault, the project I used to test had a project specific debugger  (Project options to gdb).

Works fine if using FpDebug.

[Alexander (Rouse_) Bagel]

I owe you an apology. My fault, the project I used to test had a project specific debugger  (Project options to gdb).

Works fine if using FpDebug.

Got it, yes indeed, CPU-View only works based on FpDebug

[Alexander (Rouse_) Bagel]

In the next few days, I want to add customizations and in them logging and crashdump settings to keep track of CPU-View status when parsing such errors.

[Martin_fr]

I think it makes sense to add many of my solutions to the main debugger.

It certainly does. Or will do.

I think we need to start below, with the Interface/API. In order to have your code become part of the IDE it has to access the debuggers via API. And work with all (or most) of the available backends. At least let the backends have the option to implement what is needed.

There are not enough IOTA interfaces, all debugging is laid directly on specific windows. If they let me - I could offer my implementation.

It will still take me a while to go through your code and understand it. But then it will also take good time to work on any Intf code. So I think its ok to start discussing it.

I am certainly most interested on your ideas, and maybe the implementation you have.
Trying to design an interface on my own, with little to no feedback is a receipt for oversight

There are 2 (not up to date) wikipages
https://wiki.freepascal.org/Lazarus_Debugger_Implementation
https://wiki.freepascal.org/TDebuggerIntf

Currently there are 2 packages
- LazDebuggerIntf (new)
- DebuggerIntf (old)

Most of the stuff from the old interface should eventually be replaced.

The concept of having Supplier/Monitor pairs is something that probably should survive, but needs tuning
https://wiki.freepascal.org/Lazarus_Debugger_Implementation#TDebuggerDataMonitor_and_TDebuggerDataSupplier_-_stack,_watches,_....

The new interface currently has mainly stuff for watches.

Then there is the frontend. Internally it still has a lot of old code...

IdeDebugger

I saw that you use that package. It is an internal package, with no guarantee on compatibility. It is considered part of the inner IDE and will change with it as needed.
However the plan is to provide an interface to it. (like IdeIntf for the rest of the IDE)

For the development you are probably on the save side using it. (if you publish, mind that IdeDebugger is GPL, not LGPL - that is most of its code was originally in the IDE itself, and therefore has to be GPL)


One important aspect is, that most parts of the interface should be async. Debugger may take time to respond, and the IDE should get blocked.

Generally there will be different aspects to the interface. Some may be (but not limited to)

1) Control: Issuing commands (run, step, ...)
I haven't done much thinking about that part yet

2) State (reported by the backend)
That is still based on the old code, and for once unfortunately the state is reported for each  "section" (global, watches, locals, stack). While all of them may need there events, they don't need repeated calls of the same "entered pause" (dsPause).
But nothing much concrete has yet been done.

3) Query data
Probably what we gonna look at the most.

So far some work has been done for watches. The importance here was that the backend can return the data in such a way that the frontend can format it in any way the user wants. The exceptions are memory dumps, which need to be queried separately.

I think for registers will be a similar API like for watches. So Register values can be described in a target-unspecific way, but will at the same time include the raw content of the the register.

Due to lack of a direct memory access API the mem-viewer (the one I did) uses a watch to fetch its data.

Asm is still based on really old code. Fetching asm with gdb had several challenges. And they unfortunately still manifest in the data.

4) "Attributes/Properties"
Some way of describing, what the debugger supports. But also what the target is.

A generic IDE viewer, will always need all data in a target unspecific way.  But more or less specialized views may exist for certain targets.


Before I go into details, I will wait for your initial thoughts (if that is ok with you).

And hopefully find some time to further study your code.


Couple of random notes:

The watch viewer (mine) takes %RSP and then follows the stack. In all watches (fpdebug) % can be used to indicate a register name. Because "RSP" could be a normal variable defined by the user.

Not looked in detail by your asm parser (script).
With Fpdebug lots of that could be done in the backend, when the asm is still in structured data, rather than printed. But then other backends can't do it, they would then need access to that code in some shared unit. On the other hand, I don't know if you keep track of registers over multiple statements?

[440bx]

I can't help but voice a concern about CPU-View, which is...

So far all Lazarus releases use formal releases of FPC, e.g, the current Lazarus release uses FPC v3.2.2

Personally, I am extremely interested in seeing the capabilities of CPU-View incorporated in Lazarus but, given that CPU-View requires the development/trunk version of FPC, it seems unlikely or problematic for, at least some, of Alexander's work to be incorporated into Lazarus.

At this point I'm wondering, will some future version of Lazarus require the trunk version of FPC ? (in order to use the CPU-View features) or will some of the CPU-View code be reworked by the Lazarus developers to compile with the stable version of FPC (whatever version that happens to be at the time.) ?

Just wondering how these features get into the "normal" Lazarus development.

[Martin_fr]

At this point I'm wondering, will some future version of Lazarus require the trunk version of FPC ?

No, or not what is then trunk, but what is now trunk may then be a release...

Incorporating into the IDE, can only pick what works with current + previous release. That rule wont be broken. (Though if it is just some small part of the rtl/fcl => that can be distributed as copy => like the new TProcess, which is used already)

But also it first needs to access the debugger in a more formal way (via an interface for that purpose). So before incorporating anything, such an interface must be created (that was a plan before the cpu view, but no has extra motivation / still a lot of work)

[440bx]

Incorporating into the IDE, can only pick what works with current + previous release.

That's reassuring.  Thank you for confirming that.

[Alexander (Rouse_) Bagel]

Before I go into details, I will wait for your initial thoughts (if that is ok with you).

I didn't know there were two debugger options and implemented everything based on DebuggerIntf, I need to learn what LazDebuggerIntf represents.

Regarding interfaces as I see them:
1. We need to introduce a set of interfaces under debugging windows IDisassemplerWnd, IRegistersWnd, IMemoryWnd, IStackWnd
2. The debugger itself should not know anything about the classes implementing these windows, the windows should be registered in the debugger, and the debugger should call the methods of these interfaces
3. All synchronization tasks with the worker thread should be assigned to the debugger interface. For example, now I have to call cache reset through DebugThread.BeforeContinue + TThreadWorker call before changing registers, or execute MaskBreakpointsInReadData + TThreadWorker
4. A bit later I can provide a more detailed description of all the necessary methods for each interface, but from the basic ones there should be
- procedure Init(Debugger: TDebuggerIntf); // for proper initialization without the hack with DebugBoss.RegisterStateChangeHandler
- procedure DoShowWindow(AHandled: Boolean); // to block the transfer of control and focus to other windows implementing the same interface
- procedure DoStateChange(AOldState: TDBGState);

Now in the debugger interface it is not quite clear what methods of the debugger can be called safely from the main thread. By trial and error I figured out that MaskBreakpointsInReadData must be called via TThreadWorker, I'm not sure about the other methods. Most likely, if somewhere else such synchronization is required and I have not performed it, it may lead to deadlock.
There is no easier way to get the ID of the active thread.
There is no direct access to TDebuggerIntf, I had to get it via DebugBoss.Snapshots.Debugger (but that may not be necessary after what you wrote)
Register reading is only partially done, so I had to implement it completely by myself, as I need to fully work with SIMD XMM/YMM registers both in Windows and (importantly) in Linux

And about the registers. Here I don't understand a little bit, what do you mean tracking? I react to notifications from DebugBoss->OnState and there I re-read the thread context + subscribe to TThreadsNotification.OnChange to re-update the registers when the thread switches in the debugger window.

[Alexander (Rouse_) Bagel]

I can't help but voice a concern about CPU-View, which is...

I need some time to explore the possibilities of FPC 3.2, probably it will be possible to modify the code of FWHexView (which is the basis for CPU-View Viewers) to support the stable version of the compiler.

[440bx]

I need some time to explore the possibilities of FPC 3.2, probably it will be possible to modify the code of FWHexView (which is the basis for CPU-View Viewers) to support the stable version of the compiler.

It would be really nice if your packages could use FPC v3.2.2.  I believe that would make it possible for them to be adopted more quickly which would benefit most everyone (at least those like myself who enjoy looking at the assembly code and debugging in assembler.)

I've looked at the CPU-View screenshots and would love to use it

[Martin_fr]

It would be good if you would in you posts disambiguate between backend and frontend. Debugger is a rather loose term.

It be too much to say we have 2 interfaces. We have 2 places that define part of the interface.
And important to point out: this is the interface between ONE frontend and ONE backend (more below).

The new class mainly is watch value related. The backend used to do all the formatting (print as dec, or hex, or oct...). And now there is a way to just give the raw data.

- DebugThread /ThreadWorker, are part of FpDebug. The normal frontend does not access them at all.
- Threading is introduced in LazDebuggerFp and handled inside of it. And if used through the official intf, then its not visible to the outsider.
  (Of course currently not all you needed is accessible via the proper interface)
- Some threading is also in FpDebug, but again not visible outside

The interface as far as it exists does not expose any threads. However it is meant to be called from the main thread only. I.e. it is not threadsafe, and can not be called from any random thread that the frontend or an extension may have created.


1 & 2)
The backend already does not know of those windows.-
- For the backend we need to have interfaces providing that data
- For the frontend we can have interfaces, to either replace a window, or hook into it (if/when appropriate). For watches I started an interface for Value providers. That likely will be extended, up to and including access to the canvas for custom paint.

3) As mentioned those issues are the lack of an interface. Your work around then of course encounter deep internal details....

4)
- RegisterStateChangeHandler / Init / DoStateChange
I have to do a bit of guessing here.  You mean
- so the frontend extension can initialize itself?
- or the frontend can send a command for the backend to initialize?

You do have to register somehow for that.
Either
- As current, register event handlers
- register an interface and implement that interface in your extension, and have those methods called in your extension via that interface.
- or ?

But yes, the current state is indeed to be replaced. It has to many internal values. More dedicated notifications should be added.


To start with an example (from memory / exact implementation needs to be checked)
(Assuming DebugBoss was exposed via IdeIntf or similar (TODO))

To be able to display watches the frontend registers a  IDbgWatchesMonitorIntf.

If you would replace the entire frontend (not just add/replace a window, but replace everything (the entire IdeDebugger), you would register your own. Currently there is only one monitor allowed, so if you just add a new Window the the existing IdeDebugger, then you have to interact with a future interface of the IdeDebugger (as the IdeDebugger already is the current frontend).
That currently does not yet exist. So you have to look at whatever DebugBoss.Watches offers.

So there will be yet another API to be done... DebuggerFrontEndIntf

That is imho correct. The Backend does not care that there are several Windows, and if they need or need not to know of each other. => E.g. what happens if I select an entry in the History window.
For the Backend there is only one frontend. It registers one Handler for watches.

The frontend then has to decide, if it needs to provide access for multiple parties (different windows) then that is something the frontend has to do.
Mind that currently IdeDebugger is the only frontend. And in most cases extra frontend like your CPU view will be additions to that. => but you could (not yet) replace it entirely.

By registering the IDbgWatchesMonitorIntf the IdeDebugger can also get new notifications (better than the current StateChange). But that hasn't yet been done.

Currently we have

Code: Text  [Select][+][-]

1. "Frontend (IdeDebugger)"   <>  LazDebuggerIntf/DebuggerIntf (the interface)  <> "Backend (E.g. LazDebuggerFp)"

We also need (todo)

Code: Text  [Select][+][-]

1. "Extensions" <> LazIdeDebuggerIntf <> "Frontend (IdeDebugger)"

Then if you want to actually get data (evaluate a watch) you need to call (on the backend interface)
  IDbgWatchesSupplierIntf.RequestData

- That is probably going to be handed through to the backend by the future IdeDebuggerIntf.
- You need to pass in a IDbgWatchValueIntf

IDbgWatchValueIntf carries all the info needed. The context (stack/thread), the expression, any options, and an interface to receive the result.

I opted for passing in an Interface (that the caller must implement). That allows easy extension in future.

Of course anyone implementing the interface (e.g. IDbgWatchValueIntf) will have to add new methods if it gets extended.
Therefore for most of the interfaces there is a template in form of a generic. So if the class implementing the interface inherits from that template, new methods will have default implementations.
Something like...

Code: Pascal  [Select][+][-]

1.   TWatchValue = class(specialize TDbgDataRequestTemplateBase<TBaseClass, IDbgWatchValueIntf>, IDbgWatchValueIntf)

So to get an expression evaluated as watch, you have a class implementing IDbgWatchValueIntf.
You call: IDbgWatchesSupplierIntf.RequestData(myClass);
The backend then can query the thread, stackframe, expression and more.
The backend then can call BeginUpdate, ResData, SetValidity, EndUpdate to write the result.
The EndUpdate or SetValidity acts as signal that the work has been done.

The handling of ResData calls is currently in IdeDebugger (frontend). That should maybe move into a public package. But that is part of breaking everything into interfaces, and tools packages.

You can get a memory dump with the above watches interface. But a dedicated API for that will be of interest too.

Though there are some benefits from getting the memory via watches.

If you want the memory for "MyFooInstance", then
- using a watch you can pass the variable name, and it will get the address.
- TODO: a flag could be specified if you want the variable or the data
  objects are a pointer, so the variable would be the memory containing the pointer.
  the data would be the memory pointed to.
- The backend does know the size of the data, and can return the exact amount of memory.
- The frontend can already override that size (at least extend it)

Of course if you are looking at other memory, things like the size of a variable do not matter. And then a simpler API would help, where you just give an address (still could be a variable) and the size you want.

There is no easier way to get the ID of the active thread.

I have to double check
ThreadsMonitor.CurrentThreads.EntryById[ ThreadsMonitor.CurrentThreads.CurrentThreadId  ].ThreadTargetId

That is the old interface (hence the complicated way). So changes can and should be discussed.

Though the idea to have an internal ID (just between frontend and backend) is IMHO ok. It simplifies implementing many different backends, as they are free to specify what they expect.
You can then query info using that ID. (like the OS id for the thread)

There is no direct access to TDebuggerIntf, I had to get it via DebugBoss.Snapshots.Debugger (but that may not be necessary after what you wrote)
Register reading is only partially done, so I had to implement it completely by myself, as I need to fully work with SIMD XMM/YMM registers both in Windows and (importantly) in Linux

Yes there is a lot more interface needed.

For registers a similar interface like for watches should be created. Or well, If you have different ideas, I am happy to look at / discuss them.

- Registers should be returned in a generic form, leaving all formatting to the frontend
- This may include a copy of the raw content, if a frontend exists specific for the current architecture.
- Changing the content of a register has to be implement too.


And of course threads and stacks too.... And all the rest.

Of course that will mean to change all existing backends. So its gonna take time, once a design for the interface has been decided.

And about the registers. Here I don't understand a little bit, what do you mean tracking? I react to notifications from DebugBoss->OnState and there I re-read the thread context + subscribe to TThreadsNotification.OnChange to re-update the registers when the thread switches in the debugger window.

You calculate addresses for access via registers? I haven't looked deeply at it. Is that just for
 [RIP + 123] style access?
 or also [RAX*4+123] ?

If RAX too, then based on the current value (register from context for current thread) of RAX?
Or if you have
  mov RAX, [foo] 
  mov RBX, [RAX + 123]   
In the 2nd statement would you have the value "[foo]" loaded to RAX in the first statement?

Obviously only if there is on jump from other code to the 2nd statement.

[Martin_fr]

As for the Monitor/Supplier concept.

It might be possible to allow more than one Monitor (frontend) to register with the Supplier (backend). But that may be adding unnecessary work to the backends....

IMHO it is better the frontend acts as hub. Though the frontend could replicate the exact same interface. And just distribute/forward any calls.


About state and notifications.

There is a what and a how.

** How:
- RegisterHandler
- RegisterInterface and get callbacks to methods on the interface)

** What:
There are (at least) 2 types.

* Global
Similar to the current state, but less internals.

Also there are events that come from the backend, and events that the frontend can generate itself.

Frontend:
- New Backend
- New Init (the IDE is about to start a new session), but nothing has been done yet (frontend)
- maybe some steps during init.
   for the backend they will differ
- ...

Backend:
- RUN (initial start) usually immediately before launch
- Pause
- Resume (run or step)
- Stop
- Error

- init and destroy might exist as direct response to the frontend requesting those actions.

The frontend can then add to that what kind of event is happening.

Though Pause will probably need 2 states (as currently) and it has to be seen if they can be split in the frontend or need to be done in the backend.
Though the 2nd may not be called pause.

Currently we have dsInternalPause. It is not a real pause, it is something were the debugger will continue. But the IDE should perform some tasks that it would do in a pause.
Example:
A breakpoint that is set not to pause ("break" action disabled), but records a snapshot => the Frontend must eval stack, watches etc and record that.

Of course the trigger does not have to be called "pause". That is just owed to building on top of existing code for far to long a time....


* Specific
That would be events applying to selected parts of the interface. E.g. special events for watches, or locals or stack, ....

In most cases they can be derived from global events, but in some cases they might be better tunable if the backend issues them.

Watches would just need to know
- data available => usually when the debugger goes to pause
- data no longer available
- data needs to be refreshed => E.g. a register value was changed via API, or memory was written.

The frontend would know, if it did write to memory. But if a register was changed, then maybe the backend can send invalidation for a specific thread...