Watching for exceptions (system-wide)

[MarkMLl]

I've got a suite of Windows programs, one of which is a watchdog which makes sure that it's being patted regularly (via a named pipe) by various applications.

One of the other things it does is walk all windows and windowstations, looking for popped-up exception messages. If it sees something it doesn't like, it objects- loudly (using MIDI).

Has anybody ever looked at this sort of thing for Linux (GTK2)?

MarkMLl

[dbannon]

Has anybody ever looked at this sort of thing for Linux (GTK2)?

Do you mean the interprocess communication ?  On Linux its DBus officially but I found that the FPC specific IPC unit to be very easy, very light on resources and effective. And being cross platform, it works, to my knowledge, on Windows, Mac and Linux.

https://wiki.freepascal.org/SimpleIPC

Davo

[MarkMLl]

Do you mean the interprocess communication ?  On Linux its DBus officially but I found that the FPC specific IPC unit to be very easy, very light on resources and effective. And being cross platform, it works, to my knowledge, on Windows, Mac and Linux.

The IPC side of it is trivial. Somewhat more difficult is watching out for unexpected exception windows since these are completely oblivious of any conventions. e.g. if an arbitary program suddenly fails an assertion or tries to dereference a null pointer I want to detect the resulting exception dialog(ue).

The idea- and this is something that's worked will for 20 years- is that an exception window can be detected immediately (say within 30 seconds if that's how often the tree of windows is walked) while a watchdog might not be picked up for an hour when the next scheduled activity has not taken place. Both approaches have their advantages.

I've been experimenting over the morning, and it looks as though- at least with a GTK2 app on a KDE desktop- there is no useful hint in the caption etc. However I've not got far enough in yet to be able to see any text in a window: if I could see that then I could recognise "Access violation" etc. messages.

MarkMLl

[lainz]

Try TApplicationProperties (Additional tab) OnException, so you can handle all exceptions there...

[MarkMLl]

Try TApplicationProperties (Additional tab) OnException, so you can handle all exceptions there...

Not relevant. I want to be able to see anything on the X11 display that looks like an exception window, i.e. is unexpected and unplanned, rather than monitor exceptions in the current program.

MarkMLl

[lainz]

Try TApplicationProperties (Additional tab) OnException, so you can handle all exceptions there...

Not relevant. I want to be able to see anything on the X11 display that looks like an exception window, i.e. is unexpected and unplanned, rather than monitor exceptions in the current program.

MarkMLl

Ok I see, I thought that you was creating all apps on the workstation so you can add that event in each one. But it can be any app not only yours...

[MarkMLl]

Ok I see, I thought that you was creating all apps on the workstation so you can add that event in each one. But it can be any app not only yours...

It could /potentially/ be anything, and even if they were all programs I'd written from scratch a few examples from the Windows variant of the program come to mind:

An exception raised before the appl has fully initialised.

An exception raised by a dynamically-linked library that's in-process but has its own low-level stuff.

An exception raised by an external program e.g. invoked to send mail.

At present the best I think I can do- and this is actually a reasonable starting point- is to use xwininfo (or eventually one of the X11 APIs) to detect newly-appearing windows which have a certain range of sizes and placements. I can see the window captions but unlike the Delphi/Windows case this doesn't include a predictable message, and so far I've not worked out whether e.g. the "Access violation." message is in an accessible resource.

It's a hack, but it makes the difference between catching an arbitrary error after 30 seconds and catching it after several hours at which point sorting things out can be painful.

MarkMLl

[dbannon]

I can add to you list. Low level GTK Libraries will often drop complaints to the console. And just looking at them, it quite hard to tell which ones are serious and which ones really don't matter.

Try and monitor those ...

Davor

[MarkMLl]

In the case of the programs I'm monitoring, in the Delphi versions the major cause of out-of-process exceptions was the BDE/ODBC. Since that's obviously being replaced with in-process database libraries I think I can downplay the immediate requirement for exception dialog(ue) detection a bit, but experience suggests that it's still something very much worth having.

MarkMLl

[dbannon]

I must admit I feel the place to watch for exceptions and other things going wrong is in the application itself. Then, if it detects something, it can tell someone "I feel sick", if it dies, then easy enough for a monitor app to tell its not there anymore.

Davo

[MarkMLl]

I must admit I feel the place to watch for exceptions and other things going wrong is in the application itself. Then, if it detects something, it can tell someone "I feel sick", if it dies, then easy enough for a monitor app to tell its not there anymore.

Doesn't always work. Trust me. And how can a monitor app detect the difference between a program running properly and one that's dead except for having an exception dialog(ue)?

MarkMLl

[440bx]

I don't have a solution to the problem you want to solve but, if I needed to solve that problem, the first thing I would investigate is what happens to an application when it is handling an exception condition.

Here are some things I would investigate:

1. if the exception or abnormal condition is being handled by the O/S then, _maybe_ the O/S suspends all the process threads until the exception is acknowledged.  Under Windows, that is detectable.

2. Another thing I'd look at is what does the thread stack look like ?.  Two common characteristics of a thread handling an exception is that the window in which the problem is displayed is application modal.  There are a couple common side effects of that, 1. the stack is "stable" (the stack pointer does not change until the user acknowledges the problem, that can be detected) and 2. there is usually an address on the stack to a low level routine that returns control to a ring 0 routine that will do any necessary clean up and resume the thread's execution if possible.  It's a bit complicated but it is detectable.

The "gist" of the whole thing is figuring out what changes take place in an application when it is handling an exception then, figure out how to detect those changes quickly and reliably.  Basically, you're looking for a detectable pattern in the application's state.

I cannot offer any Linux specific suggestions since my knowledge of that O/S internals is much too deficient to offer anything specific but, I think the above is probably an approach worth investigating.

HTH.

[MarkMLl]

2. Another thing I'd look at is what does the thread stack look like ?.  Two common characteristics of a thread handling an exception is that the window in which the problem is displayed is application modal.

The thread stack is probably not something that can be inspected from outside. I agree that it's worth looking at all window properties as exposed by X11, and there's also a display manager protocol (EWMH?) which exposes various window hints which might possibly have something to offer.

This is something which is far simpler on Windows than on unix+X11, or perhaps I was just lucky with Delphi. I could possibly help things along a bit if I forced exception dialog(ue) names to be something distinctive, but that still wouldn't help the case where there was an early exception (e.g. last night I was creating unix domain sockets in the .lpr file, which is something that raises an exception unless the correct capabilities have been assigned... OTOH since there's no dialogue that won't sit there waiting for a response).

MarkMLl