I might be able to provide a partial answer for X11.
I have a monitoring program here which can detect if something that looks like an exception popup is on the screen, and alert a monitoring program to- on rare occasions- wake me up in the middle of the night. Looking at the code, I ran wmctrl -lp to get a list of titles, after which I filtered looking for candidate problems. So I suggest that in the X11 case it boils down to working out what API wmctrl is using... which might not be a simple job since I find all of the Window Manager stuff to be pretty impenetrable.
The predecessor program was written for Win-32 in the W2K era but in practice only used NT-era APIs, I've got a very vague recollection that I used the windowstation (?) API to scan multiple desktops and (presumably) filtered on names etc. after that.
HTH in some small way.
MarkMLl