The office color panel shown in your screen shot, in the end, inherits from a plain old form, with many additions though, but should appear similar to the form of my demo.
Woops, yeah, I've should have looked one line further

TJvCustomOfficeColorButton = class(TJvCustomPanel, IJvHotTrack)
private
FMainButton: TJvColorSpeedButton;
FArrowButton: TJvSpeedButton;
FColorsForm: TJvOfficeColorForm;
So TJvOfficeColorButton also has a form.
My idea was: the popup windows opened from a toolbutton or a menu item, is nothing else than a "window" (in Windows terms). Therefore, it should be possible to insert another control (my color panel) into it.
If I look into TPopupMenu it inherits from TMenu (logically) and that one is dependent on the widgetset of the OS. This is not just a "form" in Lazarus which is opened for TPopupMenu. For Windows you end up in lazarus\lcl\interfaces\win32\win32wsmenus.pp. Well, with all the VistaDraw and support for different Windows-version this is going to be a hell to draw something on that window. Even if you could get a "Canvas" to draw on you need to eliminate the menu-items. Not to speak of the cross-platform aspect of it all.
It might be easier to just create a TToolButton with style tbsDropdown and hang a dummy TPopupMenu on DropDownMenu. In that dummy TPopupMenu.OnDropDown you can do the opening of the form (and cancel the dropdown of the dummy). Clicking the button itself would use the previous color. This would also be more cross-platform ready.
But you could go digging in lazarus\lcl\interfaces\win32\win32wsmenus.pp to see how that's done.