I don't know how it is in Cocoa but for the other widgetsets there is no:
GroupBox.Caption.fsBold, since
Caption is just a String.
If you mean setting:
AGroupBox.Font.Style += [fsBold];
then it might or might not work; I don't know if that's the case with Cocoa but some themed Window/Composition Managers don't allow to change some properties away from what the "theme" directs, whether it's the font, colors, etc.
However I've very little knowledge of Cocoa (and MacOS X in general) so this might not be what's causing your trouble.