BorderSpacing is not a property that affects how the borders of a control are displayed.
It is a layout property, used to increase or decrease the spacing between borders of adjacent controls (rather like the "leading" between lines of text).
You can set a design-time option that shows the extent of the BorderSpacing of a control (Tools->Options->Environment->Form Editor, in the Grid groupbox check the "Show border spacing" checkbox: this will then display a red outline -- only at design-time -- indicating where the edges of the control are situated).
The LCL controls that are based on TGroupBox all use the underlying OS widget for their drawing, and so you are restricted to what the current OS theme provides. To change e.g. colours or borderwidths your OS might offer a way to tweak the theme you use. However, many OSs set default values which are hard or impossible to change. For example Windows does not permit changes to most widget properties unless you choose the Classic theme. Instead it offers the user choices between various themes. That is why the BorderWidth property has no effect: any value you set is ignored by the underlying OS, which is responsible for the display of the widget.
People who dislike being limited by an OS theme in this way go the custom drawn route (use the LCL CD components or a library such as fpGUI, or some other alternative to the LCL).