The solution was indeed easy.
If you do not set TTreeNode.SelectedIndex to the index of the glyph that you want, then TTreeView assumes you do not want a glyph in selected mode. That actually make sense if, say the actual glyph depends on the text that you type.
All in all, it was a very easy fix.
Hope this helps other.
Also, be sure to check the site that is referenced in the TTreeView Example folder that ships with Lazarus.