Apparently, that was the problem however, I did try that before and was running into problems that really had nothing to do with the filtering of the types but was affecting my attempt to correct something.
I now have a fully working TFPGMapEx and TFPGMapObjectEx that has a TMapPair for each that can be used as an enumerator for the map.
If anyone wants a copy of my generics unit file I can post is here. It piggy backs off of FGL code.
Thank You.