Well, that is the point: if you use those different data types over the same class instance, you would use a variant, but if you can determine it is the same for the same class instance you would never use a variant, because variants are dead slow, really slow.
Which means that, although the map can be reasonably fast, you are stuck with slow code.
Note that slow code is not equal to bad code, but you should carefully consider your design.
Variants are slow, period. They are not a magic wand.