The reason why input and output are generally either 16 or 24 bit integers is that audio hardware dac's and adc's work like that.
The reason to process the data internally as floats (either single or double usually) is that the internal processing has way more precision and so keeps a cleaner signal.
But you should not mix up the two: as I already suspected, you actually do just that.
Apart from latency issues - and Fred is aware of that and is not always an issue - Fred's unifying code is excellent.
Speed is not an issue here: optimized code for buffer conversions, even on 25 year old hardware, introduces just about 2 Ms of delay in total, back and forth, which is to human ears impossible to hear. That starts at around 25 Ms.
BTW: the line in is almost always there, usually nowadays over a USB type interface.