Yes, locating the right USB device, accessing and locking it is the main problem.
IIRC, long ago, with Unix, you had serial and block devices. You could directly copy to and from them, like they were files. With some restrictions, the serial devices were like pipes. Linux didn't do that at first, although they made an effort to make it like that about a decade ago. The status- and info-files are a great example. And a few years ago they abandoned that idea again and made separate device classes for each type of interface. Flexible, yes, but you need different tools and libraries for each one.
The only thing that survived it all is the Berkeley TCP/IP stack and sockets, which are part of every OS that has internet that I know of.
Ok, the main culprit for the complexity of USB devices is Microsoft, who wanted everything to be their vision of Plug-and-Play. And it is clearly designed by a committee. But I think KISS should be used for all layers, not just the top one.