While I agree with Thaddy in principle, I must comment that Thunderbolt 4 is pretty new in the Linux world. Thunderbolt uses DNA with little or not involvement from the CPU. So, its very hardware dependent.
So, if you have, say, some storage device at the far end of that Thunderbolt connection and connecting triggers, say, a backup, your DMA channel might be saturated and that would slow the system down. But that would imply a less than perfect thunderbolt implementation in you machine. But maybe ?
So, before giving up on the hardware angle, maybe disconnect other connected devices ?
Davo