I think this is a similar problem to this: Raspberry Pi 3b+ Clicking on USB Audio Devices. Currently the only way to fix it was to swap back to a non gigabit Pi3 (previous generation) or use an old netgear 10/100mbs switch.
I found that if I changed my managed switch to force the port to 100mbs I still had the problem.
In my mind it’s a problem with the latest Pi.