I have this problem too with an iPad Pro as an endpoint.
The bedroom, late at night, is the only time that I use the iPad as an endpoint and I get constant ‘an audio file is loading slowly’ messages. But I’ve noticed that it’s a lot better after 1230 when, presumably, other people in the block have gone to bed and the bandwidth is freed up. Not completely better but much better.
It’s also better (but not perfect) closer to the router but it’s not needed then.
Roonserver and router is in the lounge of an apartment.