We have heard of a few reports that this issue appears resolved with the newest beta Apple TV firmware, and when this is released to the general public we are optimistic that it will also help resolve this behavior on your end:
I can’t comment on when Apple will release the stable version, but when they do please do let us know if updating to OS 13.2 resolves the issue on you end. Thanks!