You might be right.
This thread suggests that it’s impossible.
I tried that some time ago to run Remote over tailscale although local playback wasn’t my goal. I noticed similar behavior but thought that it doesn’t work due to ACLs. Had to be wrong.