Last update of the week, with some good news at the end–
First, thanks for the feedback earlier. It’s invaluable to have a clear picture of what is going on out in the field.
We now fully understand what is happening. Feel free to skip over the tech stuff, but I’m going to share a bit because some people will be interested.
Amazon CloudFront, TIDAL’s CDN, has a bug (and I’ve got the wireshark logs to prove it).
Sometimes when we request a portion of a media file, the CDN is sending some extra data, beyond what the HTTP headers say will be sent. This data is a duplicate of a nearby area in the media file, but in the wrong place.
We are speaking to this CDN using Microsoft’s .NET framework. This is a widely used and trusted implementation of HTTP, and is generally perfect. However, with this particular protocol violation, Microsoft’s HTTP implementation fails to notice the error, and instead returns disordered data back to the application, while pretending that everything is a-ok.
When this data makes it into our audio pipeline, it causes the symptoms that you guys have been facing.
We were able to reproduce this protocol violation using other HTTP implementations, most notably the implementation from node.js which is also widely used and trusted. Unlike Microsoft, node.js correctly flags the protocol violation and reports the error back to the application. The fact that Microsoft passes along incorrect data in this situation appears to be a bug in the .NET framework.
Separately, it’s clear based on our testing that not all of the CDN servers are equally broken, and that there are different quantities of broken servers in different regions. We’ve had the easiest time reproducing this from Canada, but have also reproduced in some other locations. Some large parts of the world are totally unaffected by this.
So, this issue is hitting people who have one of the broken .NET runtime versions, and also are coincidentally also running into one of the broken CDN servers. And that’s why we have a relatively small group of people running into issues and not tens of thousands.
One final technical point, since this has been brought up a few times–this is affecting Roon and not the TIDAL apps because they use different CDNs and media streaming techniques for their own apps and for partners like us. We use a mechanism that allows Roon full access to the complete original FLAC files, which is important to us, and we are not attempting to change this.
Next steps for us–
Based on our current understanding, we think there is a switch that we can flip from our side which is likely to mitigate this issue right now without requiring a software update to Roon. It comes with some tradeoffs, so eventually we’ll flip it back the other way, but for now it will buy us some time and hopefully bring you guys some relief before the weekend.
In the medium term, we are about 90% sure that we have a reasonable semi-permanent workaround that we could ship in a new Roon build, and that work will start next week.
In the longer term, we are hopeful that Amazon will fix their bugs! CloudFront is widely used, and we are probably not the only people running into this. We are not their customer, so we do not have a way to force this to happen, but we will pass along our information to TIDAL in case they are up for pursuing it with Amazon.
Next steps for you–
I flipped a switch just now that may help with this issue. Restart your roon core two times in a row and let me know how it goes. You just need to restart the core, rebooting the machine is not necessary. I’m hopeful that this will mitigate the issue–it seems to have done so for us.