It’s a valid concern. I’m not sure it’s as difficult to get data dumps out of companies as you would think. We’ve done it several times in the past with different players (not all at Roon…).
We have been working on the problem of broadening our streaming options since long before this latest rumor. It is a priority for us to diversify here.
As for the 24/7 trawling through a streaming service API idea…lets do the back-of-the-envelope on that.
A typical streaming service has ~60m objects (albums+tracks+artists) today.
Say it’s available in 60 countries.
On a streaming service, objects look different and have different visibility, copyright information, etc. in different countries. Lets assume we hit the same update interval that we use with bulk ingestion from TIDAL today–24 hours. That’s 3.6 billion hits per day, or 42,000 hits per second.
Not impossible, but that’s quite an infrastructure to maintain on both sides. That “doing the work once per country” thing really puts a damper on the party.
When our TIDAL update interval was wider (2-4 days), it led to widespread complaints, and even at 24h, though the complaints are much less frequent, you can still see us drift out of sync each day if you’re paying close attention. Copyright rules change day-to-day at streaming services, and displaying stuff that we are not allowed to display is a big no-no.
I think you’ve got this already, @hifi_swlon, but in case others are thinking it: here is no on-the-fly way to build the Roon+TIDAL experience. We need to efficiently answer questions like “which performances of Beethoven’s works are visible in Brazil”, and TIDAL’s data set (nor any other streaming service…) has no idea about works and performances. The only way to figure that out is to “spider” the Roon metadata universe starting with the set of TIDAL tracks available in Brazil, and pre-compute notions like “work visibility”, “performance visibility”, “performer visibility”, and “album visibility” based on the track-level rules from the rights-holders.
About 35-40% of the time we spend generating our data set is spent managing copyright data for TIDAL. It’s a surprisingly intricate and resource-intensive problem to solve given how transparent it is from the user perspective. The database tables that hold the rights data have billions of rows. Not nice stuff. It’s a challenge to manage that data efficiently when it’s nicely delivered in bulk. Dealing with it one record at a time…ouch.