Arguably nothing is as central to Roon’s mission as our deep, extensively linked metadata. Our current system was born from nearly a decade of work focused on finding new ways for collectors to browse their music, beyond what’s possible with text fields and file tags.
We’ve been extremely proud of the metadata infrastructure we launched with. It leaves your files untouched, while fetching updated and continually improved metadata for your entire collection, quietly and automatically. We’ve discussed metadata quite a bit on the community site, so we wanted to provide an update on some of the work that’s been going on quietly behind the scenes.
Moving Roon’s metadata forward
After our 1.2 release, it quickly became clear that metadata was probably the critical area where the team needed to grow. Whoever stepped into this role needed to not only understand what made our existing system work, but also needed to diagnose why, in some areas, we were falling short.
And because Roon updates the metadata in your collection automatically, any changes require extensive testing – a single new line of code can affect hundreds of thousands of albums, so verifying that every change is safe requires exhaustive validation.
Over the last few months @joel has taken the reins, getting up to speed on our infrastructure, understanding many thousands of lines of code, making changes, testing each one against content, and hunting for regressions; not to mention building diagnostic tools to allow us to isolate problems more quickly.
This work has touched our identification systems, equivalence matching, work detection, track metadata, TIDAL metadata, and more. One by one, these improvements have gone live (maybe you’ve noticed? ) but they are far from the culmination of the work planned for this area – they’re more like a giant first step.
We tested each of these changes in house for weeks, noting and investigating anything that appeared to be a regression. This is painstaking work, and we know how long people have been waiting on some of these fixes, and also some of the fixes and functionality that are yet to come.
Just remember that this first round of changes touches over a million albums – it’s easy to notice when something has gone wrong, but when a hundred pieces of logic “just work”, the entire system is nearly invisible.
What’s Changed?
The changes we’ve rolled out largely target a few troublesome areas, detailed below. We love to hear feedback, so please let us know how things are looking as you begin to notice this first wave of metadata improvements.
Track Equivalences
A significant number of metadata issues reported either here on community or by our QA team were tracked back to the system that calculates track sequencing across multiple editions of an album. Roon works hard to always surface the best data possible, even across different releases of an album that can have different track listings or the similar track listings in different orders.
This area of Roon’s metadata system has been completely rewritten to ensure:
- More accurate track titles across hundreds of thousands of releases;
- Elimination of erroneous or duplicated track titles for all reported cases, and those not reported.
Further, these changes will yield more subtle improvements:
- More consistent track level metadata within the same album;
- More consistent (and visually pleasing) display of track listings on album detail pages;
- Better matching of track level metadata such as composers and works;
- Fewer partially identified works.
Finally, we’ve improved track release information, meaning information like [12” Edit] or [Alternate Take] should appear accurately, and in many places where it was previously omitted.
Better Album Reviews
Roon processes tens (or in some cases, hundreds) of editions of a given album and, in the past, there have been times when the system has selected a review which is specific to a single release of the album, rather than focusing on the musical content (which is our current design goal). The practical effect was that, more often than we liked, all versions of an album would display a review referencing a remastered or a bonus track edition. The new review selection logic attempts to remedy this and, although it can never be perfect, we believe that the results achieved are excellent.
Missing Album Reviews
Are no longer missing! If you think that a review is still AWOL, please let us know, but bear in mind that you may need to wait for a metadata update or re-identify your album.
TIDAL search
At last, TIDAL search results are now essentially the same as those you get via TIDAL’s own app. For example, a search for “Rihanna ANTI” returns all four of the releases – regular/deluxe, explicit/sanitized – available on TIDAL (subject to being streamable in your TIDAL region).
These TIDAL search enhancements apply to user-initiated searches and to content displayed on performer detail pages. Note that the TIDAL New, Recommended, Top, and other TIDAL “Discovery” album lists remain as per TIDAL’s app without the additional versions.
Finally, a new “E” icon indicates TIDAL-marked explicit lyric content on all relevant TIDAL album thumbnails. This icon is available in the latest version of Roon 1.2, which will be released imminently, but you’ll already be getting the multiple version hits in searches.
Trackless editions
We acknowledge that it’s frustrating for our customers to find an album on allmusic.com (or a specific release of that album), but not find it in the Roon metadata database, even if the metadata on allmusic.com is obviously incomplete and doesn’t have any track information. These trackless editions are incomplete metadata records, but we recognize the value in being able to match up your album to at least some album-level metadata (e.g. for some large classical box-sets).
So we’ve removed the architectural limitation preventing us taking these and added 1.2 million trackless album editions into the metadata service which we were previously excluding.
Note that we will still not identify these albums automatically (without any track info, the risk of false positives is too high), but you can search for them using the Identification wizard and select them as your preferred metadata match. Roon will then combine your track information with the album level Roon metadata from the trackless edition.
Album Equivalences
A small improvement we’ve made to our album edition equivalence logic is the first of a number of important and far-reaching improvements we will be making in this area, leading to significantly better classical identification and metadata for both local and TIDAL albums. With immediate effect, 90k albums have more accurate edition information, with more to come.
Metadata Updates
Over the past few months, we’ve made significant architectural changes to our Metadata Cloud. Some of these changes enabled the work we have released over the last few weeks, such as tools for building internal metadata databases for testing and iteration of improvements.
These changes have also had more obvious effects, such as daily updates of Roon’s TIDAL content, ensuring new releases show up in Roon with the best possible metadata as close to their release date as our Cloud processing infrastructure will allow.
We’ve also made a variety of improvements to ensure that key metadata providers are updated on a more regular, more consistent schedule. This should also address the long-standing lag we’ve seen for extended metadata and reviews on new releases, as well as metadata corrections.
How Does This Work?
Every album in your library automatically retrieves updated metadata from our servers once a week. To see the effects of these changes you don’t need to do anything, although you can always navigate to an album’s edit screen and click Re-Identify to retrieve updated information immediately.
Will Everything Be Perfect?
That’s what we strive for, but any time a change affects a couple million albums, there are bound to be a few regressions or imperfections
Examples where metadata appears to have gotten worse are really interesting to us, as we can only test against so much content in house. We depend on this Community to let us know when we hit the bullseye and when we’re slightly off the mark, so if you see something, let us know.
In particular, cases where reviews continue to be edition-specific will help us ensure the aforementioned album equivalence changes are on the right track.
What’s Next?
The improvements listed above lay the groundwork for the future of metadata in Roon. More will follow close behind, setting the stage for the suite of changes we’ve referred to in the past as Roon 1.3 – changes like vastly improved classical metadata, expanded editing, and crowd-sourcing of corrections. These will all come, but maybe not all at once! We appreciate your patience.
Expect the pace to pick up as known deficiencies are addressed and Roon’s Cloud Metadata Service matures, ensuring a stable foundation for your collection for years to come.
Thanks everyone!