Roon Metadata: An Update

Arguably nothing is as central to Roon’s mission as our deep, extensively linked metadata. Our current system was born from nearly a decade of work focused on finding new ways for collectors to browse their music, beyond what’s possible with text fields and file tags.

We’ve been extremely proud of the metadata infrastructure we launched with. It leaves your files untouched, while fetching updated and continually improved metadata for your entire collection, quietly and automatically. We’ve discussed metadata quite a bit on the community site, so we wanted to provide an update on some of the work that’s been going on quietly behind the scenes.

Moving Roon’s metadata forward

After our 1.2 release, it quickly became clear that metadata was probably the critical area where the team needed to grow. Whoever stepped into this role needed to not only understand what made our existing system work, but also needed to diagnose why, in some areas, we were falling short.

And because Roon updates the metadata in your collection automatically, any changes require extensive testing – a single new line of code can affect hundreds of thousands of albums, so verifying that every change is safe requires exhaustive validation.

Over the last few months @joel has taken the reins, getting up to speed on our infrastructure, understanding many thousands of lines of code, making changes, testing each one against content, and hunting for regressions; not to mention building diagnostic tools to allow us to isolate problems more quickly.

This work has touched our identification systems, equivalence matching, work detection, track metadata, TIDAL metadata, and more. One by one, these improvements have gone live (maybe you’ve noticed? :slight_smile:) but they are far from the culmination of the work planned for this area – they’re more like a giant first step.

We tested each of these changes in house for weeks, noting and investigating anything that appeared to be a regression. This is painstaking work, and we know how long people have been waiting on some of these fixes, and also some of the fixes and functionality that are yet to come.

Just remember that this first round of changes touches over a million albums – it’s easy to notice when something has gone wrong, but when a hundred pieces of logic “just work”, the entire system is nearly invisible.

What’s Changed?

The changes we’ve rolled out largely target a few troublesome areas, detailed below. We love to hear feedback, so please let us know how things are looking as you begin to notice this first wave of metadata improvements.

Track Equivalences

A significant number of metadata issues reported either here on community or by our QA team were tracked back to the system that calculates track sequencing across multiple editions of an album. Roon works hard to always surface the best data possible, even across different releases of an album that can have different track listings or the similar track listings in different orders.

This area of Roon’s metadata system has been completely rewritten to ensure:

  • More accurate track titles across hundreds of thousands of releases;
  • Elimination of erroneous or duplicated track titles for all reported cases, and those not reported.

Further, these changes will yield more subtle improvements:

  • More consistent track level metadata within the same album;
  • More consistent (and visually pleasing) display of track listings on album detail pages;
  • Better matching of track level metadata such as composers and works;
  • Fewer partially identified works.

Finally, we’ve improved track release information, meaning information like [12” Edit] or [Alternate Take] should appear accurately, and in many places where it was previously omitted.

Better Album Reviews

Roon processes tens (or in some cases, hundreds) of editions of a given album and, in the past, there have been times when the system has selected a review which is specific to a single release of the album, rather than focusing on the musical content (which is our current design goal). The practical effect was that, more often than we liked, all versions of an album would display a review referencing a remastered or a bonus track edition. The new review selection logic attempts to remedy this and, although it can never be perfect, we believe that the results achieved are excellent.

Missing Album Reviews

Are no longer missing! If you think that a review is still AWOL, please let us know, but bear in mind that you may need to wait for a metadata update or re-identify your album.

TIDAL search

At last, TIDAL search results are now essentially the same as those you get via TIDAL’s own app. For example, a search for “Rihanna ANTI” returns all four of the releases – regular/deluxe, explicit/sanitized – available on TIDAL (subject to being streamable in your TIDAL region).

These TIDAL search enhancements apply to user-initiated searches and to content displayed on performer detail pages. Note that the TIDAL New, Recommended, Top, and other TIDAL “Discovery” album lists remain as per TIDAL’s app without the additional versions.

Finally, a new “E” icon indicates TIDAL-marked explicit lyric content on all relevant TIDAL album thumbnails. This icon is available in the latest version of Roon 1.2, which will be released imminently, but you’ll already be getting the multiple version hits in searches.

Trackless editions

We acknowledge that it’s frustrating for our customers to find an album on allmusic.com (or a specific release of that album), but not find it in the Roon metadata database, even if the metadata on allmusic.com is obviously incomplete and doesn’t have any track information. These trackless editions are incomplete metadata records, but we recognize the value in being able to match up your album to at least some album-level metadata (e.g. for some large classical box-sets).

So we’ve removed the architectural limitation preventing us taking these and added 1.2 million trackless album editions into the metadata service which we were previously excluding.

Note that we will still not identify these albums automatically (without any track info, the risk of false positives is too high), but you can search for them using the Identification wizard and select them as your preferred metadata match. Roon will then combine your track information with the album level Roon metadata from the trackless edition.

Album Equivalences

A small improvement we’ve made to our album edition equivalence logic is the first of a number of important and far-reaching improvements we will be making in this area, leading to significantly better classical identification and metadata for both local and TIDAL albums. With immediate effect, 90k albums have more accurate edition information, with more to come.

Metadata Updates

Over the past few months, we’ve made significant architectural changes to our Metadata Cloud. Some of these changes enabled the work we have released over the last few weeks, such as tools for building internal metadata databases for testing and iteration of improvements.

These changes have also had more obvious effects, such as daily updates of Roon’s TIDAL content, ensuring new releases show up in Roon with the best possible metadata as close to their release date as our Cloud processing infrastructure will allow.

We’ve also made a variety of improvements to ensure that key metadata providers are updated on a more regular, more consistent schedule. This should also address the long-standing lag we’ve seen for extended metadata and reviews on new releases, as well as metadata corrections.

How Does This Work?

Every album in your library automatically retrieves updated metadata from our servers once a week. To see the effects of these changes you don’t need to do anything, although you can always navigate to an album’s edit screen and click Re-Identify to retrieve updated information immediately.

Will Everything Be Perfect?

That’s what we strive for, but any time a change affects a couple million albums, there are bound to be a few regressions or imperfections :stuck_out_tongue_winking_eye:

Examples where metadata appears to have gotten worse are really interesting to us, as we can only test against so much content in house. We depend on this Community to let us know when we hit the bullseye and when we’re slightly off the mark, so if you see something, let us know.

In particular, cases where reviews continue to be edition-specific will help us ensure the aforementioned album equivalence changes are on the right track.

What’s Next?

The improvements listed above lay the groundwork for the future of metadata in Roon. More will follow close behind, setting the stage for the suite of changes we’ve referred to in the past as Roon 1.3 – changes like vastly improved classical metadata, expanded editing, and crowd-sourcing of corrections. These will all come, but maybe not all at once! We appreciate your patience.

Expect the pace to pick up as known deficiencies are addressed and Roon’s Cloud Metadata Service matures, ensuring a stable foundation for your collection for years to come.

Thanks everyone!

26 Likes

Great work and thanks for updating us on what is happening.

I’ve just had a look at some of the 449 (out of 2550) identified albums I have without a review and indeed as promised re-identifying them led to some reviews being retrieved.

Am I to understand that if I do nothing these will automatically be updated once a week and so in 7 days time there will be loads of reviews where there were none?

SJB

Congratulations @joel and @mike. I don’t understand the implications of everything Mike said but this kind of careful and thoughtful incremental improvement is exactly why I became a life member.

Edit: Identifying a number of albums I notice that Roon reports a mismatch between my files and metadata when I have a .flac extension in the name.

3 Likes

Mike it all sounds very exciting and I look forward to the enhancements.

One thing I didn’t hear in your update is “end user provided meta data”. The bulk of my library is live concerts to which there are documented reviews (newspaper articles, online blogs, etc) that I’d love to be able to Cut & Paste into a user’s notes field (for lack of a better term).

I would not want to overwrite the meta data fields Roon is updating because sometimes your updates are better than my pasted reviews and other times mine more important to me than yours. The only solution I see is a independent user’s notes field that we can edit and Roon can display. I’d venture to say that many Roon users would find this meta data field handy as they can use it for their own reasons (personal notes on the performance, clippings of reviews they’ve found, or whatever tickles their fancy).

2 Likes

I’ve been asking for this since before Roon existed. In Sooloos 2.1 I used the reviews field, which one could edit: this kludgy solution justly irritated @danny and @brian. The early discussion about text fields ended up being hijacked by discussion of the “version” field, which some people find very useful, but which has nothing to do with a text field for notes and comments on tracks, performances, works, albums, artists, composers.

I really hope that at some point the idea of a user notes field will come back.

And PDFs, while I’m at it! :wink:

As someone who ran the trial long ago, and finally coughed up the cash, I am glad to hear of these metadata improvements! I have a very large music collection and was disheartened by the number of unidentified albums and the number of review-less albums.
I have a ton of classical and can’t wait for the improved metadata.
I’ll have to wait for the updates to hit!

OK, this sounds great. But, I would still love to have a Focus that allows me to filter the albums with metadata that changed in the last x days.

Original feature request here Focus on.....updated metadata

It seems even more relevant now.

3 Likes

Thanks a lot for the update. I am testing your software and it is great to know you guys are so responsive to the user’s need.
I really hope that all the hard work you are doing will make easy to improve the editing. Apart the bigger tickets there are some simple things that could really help. Yesterday for example I had to identify an album that is part of a box set. The album was identified correctly but the songs were not matched. So I had manually to move them where they belong. Unluckily for me it was disc 5 of a 20 disc box set so I had to move each song down one step at time across 4 album before being able to match them. Now I have still the other 15 to do and I really hope this doesn’t happen again otherwise there will be a lot of clicking…

Thanks a lot

Marco

Changes sounds great overall. I’d be sorry to no longer have access to reviews of specific editions of an album. Sometimes they can be quite useful, explaining differences or the remastering process. How about have a tab with Addition Reviews?

I’d like to then be able to access your pasted reviews! It would also be good if metadata from albums not in the Roon DB were imported from users. Like you, I have tons of live albums that I have tagged myself. Lastly, a huge amount of data for live albums is available online, but not from the sources Roon uses. I wish, as I suggested long ago, that Roon could use some of these sources and/or allowed users to contribute more easily.

It would be nice if Roon could extend it Meta Data sources, but let’s face reality, they’re not going to troll the numerous Deadbase sites to get the meta data you and I want. :wink: I’d be happy enough if they just gave me a free text field of unlimited size that I can paste into. Life would be grand.

PS: I like your idea of sharing meta data. If we could import/export meta data User fields then we could share meta data libraries. Even grander.

1 Like

It is actually not that difficult. There is, for example, a single site that has all Dead live recordings (http://www.deadlists.com/). Similar sites exist for Phish, Allmans, Neil Young, Bob Dylan, etc. Additionally, http://archive.org/ and http://db.etree.org/ have huge metadata repositories of live recordings.

My hope is that Roon will provide an API for third-parties to import metadata from sites like these.

A big thanks to Joel and everyone whose been working on this.
Regards,
Mario

2 Likes

Hi, thanks for the update.

What is the ETA for the user preference regarding showing already embedded lyrics over the Roon sourced lyrics.

99% of my albums are non-english (small percentage) or English live bootleg-style editions that Roon provides no lyrics for, although the studio editions with the same exact lyrics are in the Roon database. Displaying embedded lyrics is top priority for many users that took their time to actually carefully edit all their tracks.

Personally, I have gone back and currently listen to those in songbird since I can display all embedded lyrics easily. I would like to eventually be able to use Roon for all listening needs. Is this under the forthcoming “expanded editing”?

3 posts were split to a new topic: Moloko, Statues - No review

2 posts were split to a new topic: What decrees that a composer is listed alongside the track name?