"Unavailable" music in playlists

Hi Roon gurus @support:

Setup: I use a 2013 MacBook Pro with 8GB of RAM as my Roon Core running OS Sierra. I’m running Roon 1.3 (build 196). My watched folders are primarily on a USB hard drive connected to a 2015 iMac that is connected to my Airport Extreme router. My MacBook Pro is also connected to this router via a long run of ethernet. There is a small number of files on my core machine, but the vast majority (over 40,000 tracks, 2.5TB of music) are on the USB hard drive in two folders.

Issue: When I switched from 1.2 to 1.3, almost all of my music stayed on the USB hard drive but I moved the USB hard drive to a different location on the network. Specifically, the hard drive had been attached directly to my MacBook Pro core machine and now it is attached to the iMac described above. On my MacBook Pro, I turned on Internet Sharing (File Sharing) to designate the two folders on my hard drive as watched folders and added those folders. I did not assign a static IP address to the iMac or otherwise use SMB sharing, though I have a friend who is a roon user and IT professional who says I should do that and he will be helping me with that later today.

Roon then began scanning the tracks and then imported the tracks. The total number of scanned tracks was something like 45,000 but the total number of imported tracks was about 38,600. The first thing I noticed when importing was done was that my playlists contained a good percentage of tracks that were labeled “Unavailable.” If I searched for the album, however, I was able to find it in my library but it was labeled “Unidentified,” and Roon then presented me with option to edit the metadata (I think). I didn’t do any editing. Also, in the Overview section of Roon, I noticed that the albums that showed up as recently added were definitely not the same albums that I had in fact recently added using 1.2. So something went wrong with the importation of files.

The next day, when I went to use Roon, none of my tracks were available. When I went into Storage, the two watched folders containing most of my music said “DirectoryNotReady.” So I went back into Sharing, added those folders again, and Roon began rescanning and reimporting the music. When this was finished, Roon had imported the same number of files as before. But this time there were even more files with the Unavailable label in my playlists, and the playlists themselves were truncated. For example, a playlist containing 50 tracks would only show 23 tracks. Of the tracks that did show up on a playlist, the tracks that were contained on my USB drive were labeled Unavailable. But Tidal tracks in my playlists were left unchanged.

I then tried connecting the USB drive directly to my MacBook Pro core machine (in other words, going back to the same storage setup that I had been using with 1.2), and rescanned and reimported all of the music after designating the appropriate folders in Storage. Result: same issues as above.

Note: I do have a copy of my Roon 1.2 database folder from about a month ago. I should have made a copy of 1.2 right before I changed to 1.3, but if we have to go back to 1.2, I guess I’ll just have to lose changes I made in the last month.

Further note: If you need me to send logs or PM you or send any other info, please include instructions on how to do so as I have not done this before.

THANKS!

Update: I was able to firm up the Storage connection to the two folders on my hard drive with the help of my friend. The hard drive containing my music is attached to my iMac (not my Roon Core machine), and each of my watched folders now has the SMB IP address that is assigned to the iMac. After redoing this Storage connection, my music folders were rescanned and reimported. Bear in mind that I have now rescanned and reimported my entire library multiple times over the last few days. That in itself could be a source of the problems described below. Here are the problems that remain:

  1. My playlists (I have 50 of them going back eight years – these are VERY important to me) are missing many tracks. When I say missing, I mean they don’t show up in the playlist at all. At the top of the playlist, just under the title, Roon reveals exactly how many tracks are missing (e.g., 4 of 10 tracks, 21 minutes). In many playlists, a large number of tracks are missing, sometimes more than half. But most, and perhaps all, of these tracks, are still in my library, as revealed by a search for the track. So the missing tracks were apparently scanned and imported but not recognized by Roon for inclusion in the playlist.

Please note that I have a complete list of tracks for only a small number of these playlists, so I cannot be sure which tracks are missing. The only source that I believe would have this information is my old Roon 1.2 database. As noted above, I do have a copy of that database, although it is about a month old. But it should have almost all of my playlist information. So maybe you can advise me on a way to retrieve playlist info from the old database.

  1. Other tracks in my playlists are labeled “Unavailable” and cannot be played from the playlist. When I click on the album name in the playlist I get a message: “This album was not found.” But once again, these same tracks and albums have been imported because a search reveals they are in my library and they are playable. Here is an example of the file path for a track that successfully appears in the playlist:

/Users/randysimon/Library/RoonMounts/RoonStorage_baa350c543051522e8713eb92167380dc03e3e46/Compilations/The Art Of Cecilia Bartoli/10 Mozart_ Don Giovanni, K 527 - La Ci Darem La Mano.aif

For a track that is labeled “Unavailable,” there is no file path when I click on View File Info. But when I find that same track using Roon’s search function, the full path is available.

  1. When I click on Overview and look at Recently Added albums, what I see looks nothing like the albums I had in fact added recently before changing to Roon 1.3 and changing the location of my hard drive. In addition, many albums are labeled “New” despite the fact that I had added these albums many years ago. This problem is not as important to me as the problems involving playlists. Sure, I would like to see the albums that I in fact added in chronological order, but if that is impossible, I can live with it. The main thing is that all the albums are there and playable.

  2. As far as I can tell, all of the Tidal tracks in my playlists are still there, though I am not 100% certain of this.

Please help: I’ve been patiently waiting for someone to reply for more than 30 hours.

Hey @alwayslearning - sorry for the trouble here. I think I know the source of a few of these issues, but I need a bit more information to be sure.

First, a little background on how we handle your files in Roon-- what happens is that each file you import into Roon gets a signature in the database, and Roon should be able to track that signature if you move the file to a new location, or even if you rename it.

It’s important to note that it’s of course possible to have more than one copy of a given track in your library, and Roon will track them both (along with their associated edits, playlist appearances, play history, etc).

Here’s the issue; say you have all your music in a network. Then, you make some storage changes, and now Roon thinks this folder is “temporarily offline” – this is when you’d see DirectoryNotReady. Because the directory is still configure, and Roon isn looking in the old location, the app MUST assume those files are coming back.

What you want to do is tell Roon those files are moving elsewhere, by editing the folder and pointing to the new location, or by removing the folder from Roon completely. In either case, all the files would be marked as “missing”, and when you add the folder in its new location, Roon would recognize the file signatures, match it up to the database, and re-attach all your edits, playlist appearances, etc.

Instead, if you leave the directory in that “temporarily offline” state, when you add the new folder, Roon is assuming the first location is coming back, and so all the files are considered to be new imports. It’s a bit complex, but it’s designed to be robust and allow for the movement of files to new folders – it just needs to be done the right way.

What I would recommend at this point is that you remove all storage devices in Settings, and once your library is empty, re-add just the current folders. Unless you’ve been making additional edits in the last few days, I think everything should be right where you left it, although feel free to load up the backup from a month ago if you want to be absolutely sure.

In regards to your other question:

I can tell you that we’ve seen other users struggle with similar with the performance of Apple’s SMB implementation, but we have seen this work. I am wondering why you don’t just run the Core on the iMac, if that’s where you intend to connect the USB drive. As a general rule, directly attached USB storage is going to be a lot simpler and more reliable than what you’re attempting here.

Let me know how the above sounds, and if you need some more details on any of it, and we’ll be happy to help. Thanks for your patience so far!

@Mike:

Thank you so much for your response. Let me try to respond to some of your points so we can determine where we go from here.

At first, I did not remove any of the old folders. I just added new locations on the network for the old folders. So that apparently explains why I saw the DirectoryNotReady label. This happened a couple times. It might also explain why the Recently Added albums look so different, and why so many albums are labeled as “New” that were in fact added long ago.

When I made my most recent change in the location of my folders, I removed the old folders before all files were rescanned and reimported. So, in theory, shouldn’t Roon have recognized the file signatures, and reattached my edits, playlist appearances, etc.? Or did I have to remove these folders the first time it scanned and imported the files from the new location? In any event, my playlists have all the problems I described above and I’m still looking for a fix. Here’s how my Storage settings look now:

So do you still recommend removing ALL of my current folders, then adding back just the two folders (DSD and Music)? Given that Roon did not get the playlists right when I did this most recently after removing the old folders, I’m skeptical that this will work.

And if I want to check the result against the backup from a month ago, can you give me step-by-step instructions on how to do so? If a new attempt at rescanning and reimporting does not fix the playlists, it seems that the backup holds the key to reconstructing them, though I am not sure if I would then have to do the reconstruction manually after seeing what the missing tracks are, or if there is some more efficient way to reconstruct the playlists. And then how do I get back to using 1.3? If you don’t mind, I am going to need some detail in how to proceed.

The reason I will not use my iMac as my Core machine is that I am using a special “bridged ethernet” connection in my audio setup (details on computeraudiophile.com) that requires my Core machine to be in close proximity to my streamer (SoTM sMS-200), which is in my listening room. The iMac is in the other corner of the house. And I would prefer not to directly attach my hard drive to my Core machine in the listening room, as I would rather not have a noisy (physically and electrically) hard drive in the listening room next to all the other audio gear. So I am committed to making this work in the current setup.

Thank you for your help.

Jon

One more thing: If there is a way to look at my playlists in my old 1.2 database backup, then perhaps I could just print that information out or save it somewhere, and manually rebuild my playlists in 1.3 to include the missing tracks. This would be hours of manual labor, but if it is the only surefire way to get these tracks back in my playlists, I would be willing to do it.

But I would need some step-by-step instructions for getting back into my backup copy and the best way to navigate to playlists and make a copy of the old playlists.

Patiently waiting for your expert guidance @mike.

Why don’t you run a backup, and then try this – there’s some complexity around this once the same files have been added repeatedly, but in most cases, I think that should take care of it, and it’s a pretty quick test.

if that doesn’t work, I can walk you through restoring the backup from a month ago. Let me know how it goes @alwayslearning!

@Mike:

Here are the results from removing all current folders (I could only disable the “Music Folder,” which I assume is the local Music folder on my laptop). As you can see, the most recent import of the DSD folder resulted in about 500 fewer tracks imported than any previous import. And when I use the Focus feature to search for DSD64 albums, less than two thirds of the number of albums show up, confirming that these albums were never imported. So something appears to have messed up with regard to that folder–something that hasn’t happened before. The other folder (Music) shows the same number of tracks imported as before:

With regard to playlists, the exact same problems are still present. In fact, the playlists appear to show the same number of tracks missing and the same tracks are labeled as “Unavailable.” But I bet some of those DSD tracks that are missing are missing from my playlists as well. Here is an example of a playlist that is missing 13 of its 26 tracks, with many of the tracks that were found listed as “Unavailable”:

Finally, the Roon activity monitor is still showing activity, although nothing has changed in this monitor for at least an hour and I initiated the import about 3 hours ago, so I doubt that it’s going to change the results we see above:

So it feels like I’m back to Square One. It’s probably time to try to utilize the old backup, don’t you think? Although I’m worried now that Roon is not even importing all of my music. Yikes!!

Update for @mike:

Well, Roon finished its activity and lo and behold, all of my DSD albums now appear to have been imported. Whew! The problem with the playlists remains. Looking forward to your next suggestion!

Was just reading this over and starting to respond – excellent timing :wink:

Ok, so I have a feeling some of the storage shuffling may have caused the database to reimport a lot of your content as “new” – that’s why the playlist information isn’t being linked up.

I bet you’re seeing some large numbers in Settings > Library > Library Maintenance – can you confirm?

Sorry, do you mind refreshing my memory – you have a database from immediately before the 1.3 upgrade? Or an older one? Just want to make sure I have this straight in my head before advising on the next step here.

Thanks for your patience Jon – we’re getting there :slight_smile:

Hi @mike, thanks for the quick response. Here’s what I see under Library Maintenance:

Regarding my old database, I saved Roon 1.2 on January 5, a little over a month ago. So yes, I certainly made some changes after that and before I installed Roon 1.3 (on the day it was released). But there would not be many changes to my playlists and very little or no metadata changes. So I’m thinking that’s the only source for reconstructing those playlists. But HOW we do this, I need guidance from you. Thanks.

Yup, that looks like what I was expecting. Ok!

So, everything is going to come down to the database on your Core machine. You can find information about how to find it here.

What you’re going to want to do is find your current database and rename the folder from Roon to something else, like Roon_old

Once you’ve done that, you should take the backup from a month ago and move it into place (as per the database location article). At this point, you should have a Roon folder in the same location as the folder you just renamed.

Then, launch 1.3 and wait for the database to upgrade. At this point, I’m not 100% sure whether you’re going to be seeing your library or not, but why don’t you post one more screen shot of the Storage tab of Settings before you change anything, then we can make sure we get it right this time.

Thanks @alwayslearning!

Hi @Mike, here’s the Storage screenshot:

I will follow the above instructions once you’ve seen this and give me the go-ahead.

Jon

Looks good to me.

How are the playlists looking? What about library maintenance?

@mike: Since I haven’t implemented your plan, everything is the same – playlists are still missing tracks and showing tracks as unavailable (see example above) and library maintenance shows the exact same screen as above. So I will forge ahead with your instructions.

Just to be clear, when you switched over to the backup from a month ago, you were running 1.2, and you were asked to take the 1.3 update, is that correct?

One thing not mentioned is to be sure to shut down Roon on your core before moving the database folders around.

May have been mentioned elsewhere, but would feel crappy if something bad happened and I didn’t post this!

This is a very good point, I’m going to make that a little more explicit in the documentation.

@Mike: I renamed the current database and moved the backup to the same Library location on my Core machine. I opened Roon and, at first, it asked for my login info. I provided that info and then it seemed to get stuck attempting to log me in. So after a few minutes I just quit Roon and then opened it up again. This time I got the message: “The new version of Roon you installed requires a database update. This may take longer than usual.” This seems appropriate – it looks to me like its updating to 1.3. So that’s where I am now – waiting for this update to finish. Fingers crossed.

@Mike: Sorry, I forgot to answer your question above. I’m not sure I understand your question. Last week, when Roon came out with version 1.3, I was asked to update from 1.2 and I did that. I did not do a backup right before I updated (unfortunately), but I have always had my backup from Jan. 5. So this morning, per your instructions, I put that old backup in the Library folder. I also renamed the 1.3 database I’ve been using for the last week to Roon_1.3. So I have two databases in my Library folder, but only one of them is named Roon and that is the backup from Jan. 5. My understanding is that this is the database that is now being updated to 1.3. Make sense?

Yup, that sounds right. Let me know how it goes!

Assuming things were in good shape when you made the backup a month ago, I’m sure we can figure this out.