Migrated Core, NAS Storage, Playlists Now (Mostly) Empty

Roon Core Machine

MacBookPro (headless, only running Core)
Big Sur 11.6
3.5 GHz Dual-Core i7
16 GB RAM

Networking Gear & Setup Details

Ethernet everywhere, between MBP and NAS and all audio end-points

Connected Audio Devices

Raspberry PI running RAAT client → USB → Schiit stack (among other endpoints)

Number of Tracks in Library

~60k tracks

Description of Issue

I just migrated my core from running on a Synology 218+ NAS to a dedicated MacBookPro. The migration went very well; restoring from backup worked, no issues keeping my music on the NAS, and editing each of my storage points in Roon settings after migrating to the new core worked fine based on a random testing of older music in various places. I’ve tested everything that I can think of – playback, adding new albums, albums associated with tags, etc – and everything is working as expected. Immediately after editing the storage locations, though, I did notice that 550 songs were immediately flagged for re-scan (which seems like a random number), and 3 old MP3 albums were skipped due to being corrupt (although they weren’t skipped or flagged when core was running on the NAS). I chalked it up to them just going bad at some point over the past year and half.

But one big problem: All of my playlists have been drastically truncated, in some cases dropping from 100s of files in a playlist to >40. And it’s not that they’re not that they’re showing up as unavailable, the summaries of the playlists are showing the drop. For example, on the original core I have an 80s playlist with 1397 tracks. After the migration on the new core it’s only showing the playlists has 54 tracks.

Before the migration I made a fresh backup on the original core and restored from that backup on the new core, and I just fired the original NAS core back up and the playlists are correct on that NAS; they’re only truncate on the new core after the restore.

I do notice a lot of messages like this in the new core’s logs after migrating. It logged this type of event against ~half of my tracks, 34,165 of them.

RoonServer_log.06.txt:10/10 10:37:05 Debug: [music/storage] processing reshuffle and delete for track 67:0:iTunes Media/Music/R.E.M_/Complete Rarities - Warner Bros. 1988-2011 [FLAC]/36. What’s The Frequency, Kenneth_ (Live On S.N.L. New York, NY _ 11_12_1994).flac from \iTunes\iTunes Media\Music

“\iTunes\iTunes Media\Music” is the watched folder on my NAS (still named that way from the legacy migration from iTunes when I first started using Roon). This track, and any of the others that have the same type of log entry, play fine on the new core.

Thanks!

Hey @rcrawley,

Congrats on choosing to have a dedicated device as a Roon Core - that’s a great step to take! It looks like so much of the migration process worked perfectly.

When going through something like this when you have a local library, there are a few specific steps to take to make sure Roon will read your library properly: the first one being removing the storage locations, performing the migration, and re-adding the path to the storage location.

If you haven’t, could you please try to follow the steps here:

Thanks @beka. I went through the full migration again following the below steps and same result; my playlists on the new core are all just a handful of songs when they should be in the hundreds or thousands. I’ve included two screencaps below of the playlists on the original core and on the new core.

Here’s what I did in the 2nd migration:

  • Performed a backup on the original core
  • Installed RoonServer again on the MacBookPro
  • Connected a client to the new core
  • Selected “restore from backup” instead of logging in
  • Restored from the fresh backup
  • Logged in/authorized the new core
  • Removed all storage links in the new core (completely deleted them)
  • Added the new storage links pointing back to the NAS

Roon scanned all three new storage locations for a few hours, and it when it finished it was still missing almost all of the playlist items.

One note: I didn’t change the storage at all, just migrated the Core; all of the content files are still in the same place accessible via the same NAS access.

Is there anything else I can look at to figure out why my playlists didn’t come over as part of the migration?

Thanks!

Original Core before migration:
Roon-Old-Core-Full-Playlists

New Core after migration (and after re-scanning the same storage locations on NAS):
Roon-New-Core-After-Migration

Hey @rcrawley,

Wow! Thanks for the thorough reply :nerd_face:

I was hoping to ask for two more things:

  1. Could you please check if some of the files that are missing from a playlist in Roon, can be found outside of the playlists in Roon? In other words, can you access those tracks outside of the playlist, but in Roon?

  2. Could you please navigate to Settings → Library ->Clean up library and get a screenshot. Please, do not run the process, just screenshot the window and share that with us.

Thanks a lot :pray:

Hi @beka

  1. Yes, all the songs that are in the playlist from the original core are playable manually through the new core. To test, I exported a playlist from the original core and then moved over to the new core and randomly played 10 songs from the original playlist that aren’t in the new core’s playlist, all good.

  2. Here you go:

I’m not sure what the 75 files not associated with a storage location is about; I’ve added the same NAS storage locations to the new core as I have on the original core and so far haven’t found any music that’s cataloged in the old core that isn’t available in the new core.

Thanks for the help!

Hi @beka. Just checking back on the above. Right now I’m basically stuck and unable to run off the migrated core due to the weird playlist issue. Anything that I missed in the docs or anything else I can test to help narrow down this issue?

Thanks!

No update from support so I tired a new migration one more time, this time mounting the shares on the new Mac Roon core via SMBv3 (per the recommendations here: 24/88 + 24/96 + 24/176 music still stutters, stalls and won't play - #3 by mike), but exact same result. My playlists are still showing the incorrect number of tracks (and it’s the same number of tracks every time I test a restore). The only difference is now I’m seeing that Roon thinks there are 90 files not associated with a storage system rather than 75 during the previous migration. Since the actual music storage didn’t change at all, and the new core has the same mount points as the original core (although now mounted as network drives rather than local since the original core was running directly on the NAS), I don’t follow why Roon sees files that aren’t on a storage location.

Could this be a bug with restoring from a backup not correctly handling playlists?

I’m about at my wits end with this one. Is there any other method of requesting support than just waiting on someone to respond in a forum thread with more questions? There has to be a better way for paying customers…

If you change storage points Roon can easily see the new location as new.

Try this…When you install the new version, go to storage and disable all storage locations. Then restore the backup. Go to storage and Edit the old location to the new location.

1 Like

@Rugby That seems to have worked! Thanks for the suggestion, but I did two things differently so I’m not sure which worked:

  1. I did not choose “Restore From Backup” during initial setup. Instead I went through the process of setting up a new core and then restored from backup from within the client. Previous tests I’d restored from backup from the beginning rather than logging in and creating a new system.

  2. When I edited each storage location after restore to point back to the NAS, I did one at a time (I have three) and let each one finish before starting the next one. In the previous tests I’d edited them all at once and just let them all scan at once.

So I have no idea which one made a difference but the combo worked. I wonder if there’s an issue with restoring from backup without logging in and creating a new core?

It is strange that it’s re-analyzing all of the tracks, though. Shouldn’t that data have come over with the restore?

Welp, I spoke too soon. :frowning_face: After successfully running on the new Mac core for about a week, this morning I am back to the exact same missing playlist tracks issue I saw after the initial migration. It appears to have cropped up while trying to get around yesterday’s upstream failures on Roon’s side (see Odd message "Metadata Improver: Paused" - #5 by Mikael_Ollars for details).

This morning I rebooted the Mac again b/c Metadata was still stuck for me on the Mac core after restarting Roon so I rebooted the Mac. After coming back up the metadata issue was resolved but the playlists have now reverted back to the same state of missing almost all of my playlist files, and strangely they’re missing the exact same files they were missing during all my failed migration attempts; I can test b/c my 80s playlist always only contains Paul Simon tracks when I’m seeing the failure.

As you can probably tell I’m extremely frustrated by this, but mostly b/c there’s no SLA for support when a system is completely down. Roon is hands-down the best music software I’ve ever used, but honestly I don’t know that I’m willing to continue playing a premium for software that doesn’t come with actual support; I’d prefer to use something like Plex, which I don’t like but has a larger community support base w/o paying a premium.

Hey @rcrawley,

A whole bunch of apologies going your way :bouquet: :pleading_face: . I am truly sorry that we never got a chance to circle back. Alongside regretting this, we are grateful for your patience and determination to find a solution. Thank you for engaging on our community and investing time and effort to see this through.

I wonder if you could try again the steps that imported your playlist correctly (disable and remove storage locations, migrate, add the storage locations fresh). If the missing playlists appears again, could you please grab a set of logs as soon as you notice it and upload them here?

Thanks a whole lot :pray:

Thanks @beka. I’ve migrated again using the above steps and playlists are there as expected. I’ll keep an eye on them over the next week and will DM you logs if/when it happens again.

1 Like

Hey @rcrawley,

Wow! That sounds awesome. Thanks for keeping us updated.

We’ll be in touch :pray:

@beka Ok, after another 2-ish weeks on the new core it’s now reverted back to the missing playlists. Weirdest thing I’ve seen…works great for 10-ish days (repeatably) then fails in the same way every time.

I’ve uploaded a tar.gz of my entire ~/Library/RoonServer/Logs directory. Please let me know if you need anything else.

Thanks!

Hey @rcrawley,

Thanks so much for not hesitating to let us know that the issue has returned…like clockwork :mantelpiece_clock: :pensive: . We’re sorry about that!

We got your logs — thanks for that — and I have passed them along to our technical team for investigation.

Do you remember on what day (and possibly time) you’ve noticed your playlists disappearing?

@beka I noticed on Nov 19th and posted the logs immediately when I noticed. I can’t say that I checked religiously every day but I do play playlists at least ever other day-ish so at most it wouldn’t have been more than 2-ish days before the 19th when they reset themselves.

Thanks for the confirmation on the logs and hope that helps!

Hey @rcrawley,

Millions of thanks for following up and for your patience while we’re investigating this. Your case is still with our technical team and we’ll follow up as soon as they have a chance to take a look at your logs.

We’ll stay connected :pray:

Hi @rcrawley ,

Can you please confirm if these playlists were created within Roon or if they were imported from an XML playlist?

How are you referencing the Synology location now, is it via Hostname or IP Address? Since the Synology was the Core before, it would be able to directly access the music files, but now it has to traverse the network.

@noris

  1. The playlists are a mix of playlists that I imported when I first started using Roon on the Synology NAS and of playlists I’ve since created natively in Roon.

  2. The storage locations are referencing the NAS by hostname. I added them all in the format of \nostromo<MOUNT_POINT>

Let me know if that’s what you need. Thanks!

Hi @rcrawley ,

Thanks for those details additional details here. I suggest we proceed as follows, can you please:

  1. Set your Mac Database aside:
  • Exit out of Roon
  • Navigate to your Roon Database Location
  • Find the folder that says “Roon”
  • Rename the “Roon” folder to “Roon_old”
  • Reinstall the Roon App from our Downloads Page to generate a new Roon folder
  1. Restore from backup to get the playlists back

  2. Capture a manual log set when the playlists are working as expected following the restore

  3. Capture a manual log set when the playlists disappear after the ~10 days

Let us know if you are able to gather this, thank you!