Serious import bug with Nucleus [solved how to avoid it]

I have a problem that looks like a serious bug in the database and content import function of Roon.
I found it in the migration to a Nucleus, but it is Roon functionality, seems unrelated to the OS or hardware.
Of course, we should try it on a regular Roon installation, but that is laborious.
Btw, I discussed this initially with @Danny, he suggested it is temporary behavior and will clear itself up. And at first I thought so, it was partially cleared up. But now we are several days into the life of this box and it still isn’t cleared up…

So here it is.

I got my new Nucleus. Installed a 2 TB SSD in it. Restored from a newly made database backup. My library is about 1.2 TB. I started copying the library from my Windows machine. I took quite a long time, several days, so I did it in chunks.

But I found that several albums were imported in pieces. Most of the time, track 1 was one album, and the rest was another album. Sometimes it was track 2 that was separated, and in rare occasions it was two or three albums. In most cases, the albums had the same metadata and cover, but not always.

One curious thing: the larger fragment, with all but one album, had the correct import date, but the single-track fragments all had the date of the copying.

I suspected that the copying, slow as it was, overwhelmed the database processing. And @Danny pointed out that it was probably a long time since I did a complete load, so I wasn’t used to how it behaved. He said it should clean up. And it did, partially. But I installed the albums in alphabetical order chunks, and while I have messed with things a bit, cleaning things up manually, merging albums, I did leave a few albums in the As as a test case, and they never got fixed.

But one strange thing: lots of errors in the A-B-C section of the library, the part I copied first; those partially cleaned themselves up but not fully. But there were many fewer errors in the rest of the alphabet. Not zero, I had a bunch in Vijay Iyer, but much fewer than in the beginning.

This is very troubling. Manual repair is of course cumbersome – I had many hundreds of bad albums. But also, when I do merge them, they lose meaningful metadata, tags and import date and other stuff.

By now I have it nearly cleaned up. Of course, I have impaired any scientific experiments because there were other errors in the database and content library, my own mess, so while repairing these broken albums I have also cleaned up the other stuff. So it will be difficult to do forensic analysis.

I have thought of one thing: I could take a database backup, then restore the original backup I made from the NUC. In this case, the content library is still there, instead of being slowly streamed in, so it would be interesting to see how Roon would handle that situation, if it could import stuff better in that order, content first and then database. Unconventional approach, but might be useful. And then if that causes a mess, I could restore my most recent backup. But I am a little bit worried that more things will go wrong… Plus I would lose other metadata edits I have done.

Please ponder this. It is a pretty serious problem.

I like the Nucleus otherwise. But this was painful.

Yikes, thanks for the feedback, Anders.

So, you installed the hard drive, restored the backup, and then you copied the files to the internal SSD over the network right? Was that done using the /Data share? And the Core was running while the copy was happening?

I’m going to look into this, but I’m guessing Roon was just seeing partially copied albums during the copy, and that’s why the “clumping” got messed up.

We should do some testing, and then we can document a less painful way to do this.

If you feel like experimenting, you can go into the /Data share, rename the /RoonServer folder to /RoonServer_old or similar, and then restore the backup again. That might just get everything back to how it was, and you can always switch back to the current database if not.

1 Like

Yes, I think that’s what happened.
I have seen a milder version of it in the past: while copying an album, halfway through, I was in a remote and viewed the album; it was of course messed up, being incomplete, but the look sometimes locked in the mess. Sort of like a Roon version of Heisenberg’s uncertainty principle, where the observation affects what is being observed.

Btw, I am not sure, but I think I restored the backup before installing the drive, but I don’t think that would have made a difference. In fact, i had that mess up where I first installed a 160 GB SSD by mistake, then put in the right one, but I don’t think that would have made a difference either.

Interesting problem, and not the easiest to diagnose.

I will try that experiment. I can stop the Roon software on that web control page, right. So I stop Roon, rename /Roonserver, start Roon, in a remote restore the backup, and see what happens. And then I can go back by stopping Roon, switching the names, and restarting. Right? Very neat. (Except there is no way to intelligently merge databases, with the unrelated cleanup I did…)

I think once you stop Roon and rename the database, you may need to click restart in order for the remote to connect to the new, fresh database.

Then, you can restore the backup from the log in page.

And yeah, no way to merge the databases.

Come to think of it, the whole thing may have gone better if I had kep Roon stopped during the entire file copying, and then let it absorb a static library. If the experiment confirms that, we should maybe put that in the guidance.

That’s how I’d have approached it, get the music across and then the database. In fact I’d likely have put the music on the SSD before installing it in the NUC.

1 Like

I did it, restored the original database with the SSD already populated with music, and it worked flawlessly. As far as I can tell. 1500 Tidal, 1800 local. Took maybe ten minutes.

So this is the normal behavior of the Nucleus.
We need to put this recommendation in the guidance.

Thanks, @mike.
Note the result, @Danny.

Hindsight is easy, but I’m sure this very issue (problems arising as a result of large-scale copying whilst Roon is running) has come up a number of times in the forums. I’d always understood that best practice was to stop Roon before doing such copying. Definitely needs to be documented, and not just for Nucleus.

My Rock is very quick to start analysing files during copying them to disc. Either it needs to wait until all disc activity has stopped on a per folder basis or not be so quick off the mark. Analysis of a partially copied over folder is never going to end well…?

Disable server, do the copying then re-enable it. Having said that, when copying 10-20 albums or so I’ve found Roon ingests them without hiccup.

Yep, agree on both.
Disable during large volume import.
And I have never had the problem on smaller volume imports, which is why I didn’t think to do the cautious approach on the Nucleus mass import.