Bad thinking or bug?...and a warning to other Linux users

Yesterday I updated the Linux distribution on which I run Roon Server (it’s a rolling update so no reinstall is required). On rebooting the server it entered UEFI BIOS instead of booting Linux. I powered down the server and disconnected the directly attached drives containing music (a precaution to ensure I cannot inadvertently write to a music drive whilst recovering the boot drive). I then booted the server using a USB stick containing a liveboot instance of Linux so I could check the SSD’s function, partitions, filesystem etc. for integrity to find the issue.

A little online digging led me to understand that changes had been made to the EFI Boot Stub causing any forward slashes in the EFI Boot Stub string to stop working as before. After using Linux tooling to rewrite the BIOS’ EFI Boot Stub without the forward slashes I restarted the server, checking to see if it would now boot from the SSD that houses the server’s Linux OS and Roon Server + database. The server booted correctly and apart from the timeout messages regarding inability to mount the disconnected drives everything appeared normal, or so I thought.

I shut down the server, reconnected the power supply to each of the music drives and restarted the server. It booted and mounted the music drives without a hitch. All seemed well until I tried to run a Roon client and I was presented with a sign-in prompt a brand new virgin install would encounter - enter username and email address. Roon had effectively trashed the existing database for no apparent reason other than it on a single occasion encountered no content in the folders it looks to for content …

It’s hard to see this as anything other than poor software design. There was no reason to throw the baby out with the bathwater - a simple prompt to say “Your music folder(s) are empty, shall I expunge the existing Roon database or shut down and give you a chance to investigate?” would have completely avoided the need to rebuild or restore the database.

Lesson for Linux users: if your server hosting Roon plays up make sure you disable Roon Server before doing any troubleshooting. Roon is a rather blunt instrument!

1 Like

Hope your library was backed up.

Not quite the same as I run a Rock, but I have often restarted without my NAS drive being switched on with no problems. Roon simply says the drive is unavailable but otherwise all is OK.

It sounds as though Roon thought it was a new machine. At that login prompt did you see a Restore a Backup option in the lower left corner? See the migration KB article for example.

As @Slim_Fishguttz says, I hope you had a backup.

The key difference (I think) is that in your case your system would be mounting a network share using SMB, whereas in my setup it was a local filesystem mount. It’s possible Roon understands when a SMB share is inaccessible but have not anticipated a local drive not being mounted in a Linux filesystem - the mount point would simply appear to be empty.

Roon would have shut down knowing there is content. On starting up again the content is gone. Time to enter limp mode and ask the user whether to start over or to check whether a drive or something else has failed. If it discards the database under these conditions I suspect it’ll do exactly the same thing for a failed NFS mount, because on the surface it’ll all look the same.

Let’s see if @support has any input.

If what you say is true, a warning isn’t good enough and Roon should be involved.

1 Like

hi, what linux distribution do you use?

Hi @gduffield,

Even if your music files are available, Roon should not start with a fresh database. It seems like the RoonServer directory was unavailable or removed. As long as the database directory is still available there should be no change, regardless of watched folder availability.

Has your team tried to reproduce the issue or just assumed I don’t know what I’m doing?

@dylan, I ask again … Has your team tried to reproduce the issue or just assumed I don’t know what I’m doing?

I just unplugged the internal sata drive on a Roon OS machine sitting on my desk. No internal storage, no music. Rebooted with it plugged in, internal storage returned and so did my music.

No one made a comment insinuating any such thing. @dylan only explained that Roon is quite tolerant to having audio files missing. I know it is, because I wrote most of that code. The database, however, can not be missing.

Any thoughts as to how the database could go missing in the situation I described? No changes were written to disk, the OS still booted, Roon still ran?

Seems someone else experienced a similar issue, albeit with an unavailable NFS mount. Related perhaps?

Tell me about your setup…

I don’t know what distribution you are running, if you used our installer, are running in a VM or docker instance, etc…

Where is your database located? What is that partition and how does it get mounted?

What’s in your database now? how big is it (du -sh <dir>)?

not quite… he didn’t experience a db drop. in his case, the filesystem was missing, so all the music disappeared. when it came back however something changed (or a bug was triggered) causing it to think all the files were gone for good. Then when they returned, Roon had to rescan the files and once they are confirmed to be the same files as before, all the data will return too.

In your case, it looks like the whole db disappeared, not just the stuff about the files. The difference in symptoms being that you were logged out.

I’m not near machine now so can’t report database size and location, but the install was done using your standard install script and whatever it defaults to re storage location - I vaguely recall it’s somewhere in a /var subfolder tree.

Drive is a Samsung M2 SSD, formatted ext4, mounted via fstab, distribution is Arch on X64. The only change I made from liveboot was to the BIOS’ boot stub, nothing to disk.

A restore of a fsarchiver backup of the Roon subfolder tree in /var returned things to normal.

do you have a copy of the “bad” folder in /var?

No, sorry, I killed it and ran the restore.

That’s unfortunate… and understandable.

Unsure there is anything we can do here now.

My guess is that something during the software update of Arch caused the Roon database in /var to get blown away. Unsure why it would do that, but that’s outside the domain of Roon.

If you had any record of what was in there, it would have been helpful in diagnosing whether it was corruption or deletion, but the lack of error points to deletion.

Are you running 5.x kernel?

There is a known issue with Roon scanning not finding all the content in a single scan, then losing and re-finding during the next scan with 5.x but no issue with 4.19.x.

May or may not be related to your experience.

I think the Linux 5.x issue you are talking about is probably related to audiolinux and not generically 5.x. I’ve been testing 5.x for a future Roon OS update without issues.

1 Like