RoonServer version: 2.71 (build 1683), production, linuxx64 Platform: Unraid (Docker container, ghcr.io/roonlabs/roonserver:latest), database stored on a BTRFS-formatted NVMe cache pool
Symptom: Every other night, at the scheduled 02:00 automatic backup time, RoonServer detects database corruption during the backup's validation pass and deliberately halts its own broker threads. The Core is then completely unavailable — no client can connect — until I manually restart it the next morning. Recurred on 25, 27, and 29 August 2026, always at the same point in the backup process, identical stack trace each time.
Warn: [broker] validation failed: LevelDb.Exception: Corruption: corrupted internal key in DBIter at LevelDb.Database._CheckError(IntPtr err) at LevelDb.Database.LoadPrefixedBy(Byte[] prefix, Int32 stripOffset)+MoveNext() at LevelDb.KeySpace.GetEnumerator()+MoveNext() at Roon.Broker.TinySooidDb.Vacuum(VacuumOperation op, KeySpace ks_from, KeySpace ks_to) at Roon.Broker.VacuumOperation.Run() at Roon.Broker.Database.Validate() Error: [broker/database] corruption detected: Corruption: corrupted internal key in DBIter Warn: [broker] detected corrupt database, notifying client Warn: [broker] detected corrupt database, halting broker threads Info: [loadstatus] IsDatabaseCorrupt False => True
What I've already checked/ruled out:
Backup destination disk is not full — plenty of free space. The disk holding the live database (NVMe, BTRFS) shows no active hardware fault: SMART reports no critical warnings, 100% available spare, 6% wear. A btrfs scrub completes clean, "no errors found." However, btrfs device stats on that device shows a non-zero corruption_errs: 3 — historical checksum failures recorded on that device at some point. All restarts of RoonServer have been graceful (docker restart / clean container stop), never a hardware power-cycle or unclean shutdown on my end — so this isn't being caused or compounded by how I'm recovering from it. I've temporarily disabled the automatic scheduled backup as a stopgap, since the corruption is only ever surfaced by the backup's validation pass — this stops the nightly halt but obviously isn't a real fix.
Request: Given the pattern (LevelDB corruption specifically surfacing during VacuumOperation.Run()/Database.Validate()), I'd guess a single malformed record in one keyspace is the culprit, similar to other reported cases of this exact error. Could you help identify which keyspace/database file holds the corrupt record and advise on repairing or safely clearing just that portion, rather than restoring the whole Core from an older backup? Happy to provide full log files or run any diagnostic needed — I'd prefer to share those privately rather than in this public thread if possible.
Thanks for the detail and for the disk-level work.
The corruption is in the music keyspace. Profile, playlist and radio validate cleanly every night; it always fails after several million music entries.
There is nothing we can repair here. A corrupted internal key is damage at the storage layer rather than a record we can identify and remove, and the music keyspace is the library itself, so it cannot be cleared separately.
The only route is a restore. One thing to know before you start: validation runs as part of the backup, so when it fails the backup never completes. You have no successful backup from any night since this began, and your newest usable one predates the first failure.
Please restore the newest backup you have, run a manual backup immediately, and check that it validates. If it fails, that backup already carried the damage, so go one further back and repeat.
If no backup restores and validates cleanly, then a fresh database is the only remaining option. Your library rebuilds from your files and streaming accounts, but edits, playlists, play history and favourites do not come back, which is why it is worth working back through several backups first.
Thanks @vadim I’ll work through the restores - my main concern here is that this is the second time I’ve experienced this kind of corruption in a couple of months. I had to do a full rebuild last time.
Thanks for the thorough diagnostics; they saved us a round trip.
First, some good news: this is one corruption event being re-detected, not fresh damage every night.
08/27 processed 4124425 music entries → fail
08/28 processed 4124807 music entries → fail
08/29 processed 4126378 music entries → fail
Playlist, profile, and radio pass every time, and the music count only moves by the amount your library actually grew. Validation is aborting at the same point in the keyspace each night. The bad block was written once, on or before Aug 25; the “every other night” pattern was the backup retry schedule.
The bigger finding is what surrounds it. Your logs show two terminations that were not orderly shutdowns.
Both fall inside Roon Server’s 01:00–05:00 background work window, so writes were likely in flight. Together with corruption_errs: 3, that points to the platform occasionally losing data underneath Roon Server, which would also explain why the rebuild two months ago didn’t hold. Worth sorting that out before working back through restores, or whatever you land on will corrupt again.
Two things you can stop chasing: your storage layout is correct (/Roon is mounted directly from the NVMe, not through the /mnt/user fuse layer), and the 295 CorruptFile warnings are 27 bad MP3 rips, unrelated to the database.
What would help most:
The Unraid diagnostics zip covering Aug 25–Sep 1. The syslog around 08/31 04:18 and 09/01 04:39:57 should say whether those were reboots, panics, machine check exceptions, OOM kills, or Docker actions. Any nvme0: I/O timeout, csum failed, or mce: lines are what we’re after.
What runs around 04:40 daily? Appdata Backup, Auto Update Applications, mover, parity check. If a plugin stops the container, the default 10-second Docker stop timeout is likely too short for a database your size to close cleanly.
Reset the btrfs counters (btrfs device stats -z) and recheck in a few days. Scrub only verifies what’s on disk now, data corrupted in memory before the checksum was computed gets stored with a matching checksum, so a clean scrub doesn’t fully clear the pool. A count climbing from zero would be close to conclusive.
Leave automatic backups off for now. Once we’ve seen the syslog we should have a much better idea of what’s underneath this.