· Backup functionality does not work. It is the same issue that I reported before(https://community.roonlabs.com/t/issue-with-roon-backup-on-ugreen-dxp4800plus-ref-c9h9ee/318795/6). I have tested every update that happened between then and now and still facing the same problem.
I have tried to map /RoonBackups to a different volume and location, that didn’t help.
I use Roon only to play my files, no Tidal or any other streaming.
My library contains about 137k of tracks, most are ALAC, but also FLAC and DSD formats
Here is a short animation:
Your Docker configuration is perfectly fine and is not the cause of this issue. The diagnostic logs clearly show that your backups are failing because Roon is detecting latent corruption within your database.
Whenever you initiate a backup, Roon performs a mandatory database validation first. As seen in the logs, it hits a fatal error (Corruption: corrupted internal key in DBIter) during this check and immediately halts the broker threads to prevent backing up corrupted data.
Unfortunately, this type of deep LevelDB corruption cannot be repaired. The system is operating exactly as designed by stopping the backup of a compromised database, but the only way to resolve this and restore backup functionality is to start fresh with a completely new Roon database.
To proceed, you will need to:
Stop your RoonServer Docker container.
Navigate to your mapped volume (/volume1/docker/roon) and either delete its contents or rename the folder to something like roon_old.
Restart the container to allow Roon to generate a fresh database.
Set up your audio zones and let Roon re-scan your 137k tracks from the /Music directory.
I know this is frustrating, especially with a large library, but it is the only viable path forward when the core database files are corrupted. Let me know if you run into any issues during the reset.
I’d done this process multiple times(I’m a ROON user for about 9 years) and every time end in apparently the same corrupted state. Sometimes it goes away as it happed at the end of last ticket and goes bad again after a next update or two.
It does not appear to be NAS/Docker related, it had happened on MacOS and Windows too.
As a programmer I could ask why use LevelDB, but regardless. Before I go with a nuclear approach…
Is there anything else I can do not to loose at least my listening count numbers. My multi-album merges and some other custom metadata? Could I shutdown ROON, ssh and copy some tables, then do your instruction and restore these selected tables?
My reason is obvious, it will take a day of so for Roon to recreate its database, then, in who knows how soon, it will corrupt again. There should be a better option than that.
Am I asking for too much?
mjw
(You want nice, supportive? Call a damn helpline.)
6
LevelDB is a NoSQL database, so you won’t find any tables. Restore a good backup and see how that goes.
@mjw is right. LevelDB is a key/value store, not a relational database, so there are no tables to copy out. Play counts, merges, and edits live across thousands of interdependent keys with references into the artwork and media caches; hand-copying a subset into a fresh database won’t produce a working library.
A restore is what preserves all of that. And because Roon validates the database before writing a backup, any backup that completed is known-good, a corrupt database can’t produce one. Your automatic backups were running successfully at the end of April, so there should be something usable in /RoonBackups.
Rather than starting from empty:
Stop the container.
Rename /volume1/docker/roon to roon_old , keep it until you’re happy with the result.
Start the container and restore from /RoonBackups, newest first. If that one fails validation on the next backup attempt, work backwards.
You’d lose only the listening since that backup. Tracks re-scan from /Music; the edits come back with the database.
On why this keeps recurring: from the mount output you shared in April, /Roon sits on volume1, your 17 TB btrfs pool, so I believe the database is on the spinning array. Roon highly suggests the database to be on an SSD, and this is why: constant small synchronous writes on HDD-backed NAS storage is where we see this class of latent corruption. The DXP4800 Plus has NVMe slots, remapping /Roon to an SSD volume (leaving /Music and /RoonBackups on the big pool) is the change most likely to stop the recurrence. Worth confirming the container is stopped gracefully on reboots and updates too.
Once you’re back up, run daily backups so the worst case is a day of history rather than a rebuild.
As I mentioned, I had rebuild my ROON database multiple times when the whole thing was sitting on M1 MacMini, with ROON on SSD, Music on separate SSD and it still happened many times while I used it there anyway.
I’d say that most likely it is how LevelDB writes are implemented, and not btrfs, spinning drives, etc., is the culprit.
I’ve searched the forum for LevelDB corruption and it looks like this is happened to many people in different scenarios. It is frustrating for us, since we do love ROON otherwise.
Besides reimporting all of my music, which took 4 hours, it is going to take a few days to analyze it, before I want to do anything else with it.
Wander why you recommend redoing the whole roon folder and not just the broker_4.db which seems to be where the corrupted data is.
mjw
(You want nice, supportive? Call a damn helpline.)
9
LevelDB is in-memory, and writes are performed when the memtable reaches a certain size. You should not use spinning disk for the database, and if your BTRFS storage is a pool, then this is not in my experience entirely reliable.
I suspect your experience with macOS is unrelated, but could, for example, have been caused by restoring data from Time Machine.
I have used spinning disk for databases, before solid state drives were conceived. Solid state drives are great, but not really required, if one cannot afford them, has what he has, etc.
I wish for us having a better advice and support, but alas…
In case somebody is like me, here are my steps to fix my backups, or contact me and I’ll walk you through on zoom:
Shutdown Roon docker container
Navigate to the location of the database directory that I identified as one containing corrupted data. In my case it was /volume1/docker/roon/database/RoonServer/Database/Core/c7745a826a0c4cefa17e7592548e0701/. The database name is broker_4.db, but as far as I understand it can me broker_1/2/3/5/.....db in earlier or later versions.
You need to have python to repair it. I installed it in another docker container, logged in there and mapped access to the above path from it. One of the python LevelDB libraries is called plyvel, but there are others. I used plyvel:
# repair.py
import os
import sys
import plyvel
db_path = sys.argv[1]
if not os.path.isdir(db_path):
raise SystemExit(f"Database directory does not exist: {db_path}")
print(f"Repairing {db_path}...")
plyvel.repair_db(db_path)
print("Repair completed.")
In my case db_path was /volume1/docker/roon/database/RoonServer/Database/Core/c7745a826a0c4cefa17e7592548e0701/
!!!Make sure your ROON is not running!!!
4. Started ROON container and backup works once again.
mjw
(You want nice, supportive? Call a damn helpline.)
11
Unfortunately, this won’t fix any underlying data issues with LevelDB, since this file is from the message broker not the database, which has many thousands of small files containing key-value pairs.
Unfortunate or not… it fixed it.
While my spinning disks, time machine, etc. and ROON can lead to corruption again, currently I achieved the solution to my current problem:
A correction first, because it changes what the repair did. broker_4.db is the database, not a message broker file. It is the store Roon validates before every backup, which is why repairing it made backups start working.
That is also why we cannot recommend what you did.
LevelDB’s repair does not recover damaged records. It walks the table files, discards anything it cannot parse, and rebuilds the manifest from what survives. Your backup validates now because the unreadable data is gone rather than repaired. We have never tested that tool against a Roon database and it is not something we can support.
The specific risk is not that it fails today. It is that Roon’s data is thousands of interdependent keys with references into the artwork and media caches, and a repair that silently drops some of them leaves references pointing at nothing. What that looks like later is new corruption reports, albums or artists that will not open or render, edits and merges that half-exist, and errors we will not be able to trace back to this, because from our side the database looks valid. If you come back to us in three months with something strange in the interface, neither of us will be able to tell whether it came from here.
The supported route out of a corrupt database is a restore or start from scratch. Your automatic backups were completing successfully at the end of April, and Roon cannot produce a backup from a corrupt database, so anything in /RoonBackups from then is known good. Restoring the newest one that passes gets you a database with intact references and costs you only the listening history since it was taken.
If you would rather stay on the repaired database, that is your call, and it may well run for a long time. Please take a fresh backup now so you have a restore point, and please treat any odd behaviour from here on as possibly related.
Vadim,
Thank you for the explanation, it all does make sense.
I actually did do both, from scratch and unsupported restore, and compared both results to the best of my abilities. I have not noticed any difference in number of albums, tracks, etc. One thing that I did notice, was change to Genres. Th old corrupted db had way too many more genres. Some of them with just one, two albums in it.
Another strange thing was, that when I restored my original April backup over old database, it would immediately produce a “corruption”. Meaning, I couldn’t generate a new good backup, I was getting the original error. But when I started with, an unsupported by you, restored database, and restored from the April’s backup over it. I could generate a new backup after. This I cannot explain. Of course I could generate a backup from a freshly generated roon directory and restoring April’s backup produced a good result too.
I’m very close to “go back 30 years” and start making a physical, file system level backup of roon directory, on cold Roon server. We did that with an Oracle server in the old days. But since everything seems to work right now, I’m a bit reluctant to act on this OCD impulse until next time.
Again, thank you for taking the time to explain some things about this LevelDB stuff.
