Can't restore from backup without "hacking" things

Roon Core Machine

Synology DS1621+ / Linux 4.4.180+
8 core Ryzen V1500B / 32GB RAM

Networking Gear & Setup Details

Ubiquiti ethernet switches. Everything 1Gbps ethernet linked.

Connected Audio Devices

Various devices.

Number of Tracks in Library

+12,000

Description of Issue

I have nightly backups. Sometimes for reasons unrelated to Roon, I can’t gracefully shutdown the NAS and have to hard power cycle. Not great I admit, but s**t happens. That’s why we have backups right?

When this happens, with the latest version of Roon (2.0 build 1182) Roon database or whatever gets corrupted and I loose all my settings (Tidal login, backup config, Roon endpoints, etc). This didn’t happen with older versions of Roon. But no big deal, I’ll restore from backups…

Only twice now in about 10 days the backups won’t restore without manual intervention. Basically I get an error saying the backups aren’t valid. I’m able to manually fix my backups by going to the directory containing my backups (/volume1/Roon/backup/9892a01e-ff48-b763-06c7-85996c98f7f7) and run the following commands to fix the issue:

echo -n "oban" > name
echo -n "ROON BACKUP FOLDER v2" > _roon_backup_

This is because the above files are empty.

No idea what is going on, but you can imagine concern when it happened the first time and I had to debug/figure out the fix.

Obviously in a perfect world, I shouldn’t need to restore from backups, but I don’t think it is at all acceptable that the backups are “corrupted” and need me to fix them.

I’ve recently had an issue restoring from a backup too. I ended up doing analysis from scratch because the database corrupted shortly after it was completed. I’ve no idea if this is connected but it is worth taking a look?

Have you opened a case with Synology to see why this is? I’ve used 4 different Synology models over the last decade and only had this happen a few times and that was years ago with early DSM versions (and models). Seems like you might have a larger issue if you have to continually pull the plug to power cycle.

1 Like

I have talked to Synology, and they are not at all helpful in this matter. I’ll leave it at that, since this isn’t a Synology forum.

Not sure if your problem is the same as mine Henry, but it sounds different? Are the files I mentioned above getting truncated on your system as well?

Any chance of Roon support taking a look?

Hi @Aaron_Turner ,

Apologies for the slow response here. Can you confirm if you ran any extended error checking on your drives? If data is suddenly going missing, it sounds like one or more of your drives might be heading toward failure. What happens if you try to set a new backup location, do the files get overwritten there as well?

Hi @noris,

Drives do a monthly health check and just passed with flying colors.

My (rather educated guess) is that when the Linux kernel on the NAS starts having issues (it’s definitely related to CPU utilization and makes the system unstable and somewhat unresponsive), it is preventing writing/updating files during backups. But rather than ending up with an incomplete/failed backup, it is corrupting the metadata files which seem to prevent Roon from knowing these are backups.

The expected behavior is this should operate like a relational/SQL database WAL (write ahead log) which ensures a transaction (or in this case a backup) either completes successfully or does not “exist” at all rather than ending up making all previous backups unavailable (worst possible outcome). Luckily, I’m pretty tech savvy and was able to figure out what was going on!

The real first step here is to solve this hard power cycle problem. Who knows what problems that causes with the data on the NAS. The hard power cycle could prevent cached data from being written to the drives. Absolutely awful if you are running any version of RAID.

2 Likes

@musicjunkie917 Thank you for the advice. I’ll be sure to give it all the due consideration it deserves.

Hi @Aaron_Turner ,

Are the file permissions by any chance wrong for that directory? It sounds like Roon might not have access to write to those files in particular.

It may be interesting to see what Roon logs show when the backup runs. Next time you encounter the issue, can you upload a manual set of your NAS Core logs (accessed via these instructions) to the following link and let us know?

https://workdrive.zohoexternal.com/collection/8i5239cc05950ac07456889838d9319545a82/external

Roon is reporting my backups are successfully completing and as I described above, once I fix the two metadata files I can restore. As for “are the permissions correct” can you tell me what the correct permissions are??? I don’t know what Roon expects.

Anyways, I’ve uploaded Aaron_Turner.tar.gz to your zoho file service containing the RoonServer logs.

Hi @noris I see the latest release indicates something about backups failing and users not getting notified. Any chance that has something to do with my issue/logs?

Hi @Aaron_Turner ,

No, I am not aware of the changes impacting your case. I looked over the logs and I’m seeing errors around cleaning up previous backups. I wonder if you create a new folder and set it as the new backup directory, does that work without issue?

So I shutdown Roon, moved /backup out of the way and created a new directory in it’s place.
Did a manual backup and it created /backup/RoonBackup/9892a01e-ff48-b763-06c7-85996c98f7f7 which wasn’t expected… the old backups were in /backup/9892a01e-ff48-b763-06c7-85996c98f7f7 Not sure why the extra directory or if that is a concern???

You said there were errors… can you tell me what errors to look for in the logs to make sure this fixes it? There is nothing in the UI indicating any errors and when I dig through the logs I only see some errors like:

RoonServer_log.20.txt:01/13 19:28:13 Error: [seekableurimediafile] while determining length: System.IO.IOException: FTMSI-B: timeout getting file length

which only seem to happen during playback and not backups.

Hey @Aaron_Turner,

Ben here stepping in for Noris while he is out of office. Following up on the newly created backup directory, did you run into the same issues?

This is expected behavior after moving the original backup location/folder.

Let me know, thanks! :+1:

Would you like me to upload the logs again? Without knowing what to look for, I don’t know.

Hi @Aaron_Turner ,

Are you still seeing this issue? If so, then yes, can you please upload a new log set and let us know? I saw a strange error related to your backup and I’m curious if it’s still the same one or if there’s any new information in the logs. Thanks!

Well I’ve disabled the process on my NAS which was causing the kernel to hang, so I’m not having to hard restart my NAS. This was the 2nd time I’ve found a user-land process to cause the issue (seems related to high CPU utilization).

But yeah, the only way I know of recreating this issue is to cause the NAS to hang/crash. I’d rather not do that to be honest :slight_smile: The purpose of this post was to inform Roon of this bug so hopefully you could fix it.

Well, my NAS started having it’s problem again. So I had to hard reset and sure enough, Roon forgot it’s settings and when I go to restore from backup, it says I have no backups again.

This was after over the weekend I verified backups were indeed working and I could see the 31 previous backups. So back to square one.

Had to restore:

  • /backups/_roon_backup_root_
  • /backups/9892a01e-ff48-b763-06c7-85996c98f7f7/_roon_backup_
  • /backups/9892a01e-ff48-b763-06c7-85996c98f7f7/name

from my NAS backups. then I could restore the Roon backup.