Backup problem persists after RAM upgrade to 32GB (ref#TGCBFJ) [Ticket in]

Support
1

Hi! What’s not quite right with Roon?

· None of the above quite fits

None of the above quite fits

· None of these quite match

Tell us what's going on

· Hey Noris,
the backup problem is back, even after upgrading the RAM to 32 and I‘m reducing the libary at the same time.
Regards
Uli Lennartz

Tell us about your home network

· You know

2

Hi @Uli_Lennartz,

Thank you for writing in and sharing your report! We were able to review fresh Roon Server logs, and saw the core of the issue here.

The good news first: the RAM upgrade did exactly what it needed to do. The logs from 16–18 September show no out-of-memory terminations at all, the operating system is no longer killing Roon Server. Where the machine was previously pushing ~14.9 GB against a 16 GB ceiling, it now sits at around 10 GB, with the kernel reporting roughly 64 GB installed. (Worth confirming what actually went in, since that’s more than the 32 GB you mentioned.) Either way, memory now has a very wide margin.

Which means the backup failure you’re still seeing is not the same problem, even though the error text looks similar. You can stop trimming the library, library size is no longer a factor, and reducing it further won’t change this.

What the logs show now:

Each night the backup begins with a database validation pass. It works through playlists, profiles, radio, and all ~13.8 million music entries, then fails at a consistent point about three minutes in. 17 and 18 September are identical:

[broker/database/vacuum] processed 13867798 music entries
Warn: [broker] validation failed: System.OutOfMemoryException
Error: [broker/database] corruption detected
Warn: [broker] detected corrupt database, halting broker threads

Despite the wording, this isn’t memory pressure, the exception comes out of the storage layer, and with more than 50 GB free the only way that allocation fails is if a record in the database reports an invalid size. Roon reaches the same conclusion on the very next line: corruption detected. There’s a damaged record in the database file, and validation lands on it in the same spot every night.

This also explains the daily restart.

That last line is the key one. Once the broker threads halt, they never resume, the backup flag stays stuck at “in progress,” and the media and transport threads stay paused. Your second backup destination (\Musikserver) starts on schedule at 06:00 but simply hangs, because there’s nothing left running to talk to. On 18 September the server sat in that state from 02:03 until your manual restart at 16:08. The restart isn’t a separate fault, it’s the only way out of the halt.

As for where the damage came from, the most likely explanation is the 22 hard terminations on 6 September. Database writes interrupted mid-flush leave exactly this kind of injury. The RAM upgrade removed the cause, but the damage it left behind is still there, and it won’t repair itself.

Next step, one question: When was your last successful backup? These logs only cover three days, so we can’t determine that from here. If you have a backup from before this began, restoring from it is the cleanest path forward.

Thank you!

3

Hey @Uli_Lennartz,

Just checking in on this following @benjamin ’s reply. Have you had a chance to look back at the last successful backup you had before this started, and if so, was it possible to restore from it using the Backup guide? If that route is not available, please let us know what changed most recently on the Roon Server side, and we can keep working from there. Please reply with any updates or questions.

Please note, if we don’t hear back from you this thread may close automatically soon. If the thread auto-closes and you need further assistance, please submit a reopen support request via the technical support help form below and specify that the issue should be reopened. Thank you.

4

Hi Noris,

unfortunately, the Problem is still there.

Regards

Uli

5

@noris
Can I get some help?

6

The problem persists. A backup still cannot be produced. I have to restart Roon every day.

[Signature removed]

7

Hi @Uli_Lennartz,

Thank you for your patience. We’ve analyzed the latest logs to reach our cloud diagnostic servers from the ROCK hosting your Roon Server to check for signs of ongoing corruption or OOMs.

Your logs show an urgent issue to address first. The ROCK web admin page is reachable from the internet, and outside addresses have been probing it daily since at least 27 September.

  1. In your router settings, remove any port forwarding, “Exposed Host” or DMZ entry that points to your ROCK (port 80). If UPnP port sharing is enabled, turn it off as well.
  2. Reply here once this is done.
  3. When was your last successful backup? Restoring from it is still the cleanest path to fixing the failed backups and daily restarts.

Thank you!

8

Hi Connor,

  1. The Exposed Host are closed. UPnP turned off as well

  2. I restored it form the cleanest Backup, I think…The last one was in September.

Regards

Uli

9

Hello @Uli_Lennartz,

Thank you for closing the port forwarding and turning off UPnP.

We have gone back through your logs and found the actual cause of the failed backups and the daily restarts. During each backup, Roon checks the database first, and on a library of your size that check needs more open files than Roon Server is currently allowed to use. When it reaches that limit, the check fails, Roon reports it as a corrupt database and stops, and it stays stopped until it is restarted. That is a misleading message: the database is not damaged, and restoring a backup could not fix it. We are sorry we pointed you in that direction earlier.

This is a known issue that our developers are already working on, and we have added your case to the existing report. We do not have a release date yet, and we will let you know in this thread as soon as the fix is available.

Until then, every scheduled backup will fail in the same way and stop the server. If you would like to avoid the daily restart in the meantime, please turn off the scheduled backups for now. Your existing backups stay where they are, and we will tell you here when it is safe to turn them back on.