Failed backups on Ubuntu server setup (ref#RHU1SP)[Ticket in]

Support
1

Hi! What’s not quite right with Roon?

· None of the above quite fits

None of the above quite fits

· None of these quite match

Tell us what's going on

· Backups fail. i have a headless Ubuntu server recently installed and set up. The backup runs, I see a folder created on my network share, but nothing is inside it and when I have Roon look for backups and I point it to the folder created on the share by the backup process, it says that is not a valid backup file. Also the files created are extremely small, 21bytes. It does not seem like the backup ran successfully.

Tell us about your home network

· That should not be relevant. It is a local network that has internet access and is behind a firewall.

2

Hey @bret, welcome to the community, and thanks for the clear report.

That behavior does sound like the backup is creating the destination folder, but not actually writing a valid backup set inside it. Since this is a new Ubuntu server setup, the first thing I’d like to check is the backup target itself. Roon backups need to land on a share that stays available, allows normal file writes, and is being reached through a path Roon can use reliably. Our Backup guide covers the expected backup and restore flow, so it’s worth comparing your setup against that.

Please try creating a new backup destination in a fresh folder on the share or to a local share, then run a manual backup again and leave the process open until it finishes. If it still produces only a tiny folder with no contents, the next useful check is whether the share is mounted with write access from the Ubuntu server itself, not just visible to Roon.

Could you also tell us what kind of share this is, for example SMB or NFS, and whether the Ubuntu server is connected over Ethernet or WiFi? Let us know what you see after the next backup attempt.

3

Thanks @noris (Chuck?),

I figured that since Roon created files and folders in the share that we could assume that the share and permissions and such are fine.

I didn’t belabor my original description, but originally I was backing up to the Ubuntu server in /home/roon/backups, i.e. my user’s home folder. This was similarly not successful, creating just the nearly empty folder and 21-byte file (also an empty folder called xx).

I read the backups help page that you referenced, and since it recommended NOT doing a backup to the local machine (for obvious data-safety reasons). I figured I would just try doing what the help docs say before posting here, so I created a share on my Windows computer, gave the roon user read/write permissions, and entered the details as a new backup job. I’m using SMB. \\server-ip-address\share-name and I filled in the credentials in the form.

The Ubuntu server is connected using Ethernet and my Windows computer that has the SMB share happens to be connected via Wi-Fi.

I can try doing a local backup again if that makes it simpler.

Not to ramble on too much here, but when I originally set up the backup on the local Ubuntu server to /home/roon/backups the backup job would actually stop/crash the Roon server processes. Or at least stop them from effectively functioning. Meaning using htop I could see a few roon processes still running, but very low CPU, and the UI of my Roon client software said there was a problem with the database and that I’d have to restore from backup. In order to solve that, I rebooted the server and it came back up fine.

I’ve not changed my setup from when I originally submitted my ticket. I’m doing what you suggested, I’m running a backup while watching the window…

Snapshotting databases for backup…

About two minutes later: There was an issue loading your library. To avoid data loss, restore a backup now or head over to our Help Center for more information. (Restore a backup) button.

That’s it. I’ve hit this point before and waited a while and it doesn’t come back. I’ll need to reboot the server to get it back up and running.

Are there some logs I should review and/or send you?

Thanks.

4

I think I found the problem, out of memory. I found the logs. There is a lot more in this stack trace, but I think these are the important lines. I replaced my IP address with x.x.x.x.

09/24 07:05:22 [Local 09/24 03:05:22] Debug: [broker/filebrowser/volumeshare] Use dispose, id: \x.x.x.x\roon-server-bkup
09/24 07:05:22 [Local 09/24 03:05:22] Warn: [broker/backups] unexpected error doing backup on FileBrowser.Entry: \x.x.x.x, roon-server-bkup : \RoonBackups: Sooloos.SynchronizationContextThreadException: In Broker:Media
—> Sooloos.SynchronizationContextThreadException: In Broker:Transport
—> Sooloos.SynchronizationContextThreadException: In Broker:Misc
—> System.OutOfMemoryException: Exception of type ‘System.OutOfMemoryException’ was thrown.

My library is about 182,000 tracks. The computer I’m running this on has 16GB of RAM. Should be plenty. The only thing I’m running on this computer is Roon.

I found check.sh and ran it:

xxxxxx:/opt/RoonServer$ ./check.sh

Checking to see if RoonServer can run on this machine

Checking for a supported glibc version                       [   OK   ]
Checking for Binary Compatibility                            [   OK   ]
Checking memory page size                                    [   OK   ]
Checking for ALSA Libraries                                  [   OK   ]
Checking for ffmpeg or avconv                                [   OK   ]
Checking for the mount.cifs command                          [   OK   ]
Checking for libicu                                          [   OK   ]
Checking for libstdc++                                       [   OK   ]
Checking for libtasn1                                        [   OK   ]
Testing ulimit -n 8192                                       [   OK   ]

STATUS: SUCCESS

Any suggestions?

5

I looked further into this. The backup was running out of open files. Line 38 of /opt/RoonServer/Appliance/RoonAppliance runs “ulimit -n 8192,” which caps RoonAppliance at 8,192 open files even when systemd allows more.

My library (about 182,000 tracks, a 24 GB database spread across roughly 34,600 files) needs more than that during a backup. When the limit is hit, the log shows “Too many open files,” the error surfaces as an OutOfMemoryException, and the Core goes down.

To get a backup to run, I raised the limit on the running process with prlimit:

sudo prlimit --pid --nofile=131072:131072

With that change, the open-file count peaked above 10,000 and the backup got through the stage where it had always failed.

The change is lost every time Roon restarts.

@noris Please put in a request with your dev team to raise the hard-coded limit, or stop the launcher scripts from lowering a higher limit set by systemd, in a future update.

For now, my backup is running.

Thanks,
Bret

6

@noris Do you think I’ll get a response on the above? I’m new here, I’m not sure what to expect.

My concern is that I have a band-aid fix on my backups. Next time I restart the server, my backups will be broken again.

7

Who at @roonsupport can help me with this issue or give me advice on how to set up the server so that this issue does not come up? I thought I followed the how-to on installing headless on a linux server pretty well.

If I need to re-do the installation differently I can do that. I have not put a ton of time into this installation yet.

9

@ivan Just checking if you’ve seen my question here and maybe you have an idea?

10

Hi @bret

The three launchers are Server/RoonServer, Appliance/RoonAppliance and Appliance/RAATServer, and each carries the same line at 38. Leave check.sh alone, it only tests the value.

Rather than editing them by hand after every update, here is a version that reapplies itself on each start.

1. Open the service override

sudo systemctl edit roonserver.service

2. In the editor that opens, add

[Service]
LimitNOFILE=131072
ExecStartPre=-/bin/sh -c "sed -i 's/^ulimit -n 8192$/ulimit -n 131072/' /opt/RoonServer/Server/RoonServer /opt/RoonServer/Appliance/RoonAppliance /opt/RoonServer/Appliance/RAATServer"

Put this in the blank area near the top, above the line that reads “Edits below this comment will be discarded”. Anything added below that line is thrown away and the override will come out empty. Save with Ctrl+O, then Enter, then exit with Ctrl+X. You should see “Successfully installed edited file”.

The second line raises the ceiling the service is given. The third rewrites the limit in the three launchers just before Roon starts, so an update that restores the original files is corrected on the next start automatically.

3. Apply it

sudo systemctl daemon-reload
sudo systemctl restart roonserver

4. Confirm it took

grep -rn "ulimit -n" /opt/RoonServer

The three launchers should now read 131072. check.sh will still say 8192, which is correct and expected.

To remove it later, run sudo systemctl revert roonserver.service and restart the service, then reinstall Roon Server if you want the original files back exactly as shipped.

Two things worth saying plainly: this edits files inside the Roon installation, so it is a local workaround rather than a supported configuration, and it is worth undoing once the proper fix is out. We will post here when that happens.

11

Ok I’m glad to hear that this will be fixed for everyone. Thanks.

12

Hey @bret

Thanks for performing that test, we have added this info to the investigation ticket. The ticket has been submitted for developer review, meaning that this may take some time to be reviewed.

There aren’t any additional details needed at this time, so we will go ahead and close the thread for the time being, and if any other questions arise after the developers have reviewed the case, we will re-open the thread and reach out once more.

Thanks again for your report, and thank you in advance for your patience as the ticket progresses through the development queue for review.