Bringing back topic of serious linux Roonserver problems with frequent server 'oop's' and devouring memory

[Wessel_Dirksen]

I’m sorry but a rant here, also intended as a reminder, is absolutely necessary again. The linux server issues mentioned previously as long ago as 2 years ago still continue.

Roon server keeps using more and more RAM even sitting idle

Roon Core Machine Server: NUC8i3BEH with Windows 11 and 32GB RAM and 256GB SSD Library: QNAP TS-451 with 8TB RAID5 Networking Gear & Setup Details Server and Library both use Ethernet connection to TP-Link AX11000 router, AT&T 1GB Fiber Remotes use 5G wireless Connected Audio Devices Server is connected to Denon x3700H via HDMI. Remotes use Roland UA-1610, various Google Home speakers, phones, iPad, etc. Number of Tracks in Library 55,000 tracks Description of Issue As the R…

Roon server process reaches 9.9G of RAM

Hi, I have Roon with the latest release updated on a Ubuntu 22.04.02 physical machine. The Roon server process reaches 9.9G of RAM usage and starts to drop Arc connections. I believe the release 1244 still has big memory leak issues. I restarted the server just yesterday because of this and had to do it again today.

Roon server on Linux consuming huge amount memory even when not in use

[Roon - Top memory] You can see here that RoonAppliance consumes between 70% to 78% of 16GB total RAM depending on activity. When the screenshot was taken no Roon appliance was opened and therefore not playing media. It looks to be holding a boat load of cache in memory. I am running Roon on a Manjaro (+/- Arch Linux) based home server + router using the systemd-networkd network ecosystem. The screenshot shows the system parameters. So both the Roon server and ARC work well and the machine wo…

I have tried many adjustments in roonserver.service and changing kernels which sometimes improves, but never solves the issue. A year ago there were reports of a working beta version in testing that solved it. Where is that fix now for the rest of us?

Currently resource limitation adjustments don’t seem to make a difference to the constant Roon server “freeze and forget” behavior. Memory limits are still necessary to prevent Roon gobbling up 90% of 16GB RAM on a reasonably powerful minimalistic headless server and stop it from seizing more often

This is becoming ‘abondon ship’ material as the frustration is running so high here with Roon freezing up 4 to 5x per album regardless of resolution, Qobuz, or local libraries. However AFAIK, apart from buying an expensive streamer, there really is no lifeboat. Volumio at last check isn’t cutting it either.

So help us please !! There are many here who share this experience I’m sure. If you cant fix it, then I’m sure someone in the R&D knows a few tips in how to manage a Roon linux server to minimize some of these issues. I really feel the inaction and/or silence on this is NOT what Roon as a company with a high international reputation wants for themselves.

So acknowledged, I’m angry. But I NEED YOU! Please fix it!

[Fernando_Pereira]

The problem with this issue is that it must depend on some hard to replicate, relatively rare combination of factors. Over the last several years of running Roon on Ubuntu Server (20.04 and then 22.04), 3 different systems in different locations, I had only one relatively short period with concerning physical memory growth on one of those systems. Here’s a typical memory graph for my most used server (the short drop-to-zero events are reboots (Ubuntu updates) or new Roon Server releases.

[Wessel_Dirksen]

I’ve been on the PC with Glances running through an ssh window. There are 100% CPU spikes when roonserver goes MIA. While they are brief spikes, it is enough for roonserver to be knocked unconscious for 20 to 30 seconds. Because the roonserver is located on a linux based router and file server, I suppose with my particular use case the CPU spikes could interfere with network transmission making a bigger impact on the issues with roonserver. However the SSH connection is not affected when this happens so it cannot due to a complete network freeze. The network is otherwise normally very quick in regard to latency with LAN network reporting 0ms.

I’ve been using memory limits for awhile but had to put CPU limits back in roonserver.service and gave systemd-networkd a higher priority which seems to have helped again. I thought it was fixed awhile back but now back to CPU limits.

[Service]
…
MemoryMax=60%
MemoryHigh=50%
CPUQuota=80%
CPUWeight=70

[Andreas_Bruehwiler]

You have all my support, Wessel_Dirkson. Exactly the same problem here. I have a lifetime licence and this seems to become a lifetime problem. After starting roon, it is usually good for about 24hrs and then, spikes and memory leaks start. System is unresponsive for around 30 seconds. I logged the events, it usually happens 4 times per hour, at the same minutes of the hours. This is just a short part of the log:

Would be nice if you guys @ Roon have a look at your code and find out what you are doing 4 times an hour.

And it is not only a Linux issue - running here on Windows 11 Professional.

[Fernando_Pereira]

There’s definitely something happening occasionally. Here’s the memory graph for one of my Roon servers. I reboot them for Ubuntu software upgrades, and RoonServer for new versions. I have no idea what was going on May 19-27, but nothing very different from my usual usage on this server.

[Wessel_Dirksen]

Thanks for your support Andreas. So this is also happening on Windows? Ouch!

I thought bringing back CPU limits would fix it but as you indicate, for several days after reboot it seems to be OK then starts to happen again. The problem is a bit less frequent and restores itself in about half the time with the CPU limits in place. Still unacceptably disruptive.

I have decided to give this another try with Roonserver running on it’s own exclusive x86_64 4 core machine (Odroid H4 with 8GB DDR5 RAM) with a very minimal installation of ArchLinux which would be close to a ROCK setup. No extra services will be running except roonserver (and it’s dependencies) and dhcpcd. Because in the past some have suspected that IPv6 may be part of the problem, I will also test it with IPv6 disabled if the issues come back

[Andreas_Bruehwiler]

I disabled IPv6 as well for testing, but no results.
Interestingly, the spikes go together with an increasing memory usage. As soon as you have the spikes, memory usage starts to grow. I did set up a watchdog that restarts Roon if the usage goes over 10GB. But this is a really dirty fix.
I disabled the local library (about 100’000 files) for now and let’s see if thas has an impact.

[Andreas_Bruehwiler]

Every processor spike generates around 300 entries in the Roon_log as such:

06/19 07:22:32 Trace: [library] endmutation in 16ms
06/19 07:22:32 Trace: [library] endmutation in 18ms
06/19 07:22:32 Trace: [library] endmutation in 16ms
06/19 07:22:32 Trace: [library] endmutation in 18ms
06/19 07:22:32 Trace: [library] endmutation in 22ms
06/19 07:22:33 Trace: [library] endmutation in 19ms
06/19 07:22:33 Trace: [library] endmutation in 29ms
06/19 07:22:33 Trace: [library] endmutation in 17ms
06/19 07:22:33 Trace: [library] endmutation in 19ms
06/19 07:22:33 Trace: [library] endmutation in 16ms
06/19 07:22:33 Trace: [library] endmutation in 16ms
06/19 07:22:33 Trace: [library] endmutation in 16ms
06/19 07:22:33 Trace: [library] endmutation in 18ms
06/19 07:22:33 Trace: [library] endmutation in 28ms
06/19 07:22:33 Trace: [library] endmutation in 16ms

It seems to be library related. I deleted the local library and now no problems within the last 24 hrs.

Roon, would you mind to throw some ideas in here and be a little bit cooperative?

[Geoff_Coupe]

@Andreas_Bruehwiler

Roon, would you mind to throw some ideas in here and be a little bit cooperative?

This category (Roon Software Discussion ) is not generally monitored by Roon Labs staff, so it’s unlikely that you will get any response.

Either raise a support request in Support, where the Support team can pass logs to the dev team for further investigation, or raise your concerns in Feedback, where every post is read by Roon Labs.

[Andreas_Bruehwiler]

Thanks Geoff, moved it to support.

[Wessel_Dirksen]

I have now spent 5 days with Roonserver running exclusively an Odroid H4 (based on Intel N97) with 8GB of RAM running a minimal headless installation of ArchLinux which is connected via ethernet to the LAN. The H4 is also used as the main USB audio endpoint thereby replacing a separate RPi4 running Ropieee. Roon ARC connects fine. IPv6 has been deactivated in dhcpcd.conf.

/etc/dhcpcd.conf

...
noipv6rs
noipv6
...

Roon has been playing >20 continuous hours while I’ve been working, and I’ve let it run through the nights via Roon radio, and so far it performs very well with no glitches or latency issues. I have not altered roonserver.service and RAM now stays consistently between 18% and 22%. A few tweaks were necessary to run smoothly:

Without interrupting the audio stream or getting stuck, there were moments where Roon app briefly showed the lost connection icon. Also my ssh connection got disconnected quite often. Since the default linux network buffers are rather low at 208 kb. Increasing buffers to 16384kb in sysctl solved both issues and improved latency of the Roon music UI, scrolling through music library, etc. This did not have any noticeable change to RAM being used. I tried 32768kb as well which appears to be no different to 16384kb, not better or worse.

/etc/sysctl.conf

net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.core.rmem_default=16777216
net.core.wmem_default=16777216

Also while still running smoothly there were 100% CPU spikes from time to time. The default linux CPU governor setting is ‘powersave’ where the H4 was often running at the low limit of +/- 1.4Ghz. While changing the setting to ‘performance’ definitely fixed this, it was running at a minimum of 2.5Ghz constantly which was raising the temps up a bit. I ended up setting the state back to ‘powersave’ but setting the minimum range to 2Ghz which solved the problem as well without idling so high. I have not seen any more max CPU spikes.

/etc/systemd/system/cpupower.service

[Unit]
Description=Set CPU governor to powersave with min frequency 2GHz

[Service]
Type=oneshot
##ExecStart=/usr/bin/cpupower frequency-set -g performance
ExecStart=/usr/bin/cpupower frequency-set --min 2GHz
ExecStart=/usr/bin/cpupower frequency-set -g powersave

[Install]
WantedBy=multi-user.target

I also have the following services running aside from bare essentials to run Roon Server:

dhcpcd.service for networking
cpupower.service
thermald.service
irqbalance.service
systemd-resolved.service

The moral of the story might be to only run Roon Server on a dedicated machine. I recommend the Odroid H4 as a reliable Roon Server platform for running on Linux. Not expensive, much smaller than a normal PC, silent / fanless possible with the above settings.

EDIT: I will try to run it without blocking IPv6 and see what happens to gain insight into whether it is part of the problem ornot

So all is good but there is a mystery. Qobuz shows all the media and tracks in Roon but when you select them to play it will not play/stream the selected audio saying “can’t find media” or something like that. I have never had this problem before. I researched all the forums and tried logging in/out, opened up a bunch of ports on the LAN, deleted cache folder, mDNS set in LAN zone in firewalld, etc. but Qobuz won’t stream through this new Roon server. While this could be related to router settings, nothing worked so I tried a switch to Tidal and it now just works. I’m a bit sad as I respect Qobuz for boldly opening up the streaming industry to untouched high res audio. Maybe I’m paranoid but sometimes I think the FLAC files (not MQA) used in Tidal are not as clean sounding as Qobuz. But my Roon server problem I believe is finally solved

Thanks for the support on this issue

[Wessel_Dirksen]

Putting an update in early possibly before topic is closed.

I have been playing music over the above described H4 running Roon server non-stop with IPv6 open for about 38 hours now. There is no noticeable difference compared to the past week with IPv4 only: RAM use steady at +/- 2GB from 8GB and no CPU blockages. It might be too early to know for sure but previously on my other machine RAM use would have certainly been maxing out and playback would have crashed & stopped by now. I am using the same large local library with a recent backup of the previous machine. Based on this experience IPv6 does not appear to be the cause of the server errors.