A memory leak in RoonAppliance?

[ranel]

Roon Core Machine

Thinkpad T480s with 16GB RAM, running Ubuntu 22.04.

Networking Gear & Setup Details

On WiFi, no VPN.

Connected Audio Devices

No audio devices connected when this happens.

Number of Tracks in Library

About 7k tracks, maybe half of which are local redbook files and the other half from Qobuz.

Description of Issue

Since a short while I’m suffering from what looks like a memory leak in RoonAppliance. Here’s how the memory used by my core looks over 24h during which there’s no music playing, and no connections from Roon Remotes, Bridges, or ARC, only housekeeping (this is MemoryCurrent as reported by systemctl show):

The journalctl log looks like this:

log

Mar 07 23:01:05 CAP-T480s systemd[1]: Started RoonServer.
Mar 07 23:01:05 CAP-T480s start.sh[198179]: 00:00:00.000 Warn: get lock file path: /tmp/.rnsgem0-
Mar 07 23:01:05 CAP-T480s start.sh[198179]: Initializing
Mar 07 23:01:05 CAP-T480s start.sh[198179]: 00:00:00.026 Debug: RoonServer, before attempting to start RoonAppliance binary at path: /opt/RoonServer/Server/…/Appliance/RoonAppliance
Mar 07 23:01:05 CAP-T480s start.sh[198179]: 00:00:00.036 Debug: RoonServer, after starting RoonAppliance
Mar 07 23:01:05 CAP-T480s start.sh[198179]: Started
Mar 07 23:01:06 CAP-T480s start.sh[198205]: aac_fixed decoder found, checking libavcodec version…
Mar 07 23:01:06 CAP-T480s start.sh[198205]: has mp3float: 1, aac_fixed: 1
Mar 07 23:01:10 CAP-T480s start.sh[198179]: Running
Mar 08 12:19:25 CAP-T480s start.sh[198179]: Not responding
Mar 08 12:19:31 CAP-T480s start.sh[198179]: Running
Mar 08 12:33:53 CAP-T480s start.sh[198179]: Not responding
Mar 08 12:33:58 CAP-T480s start.sh[198179]: Running
Mar 08 12:34:33 CAP-T480s start.sh[198179]: Not responding
Mar 08 12:34:39 CAP-T480s start.sh[198179]: Running
Mar 08 12:44:14 CAP-T480s start.sh[198179]: Not responding
Mar 08 12:44:19 CAP-T480s start.sh[198179]: Running
Mar 08 12:49:48 CAP-T480s start.sh[198179]: Not responding
Mar 08 12:49:53 CAP-T480s start.sh[198179]: Running
Mar 08 13:04:05 CAP-T480s start.sh[198179]: Not responding
Mar 08 13:04:10 CAP-T480s start.sh[198179]: Running
Mar 08 13:04:25 CAP-T480s start.sh[198179]: Not responding
Mar 08 13:04:30 CAP-T480s start.sh[198179]: Running
Mar 08 13:19:50 CAP-T480s start.sh[198179]: Not responding
Mar 08 13:19:55 CAP-T480s start.sh[198179]: Running
Mar 08 13:34:08 CAP-T480s start.sh[198179]: Not responding
Mar 08 13:34:13 CAP-T480s start.sh[198179]: Running
Mar 08 13:49:18 CAP-T480s start.sh[198179]: Not responding
Mar 08 13:49:24 CAP-T480s start.sh[198179]: Running
Mar 08 13:49:49 CAP-T480s start.sh[198179]: Not responding
Mar 08 13:49:54 CAP-T480s start.sh[198179]: Running
Mar 08 14:04:25 CAP-T480s start.sh[198179]: Not responding
Mar 08 14:04:51 CAP-T480s start.sh[198179]: Running
Mar 08 14:05:26 CAP-T480s start.sh[198179]: Not responding
Mar 08 14:05:31 CAP-T480s start.sh[198179]: Running
Mar 08 14:06:11 CAP-T480s start.sh[198179]: Not responding
Mar 08 14:06:27 CAP-T480s start.sh[198179]: Running
Mar 08 14:19:54 CAP-T480s start.sh[198179]: Not responding
Mar 08 14:20:26 CAP-T480s start.sh[198179]: 15:19:20.084 Debug: RoonServer, after RoonAppliance exit, exitcode: 137
Mar 08 14:20:26 CAP-T480s start.sh[198179]: Error
Mar 08 14:20:28 CAP-T480s start.sh[198179]: Initializing
Mar 08 14:20:28 CAP-T480s start.sh[198179]: 15:19:22.098 Debug: RoonServer, before attempting to start RoonAppliance binary at path: /opt/RoonServer/Server/…/Appliance/RoonAppliance
Mar 08 14:20:28 CAP-T480s start.sh[198179]: 15:19:22.107 Debug: RoonServer, after starting RoonAppliance
Mar 08 14:20:28 CAP-T480s start.sh[198179]: Started
Mar 08 14:20:28 CAP-T480s start.sh[297930]: aac_fixed decoder found, checking libavcodec version…
Mar 08 14:20:28 CAP-T480s start.sh[297930]: has mp3float: 1, aac_fixed: 1
Mar 08 14:20:29 CAP-T480s start.sh[198179]: Running

I haven’t added any local files since months, and didn’t do any significant imports on Qobuz either since about a month (when I migrated from Tidal).

I’ll be grateful for any help, because this is severely impacting my use of Roon: Even if I restart the server every morning, I cannot listen on my way back from work because the server is already in a past-10GB zombie state and ARC no longer connects. What I tried so far was removing all extensions. What I could try still, if you think it’s useful, is generating a massif report for my core over a similar period. Let me know!

[soks (Trent)]

Couple thoughts here. First is whether you can mitigate the problem in the meantime by setting a limit on the service’s memory. Steps to do that:

1. make a backup of /etc/systemd/system/roonserver.service somewhere
2. open that file to edit with sudo permissions
3. add:

[Service]
Restart=always
MemoryMax=6G

and restart the services with sudo systemctl daemon-reload. Running systemctl status roonserver should show the changes take affect. Restarting my services shows roonserver down to 1.8GB (from 6.6GB). The memory limit I put is just an estimate and perhaps you would need to alter it.

An alternative to editing the config would just be regularly restarting the service yourself.

---

Just for reference, here are some notes on how RoonService is behaving on my system (before I restarted the services for the steps above).

systemctl is accounting for more processes and totals 6.6G in memory. My library is only about 2.5K tracks, almost all Qobuz.

$ systemctl status roonserver
● roonserver.service - RoonServer
     Loaded: loaded (/etc/systemd/system/roonserver.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2023-01-08 17:31:03 EST; 1 month 29 days ago
   Main PID: 974 (start.sh)
      Tasks: 121 (limit: 18928)
     Memory: 6.6G
        CPU: 2d 20h 58min 27.017s
     CGroup: /system.slice/roonserver.service
             ├─    974 /bin/bash /opt/RoonServer/start.sh
             ├─1041909 /opt/RoonServer/RoonDotnet/RoonServer RoonServer.dll
             ├─1041945 /opt/RoonServer/RoonDotnet/RoonAppliance RoonAppliance.dll -watchdogport=
             ├─1041949 /opt/RoonServer/Server/processreaper 1041945
             └─1041995 /opt/RoonServer/RoonDotnet/RAATServer RAATServer.dll

I’m running on a similarly sized machine and see a large amount of virtual memory being used by RoonAppliance on PopOS but a reasonable resident size.

The virtual memory for RoonAppliance is 23.1GB reported by top and matched with pmap. The resident size is 1.4GB.

How does that compare with your system?

For top:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                              
1041945 root      25   5   23.1g   1.4g 253400 S   2.3   9.2 367:18.25 RoonAppliance  

And from pmap there is no discrepancy from top:

$ sudo pmap 1041945 -x | head
1041945:   /opt/RoonServer/RoonDotnet/RoonAppliance RoonAppliance.dll -watchdogport=
Address           Kbytes     RSS   Dirty Mode  Mapping
~~skipping all of this print~~
---------------- ------- ------- ------- 
total kB         24238076 1496968 1242748

[Nigelc]

You are not alone.
I have 17k files. Looking at my usage over the last year I have been been using about 2-3G consistently.
Something changed in a recent update I think?
There are a few of us with this issue.
I guess we need to say the keyword ARC get get attention!

I have actually switched back to my LMS server for the moment, I really can’t be bothered restarting my server everyday!

Memory leak causes search slowness and play stoppage

Ah yes! Came here for the same issue! Same symptoms! Using Linux in my case, running 1223 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 697599 root 20 0 24.8g 6.5g 58752 S 90.4 84.3 2799:38 RoonAppliance Using about 6.5GB of memory! Looking at the graph is the 4th when the new update hit? [image]

Excessive RAM usage

Roon Core Machine Ubuntu Server 22.04.2 Lenovo Thinkcentre P340 i7 10700t 32GB RAM Running Roon in a docker container Networking Gear & Setup Details Server connected to a Synology RT6600AX, endpoint vary mostly wireless Connected Audio Devices Linn Majik DS Bluesound Pulse Number of Tracks in Library around 10,000 albums 169,300 track 95+% of these are Tidal Description of Issue Roon has generally been running fine but over the past week it has starting consuming all a…

[Philip_Holst]

I don’t think the problem is only on Linux. I have a Windows server which has been running Roon for 2+ years. This started happening for me with the last couple of releases. For a day or two the memory usage will be stable, but after a while it will start growing. It eventually gets to a point where playback will stop and the Roon Apps will not let me do anything related to playback. I usually need to restart Roon every 4 days to avoid this.

[ranel]

Thanks for the MemoryMax trick! I didn’t know this is possible in systemd. I did the same change (with an 8GB limit) and will see how things go. Until now I just used a shell script to poll the memory usage and restart the server when needed, but this is much cleaner.

I just restarted my server so the memory usage in top / pmap is not interesting to look at yet. But since real memory usage is always between virtual and resident, I thought that the value returned by systemctl (which does fall in between the two) might be a good approximation of the actual value. It’s true that it includes a few processes in addition to RoonAppliance, but I never saw these having a significant memory footprint.

[ranel]

Thanks for linking these threads! I knew I’ll be in good company I scanned for similar issues before posting, but there’s so much traffic here I cannot even find threads I posted to a day ago… Seems to be a real issue then, and on multiple platforms judging from @Philip_Holst’s report.

[danny (Danny Dulai)]

systemd is showing you all memory, including virtual memory. These numbers are not what you think they are… they are not actually memory consumption. When you set max memory to something lower, it just takes less virtual memory because the VM subsystem is releasing/reallocating more aggressively (more CPU). You could also be running into garbage collection releases happening more often (much more CPU), but not likely at this level.

If you made a real difference with the MaxMemoery thing you set, Roon would crash when it hit that limit.

Are you crashing when you hit large memory numbers?

Let me give you an example from a ROCK system I have running here:

This is the output of top:

  PID  PPID USER     STAT   VSZ  %VSZ CPU %CPU COMMAND
23391 23351 root     S    26.8g 349.0   3  3.5 /roon/app/RoonServer/RoonDotnet/RoonAppliance RoonAppliance.dll -watchdogport=45435

Notably, it is claiming to be using 26.8 gigabytes of memory. Funny, since this machine only has a single 8 gigabyte stick of RAM, and no swap set up. I’n also running a bunch of other things on this machine, which must use some RAM. My guess is that the RoonAppliance in this situation is usually close to 4 gigabytes (about 150k track library).

Roon does spew on stats in it’s log on this, and while it’s not exact either, this is what it shows on my system:

03/10 16:34:59 Info: [stats] 27468mb Virtual, 4706mb Physical, 4288mb Managed, 467 Handles, 94 Threads

Looks better, but probably still off from reality because of garbage collection.

Memory management on modern operating systems is very complex, and the stats about memory consumption at these levels are often misunderstood because they are incomplete and not intuitive.

Moving this ticket out of the #support area as no support is needed here. Roon is operating as designed. It looks to me that your Roon is using something like 1 gigabyte of memory, and the rest of the stats are painting a bad picture.

This is another reason we don’t show information like this in ROCK or Nucleus. It creates irritation on the consumers’ end and wasted support costs on Roon’s end – all for nothing.

[ranel]

Hi @danny! You’re asking

Are you crashing when you hit large memory numbers?

Yes. Once I get to about 14 GB (on a 16 GB system) of whatever fictitious memory systemctl is showing, RoonAppliance crashes with exitcode 137:

Mar 08 14:20:26 CAP-T480s start.sh[198179]: 15:19:20.084 Debug: RoonServer, after RoonAppliance exit, exitcode: 137
Mar 08 14:20:26 CAP-T480s start.sh[198179]: Error
Mar 08 14:20:28 CAP-T480s start.sh[198179]: Initializing
...

And to be extra clear, this is without any MemoryMax tricks in systemd, vanilla Roon with no extensions.

I guess it was counterproductive to give this thread such an annoying name (“A new memory leak in RoonAppliance?”), sorry for that. But a crash with exitcode 137 after some-form-of memory usage approaching machine limits points towards this being a real issue.

[danny (Danny Dulai)]

ok, crashes on exit code 137 are error worthy

7k tracks? what audio devices? what remotes? core on the ThinkPad?

[soks (Trent)]

Quite welcome! If Roon crashes at your 16GB limit then hopefully this will at least let it restart more gracefully.

@ranel

It’s true that it includes a few processes in addition to RoonAppliance, but I never saw these having a significant memory footprint.

I don’t see them having much memory either. What I do see is my Roon logs report:

03/10 15:44:44 Info: [stats] 23472mb Virtual, 1242mb Physical, 506mb Managed, 360 Handles, 79 Threads

Which matches closely with the RSS size for RoonAppliance from pmap. I think systemctl status is accurately reporting a memory close to what’s being physically used. My current value is Memory: 2.6G, which is far lower than the virtual memory but slightly higher than the RSS of the separated PIDs summed up. What’s it sit at for yours vs. the results of systemctl show? It seems like you are actually running out of memory…

[SandsOfArrakis (The Spice Must Flow)]

I’m running Zorin Linux Pro on a Lenovo Ideapad S530. It has 16 gb of RAM built in. After a fresh reboot RAM usage is around 3 gb. But it creeps up to 13.2 gb after awhile. Whether I’m using Roon or just idling with RoonServer running.

Eventually I will have to reboot as the Roon Core becomes pretty unstable. Information pages like artist views etc. no longer opening. ARC failing to play with “something went wrong” error messages. Etc.

After a reboot of the laptop everything works again as it should.

[ranel]

7k tracks? what audio devices? what remotes? core on the ThinkPad?

Yes, about 7k tracks. The crash shown in the first post happened with no audio devices connected to the core, no remotes, no ARC, and no recent additions to the library. I disconnected everything, removed all extensions, rebooted the machine and waited. So that’s core running on an i5-8350U ThinkPad connected to WiFi and nothing else. And my home network is rather quiet, there shouldn’t be many devices coming and going.

Let me know if we can help to get this reproduced or better documented. I tried to run core under the massif profiler to see how much memory is really used and where does it go, but that didn’t immediately work. I could try harder though.

In the meantime I’ve been using the MemoryMax trick and systemd restarted my core twice because of it. It’s too early for conclusions, but I didn’t have problems with ARC connections (that’s mainly how I use Roon on workdays) since.

[Bozon]

I have Ubuntu 22.04.02 on a PC I built with a Ryzen 5 4600G and 16 GB of RAM.

I have exactly the same issues described here. Roon Server starts to consume memory to the point Arc is unable to reproduce anything when I am on the go.

The memory usage keeps growing to the point the Roon Server depleates the server memory and gets unusable. It lasts about a week and I have to restart the system.

On my case Roon core reaches about 9.9 GB.

I will try the suggested memory limit workaround in order to the service to auto-restart when the leak reaches unacceptable levels.

[Bozon]

It has been 11 days since I applied the MemoryMax setting on my system and Roon has been working fine all this time. Memory consumption never surpases 1.95 GB now and the service has been stable.
It seems that when that limit is active, Roon detects the 6GB as the maximum RAM available and stops allocating memory. The service does not crashes and restart as @danny said. I have been monitoring it and I see the same PID everyday.

[ranel]

Yes, I’ve seen this too; With MemoryMax set, the memory usage would sometimes take off and stop just under the limit, hovering there for pretty long. In my case though, the core was already dysfunctional when hovering under the limit: ARC would not connect, and I had to manually restart the core to get it working again. Maybe setting a rather tight limit (like 2x the normal memory usage or so) could help with this.

I’m saying all that in the past tense, because I have bought a 400€ Intel NUC, moved my core to it, and didn’t have a single issue since.