Roon Server High CPU on Ubuntu

[Mitch_AU (Mitch)]

Roon Core Machine

Linux Ubuntu 20.04.3 LTS
Intel(R) Core™ i5-7260U CPU @ 2.20GHz
16GB RAM

Networking Gear & Setup Details

Gigabit ethernet network. Gigabit internet connection. pfSense router.

Connected Audio Devices

5 Sonos end-points, 1 Ropieee (RPi2 with Hifiberry Digi+)
Other found but not enabled (smart TV’s etc)

Number of Tracks in Library

Approx 10k tracks located on NAS.

Description of Issue

I have been running Roon server on my Linux box without issue for several years and recently (last few months) started to have issue where roonserver service is consistently using 200-300% CPU usage. I have tried leaving it to complete any process and loads of restarts but the issue prevails.

Core and remotes all on latest 1.8 (831) version.

When I look at journalctl -u roonserver -f I can see some corrupt jpeg files but not sure this is causing the issue.

Sep 29 07:57:35 dockerhub start.sh[3985171]: aac_fixed decoder found, checking libavcodec version...
Sep 29 07:57:35 dockerhub start.sh[3985171]: has mp3float: 1, aac_fixed: 1
Sep 29 07:57:35 dockerhub start.sh[3464805]: Not responding
Sep 29 07:57:40 dockerhub start.sh[3464805]: Running
Sep 29 07:57:53 dockerhub start.sh[3985171]: Corrupt JPEG data: premature end of data segment
Sep 29 07:57:54 dockerhub start.sh[3985171]: Corrupt JPEG data: premature end of data segment
Sep 29 07:57:54 dockerhub start.sh[3985171]: Corrupt JPEG data: premature end of data segment
Sep 29 07:57:54 dockerhub start.sh[3985171]: Corrupt JPEG data: premature end of data segment
Sep 29 07:57:54 dockerhub start.sh[3985171]: Corrupt JPEG data: premature end of data segment

I have turned Background Audio Analysis off in the hope of fixing the problem but this hasn’t helped.

I am not sure if there is something here in the logs of interest:
cat /var/roon/RoonServer/Logs/RoonServer_log.txt|grep Error:

09/29 16:15:36 Error: [broker/images] addimage failed: System.Exception: failed to decode jpeg
09/29 16:15:36 Error: [broker/images] addimage failed: System.Exception: failed to decode jpeg
09/29 16:15:36 Error: [broker/images] addimage failed: System.Exception: failed to decode jpeg
09/29 16:15:36 Error: [broker/images] addimage failed: System.Exception: failed to decode jpeg
09/29 16:16:45 Error: [cast/client] [BRAVIA-4K-GB-ATV3-630f8b9a9a406da866c7729f8f4eb23a._googlecast._tcp.local] Exception writing message to stream: 
09/29 16:16:45 Error: [cast/client] [SHIELD-Android-TV-45aa02df370a484b979610d4dfa5b342._googlecast._tcp.local] Exception writing message to stream: 
09/29 16:17:15 Error: [cast/client] [SHIELD-Android-TV-13b1ca8563faf4ac6976a2ce056f27e3._googlecast._tcp.local] Exception writing message to stream: 
09/29 16:17:45 Error: [cast/client] [SHIELD-Android-TV-45aa02df370a484b979610d4dfa5b342._googlecast._tcp.local] Exception writing message to stream: 
09/29 16:17:45 Error: [cast/client] [BRAVIA-4K-GB-ATV3-630f8b9a9a406da866c7729f8f4eb23a._googlecast._tcp.local] Exception writing message to stream: 
09/29 16:17:45 Error: [cast/client] [SHIELD-Android-TV-13b1ca8563faf4ac6976a2ce056f27e3._googlecast._tcp.local] Exception writing message to stream: 
09/29 16:18:16 Error: [cast/client] [SHIELD-Android-TV-45aa02df370a484b979610d4dfa5b342._googlecast._tcp.local] Exception writing message to stream: 
09/29 16:18:16 Error: [cast/client] [SHIELD-Android-TV-13b1ca8563faf4ac6976a2ce056f27e3._googlecast._tcp.local] Exception writing message to stream: 
09/29 16:18:46 Error: [cast/client] [BRAVIA-4K-GB-ATV3-630f8b9a9a406da866c7729f8f4eb23a._googlecast._tcp.local] Exception writing message to stream: 
09/29 16:18:46 Error: [cast/client] [SHIELD-Android-TV-45aa02df370a484b979610d4dfa5b342._googlecast._tcp.local] Exception writing message to stream: 
09/29 16:18:46 Error: [cast/client] [SHIELD-Android-TV-13b1ca8563faf4ac6976a2ce056f27e3._googlecast._tcp.local] Exception writing message to stream: 
09/29 16:19:14 Error: [cast/client] [SHIELD-Android-TV-13b1ca8563faf4ac6976a2ce056f27e3._googlecast._tcp.local] Exception writing message to stream: 
09/29 16:19:14 Error: [cast/client] [SHIELD-Android-TV-45aa02df370a484b979610d4dfa5b342._googlecast._tcp.local] Exception writing message to stream: 
09/29 16:19:15 Error: [cast/client] [BRAVIA-4K-GB-ATV3-630f8b9a9a406da866c7729f8f4eb23a._googlecast._tcp.local] Exception writing message to stream: 
09/29 16:19:47 Error: [cast/client] [SHIELD-Android-TV-45aa02df370a484b979610d4dfa5b342._googlecast._tcp.local] Exception writing message to stream: 
09/29 16:19:47 Error: [cast/client] [SHIELD-Android-TV-13b1ca8563faf4ac6976a2ce056f27e3._googlecast._tcp.local] Exception writing message to stream: 
09/29 16:19:47 Error: [cast/client] [BRAVIA-4K-GB-ATV3-630f8b9a9a406da866c7729f8f4eb23a._googlecast._tcp.local] Exception writing message to stream: 
09/29 16:20:18 Error: [cast/client] [SHIELD-Android-TV-45aa02df370a484b979610d4dfa5b342._googlecast._tcp.local] Exception writing message to stream: 
09/29 16:20:18 Error: [cast/client] [SHIELD-Android-TV-13b1ca8563faf4ac6976a2ce056f27e3._googlecast._tcp.local] Exception writing message to stream: 
09/29 16:20:18 Error: [cast/client] [BRAVIA-4K-GB-ATV3-630f8b9a9a406da866c7729f8f4eb23a._googlecast._tcp.local] Exception writing message to stream: 
09/29 16:21:44 Error: [cast/client] [SHIELD-Android-TV-45aa02df370a484b979610d4dfa5b342._googlecast._tcp.local] Exception writing message to stream: 
09/29 16:21:44 Error: [cast/client] [SHIELD-Android-TV-13b1ca8563faf4ac6976a2ce056f27e3._googlecast._tcp.local] Exception writing message to stream: 
09/29 16:21:45 Error: [cast/client] [BRAVIA-4K-GB-ATV3-630f8b9a9a406da866c7729f8f4eb23a._googlecast._tcp.local] Exception writing message to stream: 

Any help faultfinding would be appreciated.
Thanks.

[beka]

Hello @Mitch_AU and welcome to the Roon community

We’re sorry to hear that after years of running Roon with virtually no issues, you’ve run into this high CPU usage. Thanks for sharing this and the steps you’ve taken to improve the situation.

Dylan, one of our technicians, explained this best here:

Memory leak in Roon?! [Solved: Update to Build 970]

Hi everyone, Our team has been tracking these reports, both here and in other places on community. I wanted to provide an update on where these issues stand. We typically see performance issues like this happening on macOS or Linux setups. Windows and RoonOS work a bit different under the hood and so those platforms tend to perform much better in this area. The good news is there is work underway to improve some low-level architecture on macOS and Linux that we believe will be a big improvemen…

While it’s not the ultimate solution, I hope it helps provide some context

[Mitch_AU (Mitch)]

Thanks for getting back to me with a response. I am not sure that I understand what you are telling me. I am having CPU utilisation issues and you linked a comment about memory leaks. Does this mean that you believe that there is also a memory leak going on that is causing a high CPU usage?

Also, assuming this is correct your recommendation to resolve the issue is to install Roon on a Windows box rather than using Linux OS?

[Andreas_Philipp1]

FWIW, I run Roon Server on Ubuntu 20.04.3 and on a machine configured similar to yours. While I certainly am experiencing the memory leak problem, I never have seen Roon with this kind of CPU utilization. Both problems don’t seem to be related.

[Mitch_AU (Mitch)]

I just had a look at free and got

Mem:       16279620     4175444     6999108       46148     5105068    12227420
Swap:       4194300     3016100     1178200

ps aux --sort=-%mem | head|grep RoonServer
root 741623 78.6 3.7 2917908 617804 ? SLl 08:43 4:47 /opt/RoonServer/RoonMono/bin/RoonAppliance --debug --gc=sgen --server RoonAppliance.exe -watchdogport=45123
Roonserver is using 3.7% of memory and 78% of CPU.

Which seems to indicate there is loads of available memory yet the CPU utilisation is 200%+. Are there other thing to look at?

[Mitch_AU (Mitch)]

@beka was there anything further you can add on this problem considering the additional information above?

[noris]

Hello @Mitch_AU ,

Thank you for providing those details!

Can you please use these instructions to access your Roon logs and upload a set for review? You can send this to me via private message if you wish for it to be more private in nature (click my name → message).

@Mitch_AU

I am not sure if there is something here in the logs of interest:
cat /var/roon/RoonServer/Logs/RoonServer_log.txt|grep Error:

This is interesting, these all appear to be cast errors, multiple times per minute. Can you please confirm, with these cast endpoints disabled in Roon or physically unplugged, do you still experience the same issue?

[Mitch_AU (Mitch)]

Interestingly, I unplugged (no power) the TV’s and media boxes and restarted roonserver. Everything seemed to start working properly with much lower (normal) CPU utilisation. However, a few hours later the roon clients became unresponsive and noticed that the high CPU utilisation had returned.

At first the high CPU thread was RemotingServer at exactly 100%

Correspondingly the following error was in the logs:

10/02 12:53:52 Error: Warning: Work item is being delayed because thread pool has reached max (100) threads @ThreadUtil.QueueUserWorkItem
10/02 12:53:52 Error: Warning: Work item is being delayed because thread pool has reached max (100) threads @ThreadUtil.QueueUserWorkItem
10/02 12:53:52 Error: Warning: Work item is being delayed because thread pool has reached max (100) threads @ThreadUtil.QueueUserWorkItem
10/02 12:53:52 Error: Warning: Work item is being delayed because thread pool has reached max (100) threads @ThreadUtil.QueueUserWorkItem
10/02 12:53:52 Error: Warning: Work item is being delayed because thread pool has reached max (100) threads @ThreadUtil.QueueUserWorkItem
10/02 12:53:52 Error: Warning: Work item is being delayed because thread pool has reached max (100) threads @ThreadUtil.QueueUserWorkItem
10/02 12:53:52 Error: Warning: Work item is being delayed because thread pool has reached max (100) threads @ThreadUtil.QueueUserWorkItem
10/02 12:53:52 Error: Warning: Work item is being delayed because thread pool has reached max (100) threads @ThreadUtil.QueueUserWorkItem
10/02 12:53:52 Error: Warning: Work item is being delayed because thread pool has reached max (100) threads @ThreadUtil.QueueUserWorkItem

Soon after RoonServer process came back at 200-300% CPU utilisation
I have messaged @noris separately with the complete logs.

I look forward to any advice you can provide. Thanks.

[noris]

Hello @Mitch_AU ,

Thank you for submitting the log files and additional details, we have forwarded these to QA for closer inspection. Once we receive the completed report from QA, we will let you know, thank you!

[beka]

Hey @Mitch_AU,

I wanted to say how grateful we are for your help in understanding this issue and investigating it

While Noris is out of office, I wanted to check in with you and see if the issue still occurs these days?