Roon Server unresponsive - Daily crashes

Nigelc · March 22, 2023, 9:51am

Roon Core Machine

OS: Ubuntu 20.04.6 LTS
Model Lenovo x230
CPU: Intel(R) Core™ i5-3320M CPU @ 2.60GHz
Memory: 8GB
NAS: QNAP

Networking Gear & Setup Details

Ethernet connected via Mikrotik switches and router.

Connected Audio Devices

Chromecast and airplay devices
RoPieee

Number of Tracks in Library

about 15k tracks

Description of Issue

The core was running fine a few days.
Then this morning the App seems to be very slow and then randomly saying “Select Different Core”
Was chewing CPU and memory

After 20 minutes is seemed to kill itself!
Is ok for a while and then slowly start chewing recourses again!
$ systemctl status roonserver
…
Mar 22 09:33:34 roon-server-2 start.sh[9642]: 05 days, 02:16:42.201 Debug: RoonServer, after RoonAppliance exit, exitcode: 137
Mar 22 09:33:34 roon-server-2 start.sh[9642]: Error
Mar 22 09:33:36 roon-server-2 start.sh[9642]: Initializing
Mar 22 09:33:36 roon-server-2 start.sh[9642]: 05 days, 02:16:44.303 Debug: RoonServer, before attempting to start RoonAppliance binary at path: /opt/RoonServer/Server/…/Appliance/RoonAppliance
Mar 22 09:33:36 roon-server-2 start.sh[9642]: 05 days, 02:16:44.377 Debug: RoonServer, after starting RoonAppliance
Mar 22 09:33:36 roon-server-2 start.sh[9642]: Started
Mar 22 09:33:39 roon-server-2 start.sh[704963]: aac_fixed decoder found, checking libavcodec version…
Mar 22 09:33:39 roon-server-2 start.sh[9642]: Not responding
Mar 22 09:33:39 roon-server-2 start.sh[704963]: has mp3float: 1, aac_fixed: 1
Mar 22 09:33:44 roon-server-2 start.sh[9642]: Running

---- A while later ----

…$ systemctl status roonserver
● roonserver.service - RoonServer
Loaded: loaded (/etc/systemd/system/roonserver.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2023-03-17 06:50:43 UTC; 5 days ago
Main PID: 846 (start.sh)
Tasks: 126 (limit: 9079)
Memory: 7.0G
CGroup: /system.slice/roonserver.service
├─ 846 /bin/bash /opt/RoonServer/start.sh
├─ 9642 /opt/RoonServer/RoonDotnet/RoonServer RoonServer.dll
├─713528 /opt/RoonServer/RoonDotnet/RoonAppliance RoonAppliance.dll -watchdogport=35459
├─713529 /opt/RoonServer/Server/processreaper 713528
└─713578 /opt/RoonServer/RoonDotnet/RAATServer RAATServer.dll

Mar 22 10:47:30 roon-server-2 start.sh[9642]: 05 days, 03:30:37.915 Debug: RoonServer, after RoonAppliance exit, exitcode: 137
Mar 22 10:47:30 roon-server-2 start.sh[9642]: Error
Mar 22 10:47:32 roon-server-2 start.sh[9642]: Initializing
Mar 22 10:47:32 roon-server-2 start.sh[9642]: 05 days, 03:30:39.941 Debug: RoonServer, before attempting to start RoonAppliance binary at path: /opt/RoonServer/Server/…/Appliance/RoonAppliance
Mar 22 10:47:32 roon-server-2 start.sh[9642]: 05 days, 03:30:39.977 Debug: RoonServer, after starting RoonAppliance
Mar 22 10:47:32 roon-server-2 start.sh[9642]: Started
Mar 22 10:47:34 roon-server-2 start.sh[713528]: aac_fixed decoder found, checking libavcodec version…
Mar 22 10:47:34 roon-server-2 start.sh[713528]: has mp3float: 1, aac_fixed: 1
Mar 22 10:47:35 roon-server-2 start.sh[9642]: Not responding
Mar 22 10:47:40 roon-server-2 start.sh[9642]: Running

After my previous support ticket…
I built a new core, on a physical machine and imported previous backup.

Any ideas?
When it is up and running properly I had no issues!

Nigelc · March 27, 2023, 4:03pm

And again…
High memory usage… Roon then kills itself

$ systemctl status roonserver
● roonserver.service - RoonServer
Loaded: loaded (/etc/systemd/system/roonserver.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2023-03-17 06:50:43 UTC; 1 weeks 3 days ago
Main PID: 846 (start.sh)
Tasks: 159 (limit: 9079)
Memory: 2.3G
CGroup: /system.slice/roonserver.service
├─ 846 /bin/bash /opt/RoonServer/start.sh
├─ 9642 /opt/RoonServer/RoonDotnet/RoonServer RoonServer.dll
├─1366868 /opt/RoonServer/RoonDotnet/RoonAppliance RoonAppliance.dll -watchdogport=35459
├─1366869 /opt/RoonServer/Server/processreaper 1366868
└─1366920 /opt/RoonServer/RoonDotnet/RAATServer RAATServer.dll

Mar 27 15:17:04 roon-server-2 start.sh[9642]: 10 days, 08:00:11.520 Debug: RoonServer, after RoonAppliance exit, exitcode: 137
Mar 27 15:17:04 roon-server-2 start.sh[9642]: Error
Mar 27 15:17:06 roon-server-2 start.sh[9642]: Initializing
Mar 27 15:17:06 roon-server-2 start.sh[9642]: 10 days, 08:00:13.573 Debug: RoonServer, before attempting to start RoonAppliance binary at path: /opt/RoonServer/Server/…/Appliance/RoonAppliance
Mar 27 15:17:06 roon-server-2 start.sh[9642]: 10 days, 08:00:13.627 Debug: RoonServer, after starting RoonAppliance
Mar 27 15:17:06 roon-server-2 start.sh[9642]: Started
Mar 27 15:17:08 roon-server-2 start.sh[9642]: Not responding
Mar 27 15:17:08 roon-server-2 start.sh[1366868]: aac_fixed decoder found, checking libavcodec version…
Mar 27 15:17:08 roon-server-2 start.sh[1366868]: has mp3float: 1, aac_fixed: 1
Mar 27 15:17:13 roon-server-2 start.sh[9642]: Running

As you can see from these screenshots
Starting to creep up

Then peaks

Then takes a while to die and restart

Any ideas?

Nigelc · March 31, 2023, 1:17pm

Build 1244
Same issue

$ systemctl status roonserver
● roonserver.service - RoonServer
     Loaded: loaded (/etc/systemd/system/roonserver.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2023-03-17 06:50:43 UTC; 2 weeks 0 days ago
   Main PID: 846 (start.sh)
      Tasks: 126 (limit: 9079)
     Memory: 2.2G
     CGroup: /system.slice/roonserver.service
             ├─    846 /bin/bash /opt/RoonServer/start.sh
             ├─1908506 /opt/RoonServer/RoonDotnet/RoonServer RoonServer.dll
             ├─1938907 /opt/RoonServer/RoonDotnet/RoonAppliance RoonAppliance.dll -watchdogport=43445
             ├─1938908 /opt/RoonServer/Server/processreaper 1938907
             └─1938954 /opt/RoonServer/RoonDotnet/RAATServer RAATServer.dll

Mar 31 12:25:33 roon-server-2 start.sh[1908506]: Not responding
Mar 31 12:26:05 roon-server-2 start.sh[1908506]: 04:04:26.210 Debug: RoonServer, after RoonAppliance exit, exitcode: 137
Mar 31 12:26:05 roon-server-2 start.sh[1908506]: Error
Mar 31 12:26:07 roon-server-2 start.sh[1908506]: Initializing
Mar 31 12:26:07 roon-server-2 start.sh[1908506]: 04:04:28.236 Debug: RoonServer, before attempting to start RoonAppliance binary at path: /opt/RoonServer/Server/../Appliance/RoonAppliance
Mar 31 12:26:07 roon-server-2 start.sh[1908506]: 04:04:28.263 Debug: RoonServer, after starting RoonAppliance
Mar 31 12:26:07 roon-server-2 start.sh[1908506]: Started
Mar 31 12:26:09 roon-server-2 start.sh[1938907]: aac_fixed decoder found, checking libavcodec version...
Mar 31 12:26:09 roon-server-2 start.sh[1938907]: has mp3float: 1, aac_fixed: 1
Mar 31 12:26:10 roon-server-2 start.sh[1908506]: Running

Nigelc · April 1, 2023, 11:06am

At least once a day now.

Nigelc · April 3, 2023, 5:56pm

And again!

Am I on some sort of support blacklist?

I know you guys are busy… But you are still answering new support questions, just not mine!

connor · April 11, 2023, 6:55pm

Hi @Nigelc,

Thank you for your patience as the team worked through an unexpected uptick in requests to reach your report.

First off, have you recently refreshed RoonServer? If not, start there:

Make a Backup of your current RoonServer Database
Exit out of RoonServer
Navigate to your RoonServer’s Database Location
Find the folder that says “RoonServer”
Rename the “RoonServer” folder to “RoonServer_old”
Reinstall the RoonServer App from our Downloads Page to generate a new RoonServer folder
On the Roon Remotes, press “Use another Core” and connect to the new database

How are your switches configured? Diagnostics indicate your Core is receiving unexpected or incomplete AIrplay discovery requests repeatedly. Are your Airplay devices on a different switch than the Core? Is the Core connected to the main router?

Let us know how things perform after refreshing the database and we’ll work from there. Thank you!

Nigelc · April 11, 2023, 7:27pm

Hi @connor

Yes, fresh install recently and imported the backup from the old system. I was getting the same issue on a different system. Hence the new server.

My network is flat. Class C. No vlans or additional routing. But some Airplay devices will be on different switches.

I don’t really use airplay devices. So I disabled them as Roon endpoints. Are you able to identify which device/s it is? I did see this in the logs, but was unable to match it to an address or device?

Without knowing which airplay device, it’s hard to say if the Core is on the same switch or not

Cheers,
Nigel

Nigelc · May 2, 2023, 9:02am

Still doing it!

Nigelc · May 4, 2023, 5:34pm

I no longer see the airplay requests in the logs (maybe the last update?).

But it still keeps glitching!
It is pretty unusable for listening for than 30 minutes!

$ systemctl status roonserver
● roonserver.service - RoonServer
     Loaded: loaded (/etc/systemd/system/roonserver.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-04-26 07:47:30 UTC; 1 weeks 1 days ago
   Main PID: 920 (start.sh)
      Tasks: 150 (limit: 9079)
     Memory: 7.0G
     CGroup: /system.slice/roonserver.service
             ├─   920 /bin/bash /opt/RoonServer/start.sh
             ├─   953 /opt/RoonServer/RoonDotnet/RoonServer RoonServer.dll
             ├─485044 /opt/RoonServer/RoonDotnet/RoonAppliance RoonAppliance.dll -watchdogport=43609
             ├─485045 /opt/RoonServer/Server/processreaper 485044
             └─485094 /opt/RoonServer/RoonDotnet/RAATServer RAATServer.dll

May 04 11:12:47 roon-server-2 start.sh[953]: Not responding
May 04 11:12:52 roon-server-2 start.sh[953]: Running
May 04 12:57:47 roon-server-2 start.sh[953]: Not responding
May 04 12:57:52 roon-server-2 start.sh[953]: Running
May 04 13:27:50 roon-server-2 start.sh[953]: Not responding
May 04 13:27:56 roon-server-2 start.sh[953]: Running
May 04 15:12:47 roon-server-2 start.sh[953]: Not responding
May 04 15:12:52 roon-server-2 start.sh[953]: Running
May 04 16:42:45 roon-server-2 start.sh[953]: Not responding
May 04 16:42:50 roon-server-2 start.sh[953]: Running
$ free -h
              total        used        free      shared  buff/cache   available
Mem:          7.5Gi       7.1Gi       121Mi       0.0Ki       304Mi       137Mi
Swap:         4.0Gi       1.0Gi       3.0Gi

Nigelc · May 21, 2023, 5:06pm

Ugh I give up with “support” here!

@Bozon Did you ever get it fixed?

Nigelc · June 20, 2023, 6:20am

I worked around the problem. So not using Ubuntu anymore.
Still a little underwhelmed by the support responses.

system · August 26, 2023, 12:19am

This topic was automatically closed 45 days after the last reply. New replies are no longer allowed.

Stephen_Howe · December 3, 2023, 4:55am

was this fixed by using another OS tempted to install on my windows server instead of ubuntu to see if that fixes it

Nigelc · December 10, 2023, 10:51am

Yes I used a Mock Rock install. Been fine ever since
@Stephen_Howe