Updated on 4/22/2024
Not sure if this is a solution or just a workaround but I have been able to stop the reachability events without any adverse effect on Roon. Hopefully, RoonServer will be more stable now but that’s TBD. Here’s a copy of the latest post so it isn’t necessary to scroll to the bottom of this thread to get to the punchline:
I’ve been spending some time looking through the Network Manager logs in trace mode and finally come to the conclusion that it is Network Manager that is the source of the erroneous reachability changed messages. To test this, I disabled Network Manager completely while leaving the static IP configuration unchanged (i.e. manually configured static IP, manually configured DNS, etc.)
The result was the complete elimination of the reachability events described in my initial post!
So far, I have observed no ill effects on Roon or anything else as a result of this change. However, I have not tested long enough in this configuration to know if the occasional RoonServer crashes are also gone. And network traffic between my server and Roonlabs looks like it may be reduced by close to 99% based on the elimination of the 1 per minute re-authorization exchanges that have been happening (I haven’t directly snooped the network to verify that but it seems likely given what I’m seeing in the logs).
As for what exactly is going wrong in Network Manager I can’t say for sure. Symptomatically, NM appears to adjust the routing table entry associated with my wired interface by dramatically increasing the routing metric. Then, it changes it back to its original reasonable value moments later. But the routing table change is associated with a network change event that must be filtering down to RoonServer. It’s not clear if the metric change is a cause of the problem or just the first visible manifestation of some other change that isn’t reported in the logs. I’m not really motivated enough to dive into the Network Manager source code to track it further at this point.
I’ve changed the title of this thread to be more specific about the issues I’ve encountered. I’m also correcting an error in the original post related to the rate at which the “reachability changed” messages occur (thanks to @Andreas_Philipp1 for catching that!), I’ll also be updating with actions taken since the original posting and the current state of play. In all likelihood these edits will not occur all at once as I have limited time available to post. – Jeff
I’ve read the various threads and elsewhere about running RoonServer on Ubuntu and some conclude happily and others never find a solution other than trying a Fedora or some other Linux. I’m familiar with a variety of *nix systems but, oddly not Linux. So it is very possible that the anomalies I am seeing are due to installation or configuration errors.
I’m posting here first to see if I’ve made any obvious errors or if the issues I’m seeing are well-known and solvable. If that doesn’t yield a solution I’ll put a ticket in for Roon support. Ok so let’s start with the basic environment:
Server Hardware
Previously I ran RoonServer on a Synology 916+ with DSM7.2. However, my Macbook Pro 15,2 developed a broken screen and I decided to repurpose it as a RoonServer. Roon was running on well on the Synology but as I started to use more DSP functions to tune my headphones I could see the load creeping up substantially. I thought the core i7 in the macbook would be great for Roon.
I like MacOS but I do not trust it as 24/7 server OS. Too many software updates requiring reboots for example. After reading the official Roon comments on Linux I chose to install Ubuntu. I may have chosen…poorly.
Updated 4/12/24:
System:
Host: jeffsubuntuserver Kernel: 6.6.25-t2-jammy x86_64 bits: 64 Console: pty pts/3
Distro: Ubuntu 22.04.4 LTS (Jammy Jellyfish)
Machine:
Type: Laptop System: Apple product: MacBookPro15,2 v: 1.0 serial: <superuser required>
Mobo: Apple model: Mac-827FB448E656EC26 v: MacBookPro15,2 serial: <superuser required>
UEFI: Apple v: 2022.100.22.0.0 (iBridge: 21.16.4222.0.0,0) date: 02/14/2024
Battery:
ID-1: BAT0 charge: 46.5 Wh (95.3%) condition: 48.8/58.0 Wh (84.1%)
CPU:
Info: quad core Intel Core i7-8569U [MT MCP] speed (MHz): avg: 475 min/max: 400/4700
Graphics:
Device-1: Intel CoffeeLake-U GT3e [Iris Plus Graphics 655] driver: i915 v: kernel
Display: server: X.org v: 1.21.1.4 with: Xwayland v: 22.1.1 driver: X: loaded: modesetting
unloaded: fbdev,vesa gpu: i915 tty: 214x27 resolution: 1680x1050
Message: GL data unavailable in console. Try -G --display
Network:
Device-1: Broadcom BCM4364 802.11ac Wireless Network Adapter driver: brcmfmac
Device-2: ASIX AX88179 Gigabit Ethernet type: USB driver: cdc_ncm
Device-3: Realtek RTL8153 Gigabit Ethernet Adapter type: USB driver: r8152
Drives:
Local Storage: total: 931.84 GiB used: 17.22 GiB (1.8%)
Info:
Processes: 326 Uptime: 2d 3h 1m Memory: 15.48 GiB used: 2.74 GiB (17.7%) Init: systemd
runlevel: 5 Shell: Bash inxi: 3.3.13
I installed Ubuntu via the Roon easy install script and that went fine. Check.sh runs successfully. And this system does play music. But it also crashes a couple times a day and restarts which is not good.
A few other fun facts:
- No music plays directly on the server
** i.e. the server is not a zone and I don’t rely on linux audio drivers - WiFi is disabled
- My initial network setup had the wired adapter connected to my Cisco SG 200-08 8-port switch. Because Roon doesn’t recommend smart switches I have since connected the 1gb ethernet adapter directly to one of the 1Gb ports on my Orbi RBR50 router.
- Over the last few days I have also replaced all ethernet cables in my system with new cat6 cables. I had a few older cat 5e cables in the mix and these were removed. In an abundance of caution I also replaced the existing cat 6 cables with new ones as well.
- There are 2 wired ethernet adapters available. One uses a RealTek 8153 and the other uses an ASIX AX88179. Currently the ASIX adapter is connected to the network. The reachability issues occur with both the RealTek and ASIX adapters.
Network:
Device-1: Broadcom BCM4364 802.11ac Wireless Network Adapter driver: brcmfmac
IF: wlan0 state: down mac: a4:83:e7:8b:d6:89
Device-2: ASIX AX88179 Gigabit Ethernet type: USB driver: cdc_ncm
IF: enxf8e43bb75797 state: up speed: 1000 Mbps duplex: half mac: f8:e4:3b:b7:57:97
Device-3: Realtek RTL8153 Gigabit Ethernet Adapter type: USB driver: r8152
IF: enx00e04c6814bb state: down mac: 00:e0:4c:68:14:bb
- The server now has a fixed ip address rather than relying on DHCP
- There are 3 zones in use:
** A Bluesound Node 2 connected to the server through the cisco switch via 1Gb ethernet
** An NAD T777v3 connected via ethernet to a Netgear Orbi satellite (RBS 50)
** A WiiM Pro Plus connected via WiFi
All these zones use RAAT as transport. There are also 3 Homepod Minis that use Airplay but I have not attempted to play music in those zones since i moved RoonServer to Ubuntu.
Music files reside on the Synology DS916+ which is connected to the RoonServer via 1Gb ethernet through a Cisco SG200-08 Gb switch. My library is small by Roon community standards with about 9150 tracks. Tidal and Sirius XM are connected services. Roon ARC works correctly and I can access it from outside my network.
Ubuntu Anomalies
For now, I’ll concentrate on just three of them:
Invalid Atom
● roonserver.service - RoonServer
Loaded: loaded (/etc/systemd/system/roonserver.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2024-03-30 20:26:00 EDT; 5 days ago
Main PID: 2040 (start.sh)
Tasks: 112 (limit: 18931)
Memory: 3.1G
CPU: 11h 50min 18.652s
CGroup: /system.slice/roonserver.service
├─ 2040 /bin/bash /opt/RoonServer/start.sh
├─264986 /opt/RoonServer/RoonDotnet/RoonServer RoonServer.dll
├─411997 /opt/RoonServer/RoonDotnet/RoonAppliance RoonAppliance.dll -watchdogport=35917
├─411998 /opt/RoonServer/Server/processreaper 411997
└─412043 /opt/RoonServer/RoonDotnet/RAATServer RAATServer.dll
Apr 02 23:38:00 jeffsubuntuserver start.sh[265022]: has mp3float: 1, aac_fixed: 1
Apr 02 23:38:04 jeffsubuntuserver start.sh[264986]: Running
Apr 04 19:16:08 jeffsubuntuserver start.sh[265022]: atom with <8 bytes is invalidatom with <8 bytes is invalidatom wit>
Apr 04 19:16:14 jeffsubuntuserver start.sh[264986]: Not responding
Apr 04 19:16:44 jeffsubuntuserver start.sh[264986]: Error
Apr 04 19:16:46 jeffsubuntuserver start.sh[264986]: Initializing
Apr 04 19:16:46 jeffsubuntuserver start.sh[264986]: Started
Apr 04 19:16:47 jeffsubuntuserver start.sh[411997]: aac_fixed decoder found, checking libavcodec version...
Apr 04 19:16:47 jeffsubuntuserver start.sh[411997]: has mp3float: 1, aac_fixed: 1
Apr 04 19:16:49 jeffsubuntuserver start.sh[264986]: Running
Notice the 3rd start.sh entry: “atom with <8 bytes is invalid” I have no idea what this means or what causes it or if it’s even a problem.But I see this message frequently when RoonServer needs to restart.
Dotnet failures
/var/crash
contains this file:
_opt_RoonServer_RoonDotnet_dotnet.0.crash
The first section of the crash log is human readable:
ProblemType: Crash
Architecture: amd64
CrashCounter: 1
Date: Thu Apr 4 19:16:08 2024
DistroRelease: Ubuntu 22.04
ExecutablePath: /opt/RoonServer/RoonDotnet/dotnet
ExecutableTimestamp: 1712062388
ProcCmdline: /opt/RoonServer/RoonDotnet/RoonAppliance RoonAppliance.dll -watchdogport=35917
ProcCwd: /opt/RoonServer/Appliance
ProcEnviron:
SHELL=/bin/sh
LANGUAGE=en_CA:en
LANG=en_CA.UTF-8
PATH=(custom, no user)
ProcMaps:
55e58a970000-55e58a974000 r--p 00000000 103:03 1969222 /opt/RoonServer/RoonDotnet/dotnet
55e58a974000-55e58a98a000 r-xp 00004000 103:03 1969222 /opt/RoonServer/RoonDotnet/dotnet
55e58a98a000-55e58a992000 r--p 0001a000 103:03 1969222 /opt/RoonServer/RoonDotnet/dotnet
55e58a992000-55e58a993000 r--p 00021000 103:03 1969222 /opt/RoonServer/RoonDotnet/dotnet
55e58a993000-55e58a994000 rw-p 00022000 103:03 1969222 /opt/RoonServer/RoonDotnet/dotnet
etc. It goes on for a while with standard dump info followed by many megabytes of hex dump. I must say that these dotnet crashes are less frequent after the lastest update but they still happen. I have no idea how to track this further.
The Great Reachability Heartbeat
Now for the strangest one of all. I looked at RoonServer logs file over a 3 day period and noticed many, many errors of the type Trace:` [remoting/brokerserver] network reachability changed. Kicking off discovery cycle.’ This is followed by a flurry of network activity that pings every zone and then communicates authorizataion info back to roon labs via api. Over that 3 more than 3 day span I counted the number of “reachabilty” entries in the logs:
updated 4/13/2024
Over the entire span of almost 4 days RoonServer was repeating this discovery process at 60Hz! That just cannot be right! So I looked at the RAATServer logs which actually grow much more slowly than the RoonServer logs. For those I was able to collect data over a longer period of time:
I’m going to end this here because this post is long enough and I’m sure I haven’t included everything that folks may want to see. So if you have any ideas on what could be happening here I would be highly appreciative!
addendum 4/13/2024:
In spite of the actions outlined above, namely:
- Tried.2 different ethernet adapters with different chipsets and different drivers
- Recabling my entire system to ensure new cat6 cables are in place everywhere
- Having the wired connection from my server bypass the smart switch and and connect directly to the Orbi router integrated switch
- Replacing DHCP with a fixed IP address
The results are basically unchanged. I still get frequent reachability events at about a rate 1/minute. Roon mostly ‘works’ despite this – music plays well in all zones and ARC works as expected. However I do suffer from Roon restarts that occur approximately 1/day.