Hi, have you experienced more crashes of Roon? I am now going near 48 hours and it’s been very stable.
Here is a nice example of how to configure the ethernet interface to be managed by networkd instead of NetworkManager on Ubuntu Desktop. If you haven’t done so, you could try this and see if you can get rid of the problem while Roon figures out how to make Roon live in peace with Ubuntu’s NetworkManager - if this in the end is the cause of the problem we’re experimenting.
@Andreas_Philipp1 I had a 36 hour period without a fault being logged. Then I was listening last night and at about 8:20 the music cut. I am going to implement your suggestion this evening.
I wish you good luck! Once you have reconfigured your ethernet connection to be managed by networkd instead of NetworkManager, you should probably stop and disable NetworkManager for good. If you don’t need easy WiFi setup on this server, doing so should be safe. Anyway, it could always be re-enabled.
sudo systemctl stop NetworkManager.service
sudo systemctl disable NetworkManager.service
sudo systemctl stop NetworkManager-wait-online.service
sudo systemctl disable NetworkManager-wait-online.service
sudo systemctl stop NetworkManager-dispatcher.service
sudo systemctl disable NetworkManager-dispatcher.service
sudo systemctl stop network-manager.service
sudo systemctl disable network-manager.serviceI actually squeezed this in while drinking my tea for the morning so now I am on a fully configured networkd static IP address and NetworkManager is disabled. I have to go to work today but I have started Roon Radio to play from my local library while I am gone. I can check the logs remotely. Thanks for the quick links, it took me all of 2 sips from my tea cup to get this done.
Thanks for the updates here, @Andreas_Philipp1 & @Robem. Our QA team is looking into this internally, but please keep us informed on if the changes you made helped — It’ll help to confirm that we are on the right track.
Ubuntu Linux Desktop - How to configure your ethernet interface
I just want to document progress…
It has now been over 24 hours since the last crash was recorded in the syslog and apport log files, while this timeline is not unheard of by the evidence seen over the past week, I am definitely in the period where I would have expected a crash by now or for it to occur at any moment. Roon Radio has been playing local files without interruption for about 14 hours now as well.
The stability so far is looking promising but there’s now always the chance that I have jinxed myself. I will update again in due course.
In my case it’s been now about 60 hours and all is well. Will report again tomorrow.
Thanks for the progress report!
I wish I could help you with your SonicTransporter… but I have no idea which Linux it uses and how it is configured.
We are trying to figure out a workaround for Ubuntu Linux which uses a network management tool called NetworkManager. This seems to cause problems with Roon, resulting in disconnects (Roon core server disconnects shortly from the network and reconnects seconds later).
Hopefully the QA team of Roon can find the root cause of the problem and a solution to it. This maybe will be of help to users of SonicTransporter, too.
I am now at 48+ hours without recording a crash in the log files and the music has been playing non-stop since I pressed play over 36 hours ago. I’m now feeling a level of confidence that this workaround successfully addresses the disconnects that I was experiencing.
Running over 80 hours without memory segmentation faults, I think we can draw some conclusions for users of Roon on Ubuntu Linux 18.04 and later:
You should use Ubuntu Server. It is light, it is fast, and it is very stable. Installing cockpit on top of Ubuntu Server you’ll have a nice and easy environment for remote administration of a headless server. Roon runs perfectly stable on Ubuntu Server.
Ubuntu Desktop uses NetworkManager as default network rendering tool, both for WiFi and Ethernet. There seem to be conditions under which Roon doesn’t play well with NetworkManager. This can cause memory segmentation faults which are experienced by sudden interruptions of playback and disconnection of the Roon remotes from Roon core. After 10 seconds or so, Roon core comes online again. These events can be more or less frequent, from one every 1-2 days to several per hour.
If you must use Ubuntu Desktop for your Roon server, you should for the moment being refer to the experience and instructions reported on this thread and configure it in such a way that it won’t use NetworkManager as a network rendering tool. This is easily done with netplan, and a reconfigured Ubuntu Desktop can run as stable as Ubuntu Server.
This seems to be really good detective work. Interestingly there was a small update to the Sonicorbitor Software yesterday and I got 9 hrs play before drop which is the best in nearly 3 months. Might just be random. I’m not sure how to apply the NetworkManager mods to the Sonic transporter though…
Lawrence, I think you shouldn’t. SonicOrbiter is a custom OS and we don’t know how it manages networking. But you could make the developers of SonicOrbiter aware of the problems which seem to affect NetworkManager and Roon.
Okay, I have done some additional digging through my log files and have the following timeline…
19th October 2020:
Announcement that [Roon 1.7 (Build 667) is Live!](Roon 1.7 (Build 667) is Live!) on the Roon website.
22nd October 2020:
First evidence of SIGSEGV error in my journal files.
$ grep "SIGSEGV" journal.txt
Oct 22 05:01:00 Roon-Media start.sh[19765]: Got a SIGSEGV while executing native code. This usually indicates
Oct 22 05:01:00 Roon-Media start.sh[19765]: Got a SIGSEGV while executing native code. This usually indicates
1st November 2020:
Upgraded my core from Ubuntu 18.04 to 20.04
Aug 18 07:27:10 Roon-Media kernel: Linux version 5.4.0-42-generic (buildd@lgw01-amd64-023) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #46~18.04.1-Ubuntu SMP Fri Jul 10 07:21:24 UTC 2020 (Ubuntu 5.4.0-42.46~18.04.1-generic 5.4.44)
Nov 01 15:06:36 Roon-Media kernel: Linux version 5.4.0-52-generic (buildd@lcy01-amd64-022) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #57~18.04.1-Ubuntu SMP Thu Oct 15 14:04:49 UTC 2020 (Ubuntu 5.4.0-52.57~18.04.1-generic 5.4.65)
Nov 01 15:34:40 Roon-Media kernel: Linux version 5.4.0-52-generic (buildd@lgw01-amd64-060) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #57-Ubuntu SMP Thu Oct 15 10:57:00 UTC 2020 (Ubuntu 5.4.0-52.57-generic 5.4.65)
Nov 19 19:53:26 Roon-Media kernel: Linux version 5.4.0-53-generic (buildd@lcy01-amd64-007) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #59-Ubuntu SMP Wed Oct 21 09:38:44 UTC 2020 (Ubuntu 5.4.0-53.59-generic 5.4.65)
From this I can ascertain that the symptoms were present prior to my upgrade to Ubuntu 20.04. It also appears as though the update to Roon build 1.7 (667) introduced the problem into my system as evidenced in the timeline above.
@Andreas_Philipp1 As it stands today I agree with your conclusion but I believe that there were changes to the code in Roon 1.7 667 that are the root cause of these crashes. I believe that Ubuntu Desktop is a viable product to use with Roon Server and the QA team needs to perform some root causes analysis and fix their code. I also believe that this is likely impacting other users on Linux based platforms but they don’t necessarily have the luxury of being able to troubleshoot in the same way due to being locked down (think Nucleus, SonicTransporter etc.). Hopefully the QA team will be able to come up with a resolution for this as a priority. As my current setup is only stable because I have applied a workaround I will not be marking this question as solved until the root cause is addressed.
I agree on all points. What we have implemented is a workaround, not a solution. And it doesn’t help people on other platforms who might experience similar problems. Let’s hope Roon QA can find the ultimate root cause of the problem and make Roon a more stable software. Thank you for your help digging in your logs and trying out the workaround. There are already other users trying out the same.
And thank you for making sense of the logs and bringing the workaround to the table.
I am now at over 72 hours without recording a crash in the log files, music has been playing constantly without a break for about 60 hours. I am declaring with significant confidence that the changes made to my network configuration are key to the crashes not occurring any more. I have reviewed log files going back to April 2020 and they tell a story. The crashes started within a couple of days of 667 being announced, they were not present on my system (as proven by the log files) prior to October 22nd 2020. My core then underwent an upgrade from Ubuntu 18.04 to 20.04 on November 1st 2020 and the crashes remained, I also performed a kernel upgrade last week and there was no improvement. It wasn’t until I moved the network configuration from NetworkManager to networkd that the server stabilized. Hopefully the QA team can come up with a resolution to this issue quickly, if I was a betting man I would wager that these crashes are being experienced by more Roon customers than the handful detailed in these support posts over the course of the past week or so.
More users affected:
I wonder if those reporting disconnections on ROCK are experiencing something related. Would be nice to have some feedback from Roon @support.