Roon Core Machine
Roon Core 1.8 build 806
Ubuntu Server 20.04 LTS x86-64
ffmpeg version 4.2.4
Core i7-3537U ~3GHz 2 core/4 thread CPU, 12GB RAM, SATA SSD, 1Gb ethernet
Library accessed over NFS mount
Networking Gear & Setup Details
Asus AC-68U router
TP-Link Gigabit unmanaged switch
File server = SFF PC running Ubuntu Server 20.04 LTS + 8 bay disk array / 12TB useable btrfs filesystem, 1Gb ethernet.
File server & switch connected to router
Core connected to switch
Connected Audio Devices
Home Theatre = Nvidia Shield + Denon AVR
Hi-Fi = Pi3B+ running Moode Audio & Roon Bridge + Topping D70s DAC
Desktop = PC (either Win10 + Roon Desktop or Fedora 34 + Roon Bridge) + Topping D50s DAC
Mobile = Samsung Galaxy Tab S7+ tablet, Samsung Galaxy S9, iPhone, Huawei Mediapad T3 10
Others = Chromecast Audio, Pi3 + Allo Boss running Libreelec/Kodi via Airplay
Library Size
Files = >9000 albums, ~101000 tracks, files shared via NFS
Tidal Premium = >1200 albums / ~12000 tracks saved in collection
Description of Issue
I’ve had my Roon Core up until recently running on my home server/nas, Roon Core has always been bit laggy on that with background analysis enabled so I lived without whatever the analysis brings. Roon Core was stable on that system and it’s never crashed.
This week I decided to stick Roon Core on a Core i7 laptop that no longer gets any laptop use, not a recent i7 but faster than my server (i7-3537U ~3GHz) and it’d be dedicated to Roon Core rather than sharing resources on the server. It accesses the music library shared from the server via NFS over 1Gbps ethernet. The main “control surface” tends to be Roon Remote running on the Galaxy Tab S7+.
Initially I set up the new Core running on Fedora 34 in multi-user mode (aka no desktop) as it was already installed on the laptop and whilst Roon benefitted from having a faster core I found that the Core is regularly crashing when background analysis is running.
Initially this was accessing the server over AC WiFi and whilst I found no evidence of network connectivity issues to eliminate possible WiFi problems I cabled it up to a 1Gb ethernet switch but it’s made no difference.
After a bit of searching found a comment on the forums suggesting that Roon Core might not play nice when network interfaces are managed with NetworkManager. Rather than spend time changing up the network management on Fedora I swapped in a different SSD and did a fresh install of Ubuntu Server 20.04 LTS + Roon Core + dependencies just like my file server. Guess what, same crashing.
There’s been just a few reports of skipped file imports in Roon’s GUI, mainly TIFF image files but if the crashing was due to the odd corrupt/unreadable file it wouldn’t be crashing this often…
$ ls -ltr /opt/RoonServer/Appliance/mono_crash.mem.*.1.blob
-rw-r--r-- 1 root root 10000001 Jul 22 10:23 /opt/RoonServer/Appliance/mono_crash.mem.56407.1.blob
-rw-r--r-- 1 root root 10000001 Jul 22 10:23 /opt/RoonServer/Appliance/mono_crash.mem.56368.1.blob
-rw-r--r-- 1 root root 10000001 Jul 22 11:39 /opt/RoonServer/Appliance/mono_crash.mem.60256.1.blob
-rw-r--r-- 1 root root 10000001 Jul 22 11:39 /opt/RoonServer/Appliance/mono_crash.mem.60216.1.blob
-rw-r--r-- 1 root root 10000001 Jul 22 12:56 /opt/RoonServer/Appliance/mono_crash.mem.64334.1.blob
-rw-r--r-- 1 root root 10000001 Jul 22 12:56 /opt/RoonServer/Appliance/mono_crash.mem.64294.1.blob
-rw-r--r-- 1 root root 10000001 Jul 22 14:12 /opt/RoonServer/Appliance/mono_crash.mem.68205.1.blob
-rw-r--r-- 1 root root 10000001 Jul 22 14:12 /opt/RoonServer/Appliance/mono_crash.mem.68165.1.blob
-rw-r--r-- 1 root root 10000001 Jul 22 15:29 /opt/RoonServer/Appliance/mono_crash.mem.71647.1.blob
-rw-r--r-- 1 root root 10000001 Jul 22 15:29 /opt/RoonServer/Appliance/mono_crash.mem.71607.1.blob
-rw-r--r-- 1 root root 10000001 Jul 22 16:45 /opt/RoonServer/Appliance/mono_crash.mem.75344.1.blob
-rw-r--r-- 1 root root 10000001 Jul 22 16:45 /opt/RoonServer/Appliance/mono_crash.mem.75304.1.blob
-rw-r--r-- 1 root root 10000001 Jul 22 18:01 /opt/RoonServer/Appliance/mono_crash.mem.78810.1.blob
-rw-r--r-- 1 root root 10000001 Jul 22 18:01 /opt/RoonServer/Appliance/mono_crash.mem.78770.1.blob
-rw-r--r-- 1 root root 10000001 Jul 22 19:18 /opt/RoonServer/Appliance/mono_crash.mem.82669.1.blob
-rw-r--r-- 1 root root 10000001 Jul 22 19:18 /opt/RoonServer/Appliance/mono_crash.mem.82629.1.blob
-rw-r--r-- 1 root root 10000001 Jul 22 20:34 /opt/RoonServer/Appliance/mono_crash.mem.85811.1.blob
-rw-r--r-- 1 root root 10000001 Jul 22 20:34 /opt/RoonServer/Appliance/mono_crash.mem.85772.1.blob
-rw-r--r-- 1 root root 10000001 Jul 22 21:51 /opt/RoonServer/Appliance/mono_crash.mem.89021.1.blob
-rw-r--r-- 1 root root 10000001 Jul 22 21:51 /opt/RoonServer/Appliance/mono_crash.mem.88980.1.blob
-rw-r--r-- 1 root root 10000001 Jul 22 23:07 /opt/RoonServer/Appliance/mono_crash.mem.92121.1.blob
-rw-r--r-- 1 root root 10000001 Jul 22 23:07 /opt/RoonServer/Appliance/mono_crash.mem.92081.1.blob
-rw-r--r-- 1 root root 10000001 Jul 23 00:23 /opt/RoonServer/Appliance/mono_crash.mem.95872.1.blob
-rw-r--r-- 1 root root 10000001 Jul 23 00:23 /opt/RoonServer/Appliance/mono_crash.mem.95832.1.blob
-rw-r--r-- 1 root root 10000001 Jul 23 01:39 /opt/RoonServer/Appliance/mono_crash.mem.99099.1.blob
-rw-r--r-- 1 root root 10000001 Jul 23 01:39 /opt/RoonServer/Appliance/mono_crash.mem.99060.1.blob
-rw-r--r-- 1 root root 10000001 Jul 23 02:56 /opt/RoonServer/Appliance/mono_crash.mem.101986.1.blob
-rw-r--r-- 1 root root 10000001 Jul 23 02:56 /opt/RoonServer/Appliance/mono_crash.mem.101945.1.blob
-rw-r--r-- 1 root root 10000001 Jul 23 04:12 /opt/RoonServer/Appliance/mono_crash.mem.105202.1.blob
-rw-r--r-- 1 root root 10000001 Jul 23 04:12 /opt/RoonServer/Appliance/mono_crash.mem.105163.1.blob
-rw-r--r-- 1 root root 10000001 Jul 23 05:29 /opt/RoonServer/Appliance/mono_crash.mem.108287.1.blob
-rw-r--r-- 1 root root 10000001 Jul 23 05:29 /opt/RoonServer/Appliance/mono_crash.mem.108247.1.blob
-rw-r--r-- 1 root root 10000001 Jul 23 06:45 /opt/RoonServer/Appliance/mono_crash.mem.111519.1.blob
-rw-r--r-- 1 root root 10000001 Jul 23 06:45 /opt/RoonServer/Appliance/mono_crash.mem.111481.1.blob
-rw-r--r-- 1 root root 10000001 Jul 23 08:01 /opt/RoonServer/Appliance/mono_crash.mem.114540.1.blob
-rw-r--r-- 1 root root 10000001 Jul 23 08:01 /opt/RoonServer/Appliance/mono_crash.mem.114501.1.blob
-rw-r--r-- 1 root root 10000001 Jul 23 09:18 /opt/RoonServer/Appliance/mono_crash.mem.117649.1.blob
-rw-r--r-- 1 root root 10000001 Jul 23 09:18 /opt/RoonServer/Appliance/mono_crash.mem.117610.1.blob
-rw-r--r-- 1 root root 10000001 Jul 23 10:34 /opt/RoonServer/Appliance/mono_crash.mem.120645.1.blob
-rw-r--r-- 1 root root 10000001 Jul 23 10:34 /opt/RoonServer/Appliance/mono_crash.mem.120605.1.blob
-rw-r--r-- 1 root root 10000001 Jul 23 11:50 /opt/RoonServer/Appliance/mono_crash.mem.124204.1.blob
-rw-r--r-- 1 root root 10000001 Jul 23 11:50 /opt/RoonServer/Appliance/mono_crash.mem.124165.1.blob
-rw-r--r-- 1 root root 10000001 Jul 23 13:06 /opt/RoonServer/Appliance/mono_crash.mem.127639.1.blob
-rw-r--r-- 1 root root 10000001 Jul 23 13:06 /opt/RoonServer/Appliance/mono_crash.mem.127598.1.blob
-rw-r--r-- 1 root root 10000001 Jul 23 14:23 /opt/RoonServer/Appliance/mono_crash.mem.130551.1.blob
-rw-r--r-- 1 root root 10000001 Jul 23 14:23 /opt/RoonServer/Appliance/mono_crash.mem.130591.1.blob
When looking at the timestamps of the dump files it seems to be consistently crashing every ~76 minutes so looks like the cause may be something more programmatic.
Currently I have background analysis set to “Fast (2 cores)”.
It’s now been analysing for a couple of days on the Ubuntu install now, in terms of hardware the CPU temp is stable at ~70C under load and ~40C when not.
Network connectivity is a stable 1Gbps, no system/filesystem/read issues logged by the file server and the Core’s network read rates are between 50 to 400 Mbps during analysis (which is still running).
I’ve tried to eliminate any potential networking issues by logging into a shell on the Core systsem then from there logging into a shell on the file server and running htop.
I’ve been playing tunes on Roon when the Roon Core crashes and whilst Roon’s GUI will drop to the pulsing graphic I can see htop running on the server from a shell on the core happily updating so no network problems between the two.
A typical crash shows up in the system journal like this…
Jul 23 14:23:33 diablo start.sh[130551]: =================================================================
Jul 23 14:23:33 diablo start.sh[130551]: Native Crash Reporting
Jul 23 14:23:33 diablo start.sh[130551]: =================================================================
Jul 23 14:23:33 diablo start.sh[130551]: Got a SIGSEGV while executing native code. This usually indicates
Jul 23 14:23:33 diablo start.sh[130551]: a fatal error in the mono runtime or one of the native libraries
Jul 23 14:23:33 diablo start.sh[130551]: used by your application.
Jul 23 14:23:33 diablo start.sh[130551]: =================================================================
Jul 23 14:23:33 diablo start.sh[130591]: =================================================================
Jul 23 14:23:33 diablo start.sh[130591]: Native Crash Reporting
Jul 23 14:23:33 diablo start.sh[130591]: =================================================================
Jul 23 14:23:33 diablo start.sh[130591]: Got a SIGSEGV while executing native code. This usually indicates
Jul 23 14:23:33 diablo start.sh[130591]: a fatal error in the mono runtime or one of the native libraries
Jul 23 14:23:33 diablo start.sh[130591]: used by your application.
Jul 23 14:23:33 diablo start.sh[130591]: =================================================================
Jul 23 14:23:33 diablo start.sh[130551]: =================================================================
Jul 23 14:23:33 diablo start.sh[130551]: Native stacktrace:
Jul 23 14:23:33 diablo start.sh[130551]: =================================================================
Jul 23 14:23:33 diablo start.sh[130551]: 0x5098ec - /opt/RoonServer/RoonMono/bin/RoonAppliance :
Jul 23 14:23:33 diablo start.sh[130551]: 0x509d17 - /opt/RoonServer/RoonMono/bin/RoonAppliance :
Jul 23 14:23:33 diablo start.sh[130551]: 0x470941 - /opt/RoonServer/RoonMono/bin/RoonAppliance :
Jul 23 14:23:33 diablo start.sh[130551]: 0x425704 - /opt/RoonServer/RoonMono/bin/RoonAppliance :
Jul 23 14:23:33 diablo start.sh[130551]: 0x4257ba - /opt/RoonServer/RoonMono/bin/RoonAppliance :
Jul 23 14:23:33 diablo start.sh[130551]: 0x7f3853ab73c0 - /lib/x86_64-linux-gnu/libpthread.so.0 :
Jul 23 14:23:33 diablo start.sh[130551]: 0x7f3853930364 - /lib/x86_64-linux-gnu/libc.so.6 : __libc_malloc
Jul 23 14:23:33 diablo start.sh[130551]: 0x8ec2ab - /opt/RoonServer/RoonMono/bin/RoonAppliance : monoeg_malloc
Jul 23 14:23:33 diablo start.sh[130551]: 0x8eaacb - /opt/RoonServer/RoonMono/bin/RoonAppliance :
Jul 23 14:23:33 diablo start.sh[130551]: 0x8eaf01 - /opt/RoonServer/RoonMono/bin/RoonAppliance : monoeg_g_utf8_to_utf16
Jul 23 14:23:33 diablo start.sh[130551]: 0x75f396 - /opt/RoonServer/RoonMono/bin/RoonAppliance : mono_string_new_checked
Jul 23 14:23:33 diablo start.sh[130551]: 0x7ae829 - /opt/RoonServer/RoonMono/bin/RoonAppliance :
Jul 23 14:23:33 diablo start.sh[130551]: 0x70dd7a - /opt/RoonServer/RoonMono/bin/RoonAppliance :
Jul 23 14:23:33 diablo start.sh[130551]: 0x6faca9 - /opt/RoonServer/RoonMono/bin/RoonAppliance :
Jul 23 14:23:33 diablo start.sh[130551]: 0x40941ad2 - Unknown
Jul 23 14:23:33 diablo start.sh[130551]: =================================================================
Jul 23 14:23:33 diablo start.sh[130551]: Telemetry Dumper:
Jul 23 14:23:33 diablo start.sh[130551]: =================================================================
Jul 23 14:23:33 diablo start.sh[130591]: =================================================================
Jul 23 14:23:33 diablo start.sh[130591]: Native stacktrace:
Jul 23 14:23:33 diablo start.sh[130591]: =================================================================
Jul 23 14:23:33 diablo start.sh[130591]: 0x5098ec - /opt/RoonServer/RoonMono/bin/RAATServer :
Jul 23 14:23:33 diablo start.sh[130591]: 0x509d17 - /opt/RoonServer/RoonMono/bin/RAATServer :
Jul 23 14:23:33 diablo start.sh[130591]: 0x470941 - /opt/RoonServer/RoonMono/bin/RAATServer :
Jul 23 14:23:33 diablo start.sh[130591]: 0x425704 - /opt/RoonServer/RoonMono/bin/RAATServer :
Jul 23 14:23:33 diablo start.sh[130591]: 0x4257ba - /opt/RoonServer/RoonMono/bin/RAATServer :
Jul 23 14:23:33 diablo start.sh[130591]: 0x7f4b46bb43c0 - /lib/x86_64-linux-gnu/libpthread.so.0 :
Jul 23 14:23:33 diablo start.sh[130591]: 0x7f4b46a2d364 - /lib/x86_64-linux-gnu/libc.so.6 : __libc_malloc
Jul 23 14:23:33 diablo start.sh[130591]: 0x8ec2ab - /opt/RoonServer/RoonMono/bin/RAATServer : monoeg_malloc
Jul 23 14:23:33 diablo start.sh[130591]: 0x8eaacb - /opt/RoonServer/RoonMono/bin/RAATServer :
Jul 23 14:23:33 diablo start.sh[130591]: 0x8eaf01 - /opt/RoonServer/RoonMono/bin/RAATServer : monoeg_g_utf8_to_utf16
Jul 23 14:23:33 diablo start.sh[130591]: 0x75f396 - /opt/RoonServer/RoonMono/bin/RAATServer : mono_string_new_checked
Jul 23 14:23:33 diablo start.sh[130591]: 0x7ae829 - /opt/RoonServer/RoonMono/bin/RAATServer :
Jul 23 14:23:33 diablo start.sh[130591]: 0x70dd7a - /opt/RoonServer/RoonMono/bin/RAATServer :
Jul 23 14:23:33 diablo start.sh[130591]: 0x6faca9 - /opt/RoonServer/RoonMono/bin/RAATServer :
Jul 23 14:23:33 diablo start.sh[130591]: 0x41202a82 - Unknown
Jul 23 14:23:33 diablo start.sh[130591]: =================================================================
Jul 23 14:23:33 diablo start.sh[130591]: Telemetry Dumper:
Jul 23 14:23:33 diablo start.sh[130591]: =================================================================
Jul 23 14:23:33 diablo start.sh[130591]: Pkilling 0x7f4b437fe700 from 0x7f4b137fe700
Jul 23 14:23:33 diablo start.sh[130591]: Pkilling 0x7f4b423aa700 from 0x7f4b137fe700
Jul 23 14:23:33 diablo start.sh[130591]: Pkilling 0x7f4b42c45700 from 0x7f4b137fe700
Jul 23 14:23:33 diablo start.sh[130591]: Pkilling 0x7f4b4197a700 from 0x7f4b137fe700
Jul 23 14:23:33 diablo start.sh[130591]: Pkilling 0x7f4b41578700 from 0x7f4b137fe700
Jul 23 14:23:33 diablo start.sh[130591]: Pkilling 0x7f4b4337e700 from 0x7f4b137fe700
Jul 23 14:23:33 diablo start.sh[130591]: Pkilling 0x7f4b135fd700 from 0x7f4b137fe700
Jul 23 14:23:33 diablo start.sh[130591]: Pkilling 0x7f4b4698d780 from 0x7f4b137fe700
Jul 23 14:23:33 diablo start.sh[130591]: Pkilling 0x7f4b40bbb700 from 0x7f4b137fe700
Jul 23 14:23:33 diablo start.sh[134051]: Could not exec mono-hang-watchdog, expected on path '/opt/RoonServer/RoonMono/etc/../bin/mono-hang-watchdog' (errno 2)
Jul 23 14:23:33 diablo start.sh[130591]: Entering thread summarizer pause from 0x7f4b137fe700
Jul 23 14:23:33 diablo start.sh[130551]: Pkilling 0x7f37107ff700 from 0x7f384a13e700
Jul 23 14:23:33 diablo start.sh[130551]: Pkilling 0x7f384848e700 from 0x7f384a13e700
Jul 23 14:23:33 diablo start.sh[130551]: Pkilling 0x7f38487bd700 from 0x7f384a13e700
Jul 23 14:23:33 diablo start.sh[134054]: Could not exec mono-hang-watchdog, expected on path '/opt/RoonServer/RoonMono/etc/../bin/mono-hang-watchdog' (errno 2)
Jul 23 14:23:34 diablo start.sh[768]: Error
Jul 23 14:23:36 diablo start.sh[768]: Initializing
Jul 23 14:23:36 diablo start.sh[768]: Started
Jul 23 14:23:37 diablo start.sh[134057]: aac_fixed decoder found, checking libavcodec version...
Jul 23 14:23:37 diablo start.sh[134057]: has mp3float: 1, aac_fixed: 1
Jul 23 14:23:39 diablo start.sh[768]: Running
The service restarts itself without any interaction from me and it’s back within a minute.
There’s no errors showing up in the Roon server logs under /var/roon/, only shows playback and analysis messages until the Core crashes and the logs rotated…
# tail RoonServer_log.01.txt
07/23 14:23:32 Trace: [library] endmutation in 24ms
07/23 14:23:32 Trace: [analysis] analyzing trackid=20255794 url=/mnt/nas/Music/Metal, HC/Exhumed/Exhumed - Platters Of Splatter (2004) [FLAC]/CD1/24 Exhumed - Blood And Alcohol.flac
07/23 14:23:32 Trace: [analysis] analyzing trackid=20256818 url=/mnt/nas/Music/Metal, HC/Exhumed/Exhumed - Platters Of Splatter (2004) [FLAC]/CD1/29 Exhumed - Exhume To Consume.flac
07/23 14:23:32 Warn: [Tablet] [zoneplayer/raat] long rtt sync Default Output: realtime=2345897500000 rtt=37500us offset=-13317500us delta=25500us drift=53000us in 771.9025s (68.662ppm, 247.181ms/hr)
07/23 14:23:33 Trace: [Tablet] [LowQuality 32.2x, 24/44 TIDAL AAC => 24/96] [100% buf] [PLAYING @ 0:25/3:42] Stormy - Madlib / Morgan Adams Quartet Plus Two / James B. Cobb Jr. / James Cobb / Buddy Buie / J.R. Cobb
07/23 14:23:33 Trace: [analysis] analysis completed in 1796ms for 57s of FLAC 44.1kHz 16bit audio. loudness=-7.10995167293667LUFS dbTP=-0.0581638643714277 noisestartms=52 noiseendms=54340 isdts=False mqa=0 for track 20255794 url=/mnt/nas/Music/Metal, HC/Exhumed/Exhumed - Platters Of Splatter (2004) [FLAC]/CD1/24 Exhumed - Blood And Alcohol.flac
07/23 14:23:33 Trace: [library] finished with 1 dirty tracks 1 dirty albums 1 dirty performers 0 clumping tracks, 0 clumping auxfiles 0 compute tracks, 0 deleted tracks, 1 tracks to (re)load, 0 tracks to retain, 0 auxfiles to (re)load, 0 auxfiles to retain, and 3 changed objects
07/23 14:23:33 Trace: [dbperf] flush 4708 bytes, 2 ops in 1 ms (cumulative 10583530 bytes, 4368 ops in 59337 ms)
07/23 14:23:33 Trace: [library] endmutation in 26ms
07/23 14:23:33 Trace: [analysis] analyzing trackid=20259378 url=/mnt/nas/Music/Metal, HC/Exhumed/Exhumed - Platters Of Splatter (2004) [FLAC]/CD2/09 Exhumed - Quagmire Of Flesh.flac
Any suggestions?
Cheers