Nucleus One crashes and Roon Server restarts intermittently (ref#052PX3)

Support Nucleus Support
1

Select your Nucleus model from the list below

· Nucleus One

I'm having trouble with my Nucleus hardware

· Other

Select the option that fits best

· The Nucleus boots up and I can connect to it in the Roon app without issue, but I have a question about configuration/storage/attached devices

Describe the issue

Got my Nucleus One last week and set it up. It's pulling music from a streaming-optimized Synology NAS via smb, streams multi channel and hi res music to an Anthem MRX (hdmi) and a Parashound HINT6 (usb), respectively; ARC works great too. Everything worked great the first 3 days or so.

This week, I notice that it crashes once or twice daily and the Roon Server Software restarts. As of now, the OS and Roon Labs Software have been up for 6 days, 20 hours, while Roon Server's been running 18 minutes, 34 seconds. Reviewing the logs, it restarted without a preceding fatal error or user action, which has always been the case. No UnhandledException or CrashReporter entries were logged prior to the restart. What could be the reason and what should I do?

Thanks in advance!

2

Hey @Emile_Zafirov,

Thanks for writing in and for sharing such a detailed report - its good to see you on the community as well!

We were able to review a fresh diagnostic report from your Nucleus, and found a few out of memory related errors that could be causing the symptoms you describe:

(none) user.warn kernel: [443082.266325]  oom_kill_process+0xf3/0x180
(none) user.warn kernel: [443082.266286] .NET ThreadPool invoked oom-killer:
(none) user.err kernel: [443082.266717] Out of memory: Killed process 15559 (RoonAppliance) total-vm:9518468kB, anon-rss:3226236kB, file-rss:192kB, shmem-rss:0kB, UID:0 pgtables:9436kB oom_score_adj:0
Apr 23 21:30:53 (none) user.notice roonserver/run: Error

Based on your admin though, you only have ~40k or so tracks, which is well within the track limit the Nucleus One can handle. We’ll need to take your case to our development team for further discussion.

There’s always the option to increase the RAM for your Nucleus, and we can certainly share the step by step on how to do so, but we’ll definitely want to get some thoughts on what could be causing this from the team.

We’ll follow up with more information as soon as possible - thanks Emile! :raised_hands:

4

Thanks a lot, Benjamin! I am using NAS-based storage which I imagine is a bit more RAM-intensive to analyze, index & cache files so increasing the RAM is something I’d want to do but certainly not as the first thing to throw against the problem. Standing by to hear from you.

5

Roon Server crashed a couple more times tonight, in case you’d like to pull more Linux OS crash logs. Background: I added a handful of new albums while playing 24/96 music and then tried to scroll through the albums library covers (3110 albums).

6

Hi @Emile_Zafirov,

Thank you for your follow-up. We’ve reviewed the new diagnostic set and see the same pattern of OOMs at the timestamp you shared.

It’s not usual for RoonAppliance to kick an OOM at the BIOS level without RoonServer logging any memory or processing constraints prior. We’d like to request the following:

  1. At your convenience, please navigate to the RoonServer Database Location, zip up (compress) the entire RoonServer folder itself, and upload it to our secure QA database collection server here: DB upload

  2. I recommend you create a Backup of your Roon database and reinstall RoonOS from the Nucleus web admin afterward. See if this helps.

We’ll follow up as soon as we have more information to share. Thank you for your patience.

7

Thanks Connor. I am currently traveling out of town but will do both those things on Thursday night when I am back home. So you know if you don’t hear from me for a few days.

8

Hi @Emile_Zafirov,
Understood we’ll be here when you are ready to resume troubleshooting.

9

Hi Daniel, Roon Server directory uploaded. Standing by to hear from you!

10

Hey @Emile_Zafirov,

Thanks for sending that over! While our team takes a closer look, I wanted to check in and see if you also made a fresh backup and performed a fresh RoonOS reinstall via the webUI of your Nucleus?

If so, let me know if the crashing continues! Thank you :raised_hands:

11

Yes, I did that too.

As we were exchanging messages on this support request, I noticed a pattern that worked 100% of the time to cause the Roon Server to crash and restart: open the Albums view of my 3113 albums and scroll down 3-4 letters (e.g. from artists with A to artists with D). I rebooted the Nucleus before I did the Roon OS reinstall and it handled the test. I backed up the DB and reinstalled the OS, after which I repeated the test. It seemed ok till about D, then went on to J… and it crashed again.
Please let me know your thoughts or what I should do next.

12

Hi @Emile_Zafirov,
Thanks for providing the reproduction steps. Could you let us know the approximate date and time the crash last occurred? If that’s not available, please try to reproduce the crash again and share the timestamp with us. That will help us pinpoint the relevant entries in your logs for further investigation.

13

Just repeated the same steps and it crashed. 6:07 pm EST. Several times between noon today and now.

15

Hey @Emile_Zafirov,

Thanks for giving that a try! It’s clear the Nucleus is crashing due to being out of available memory, which still doesn’t add up considering your library count is ~50k tracks and about 3200k albums.

Our development team is scheduled to take a closer look at your case early next week. In the meantime—and I know this isn’t a perfect solution—you may be able to prevent the crashes by enabling or importing only about half of your library for now.

Thank you for your continued patience! :pray:

16

Hey @benjamin,

Just to confirm that crashing still continues every time I scroll a little faster through album covers but also spontaneously, without me triggering it - I just notice the up time of the server vs the OS.

Standing by for receiving your guidance on resolving this. Again, I would like to upgrade the RAM at some point, but hopefully that would be to expand the capacity of the Nucleus beyond my current number of albums and to mitigate higher RAM utilization due to using a NAS, not to be the only remedy for the current issue.

Thanks!

17

Hi @Emile_Zafirov ,

Just a thought here, if you navigate to Roon Settings → Library → Clean Up Library, do you see a large number of tracks listed as needing cleanup? If you create a Backup of your database and run the cleanup, does that help?

18

Negative. 18 deleted files, 0 files not associated with storage location and 0 with disabled storage. I cleaned up a few days before I posted here but it was 2-3 albums only - wasn’t bad at all.

19

Hi @Emile_Zafirov,

Did you give my above response a try as well? This would confirm if upping the RAM is the best next step here. :+1:

Also, can you confirm, is there anything particularly different than a standard install regarding your Roon library? Are you making use of a lot of tags/playlists?

21

Moved 1 TB of files and went from 3100 albums / 40,700 tracks to 1530 albums / 20,100 tracks.

Scrolling through the album art crashed the Roon server almost immediately (~4:16 pm EST) but after it restarted, I’ve been scrolling up and down, with and without focus filters applied, without further crashes. That while playing 2 channel DSD64 at the same time.

22

Update:
After slashing the library in half and confirming it runs ok, I added back 300 albums - to 1850 albums / 24,100 tracks. The scroll test crashed the server initially but after recovering, the server stayed on through multiple scrolls.

Added a few more albums - to 2130 and 27,700 tracks. Initial crash, then recovery and stable scrolling.

Added another 3,400 tracks. The pattern repeated.

Added another 4700 tracks to get to 35,800 tracks. No crash this time.

Added back the remainder of the tracks - to 40,700. Scrolling crashed the server twice after adding the tracks but it seems things have stabilized for now.

Please let me know what you see in terms of memory utilization.
While I am happy with the outcome (fingers crossed), I will keep an eye for a few days and I’d like to upgrade RAM for a bit of future proofing. This experiment took good 8-9 hours over the last two days and I still have some cleanup to do.

23

@benjamin
It seems that my reporting of successful resolution was a bit premature.
I’ve been doing my testing exclusively using a Roon client on a desktop PC, assuming that the server is the culprit and if it loads/caches all data it needs, album scrolling experience will be the same using any remote / client app. I can still scroll all albums using the PC, however doing the same on my iPad and iPhone crashes the server every time.