RoonAppliance: System.Private.CoreLib System:OutofMemoryException

[crom]

Core Machine (Operating system/System info/Roon build number)
My setup is a Roon Rock running on an Intel Nuc:
NUC8I7BEH3
16GB RAM
128GB NVMe drive
Roon 1.7 build 528

Network Details (Including networking gear model/manufacturer and if on WiFi/Ethernet)
All Ubiquiti stuff… nothing has changed in the last 3 months except I’ve updated to the latest Roon.

Audio Devices (Specify what device you’re using and its connection type - USB/HDMI/etc.)
I play through HQPlayer.

Description Of Issue
Hi guys, I have a recurring problem where I am happily playing music and then Roon just crashes. I get the spinning Roon logo for about 5 minutes and then it returns to the screen that asks me if I’d like to try another core. This loops until I reboot the entire roon rock server (just restarting the rock server software doesn’t do anything… it just repeats the above loop).

I took at look at the logs and saw this:

Local Time:            03/15/2020 21:22:29 +00:00
Device Serial Number:  B1C61871-6F03-45D5-808C-2EB77C4C7635
User Id:               e60b9b14-4815-4e50-968d-1f158fa1c5d5
Roon Version:       1.7 (build 528) stable
OS Version:            Firmware 1.0 (build 186) stable
Mono Version:          unknown

Application Domain:    RoonAppliance
Assembly Codebase:     file:///roon/app/RoonServer/Appliance/RoonAppliance.dll
Assembly Full Name:    RoonAppliance, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null

   Exception Source:      System.Private.CoreLib
   Exception Type:        System.OutOfMemoryException
   Exception Target Site: Thread.StartInternal
   Exception Message:     Exception of type 'System.OutOfMemoryException' was thrown.
   Exception Data:        none

   --[ Stack Trace ]------------
   System.Threading.Thread.StartInternal()
       System.Private.CoreLib.dll, N 0
   Sooloos.Storage.DirectoryStorage._SpinWorkQueue()
       Roon.Storage.Directory.dll, IL 111, N 306
   Sooloos.Storage.DirectoryStorage.<_SpinWorkQueue>b__99_1()
       Roon.Storage.Directory.dll, IL 13, N 34
   ThreadUtil._WorkerThread()
       Base.dll, IL 182, N 382
   System.Threading.ThreadHelper.ThreadStart_Context(Object state)
       System.Private.CoreLib.dll, IL 57, N 50
   System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
       System.Private.CoreLib.dll, IL 156, N 58378413
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       System.Private.CoreLib.dll, IL 17, N 33
   System.Threading.ThreadHelper.ThreadStart()
       System.Private.CoreLib.dll, IL 45, N 46

03/15 21:22:29 Error: 
Local Time:            03/15/2020 21:22:29 +00:00
Device Serial Number:  B1C61871-6F03-45D5-808C-2EB77C4C7635
User Id:               e60b9b14-4815-4e50-968d-1f158fa1c5d5
Roon Version:       1.7 (build 528) stable
OS Version:            Firmware 1.0 (build 186) stable
Mono Version:          unknown

Application Domain:    RoonAppliance
Assembly Codebase:     file:///roon/app/RoonServer/Appliance/RoonAppliance.dll
Assembly Full Name:    RoonAppliance, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null

   Exception Source:      System.Private.CoreLib
   Exception Type:        System.OutOfMemoryException
   Exception Target Site: Thread.StartInternal
   Exception Message:     Exception of type 'System.OutOfMemoryException' was thrown.
   Exception Data:        none

   --[ Stack Trace ]------------
   System.Threading.Thread.StartInternal()
       System.Private.CoreLib.dll, N 0
   Sooloos.Storage.DirectoryStorage._SpinWorkQueue()
       Roon.Storage.Directory.dll, IL 111, N 306
   Sooloos.Storage.DirectoryStorage.<_SpinWorkQueue>b__99_1()
       Roon.Storage.Directory.dll, IL 13, N 34
   ThreadUtil._WorkerThread()
       Base.dll, IL 182, N 382
   System.Threading.ThreadHelper.ThreadStart_Context(Object state)
       System.Private.CoreLib.dll, IL 57, N 50
   System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
       System.Private.CoreLib.dll, IL 156, N 58378413
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       System.Private.CoreLib.dll, IL 17, N 33
   System.Threading.ThreadHelper.ThreadStart()
       System.Private.CoreLib.dll, IL 45, N 46

My system has been running completely wonderfully for a few months now and I’d love to return to the idyll… can the Roon guys suggest a solution please?

Thanks,
Crom

[crom]

Hi @dylan, are you able to help me with this please?

[noris]

Hi @crom,

Have you tried reinstalling ROCK OS and/or the Roon Database yet? I would try the following:

1. Create a Backup of your Roon database
2. Open the ROCK Web UI and click on the “Reinstall Operating System” button
3. Set your old ROCK database aside:

• Stop RoonServer from running in ROCK’s WebUI
• Navigate to your Roon’s Database Location
• Find the folder that says “RoonServer”
• Rename the “RoonServer” folder to “RoonServer_old”
• Restart the RoonServer in the WebUI to generate a new Roon database folder

4. Restore your database from a backup

Can you give those steps a try and let me know if it helps?

[crom]

Hi @noris thanks for jumping in. This is quite an undertaking as my DB is Over 60gb in size! Is there anything less drastic to try to get to the root cause (ie either find out what caused the db corruption? Or to find out what else may be to blame)?

[BlackJack]

@crom

my DB is Over 60gb in size!

@crom

Exception of type ‘System.OutOfMemoryException’ was thrown.

@crom

find out what caused the db corruption? Or to find out what else may be to blame

Add more memory might be a way to try? How many tracks are in your library? See also:

My experiences with Rock and a 500k+ tracks library

EDIT 4/2018: this post was written in June of 2017, and since then, we’ve made a number of optimazations to Roon. The number that was previously stated as 300k is probably closer to 550k now. First, some context It’s important to clarify: libraries of this size are extremely exceptional, and will always require special care. The group of people with 300k+ track libraries represents a small handful of people amongst our user base. Off the shelf solutions simply do not apply for people in this…

[crom]

Great link @BlackJack, thank you. I didn’t realise roon was .net. Looks like I should be using Windows 10 and not Rock/linux. As a test I copied the entire DB across to a ubuntu machine - 32GB 9900k Intel chip (from memory) …decent and fast box with a good fan… it repeatedly crashed the entire machine as soon as I started up Roon. Lends credibility to the Mono problem/limitation that the post you link to points out.

I haven’t used Windows in years but I think I’ve got an install disk kicking about somewhere. I’ll give it a go and report back.

[BlackJack]

AFAIK are the Nucleus(+) and ROCK no longer based on Mono since Roon 1.7. They should use now .NET, so Windows is no longer preferred – if it ever was. I linked the post because of the hardware recommendation for large libraries. But keep in mind that your Ubuntu is neither a Nucleus(+) nor ROCK so still using Mono.

[crom]

Mono is just the Linux version of .net so unless .net is now natively available on Linux then I would think rock/Ubuntu etc is still all mono-based. .not sure.

I knew about the hardware reqs but the post you linked to indicated the less than optimal memory management within mono…hence roon team’s recommendation of using native .net under windows.

[noris]

Hi @crom,

Let’s start off with a more basic questions first:

• How large is your library? What is the amount of tracks reported in Roon -> Overview?

• What are you doing immediately before this issue occurs? Are you accessing any specific areas of the Roon app?

• Is the behavior the same on multiple Roon Remotes/Clients?

[crom]

Hi guys, an update on this:

I had a spare NVME stick and another 16Gb RAM, so I found my old windows 10 key and installed windows 10 pro on it and then installed RoonServer. I then spent 2 days trying to get a backup to restore…then gave up and spent 2 hours getting the old drive to mount in windows (if anyone’s in the same boat then use http://www.diskinternals.com/linux-reader/ and ‘save’ the entire roonserver directory straight onto a clean windows roonserver install).

This worked a treat and I am now listening to the Roon database that was constantly crashing under RoonRock, on Windows 10…not crashed in an hour of playback…I’ll report back once I’ve got more hours under my belt.

I seem to remember from my Windows day that Windows Server 2012 sounded better than Windows 7, so I might swap out to that OS at some point but it looks like the post that @BlackJack pointed me to sorted me out. For a big library like mine, Windows is the way to go.

For the record:

• Number of tracks: 332994
• No specific and repeatable use of the Roon app triggered the crash…just playing music…it even crashed on both the start and half way through playing a track.
• Yes, behaviour common to multiple remotes.

Thanks @noris I’ll update this thread in a couple of day.

Crom.