Memory requirements for large libraries plus an observation on restore

I had 8 posts on that thread, however that thread was not about technical issues. This thread is about technical issues so we’re discussing them. Besides if one has a subscription to either Tidal or Qobuz, then one has (access to) a very large music collection, way larger than my music collection. Maybe that’s what irks people music streaming services - one doesn’t own the music.

Doesn’t bother me all that much since music is about listening. I’m purposefully leaving aside the economic issues since that is topic for another thread.

You did…I’m sorry.

I just went back to my Windows based Roon Core, or Roon Server as it is called these days.

While the downsized setup with ROCK does work pretty well, I am not happy with the results of my database restore, i.e. “adding music to library”, adding and identifying tracks that have already been added and identified before after successful restore. There seems to be something broken with the backup/restore process.

I also do not like the complete absence of any kind of monitoring tools. Coming from an IT background I sort of want to know what is going on and how things are going.

After realizing that Roons database completely resides in memory (and dumps the database to disc just to save it in an efficient manner) it came to my mind that windows memory swapping might be a reason for bad performance after a given period of time.

While 64GB of main memory is more than enough for my databases Windows might still decide to start swapping for no reason at all, which in case might lead to the dramatic performance drop after 24 hours I have been experiencing and which caused me to do an automatic restart of roon every morning.

I hence deactivated disc swapping. Eager to find out if that helps.

Please let us (me) know if this helps since that may be the reason why Roon performance slows and then requires a restart.

Memory usage is a bitch. During startup roonappliance went up to 14GB before dropping below 9GB. So 32GB should be more than enough for my library. Until it is not.

Edit: This translates to a backup size of close to 18GB.

1 Like

24 later roon is going nuts again. I see CPU usage between 15 and 20 percent, local disk at a rate of 0.5 to 1.5 MB/s and network (music files) access at a rate of 20 to 65 MB/s. Memory usage goes up from 9GB gradually in the process.

First thought was this to be a rescan of my files, but that does not fit the pattern of rescan. Manual rescan is much more resource friendly, generating 3 percent CPU (max), 0.1 to 1.5 MB/s local disk and 5 MB/s (max) network (music files).

It can be easily seen that the “error condition” goes along with high cpu and high throughput on file storage. It seems files are not just “scanned” but read completely. Maybe for analysis done several times before.

Something is really flawed here.

P.s.; After realizing the windows setup ist still broken I came back to the (also broken) Rock setup. Gave up on this after waiting for almost 20 minutes for a client screen to interact with.

I will reactivate my daily roon restart routine and call it a day.

What you can do on Windows and can’t do on Rock, is to look at your Roon server log file in real time, as new log entries are written to it. I do this on Linux all the time, and find it very useful to see what the server is up to when these CPU and memory peaks appear… I now know what processes make RAM and CPU usage go sharply up, but see for yourself…

I haven’t used Windows for a very long time, but a quick Google search comes up with this Powershell command as a rough equivalent to the Linux tail -f command:

Get-Content filename -Wait -tail 1

Would you be willing to share your knowledge. Went through the logs in august without any clue. Had a whole lot of "Warn: bad track/media “CD”. Looking at these files they turned out to be perfectly ok.

You must observe the log as you are using the system; on my iMac remote I have basically two windows side by side; the Roon app is one (used as a Roon Remote), and a terminal session, connected to my Roon server, is the other…

During ‘normal’ playback and browsing on the Roon app, all is rather well-behaved, and apart from short and expected CPU spikes, nothing impacting is happening. And the latest release does a great job with garbage collection under this usage pattern.

From time to time, and at quite unpredictable moments, Roon starts on the server massive ‘storage library’ processing; this occurs independently for the Qobuz and Tidal collections saved into the database. These processes are very RAM and CPU intensive. Thing is, memory usage never goes down again to a level as before such storage library processing occurs. The log file also reports the number of open file handles, which also go steeply up during this processing, without apparently being released at the end… they stay up.

This processing, which keeps a processor core on the server at 100%, has a notably negative impact on all interaction with the remote’s GUI… page loads, time from clicking on a ‘Play’ button until music reproduction eventually starts, etc etc… all becomes very very slow.

Of course, this storage library processing vastly depends on the number of albums/tracks from Tidal and/or Qobuz you have saved into your Roon database. With smaller libraries, this is done much faster, with larger libraries it takes ever longer, and the impact is ever more felt. My local collection of about 56.000 files has not changed in months, and I don’t ever notice any processing of local files going on. Maybe for those who have vast and growing local file collections, there is a similar storage library process, which I don’t feel, as my local library is rather small and doesn’t evolve.

To me the most interesting observation has been that these storage library processes run very fast after a recent restart of Roon server, but slow down ever more, after Roon has been running for some days. A process that may take initially after a restart about 10 minutes, after two or three weeks of uptime can take hours, all the while keeping a processor core at 100% and such making all GUI interaction insufferable.

It has occurred to me that there may be a massive memory fragmentation, maybe because objects on the big object heap don’t get adequately garbage-collected. I don’t know, this is just my amateur way of trying to interpret what I do observe.

There is one more observation which I find interesting. After a recent restart of Roon server, the nightly backup of Roon’s database brings resource usage massively down again… file handle count and reported RAM usage go down, often even below the initial usage reported after a server restart. But, this is not the case anymore after some days of server uptime. Now the file handle count and RAM usage go down ever less, and, in my case, after a week or so, the backup process has no impact anymore at all on RAM usage and reported file handle count.

I am now running a database with about 300k tracks, of which 56k are local, the rest divided between Tidal and Qobuz, on a system with 16GB RAM and a Core i5 8600K processor. I can’t remember having seen RAM usage go above 10G; and when this occurs, user interaction with the system has been very slow long before, so a server restart is due.

These are my observations, and I at times have tried to communicate this to Roon support. I don’t feel that there was much interest in these observations, and my personal opinion is that the developers probably know about all this, but to address it would require something similar to a transplant of lungs, heart and liver, all at the same time…

2 Likes

Thank you @Andreas_Philipp1 for sharing. I will probably go back to my autorestart routine. My library is mostly local, so roon should just leave it alone after full analysis. I does not, which adds to the problems.

Restarting every morning takes about five minutes until roon is ready to take commands again, so while not being a solution it does not pose a problem.

I share your impression that roon is not interested in solving this problem. Not sure why this is. Time would have been better spent with a rock solid platform (pun intended) than fancy stuff like arc, which does not work well when rock oscillates. But then, what do I know.

1 Like

After the recent release, I could let my server go on for three weeks. The last 2-3 days of that were, let’s say, rather difficult. So I decided upon a weekly restart, which is quite acceptable to me.

Oh I think Roon might be very interested in solving this, but given the platform and the development tools used for the system, giving a complete solution might be prohibitively costly in terms of development effort.

I actually was very disappointed at what Roon pushed as ‘Roon 2.0’… there was nothing ‘2.0’ about this, it was just a release that put another hen in the coop… Arc. I would have expected something more similar to a more efficient reimplementation of parts of Roon’s core. And, of course, a solution to some long-standing annoyances as the infamous in/out of library problem when working with albums. To this day I nearly daily find instances where Roon doesn’t correctly and in context show that a given album is already saved into the database. This I find very disappointing, as I know very well that Roon’s senior staff is aware of the problem.

I suppose that the ARC development was largely driven by marketing considerations, and that the coincidence with a much asked-for feature was just that, a coincidence…

1 Like

That’s sorta the whole point of the Nucleus , it’s black box , the NUC/ROCK is just a cheaper Nucleus implementation. Some people want a “Hi Fi Component” approach , plug it in and go , no twiddling.

Personally I am trying (and failing) to cut down my screen time as it’s not good for my neck !!

Each to his/her own

I have a coffee machine that demands my attention for just the right amount of time for my Roon setup to start - Coincidence ? :smiling_imp:

1 Like

I get that, and I would love to look at it that way. I cannot, cause it does not work.

Anyway, if you look at it the way you look at cars, there is always some instrumentation just for fun. I mean, who needs rpm or turbocharger pressure? My wife ordered both as addon in her beetle convertible, just for the looks.

So some kind of dashboard would fit the concept, maybe fed to a mobile app.

Warn: bad track/media…

Got lots of these in the logs. Tracks or media listed are not bad at all but play perfectly. No explanation of its meaning all over the roon community.

I would be so much better if I simply knew what is going on.

What you could do, if so motivated, is to raise this as a support issue in the Support category of the forum and give specific examples of tracks that given this message. I suspect Roon Labs would be interested in this data for further investigation. If so, the support team will ask you to upload the specific examples for further analysis.

I will take another shot at Rock, as the machine I bought just for running roon is just sitting there.

This time I will not try to restore a backup but build the database from from scratch. I started this morning at 09:35 and hope it will be done in a week or so. Maybe faster. It will do a lot of adding and identifying, but that also happened when I restored the database. Maybe restore simply does not work between different platforms. Who knows.

Three hours later roon has added almost 25 percent of the library to the database. That is a decent speed and faster than restoring a backup. It will take half a day, not a week.

While I will be loosing my listening history I will probably end up with a clean database. Oh well.

Roon Labs will also support ROCK/NUC systems using an Intel NUC that is on their supported list. They will also support Roon on these systems:

I hear you, @Lebowski. Mainly two reasons why a Nucleus is not an option for me.

  • I do not see how a nucleus would behave different from the system I have set up.
  • I do not feel that I need a turnkey system. It is so much more fun to play with hardware from time to time.

Having said that I would like to say that my initial post was about two questions:

  • what is the optimum memory size for large libraries hosted by Rock
  • what is that strange behaviour of reanalyzing everything after restoring a valid backup.

Before trying Rock I had (and still have) a Windows installation that worked quite well when rebooted daily, so I am happy with what I got when using Roon (except 8:00 to 8:10 am during reboot).

Just a Random thought. Had you thought of swapping your windows and Rock PC’s OSs?

I strongly suggest maximising the RAM on ROCK machine.

I am running a NUC13 i7 with 64GB of RAM and iot makes my old Dual 16 core Xeon server on Windows look like an absolute slug.

I love Roon Rock.

My library is a relatively modest 200K tracks. Physical Music stored on NAS and accessed over 2.5Gb ethernet connection.

I recently purchased a Grimm MU1 and tried running Roon Core on it but quickly went back to my Rock NUC much more resposive io.e. Night and day difference

Regards

Andrew