My experiences with Rock and a 500k+ tracks library

Now you tell me! Whilst my collection is nowhere near that it has started me thinking ; what are ROCK’s limits in this respect and how do you actually count track numbers? I can easily count files but these include cover art etc so the number of tracks could easily be 10% lower and not so easy to find.

Could you link to the topic please?

Nope, I will not.
I go on vacation and turn on a Pc in 3 month again.
Happy summer.

1 Like

Ok thank you for your ‘help’…

Cheers will check it out.

A fascinating thread that is nearly a year old though. Anyone from Roon care to comment on what improvements have been made to the Linux/ROCK build in this performance area?
Before choosing ROCK I had already discounted OSX due to the many issues people seem to have with it with Roon. So ROCK or WIN10 it was. Thankfully ROCK was released towards the end of my free trial so the decision was made for me (WIN 10 is still rather update heavy and I could do without that for something that should just be an appliance).

EDIT 4/2018: this post was written in June of 2017, and since then, we’ve made a number of optimazations to Roon. The number that was previously stated as 300k is probably closer to 550k now.


First, some context

It’s important to clarify: libraries of this size are extremely exceptional, and will always require special care. The group of people with 300k+ track libraries represents a small handful of people amongst our user base.

Off the shelf solutions simply do not apply for people in this situation. For people with huge libraries, we have always recommended the same thing: Get the fastest consumer-grade i7 you can, at least 4 cores, 16GB of RAM, a 256GB+ NVMe SSD, a big fan to cool the thing, a nice closet to put it in, and a locally connected storage array for the content, and then run Windows 10.

For the rest of us who don’t have extreme libraries…

Consumer operating systems like Windows, OS X, and most popular Linux distributions do not create the best headless appliance experience. They have frequent software updates–some of which can’t be skipped. They upgrade packages one at a time–which creates a risk of unintended breakage. They have a HUGE footprint of unnecessary stuff consuming resources on disk, in RAM, and running in the background. This makes them good general purpose OS’s, but not the best for a media server appliance.

This is why we created ROCK. It doesn’t contain extra junk unrelated to the task at hand. It is not general purpose. It doesn’t have a package manager upgrading packages all the time, and it doesn’t run a bunch of stuff in the background–just the minimum needed to fully experience Roon. When ROCK is updated, the whole OS is updated atomically and in a “brick-proof” manner, just as appliances should be updated.

ROCK is very different than what you’d get if you started with a full-sized operating system like Ubuntu, Debian, Fedora, OS X, or Windows Server and tried to trim it down as much as possible without breaking it. The total installed size of ROCK is under 70 megabytes, including the kernel, root filesystem, and drivers. This is a small percentage of the minimum size of an OS X, Windows Server, or Ubuntu install. I think 70MB is actually smaller than the Windows Kernel alone :stuck_out_tongue:

By starting clean, and avoiding all of that extra stuff in the first place, we can create an appliance-style experience, make things smoother, snappier, more trouble-free, and easier to test and support.

ROCK, Roon, and mono

The Roon Core–on all non-windows platforms–utilizes the mono runtime, just as it does on other Linux (and OS X) platforms. Mono is shipped as part of Roon, not as part of ROCK, and ROCK uses the same Roon packages as any other Linux user is using.

When it comes to library management at extreme library sizes, the limitations that people encounter primarily result from mono’s performance characteristics, not Roon’s (or ROCK’s).

The thread I wrote above goes into a lot more detail, but I’ll quote the most relevant part:

.NET was created by Microsoft, so it should not be a surprise that the best performing implementation of .NET is currently available on Windows.

For non-windows platforms, we use a community-developed .NET implementation called mono. This is a mature and stable project that’s been around nearly as long as .NET has. For almost its entire life it’s been stewarded by various businesses associated with the project’s founder, including Ximian, Novell, Xamarin, and as of recently Microsoft.

Yes, Microsoft. Microsoft acquired the company that stewards the mono project recently, with the expressed intent of leveraging mono’s technology in order to directly support .NET on all platforms. So the prospects for .NET on Mac, Linux, Android, and iOS are looking pretty bright.

Today, mono is good enough that virtually all of our users don’t have to care about what platform they run on. Looking towards the future, there is almost certainly going to be convergence between Microsoft and Mono’s offering, because they are now being funded and developed under one roof.

Unfortunately, when a Roon library gets up into the several-hundred-thousand-tracks range, it pushes mono’s memory management infrastructure so hard that bad behavior starts to result. Symptoms might include laggy screen loading, slow focuses/searches, audio dropouts, or other stuff like that. When the framework gets messed up like this it is literally standing in the way of the code we need to run to get things done.

A catalyst for this problem is the richness of Roon’s data model–500,000 tracks means we’re also keeping track of 50,000 albums, 750,000 performers, 20 million credits, and so on. Compared to most other music apps, we manage at least 10x more data–so we are starting from a more difficult position.

Both mono and Roon have gotten a bit faster since I wrote that, but mono is still not yet on par with Microsoft’s .NET runtime when it comes to the extreme memory management scenarios that arise when managing the largest libraries.

Be careful trying to generalize these ideas…

Some of you, after reading that, are taking away the idea that “Windows performs better”. When it comes to managing extreme library sizes there’s definitely situations where it does, but in other areas, the same reasoning doesn’t hold.

We see better performance/stability out of the Linux scheduler than we do on Windows. Code related to DSP or communication with audio devices is generally a lot simpler and bit faster–both because the compilers we use on Linux seem to do a better job at optimizing our audio processing code, and because the audio subsystem on Linux is more streamlined and simpler. Unlike with library management, these areas of the system do not make such heavy use of .NET/mono, so they are not subject to mono’s performance characteristics.

On top of that, ROCK adds an extra layer of benefits. First and foremost, the benefit of having a very small root filesystem, RAM footprint, and set of background processes/components. This creates a very quiet operating environment, where Roon has no worries about outside interference, or competing for system resources with other stuff–It’s difficult to get near this point when starting with a general purpose OS, even after heavy trimming.

In short, ROCK remains our first choice for all but the most extreme library sizes–the ones that are so large that they would require careful hardware/platform choices regardless of the circumstances.

In Closing…

It’s unfortunate that this a handful of people with extreme library sizes aren’t able to fully benefit from ROCK, but the experiences of these outliers should not necessarily inform the decision making of the rest of us. Their situations are unique enough that special handling is required regardless.

On all platforms, there is a point where increasing the library size to the extreme results in performance degradation. On mono-based platforms–OS X, ROCK, other Linuxes–that tipping point comes a little bit earlier. The work to close this performance gap is happening as part of the mono project–not as part of ROCK or Roon. As mono gets better over time, the gap will shrink, and the “tipping point” will continue to increase.

But–if you’re on the right side of that tipping point–and most people are–we believe that ROCK is the better choice.

30 Likes

Thanks Brian, very informative as always.

Seems things are never as simple as many people want to make them.

A post was merged into an existing topic: ROCK and network security

Thanks for the insights @brian. How has Mono evolved these days? Has MS done some improvements?

Yeah, there has been steady improvement over the last couple of years, but the garbage collector still starts to turn into a bottleneck in some of these extreme scenarios.

Hi,

Any plans on using native decoders (supplied by the OS) ?
In particular, ALAC is rather buggy, to the point it makes the entire OS unstable.
The tracks are fine, they would play fine beforehand, play fine after that (that - is an OS reboot).
Appears to me some memory corruption issue somewhere.

The worst that a buggy codec in Roon could do is crash itself. Roon is physically incapable of trashing memory outside of its own process space–this is enforced by the kernel. If the whole OS is going unstable, something else is going on. Our ALAC codec comes straight from Apple in source code form. It’s the same one that pretty much everyone uses.

OS-provided codecs are never our first choice. When we use them, it is generally for licensing reasons–to avoid shipping code that is protected by patents. Currently, that means we get AAC/MP3 from the OS and ship the rest, but we will likely shift to bundling our own MP3 codec soon since the patents recently expired.

Some of the most difficult stability issues that we’ve faced result from poor 3rd party codec implementations. We are aware of about half a dozen defects across the AAC/MP3 codecs that come with Mac and Windows and have put some pretty nasty workarounds in place to mitigate them. It would have been much easier to deal with these situations if we were building the same codecs from source and could just go in and fix the bugs directly.

If you have particular files that cause problems with Roon, tag our support team and let them know. We are generally pretty good about tracking that stuff down when it comes up.

Hi,

I have files that play fine for years on several devices, including ROON, and all of a sudden, you hear screaching noise.
And yes, it does make OS unstable in that case.
"The worst that a buggy codec in Roon could do is crash itself. "
I wish that were true. I obviously don’t know how ROON is written, I can only observe what it does.
The only way to recover in these cases (more than one, sadly), is OS reboot.
After that - you can play the same files with no issues, as you could for years beforehand.
“We are aware of about half a dozen defects across the AAC/MP3 codecs that come with Mac and Windows and have put some pretty nasty workarounds in place to mitigate them.”
That may be the case, but as an end user, I need to play my files, properly ripped and playable everywhere else.
Which means, maybe you need to use their workarounds, or simply advise that certain codecs are unstable.
Such information would be invaluable, so that I, and other users, could avoid problematic pieces of code.

And as for Apple shipping buggy ALAC/AAC implementation - I find it hard to believe. The internet would be full of people screaming about it.
All I want is to use ROON (which I like, just look at my user name), and not concern myself over this codec or that one.
And worst - I REALLY do not wish to run into a situation (happened multiple times on multiple OSs) to hear a screeching sound in the middle of a song and find that ROCK stopped responding or Windows remote desktop all of a sudden stopped responding reliably).
I’m a software developer, so can distinguish between various causes of things.

Many thanks in advance!

If this is true, your OS has a bug–it’s the kernel’s job to protect itself and other apps from badly behaved applications. You’re a developer, so I don’t have to explain how virtual memory and protected mode works.

The bugs on OS X that I’m aware of are all in the MP3 codec. Their AAC/ALAC stuff is fine. Windows has some bugs with AAC and some with MP3.

Sure, I appreciate that–but understand that backseat driving technical decisions isn’t necessarily going to get a “sure, we’ll do it” response. You do not have the visibility into our system to know that “not using OS provided codecs” is the root cause. And with only a vague problem report and no way to reproduce the issue and investigate–neither do we.

This isn’t a problem description that I’ve heard before. Have you reported this to our support team? They are well equipped to verify that they can reproduce your problem and then put it in front of someone who can fix it.

1 Like

Hi, yes.
I did.

Please forgive me my frustration, but I first installed ROCK and ROON, both provided by you. So, understandably, I was a little upset over the situation.
I’ve replaced ROCK with Windows 10 since then.

I totally get it, believe me. I’m not trying to tell you how to do things. I also naturally, am trying to find the root cause, so that the final software is 100% wonderful (it is 95% - 98% right now :smile: ).

All I can, and am trying to do, is to provide you folks with enough information to try and zero in on the root cause.

Please forgive me my forward way of speaking, it is that I’m trying to make ROON perfect, is all. In a fairly limited way :smile:

Best regards,
Michael

Cool. I’ll check in with them and see how it’s going.

Thank you so much, Sir!
Please find the original topic below:

That’s why I love you guys so much!

Thank you!
Michael

Thank you for all your patience!
I love ROON so much, and want to enjoy music with the great sound quality and fantastic ease of use you guys provide!

Thank you again!

Michael

Hey, @brian et al - I wanted to check in and ask what the current best practices are for setting up a Roon server to handle an abusively large library (640,000 tracks and growing). They may not have changed since the recommendations earlier in this thread, but I thought I should ask fresh before ordering hardware.

My home Roon setup (with a moderate 40k tracks in 3100 albums) remains snappy running on an Ubuntu-based server (Haswell-era Xeon, music in a local ZFS pool). But the library at the work radio station, the result of years of collective scrounging, now runs over a half-million tracks and… Roon often gets sluggish.

There was a significant speedup somewhere a few releases into Roon v1.3 - thanks to whoever did that work, unless it came mostly from inherited Mono improvements - such that searches no longer hang for minutes at a time, but now usually just seconds. But… I’m wondering if better optimization of the hardware (and if absolutely necessary, OS) environment would offer noticeable further improvement.

The current station RoonServer instance is running under an Ubuntu LTS on a machine with a single SATA-interfaced SSD and an Ivy Bridge-era Xeon, with the track storage CIFS-mounted from a NAS. Clearly, there are multiple places for improvement there, but I’d love some guidance about what to prioritize.

I plan to build a machine for this with internal spinning drives for the music store, just to reduce network dependencies (and RoonServer seems to notice tracks added to the tree more reliably when the storage is local). The NAS will still be the master music repository, but I’ll rsync changes into the RoonServer machine as necessary.

It sounds as if high-end desktop CPUs are favored over server ones (which seem to lag a generation behind the desktop releases). When choosing CPUs, how would these characteristics rate in order of importance?

  • maximum single-threaded speed
  • overall multi-threaded throughput
  • number of available threads
  • number of available distinct CPU cores
  • amount of on-chip cache

Have any of the new AMD CPUs been outperforming Intel when running RoonServer?

I’m assuming that an SSD with a faster interface than SATA would be in order for the Roon database.

Would it also be useful to direct the Roon logs to a separate drive?

As for the OS for this to run on… I have a strong preference for something Linux-based (for maintainability, stability, and ease of setting up file synchronization); but if you-all believe that Windows and native .NET still have an edge over Linux and Mono for this size library (and that said advantage is likely to remain for at least a year or two), I’ll bite the bullet and build a damned Win10 Pro machine.

If something Linux-based might be near parity with Windows in performance - is ROCK flexible enough to set up in this expected shape, or would a general-purpose Linux distribution work better?

Is there any advantage to more than 16GB or so or RAM? I figure once I start researching motherboards, I’ll aim for a configuration which fills all the parallel banks.

And… what have I forgotten to ask?

Thanks!

2 Likes

Maximum single-threaded speed in the core is the main determinant in user experience snappiness.

Multiple cores helps with running more zones in parallel with more DSP and will complete stuff like audio analysis faster. I wouldn’t go beneath 4 cores/8 threads for this use case. You could do more, but added value will diminish depending on how much you value those aspects.

I understand the preference for Linux (and wouldn’t want to run Windows if I could avoid it), but in your shoes I’d do an A-B test and come to a personal understanding of what the differences are before making up my mind. It costs nothing but a few hours to install Windows without activating and learn the situation.

Here’s my feeling: Despite ongoing improvements to mono and more and more code from Microsoft.NET being merged in, Microsoft’s garbage collector is still better at handling the massive in-memory object count that comes with a library of that size. For most libraries the gap is negligible–there’s just some design choices in that GC that start to become more painful around 4-5mm objects on the heap, and really large music libraries can push things that high.

I don’t think Microsoft’s GC is ever going to make it into Mono. I think instead they’re going to keep making .NET Core better until Mono is displaced. At some point they will reach a point where we can switch to .NET Core. I have a hard time imagining Microsoft doing that and us being able to start that effort until a least a year from now, but it could be more depending on whether .NET Core 3.0 is actually viable or we need to wait another iteration.

In other words: if you didn’t have a strong preference for Linux, Windows/Microsoft.NET is the safer recommendation at this unusual scale. Since you do, it may be worth the extra few hours to try it both ways and see, just in case the Linux performance on a fast machine is good enough and you can avoid the Windows headache.

As for other hardware choices:

  • High-End Desktop vs Server CPU: High-End Desktop. Especially now that there are Threadrippers and i9’s, there’s no reason to buy a Server CPU for something like this unless you need Server motherboard features like redundant power, remote admin, etc. 4 Cores/8 threads is probably fine. You could go up to 6-8 cores especially if you see yourself running more zones/dsp.

  • Intel vs AMD: We test, develop, and ship hardware based on Intel, so there’s some passive bias. That said, AMD might get you more performance for the same $. I have a very, very fast AMD machine here (built for batch processing/machine learning) that runs Roon very well. I wouldn’t worry about it too much.

  • NVMe SSD is a good idea. Larger sizes tend to be faster, too up to a point. Samsung 960 Pro is very good.

  • Definitely store the music on local (USB3 or SATA) drives and stop watching the NAS. This will help more than you think.

  • For 640k tracks I’d consider going to 32GB RAM. Even if Roon doesn’t use it all, you’ll benefit from extra filesystem caching + more breathing room.

  • If you wanted to try ROCK, NUC7i7DHNE is the best NUC right now. You can hang USB drives off of it and it will support the storage. It can take an NVMe SSD and 32GB of RAM. So it could work.

  • If you want to build an even bigger machine, run Ubuntu or Windows. Getting a shell on ROCK is more difficult if you want to do sysadmin stuff like setting up cronjobs, rsync, etc. So that may push you in other directions.

That covered what I can think about right now. Any more questions/clarifications, let me know. I’d like this build to be a success for you.

18 Likes