Roon on macOS not working with huge library

2 posts were merged into an existing topic: Mac OS Sierra is out [Roon testing in progress]

I have been experiencing this problem and reporting it since the spring. I hadn’t seen this acknowledgement by Roon before today and am glad to see Roon is working on it.

Yes, I have a very large library on OSX and my files do the disappearing act from time to time. I noticed that when Roon is watching a folder it really does not like anything in the folder to change! Nothing added or removed. Or, heavens forbid, that a file which was put into iTunes’ “Automatically add to iTunes” folder gets moved into its proper Artist/Album folder when iTunes starts up and is set up to organize files.

There are two unpleasant workarounds. One is to quit Roon before adding/subtracting files, then restart Roon and let it go through the scanning process. The other is to Disable the Watched Folder while adding/subtracting, and then re-Enabling that folder. Again, long scanning process.

Roon people, any updates on progress on fixing this?

Thanks.

We have a few clients with very large libraries – 25k to 100k albums. What factors contribute most to fast performance? Can we rank them such that the ones with the most impact are at the top of the list?

Factors I can think of are as follows, but I don’t know which ones have the greatest impact:

  • CPU speed
  • Number of CPU cores
  • RAM
  • SSD vs. spinning hard drive
  • RAID configuration vs. single drives
  • Thunderbolt connection vs. USB3 vs. USB2
  • Size of library
  • Mac OS

Thank you in advance.

Assuming you read Brian’s post further up the chain?

It seems the basic needs for extremely large libraries would be:

Use a fast windows machine
Make sure OS and Roon database is on SSD
Sufficient RAM.

How all the intricacies rank beyond that, I’m not sure anyone’s benchmarked such things?

Thanks for this. I read that, but we are on Mac OS, and not planning to change just for Roon. I have been experimenting with tiny libraries. With a very small library of, say 100 albums, it is quite easy to tell what is going on. Searching the database appears to occur within the Roon app, on the drive where the core lives, using cached data. I can very quickly find the album or track I want to play. The only lag is waiting for a spinning hard drive to spin up, in the event that this drive is dormant (i.e. in energy saving mode).

Remaining questions:

  1. This large library problem only happens on Mac OS? What is the ETA for fixing this?
  2. Would storing music on SSDs improve search speed? Based on the above, I assume no.
  3. Is 16GB RAM sufficient for a very large library?
  4. Would processing speed help? If so, which helps more – RAM vs. CPU speed?

Thank you.

Windows was a suggestion for your clients with large libraries - I’m not sure OS X performance is so much ‘a fix’, more incremental improvements of Roon to handle huge libraries on that platform. Beyond that, I think you’d need one of the developers to talk about specifics. 100k albums is huge - I think if your clients wanted ultimate performance, they’d have to consider the most appropriate platform. Also, are they using RoonServer?

As far as I know search speed is database related rather than music media related.

For the most part the critical item here is CPU speed. Due to the way that the Roon database works adding more cores isn’t going to have any real impact on search performance. If the Roon Core is used for other tasks then having an extra CPU core or two available for those tasks will reduce contention.

Important only in the sense that you never want Roon to run out of available physical memory. When that happens the OS will swap some memory blocks to the system disk and that can really kill performance. 8GB is sufficient for the wast majority of libraries, but by all means have at it!

Next to CPU speed this is likely the most important factor but only for the database. Roon is constantly hitting its database and depending on the search query it may be hitting the disk vs. memory. These are tiny random transactions and they are most suited to SSD.

On OSX the database is stored in the user’s home directory ~/Library/Roon. One option is to swap out the system disk for an SSD, but depending on the Mac this can be a pain. It is possible to relocate the Roon folder to a different drive and use some of the underlying features of the OS to point Roon to the right place, but there are several ways in which this can break so I wouldn’t recommend it.

For the database… use SSD. For the media use whatever makes you happy, but it’s not going to make a difference in the interactive application performance. Depending on the number of drives a RAID5/10 configuration can improve the speed of moving the entire library into place, but it’s not going to make a bit of difference in terms of performance.

For reference 2 channel DXD has a transfer rate of about 16Mbit/sec (2 MByte/sec) if your drive can’t hit that then you have much bigger problems!

I would avoid USB2 as it can be painfully slow. For what Roon is doing either USB3 or Thunderbolt are more than adequate for media or the database. I don’t see much of a point in the added expense of Thunderbolt.

The bigger the library the larger the database and its associated indices. A large database on its own shouldn’t present a problem, but it is going to need more resources in terms of CPU speed, memory, and database IOPS.

Of the three platforms on which Roon is available OSX has the slowest overall performance. This is due to a few factors but the two most critical are the way that OSX prioritizes tasks (as well as the number of things running in the background) and the Mono (.NET) execution environment that Roon relies upon.

If I were spec’ing out a Core for a client with a large library I’d likely go for an I7 CPU running at a reasonably high speed, 16GB of ram, and the fastest SSD I could find. I would not spec any Mac hardware for this task (unless it was running linux or Windows). I prefer linux as the OS, but a good stripped-down installation of Windows works just as well.

Ultimately, with 1.3 and the release of the Roon Core Kit I doubt that I will spec anything but that to run the Roon Core.

  1. While Roon is constantly optimizing their code this issue is out of their hands as it’s more specific to the OS (and they didn’t write that part).

  2. No.

  3. Most likely, but people here with 500K+ track libraries would be able to tell you for certain.

  4. Processing speed would help, but it’s not the silver bullet. Sorry, but OSX just isn’t the right platform for the Roon Server when the track count goes up. For a LARGE library the best practice is to run it on a dedicated machine with the most appropriate OS. When customers have no trouble spending $1000s on tweaks they can easily afford to spend a few hundred to get their playback “transport” dialed in.

7 Likes

Wow…that is a great response. No stone left unturned there.

I fully support the notion that if you have 100k albums, then that represents a ton of time if not money to accumulate. A decent server or NUC is less than the price of the HDD to store (and backup) that quantity of media. The benefits that Roon brings to libraries of all sizes would make a spend on a server well worth it.

An powerful i7 server with Linux Roonserver or a NUC with dedicated Roon Core Kit will be the best bet here. No need to move off whatever platform you are already on. Just add the dedicated server and run Roon clients as you have today.

Seriously big libraries need to be taken very seriously with dedicated kit.

Thanks for all the replies. I wonder what architecture the big streaming services like Tidal, Spotify and others use to make their enormous databases searchable. Surely there is an industrial-strength approach … Perhaps what they do is proprietary and requires a lot more horsepower. Hard to say. For now, I’ve guided clients on Mac OS to use Roon and Tidal, and only add the artists they want to hear to a watched folder. By the time they get through everything, we’ll be in the year 2050. :slight_smile:

It’d hardly be viable setting up a server farm to run Roon at home, nor is it required. My library sits at the lower end of the album count you mentioned. Roon Server is running on an i5-4690S with 16GB DDR3/1600 (8GB would suffice) running Linux. Roon’s database is on a dedicated F2FS formatted SSD, music is stored locally on 5900RPM NAS BTRFS formatted drives without any RAID config. Performance is good. With a bit of luck you’ll see search speed enhancements with 1.3. At 100k albums I’d recommend your clients start listening and forget collecting. :grinning:

I’ve received that message running Roon on a Mac Mini to an endpoint of an ASRock BeeBox (an Atom NUC) and I only have 300+ albums. Media is on a NAS and is supplied via a GB network. It’s that POS Mini and its operating system.

Dell R710 server with dual Xeon processors, running 16 cores at 2.6 GHz and 64 meg of ram. Used on EBay for about, with shipping, $350.

1 Like

Highly recommended! We run them with Freenas on them.

300 albums is a small library. On my Mac, I have over 3000 and roon works flawlessly. My Mac mini is running OS X server plus minimserver along with roon, with 26TB of disk attached.

1 Like

Hi @Brian and @support, it’s been 4 1/2 years since your very informative post about the problem of disappearing libraries, and MacOS tricking Roon into thinking drives are mounted when they aren’t.

I’ve been a Roon user for a long time and continually suffer from this phenomenon. Yes, my library is massive, over 500,000 tracks. The core is on a Mac Mini M1 with 16Gb of RAM. Library is SMB-mounted on a Drobo 5N2. Laugh at me! I know I’m in an eddy of problematic hardware.

Still, the odd thing about it is, that when I see that my track count is doing a nosedive, I can still see that my Drobo really is still mounted. I can play the track in iTunes (now Apple Music). The drive is mounted, and Apple Music can see it. Roon can’t, though. I’ve wondered why my track count doesn’t suddenly drop to zero, as it would if I, say, powered down my NAS.

Mostly, though, I wonder whether in the past 4 1/2 years you have identified anything I can do short of replacing the core with a Windows box to stop this from happening.

I should mention that I am also a perpetual sufferer of the skipping tracks phenomenon, where I could start playing an album in my library, and Roon will go down skipping track by track, sometimes landing on a track late in the album and starting there, or just skipping all the way to the end without playing anything.

Thank you.

I owned a Drobo for years. It is the single worst piece of NAS hardware I have ever seen, and it gets worse as you put more data/files on it. I remember spending hours debugging with it years ago, and after staring at it under a microscope trying to understand the bad behavior, I concluded that it was so irregular that it couldn’t be supported with any confidence.

Something about the way they bookkeep for the flexible space/filesystem mechanism is not done well, and leads to a cumulative performance problem that occasionally escalates into “hard” failures felt by the OS or software running on it.

My thought at the time was–the filesystem momentarily glitches around one of these bad I/O operations, the OS notices, and notifies Roon, and some small % of the time, Roon sees the directory as empty. Empty is not the same as “offline”. I think the empty condition is often extremely brief, but if we get unlucky with the timing and observe it in that moment, the problem starts.

We’ve done some mitigations over the years and we see very few reports of this behavior now compared to before. It used to be one of our top support requests, and this is the first time I’ve been made to think about it in years. I don’t think we are going to take that work any further, especially not for hardware that has known problems.

With 500k tracks (this is well into the 99.x percentile library size), a laptop is not the best choice. I know that the M1’s are very fast, but they are not the best way to run Roon, and particularly with the memory requirements of such a large library, you’re going to be in some pain sharing 16GB with the rest of the stuff you will be running on the Mac.

My recommendation would not be a Windows machine–either use a Nucleus+ or a Core i7 NUC running ROCK. I would avoid the NAS–get a very large hard drive or two that can hold everything. Either put it inside the NUC/Nucleus if you can fit all of your music on it, or connect via USB. You don’t need RAID for music files–the performance benefits of RAID have no value in this context, and RAID is not backup.

Keep the NAS as a backup, and set up an automated sync in one direction or the other over SMB so you have a second copy on the Roon system.

Do that, Roon will perform better, your laptop will perform better, and you’ll never see this issue again.

1 Like

This is all really good advice. Just to clarify, my Core is not on an M1 laptop but an M1 Mac Mini. Of course this has nothing to do with the Drobo, which I am sure is the source of so much of my tsuris (pardon the Yiddish). (def from OED: “Problems or difficulties; trouble”)

thank you for your great response.

@Janis

I have the same problem with you. I have 37K album, configured with 6-core 3.2G I7 and 64G memory, but it will slow down inexplicable when using it. Normal operation is very fast, which is its due performance.

That sounds like a lot of data, but it’s not.
Most of our use scenarios are albums as a unit, and the viewing area can only accommodate dozens of pieces. I am a developer, and I can confirm that tens of thousands of data sets are not huge, and it is not a problem for computers or databases.

Slow, mainly because Roon did not pay attention to this amount, the program details did not do bit. The test data set may be small and the network connection fast. Regardless of the query, the transmission can be in tens of milliseconds if normal. Roon caches a large number of images and does not need to transfer images.

For example, I have a question here, but no one answered it for a long time, because their data are very small, there is no problem.
The problems mentioned in the post are definite and have good solutions.
Also keep an eye on your Log to find the problem.

https://community.roonlabs.com/t/macos-roon-server-slow-down-randomly-with-time-at-any-time/183515

It’s an expensive piece of paid software, and users have reason to demand better,
Expect official developers to focus on issues related to larger databases.

@Carl @brian

Here, for example, why is it so slow, because the implementation method is not considered, we don’t need to see all the results at once, still quite usable, it allows asynchronous parallelism, it doesn’t have to wait, etc… See it all.

And so on, is a test set, implementation method problem.

I suggest to subscribe to TIDAL or QUBUZ

Btw, I moved from having my mac (with music stored on external disks) as my core to a sonictransport as core, SMB mounting the music from my mac.
So far this is working exceptionally well for me. Except that copying new music onto the drives doesn’t trigger roon to update its database. I have it set to scan every 24 hours, or do a force rescan if I want it to find something sooner.

I tried other variations like a USB drive on the sonictransport but find this setup to be the best.