Roon on macOS not working with huge library

I reached 32k albums, then server because of unknown reason started to delete imported files.
After seeing some 2k albums to disappear one after another, I restarted the server.
For some moment it stopped deleting albums, but then again started. It deleted from 31k some 10k albums down to 21k, and then because also of some other unknown reason stopped further deleting.

I had to restart it few times to get all SMB3 shares properly established, it was missing one or another each time.
Now it has reconnected deleted albums, and has reached already 33k.
After two days of busy work dropouts again started after each 5 seconds.

Today I updated server to the latest build 157, and rebooted the OSX and restarted Roon.
At current moment it is actively adding new albums, and playing w/o dropouts still 2 hours after the restart. Will keep you informed if and when they again restart. I hope not… :slight_smile:

Janis

Roon build 157 on OS X was working for 1 day. Then crashed and stopped adding albums. After restart another day of fine work without dropouts while adding albums.

Now again large deleting of added albums!
After reaching 39k albums added (it took three days), Roon decided to delete around half of them - 18k.
Now only 21k are left in the library.

No NW issues visible in setup, all watched folders are SMB3 connected, in status of watching for new file in real time.
Dropouts are now again happening each 5 seconds.

What to do?

  • move Windows Roon server
  • wait for Roon to fix OS X issues, but when
  • drop it, and use JRiver

FWIW I have found Jriver unreliable/unstable and under featured in the past on OS X compared to on a windows platform…as much as I hate to run a windows box it seems to be a more stable option for both Jriver and Roon in my setup.

Latest update.
After launching new scan last night, Roon server started to add new files, but finally deleted other 11k albums. So, now number of albums is down from 39k to 9k…

Hi Janis,

I’ve not see any comments from the Roon guys, so I’m leaving an @support flag for them to follow this up with you.

Hey guys, @eric just brought this to my attention, so I thought I’d come and explain where things are.

Disappearing Tracks

We’ve investigated this about half a dozen times over the past year. It’s actually more of a problem with OS X than with Roon (which is why Roon on other platforms does not suffer this symptom), and it is relatively rare, meaning we have many users using network storage on Mac without this symptom, and only a handful who see it. This suggests that it’s in some way correlated with setup details or habits of those users.

The script is basically like this:

Roon: Hey OS X, is the network folder mounted?
OS X: Yup. Mounted
Roon: Ok, tell me the directory listing for my watched folder
OS X: That folder is empty, buddy
Roon: Really? That seems wrong. Are you sure the network folder is still mounted?
OS X: Yup, mounted.
Roon: Ok, tell me the directory listing for my watched folder again
OS X: That folder is still empty, buddy
Roon: Double-check that it’s mounted for me, OK?
OS X: Yup, it’s still mounted. Jerk.
Roon: OK, I’m gonna trust you on this…

Anyways, it’s clear to us that OS X sometimes lies to us about the status of mounted network folders, the contents of their directories, or both.

The trouble is: if Roon thinks the network folder is mounted, but sees the directory as empty, it’s going to assume that those files are actually gone. And then the album count starts ticking down.

We have spent a lot of effort trying to provoke this situation in the interest of developing a workaround, and the chief barrier has been the difficulty we’ve had reproducing it in house, which makes the process of developing and testing a workaround a lot more murky.

We are working on some general improvements to folder watching infrastructure right now–part of that work includes our fifth or sixth attempt to work around this issue. This workaround is more crude than the past ones (which could be summarized as “making the above conversation even more annoying and pedantic and drawn out to try to be extra sure”):

Basically, we’re planning to write a hidden file into the root of your network folder, and if we ever list that folder and see the file missing, we’ll know not to trust what’s going on.

This breaks some fundamental assumptions of watched folders (mainly, that they are read-only), so we’ve avoided the it so far, but it’s fairly likely that it will work. Work on this is ongoing.

Performance of very large collections

@RobertS and @Janis, you have something in common: your library sizes are in the 99.9th percentile. Congratulations :slight_smile:

@RobertS’s observation about Windows is astute–extremely large libraries do work better on Windows.

This is a tricky thing to say out loud, because it isn’t really our intent to endorse one platform over another, and the vast majority of our users will truly never feel the difference–but in the interest of openness, I’m going to explain what’s going on here.

Roon is built on top of a framework called .NET. This framework is powerful, flexible and mature, and it’s suitable for building complex, demanding, long-lasting software on every major desktop, mobile, and server platform in use today.

Our choice to use .NET is what enables us to keep releases on five platforms in sync without having five copies of the software. It gives us access to a great ecosystem, and high quality tools to work with. There are not very many alternatives for a company/product like ours that tick the same boxes.

.NET was created by Microsoft, so it should not be a surprise that the best performing implementation of .NET is currently available on Windows.

For non-windows platforms, we use a community-developed .NET implementation called mono. This is a mature and stable project that’s been around nearly as long as .NET has. For almost its entire life it’s been stewarded by various businesses associated with the project’s founder, including Ximian, Novell, Xamarin, and as of recently Microsoft.

Yes, Microsoft. A few months ago, Microsoft acquired the company that stewards the mono project, with the expressed intent of leveraging mono’s technology in order to directly support .NET on all platforms. So the prospects for .NET on Mac, Linux, Android, and iOS are looking pretty bright.

Today, mono is good enough that virtually all of our users don’t have to care about what platform they run on. Looking towards the future, there is almost certainly going to be convergence between Microsoft and Mono’s offering, because they are now being funded and developed under one roof.

Unfortunately, when a Roon library gets up into the several-hundred-thousand-tracks range, it pushes mono’s memory management infrastructure so hard that bad behavior starts to result. Symptoms might include laggy screen loading, slow focuses/searches, audio dropouts, or other stuff like that. When the framework gets messed up like this it is literally standing in the way of the code we need to run to get things done.

A catalyst for this problem is the richness of Roon’s data model–500,000 tracks means we’re also keeping track of 50,000 albums, 750,000 performers, 20 million credits, and so on. Compared to most other music apps, we manage at least 10x more data–so we are starting from a more difficult position.

That said, there are definitely some areas where Roon could be optimized, and some specific performance oriented projects are planned. We are just wrapping up a round of optimization on RAAT–our next scheduled bit of performance work is focused on improving library related performance, with a focus on mac and linux, which should help here.

That said, I’m not totally certain whether it will be enough for libraries this large–the only way to find out will be to test once the work is done. Hopefully some improvements from Mono/Microsoft will appear in parallel with our work.

In Summary

Looking towards the future, it’s fairly clear that these problems will be behind us at some point, but that that point is not today.

In the immediate term, @RobertS’s solution–a powerful Windows machine–is probably the most viable workaround for supporting extremely large libraries like these with good performance.

Unfortunately, they are some of the more difficult and long-standing issues that we are doing battle with, and the solutions are not clear or crisp.

I hope this helped explain things, and I’m sorry that we don’t have an immediate remedy for these problems.

17 Likes

Hi @Brian,

Many thanks for that frank and insightful post, I’m sure it appreciated by the community.

Given Roon’s “intent [not] to endorse one platform over another” and that even though this might pain you further, may I suggest that whilst this situation exist for extremely large libraries that a guidance note is added to the Roon Knowledge Base FAQ: What are the minimum requirements?

3 Likes

Hi @brian,

Thanks you for the very candid and clear explanation! That certainly helped to understand what’s going on with macOS version.
:wink:

Based on that I just put together Win 10 server on Intel Skull Canyon NUC6i7KYK, with i7, 32GM RAM, and Samsung 950 Pro 512 GB PCIe Flash SSD.
Already in two days 50k albums imported, and not a single case of dropouts!
Also I can confirm that Windows version is MUCH faster that macOS on large library. Now Roon server is blazing fast in any function - launch (2-3 min vs. 5-10 min), scanning folders with ~1mil AIFF files on Synonoly NAS (30 min vs. >1 hour), pulling metadata and adding 50k albums (2 days vs. 1 week), searching in the music library (5-10 vs. 30-300 seconds), and upon demand audio analysis starting playback (1-2 vs. 5-10 seconds).
Depending on the task Roon server on Win 10 is 5-10x faster than on macOS with +/- similar HW!!!
:heart_eyes:

The only downside that I observed with Roon server running on Win 10 - the App is very regularly crashing (3-4 times per day completely; and once every two hours becoming non-responsive, but then again recovering). If somebody in Roon is interested, I can provide the log file.
Adding 50k+ albums is heavy workload, and after whole libabry will be completely imported, the server might become more stable. I will continue to report progress. However, what I have discussed with other users with very large libraries on Win 10, they also have issues with the regular daily or every second day crash of the Roon server app on Win 10, even when the whole library audio analysis is fully completed.
In this respect masOS App was much more stable. While it was having dropouts, and wasn’t useful to search music, it almost never crashed. However, this unfortunately is common issue of Windows OS and any SW running on it.
To deal with this issue I have put on the server small App called “Restart On Crash”, which is looking after the Roon.exe. If Roon is crashing or becoming non-responsive for longer time, it is completely closed and relaunched. This is not 100% bulled proof solution, but some 2/3 of cases are automatically taken care and Roon is nicely restarted.

Along these lines besides improving Roon server stability on Win 10, it would be great to see if Roon team could add two features in some of the future releases:

  • switch in the set-up to enable automatic launch of Roon upon start of computer (it might be very helpful for those people who are not too fluent in IT)
  • functionally and switch in the set-up to check if Roon is responsive, and automatic relaunch of Roon if it has become non-responsive or crashed

@brian, I would appreciate your input in HW requirements to run very large 50-100k album server. What Motherboard, CPU (cores & clock), RAM (size and which DDR4 speed), which type of Flash storage/ in RAID 0 or not for the Roon App and database, what power supply, 1GB ETH or multiple 1GB ETH link aggregation between the server PC and NAS? Which of these elements are the most important for max fast search and stable Roon server operations?

Thank you so much for making this wonderful Server and Apps - experience of music exploration within the simple and beautiful UI, search with rich interconnections, and high quality playback is just MAGIC!!!
:stuck_out_tongue:

Janis

Great feedback!

With Roonserver, this is already a setting (accessible via the system tray icon)

:scream: :slight_smile:

Yes, on macOS, if one knows where and how.
In Win it is somehow more complex.

For basic user would be great help to simplify life, by putting it in Roon own settings. IMO

Hi @Janis,
Great that you have made progress here. The NUC spec you list is pretty good and the speed of the SSD being the thing that helps most with database load, boot-up etc. 2500mbps seems about as quick as you can get just now…and that is quick.

I have something slightly less powerful (but similar) running about 30k albums.
My SSD is about 1200 read speed so half yours. A full NUC shutdown and restart is about 20 seconds. Roon start and scan of local drive is about 1min.

I don’t get any crashes at all though on the NUC. Can you confirm if you installed roonserver.exe or just roon.exe to run as the Roon core ?

Roonserver.exe doesn’t carry the overhead of graphics requirements so you might find that just helps a bit. Failing that you may want to let ALL the background tasks complete and then re-assess stability. It could take you 1-2 weeks to complete the Background Audio Analysis for example.

It was Roon Server running on Windows that René was describing… Right-click on the running Roon Server icon, and you should see the menu item to select auto startup…

Thank you @ncpl for raising this point!
It is roon.exe not roonserver.exe actually.

I would try to migrate to Roonserver, but how to do correctly upgrade and connect Roonserver to the library database I have created in Roon?

See if this gives you what you need.

https://community.roonlabs.com/t/how-do-i-move-my-library-over-to-roonserver/3195

This just occurred on my Roon Core, running via Roon Server on a Mac Mini 2011, SSD. My track count was stable at about 65,000 (same count as when running Roon Server on a Synology, pointing to the same Synology media folder) and then just started progressively dropping, then re-importing.

2 Likes

Hi @Michael_Merline ---- Please see Brian’s response above. Thanks!

-Eric

Thanks Eric. I’m moving to a custom-build PC running a minimal Lubuntu installation in the longrun.

I just need to figure out the components list. The processor is the main sticking point—i5 or i7 for a headless PC running Roon Core (<100,000 tracks) and a Plex server? I’d love any input you might have.

@ncpl and @philr, that you for guidance to move from roon.exe to roonserver.exe.

It all went super smooth, after move to headless core, the server imported remaining 10k albums smoothly and has been running for two days w/o any crashing or responsiveness issues.

Now I started to perform audio analysis. Sometimes it gets stuck on some file and doesn’t move forward. However, after turning it off and on again, it continues to make progress. Do you know what is causing this issue? Can I see on which file it is getting stuck?

Janis

Sorry to jump in on this support topic, but once again I’m just fascinated by the library sizes mentioned here - 40k and 70k albums just seem unfathomable (as a circa 1k album man).

If you’re in any way inclined, I’d be really interested to hear more about your background and how you managed to build such vast collections…

I found this happen also. Best way forward at the moment is to look at the logs. Search for [analysis] from the bottom of your log files for periods where you know the analysis stalled. I saw several files/tracks repeatedly appear here.

A bit of trial and error revealed one album that had files that Roon didn’t like. They weren’t flagged as corrupt but dbpa also couldn’t convert them. It was free download sampler album so I binned it.

Analysis completed after that.

Hope that helps