Large Collection (280,000 Tracks) Slow search

Your de-duping does sound unusually thorough. I cannot say if that is causing a performance problem. Maybe roon knows? Anecdotally unusual library structures do seem to throw roon.

could be related to the “Cloud Search” move and changes in V 2.0

Out of ideas otherwise …

This sounds like perhaps an issue with the directory structure of the library being hard for Roon to traverse in combination with a mobile class CPU not being capable enough to handle a library of this size in a timely manner especially if the folder structure is inconsistent. At a minimum I would try and explain your directory structure a little bit. Something such as Artist/Album/DISC-TRACK_TITLE.flac would make the most sense. If we are dealing with a ton of nested files and folders perhaps Roon is struggling with this and using some free software like MP3tag to reorganize the folder structure and then completely recreate the Roon database would probably help. Scanning such a large library on a laptop class CPU should take a day or so. I’m hesitant to think this is a hardware issue given the symptoms. This seems like a corrupted database combined with an issue relating to the directory structure.

One idea which might help to identify the problem at least a bit:

When experiencing slow search, delayed composition lists and other issues with Roon running on my previous, underpowered NAS, I tried to deactivate one music folder after another and checked if performance would improve noticeably. Surprisingly, this was the case with just 2 folders out of 20. Both seemed to have an unusual structure with many tracks and compositions being attached to one and the same artist/composer as well as a strange hierarchy of folders with multiple layers. In my case I identified the folders with operas and classical music editions as the main source of problems.

I still have no idea if it was the number of tracks per album (up to 2400) or discs per album (up to 350) or folder hierarchy or sheer amount of data (about 15TB on HDD) making it difficult for Roon to proceed searches. In the end of the day I let these 2 deactivated and made a selection what I really wanted as a part of my collection which I search on a daily base. I kept more almost 3/4 of the tracks active but the system was reacting much much faster. Moving to a faster CPU eventually Roon seems to be able to handle the full collection (120k tracks in total) but the difference in speed is still noticeably so I usually run Roon with the 75k tracks of the core collection.

Really cannot say if your problem has a similar origin but this folder deactivation process might help anyways.

2 Likes

Directory structure is album artist / album / disk number / track name.flac. When album artist is “Various Artists” the album goes in a separate directory - otherwise the same structure (without the “various artists”). (The prime directory for named artists is “Named” and the prime directory for Various Artists is “Various.”) The processor is an Intel i7 - the computer is a NUC10. If I must upgrade from there, I’ll do it (as I’ve been doing since I started this discussion months ago). I have mp3tag and have used it for many years to transfer files to my Roon Core. If my file structure is “bad,” and there is a preferred file structure that is recommended by the Roon techs I hope they make that clear and I’ll find a way to re-do everything.

What CPU are you running? I have only two primary directories - Named and Various. There are hundreds of subdirectories under those (Album artist and album names), and some have disk number subdirectories as well. I have not sorted files by genre for my file structure.

I would suggest taking a database backup and then try re-creating your database. You’ll have the backup in case it doesn’t help. Rather than deleting the database you can rename the folder as well so you can name it back if nothing changes without even having to restore your backup

Someone can correct me if the path is wrong but I think you would stop Roonserver from the ROCK webui and the rename the RoonServer folder to RoonServer.old. See here: https://help.roonlabs.com/portal/en/kb/articles/database-location#ROCK_Core

Then reconfigure Roon, rescan your directories and see if the situation improves. If it doesn’t you can easily revert back to the old config but at least this will help you narrow down whether you have a software or hardware issue. If you junk the database and start fresh you’ve essentially ruled out Roon itself as the problem and isolated the issue to something in your environment such as hardware or an external software issue like the directory structure. What you describe for directory structure doesn’t seem unusual so perhaps you have some corrupted files. At least you know if it is still slow after a fresh db that you’ve ruled out Roon as having an issue.

@Ronald what happens if you deactivate the ´Various´ folder in storages?

NUC10 with i7 - is it one of those NUCs usually running at very low frequencies like 1Ghz to keep thermal power low? In the lab it is still fast enough, maybe double the benchmark speed of my CPU, but for a library of such size (280k tracks you said?) it might be underpowered. Do you have a chance to monitor the CPU´s behavior while executing CPU-heavy operations in Roon? When I had my underpowered Celeron still I noticed it jumping to 100% CPU usage for 5 or 10 or even more seconds while processing a search or compiling a composition list or starting a stream. I took that as a clear sign the CPU has reached its limits.

Honestly it’s most likely the CPU choking under that size of library or a messed up database. To troubleshoot properly you want to test the free hypothesis first. It doesn’t cost any money to try a clean database and reimport. You’ll know very quickly if the import is much faster after a fresh start. Otherwise then look into eliminating folders or perhaps pointing Roon to a smaller subset of files to see if it can handle a smaller library with that hardware. If you slowly start having Roon watch more and more folders and files and at some point it starts to break then look into hardware.

I am running both a passively cooled low-TDP Celeron which is rated at about 50% the benchmark speed of your i7 (if I am not mixing up the Core models) and a Core i5 Gen12 which should reach almost double the speed of your i7. Maybe for complicated lasting Roon operations even more as it is running at high base frequency. Not sure about that as I am not a CPU expert.

Both CPUs feel comfortable with 120k tracks only varying in snappiness of searches and compiling lists. But my library is just less than half the track count of yours and Roon seems to demand CPU power exponentially with increasing number of tracks, interdependencies and compositions being prescribed to one composer.

Even with the powerful i5 Gen12 I do notice the difference in reaction time between the 2 aforementioned folders with extraordinarily mean structure either being activated or deactivated.

This NUC was not bought for purposes of keeping thermal power low, no. I don’t know how to monitor the CPU running under the Linux platform upon which ROCK is based.

Did you notice any problems before Roon moved to the cloud…?

If anything Roon moving their search to the cloud would put LESS stress on your CPU not more. That isn’t the issue otherwise it would be affecting many more people.

I had zero problems before Roon moved to the cloud. And that was before I upgraded my computer and my storage drive.

Did you notice any problems before Roon moved to the cloud…?

Sorry, no experience with that as I did not own the powerful i5Gen12 at this point in time but just the significantly underpowered Celeron.

Like I said I would try to rule out Roon itself as the issue by testing with a clean database. If that doesn’t resolve the issue you’ve essentially isolated the problem to your environment. If this were a systemic issue with search many other users would have the same problem.

FYI from early-access Roon 2.0 Build 1275 and ARC Build 180 is Live!

There is probably some movement in this matter?

1 Like

Can I just reset the database from the web interface?

I’m not using ROCK so it’s hard for me to comment on that. Just make sure you take a reliable, current backup before doing so. If it doesn’t fix the issue you’ll want to revert back.

I’ve just changed these settings to “off.” Should this make any difference? Where should these be set?