You may find additional performance improvements by splitting your top level directories further. For example, my top level directories are mostly the main genres, pop, classical, jazz, soundtracks etc. I got further performance improvements by splitting out any large box-sets into a new top level directory “Compilations”. I also split a large Classical directory into 5 top level directories, Classical_0, Classical_1, Classical_2, Classical_3 and Classical_4. In other threads, others with large libraries have found performance improvements by splitting large directories by alphabetical range. E.g. A-E, F-J etc.
Restructuring larger libraries like this obviously makes them more difficult to manage but above a certain library size there seems to be little choice.
Creating A-C, D-F etc directories did the trick. So! much faster now!!! Thank you ever so much for the suggestion!
EDIT: somehow the snappiness of the interface is now back to when I only had a 1TB drive attached. Pure genius…
What I do not understand is that it took adding to the complexity of the folder structure in putting albums back into nested folders (alphabetically, 3 letters a folder)
I guess that is a common issue, and I want to highlight that once you reach a certain limit of tracks/references/folders/artists/composers/complications roon is seemingly causing your CPU usage to spike, it seems to get slower and slower with just a few problematic albums added. Sluggishness is not proportional to tracks in your library but there is a tipping point.
And some parameters of albums as well as a certain library structure seemingly make things worse. I have a particular problem with classical boxsets (samplers, operas, oratorios), as hundreds of artists can be assigned to one album itself containing hundreds of tracks. It seems roon has to go through all the references which add up exponentially when opening artists´ or composers´ pages or compiling a composition list causing even a powerful CPU core to spike for some seconds.
The total size of my library is similar to the OP´s one (adding up to 17TB of data) but I keep two thirds of it disabled most of the time to have snappy roon. With a less powerful CPU (Nucleus One class) I already notice (acceptable) delays once the active library size has exceeded 50,000 tracks ending up in sluggish roon once you reach 80,000 tracks. A bit early I think.
Have read that recommendation several times but failed to find a definition what is an ideally flat structure or what is best practice when it comes to folder structure. Could you elaborate on your experience with the matter please? Particularly how many tracks/files and volume of data you have within a typical top-level folder.
I already have split my core library into 7 top-level folders (on one volume admittingly) yet two of them being still big (one is exceeding 2TB with 500+ (artist) subfolders, another one 4TB+). Do you think it makes sense to split them further, closer to your 15 to 20?
I haven’t seen any hard and fast rules either. Probably for two reasons. Larger libraries where this performance effect starts to be noticed are not common amoung roon users. Also, although it has changed now with the sale to Harmon, for the longest time, discussion of folders has been a taboo topic. Personally I had noticed this effect within a few months of my initial roon purchase 7 years a go.
You keep on mentioning the physical size of disks. My experience is that this has no relevance. What matters is the number of directories. Both horizontally (how flat) and vertically (how deep). Both dimensions have an effect. I now have a number of physical volumes all of which I have structured in a similar way:
I have 3 physical volumes and 18 top level level directories distributed over them. You can see that not only is the directory structure quite flat but nothing runs very deep either. Most of these top-level directories have no more than 600/700 or so sub-directories which seems to be a tipping point where I notice performance declines that I am motivated to do something about. On that basis I actually have a couple of directories that are overdue being split as the performance has started to irritate but I haven’t got round to it yet. All those years ago I started splitting large Classical directories arbitrarily but I would do it alphabetically if I was doing it again.
If your larger directories are significantly larger than my examples I would try sub-dividing them further. But to be honest 80,000 tracks is not a large library by the standards here and I would not expect it to trigger performance issues significant enough for you to disable two thirds of your library. I have also never experienced what you describe of one or two “problem” albums triggering some kind of dramatic performance decline. I would look elsewhere for the source of the problem.
The most obvious thing that you have mentioned in several posts is your merger of thousands of compound artists instead of splitting them with delimiters. I’d say that was a very unusual way to organize roon and its effects on roon are likely unpredictable. There are no guarantees of course but that is where I would look first. I can understand why you are reluctant but I haven’t seen anyone else post that they have disabled most of their library to get acceptable performance so something has gone drastically wrong and the remedy will likely be equally drastic.
That was a misunderstanding. I have 180k+ total tracks and keep roughly 2/3 of if disabled most of the time for performance and quality-of-tags reasons. So my core library which is enabled varies between 60k and 80k tracks.
80k is surely not big, but I notice increasing delays (particularly opening artists´ or composers´ pages and composition lists) with every complicated album enabled making the active library exceeding 60k while being classical-heavy. If I use it with 80k mainly pop, rock or jazz, it stays snappy.
That’s why I have focussed on optimizing metadata, tags and identification in the past when optimizing my library while neglecting the folder issue.
No dramatic decline but noticeable in case of very big albums, typically boxsets with hundreds of artists assigned and thousands of tracks.
Looks very similar to my method except from the fact I have only one physical volume and use a 4th layer either for ´Disc1´, ´Disc2´or sub-categories in ´Compilations´ for a limited number of albums.
Will try to move up Compilations to top-level folders of their own as you suggest. Very helpful, thanks!
Meaning 600 to 700 sub-directories in the second layer alone, or 600-700 sub-directories in total per top-level folder? If the latter, some of my folders are exceeding this by far and I would consider a total overhaul of the folder structure.
Sounds like a reasonable theory but I have no idea how to counter this problem anyhow differently. I started merging ´ghost artists´ from the very beginning of my roon journey and attempts to properly tag affected albums, cleaning them out and un-merging artists was not successful in most of cases. I assume that many of these originate from TiVo data.
I typically limit the second level directories to 600/700. E.g. Classical_0. But I routinely break my own rules and usually only restructure larger directories when someone in the household complains.
On your second point:
It will not be easy for sure. Unmerging artists in roon is not what you need to do. Or to be honest any editing in roon at all with a larger library if you can avoid it which seems to be your instinct. You need a 3rd party tagger like mp3tag which can batch replace tags at source. That is comma, hyphen etc. delimiters that roon doesn’t recognise with a semi-colon that it does.
For example, Opera metadata is plagued with this problem but you can use mp3tag’s scripting language to automate the substitution.
I would experiment with a few albums until you can see what the side effects are as you probably have built in other dependencies into the way you are using roon at this stage. Also, sometimes you want to retain the comma where that is part of the normal artist name or at least how roon has equivalenced it. You will have to experiment and fix the side cases you encounter manually or just make another script and run it across your entire library in one go as I do. Over the years I have found it necessary to build up a library of hundreds of mp3tag scripts I can draw on or slightly modify:
Remember. You can batch up these substitutions for entire directories with hundreds of folders or even your entire library which I routinely do. I also use roon’s’ editing features like artist merge as you do but much more sparingly than you seem to have done. Mostly it is to catch equivalences with more obscure artists that roon has missed. I don’t think a lot of these roon editing features scale very well with larger libraries and were probably not designed with the edge cases of larger libraries in mind.
None of my edits happened overnight but I am sure the net effect is that under the hood roon has to work much less hard to assemble displays and searches so for the most part I do not experience the performance issues I see reported. I still get issues with certain types of searches but not enough to be motivated to complain.
Thanks for the suggestion, using MP3tag is exactly what I have tried already but instead of manually improving tags I have replaced all tags by fresh metadata sets sourced from MusicBrainz per album. I doublechecked that none of the confusing ghost tags (coming from iTunes, FreeDB or discogs) has survived, like several names separated by commata, remained in the files. Unfortunately the problem in roon persisted bringing me to the conclusion that these seem to originate from roon´s own metadata.
Which means you have per volume
6-7 top-level folders (Like ´Classical´, ´Pop´)
max. 5 second-level folders per top-level folder (like ´Classical_0´, ´Classical_1´…)
a maximum of 600-700 album folders per second-level folder containing no sub-folders at all
?
Maybe that is my problem as I have
7 top-level folders like ´Classical Hi-res´, ´Pop FLAC´
Up to 600 second-level folders mainly composers, main artists or ´Compilations´ per top-level folder
Up to 500 third-level folders representing albums per third-level folder in some cases
Fourth level for disc numbers
Will try to move Compilations and other chunky second-level folders like ´Bach Johann Sebastian` to first level as a first step. Thanks again!
Yes. The depth of the directory tree seems to matter as well. So I would try removing a layer if possible.
I have no Composer level for my Classical directories. So I only have 3 layers including CD1, CD2 etc. as there are so many box-sets and doubles and triples.
In the case of Pop and Jazz I do have an Artist layer but doubles and tripples are much rarer and I have moved all larger compilations into a separate Compilations tree. It is a long time ago now so my memory is hazy but I think I did initially have a Composer layer in my Classical Library but had to remove it. So to navigate I just prepend the composer to the Classical folders:
I’m not sure that restructuring your folder structure will entirely cure your issues but it should help. You may still need to revisit how you deal with compound artists but it is difficult to say.
