I have a feeling that the disk IO is completely pegged. He is asking Roon to scan thousands of directories over USB and is wondering why things are slow. At a bare minimum that drive needs to be connected over SATA.
I also want to add that latency is much higher over USB than over SATA. Therein lies the problem with a library this large.
Yes, I hadnât thought of that.
A simple fix would be to restructure that VA drive. I experienced a very similar problem early on with roon. I had a huge dustbin of a directory of 80âs mp3âs that mucked everything up. That directory ran to just a few thousand not 16k. When I broke the directory into smaller 100 mp3 chunks the problem went away.
Yes, he either needs to simplify his directory structure or not expect an external hard drive to function like one connected to an internal bus. It is unreasonable to expect mobile grade hardware with external USB connected spinning hard drives to be a performant environment with a library not only this large but with thousands of subdirectories containing a lot of tiny little files. Lots of small files use more IOPS than fewer large ones. A spinning HDD connected over a high latency bus such as USB canât possibly be expected to be performant with this type of configuration and directory structure. OP may be interested in this article that sort of explains what is likely going on here: What is IOPS (input/output operations per second)? | Definition from TechTarget
OP really needs a home server running a Roon library this large without any changes being made. An Intel NUC with USB hard drives isnât meant for this.
1 Like
That is exactly what I am using. But I take your point. The OP needs to upgrade to an environment closer to an enterprise one if his current way of organising his data is important to him. This is not a path that suits all. I have taken an opposite path precisely because I have no interest in self-supporting a small enterprise grade data center at home. Instead I have chosen to work within the limitations of consumer grade electronics. It does mean I must make compromises with what is possible but I am ok with that,
He could even purchase a prebuilt desktop PC that has a few empty hard drive bays and a few extra SATA ports. He can simply shuck his two external drives and install them internally in a real desktop computer and not an Intel NUC. That will likely alleviate his issues, no datacenter required. 
Not to mention a computer running Windows will be a lot easier to troubleshoot. All he has to do is take a quick look at taskmgr if there is an issue.
Yes. I can see how that would work. The complication is the cooling and fan noise. I ran a fanless desktop with a low TDP for years but just wearied of the maintenance and support. Many here probably enjoy that. By the sounds of things it comes easy to you but its not everyoneâs cup of tea.
I run mostly Noctuaâs in my 12 bay home server and itâs pretty quiet. Most modern PCs with PWM fans can be controlled to spin down to a lower RPM when at idle. It just doesnât seem that this use case is suited to this hardware configuration. Now if he went out and got a large SSD to install in the NUC (they do sell 8 TB SATA SSDâs now) that would probably help too. The idea is fast internal storage. I see USB as being the issue here. If he can get his media off of those external drives and onto an internal one heâd see an performance increase. The issue is really storage expansion with these NUCs isnât great. It would really depend on how many 2.5 inch bays his NUC has and how much storage he needs.
I just looked back in the thread, he is running a BXNUC10i7FNH1. That supports both an M.2 SSD and a 2.5 inch SATA SSD. My advice would be to max out the storage in this with two large SSDs and see if that helps. Purchase from Amazon and if no improvement return them!
The M.2 is used for roon. I donât think it is available for data.
But yes, an internal 2TB SSD may work for the problem drive. It looks like there are +/- 65k tracks. At an average of 25MB per redbook flac that is about 1.6TB. If there is a lot of high rez then the sums may not work.
But personally i would just try reorganising that massive directory first. The other USB attached drive appears to work just fine. What is different is it is not a massive single directory.
Yes, as long as he narrowed the issue down to that specific directory and Roon is otherwise performant without that directory being watched then yes Iâd first go and reorganize that.
Remember, at the encouragement of the community here, I purchased a single 22 TB drive to replace the two drives I had been using. All files are now held there on that one drive. I have two directories on that drive - one âVariousâ and one âNamed.â Inside of the âVariousâ directory are sub-directories of each album by name. Inside of the âNamedâ directory are sub-directories of album artists, under which are sub-sub-directories of album names. If it is a multi-disk album, the separate disks are also under the album names as sub-directories. The âNamedâ directory is working fine now. I have disabled the search of the âVariousâ directory.
@Ronald Glad that you were able to narrow down the problem by deactivating the one folder causing issues. I had experienced the very same thing on a smaller scale with my 2 classical music folders.
Cannot offer a solution how to organize the ´Various´ folder to make the system run smoothly although. I would not be sure that solely the folder structure is to blame. The sheer number of interdependencies, connections between one artist and a lot of tracks or lots of artists with one track each - such things according to my experience bring Roon to exponentially increase the search activity and might bring an insignificant number of tracks pushing the system to its limits. And it is still not clear if it is the CPU power itself or the attempts to access too many things on the HDD is the true bottleneck in your case. Maybe there is a chance to monitor the CPU usage over time while performing searches.
I donât believe there is a way to ssh into ROCK and look at htop to see what is going on.
Apologies. Its a long thread and I missed that.
It does all seem to be pointing at your VA folder structure. You may want to consider restructuring it. There are no guarantees but there doesnât seem to be much to loose. What I would do is start by splitting it into 5 to 10 segments. Then add each segment as a separate storage location in roon and activate one by one. If the performance problem returns subdivide again.
Your big folder may have some kind of structure you could use as the basis for segmenting. Genre maybe? Or Decades? Alphabetic ranges would work just as well and maybe your only realistic option with such a large directory. So from your screen shot you could copy everything non-alpha into a sub-directory called â_Non Alphaâ (the underscore should force the directory to the top of a list followed by A, B, C etc.) Map that fist segment as a storage location in roon and activate it and see how you get on. It shouldnât take very long and if it doesnât work you will have at least ruled out the folder structure as the issue. If it does work you can carry on adding segments.
Because I do not like Roon 2.0 always on internet requirement. My internet is spotty with frequent outages. I tried ARC but I donât have a use for it. So, I decided to stay with 1.8. much better experience.
1 Like
Sounds like a reasonable plan. Iâm going to give it a try.
I ran Roon on such a desktop until I migrated to NUC more for convenience than anything else.
It was built by our local shop with a new SSD and nothing else, I added HDD from my old PC . It still covers my video needs and holds a duplicate of the Roon library managed by JRiver
it is quite old now i7 7700 / 16 gb / 256 SSD OS etc then 2x 4 Tb, 3 x 3TB HDD
It ran JRiver / Roon & SQL Server with no issues. the Roon library was around 200k but no messy folder structures
MY NUC/ROCK has just over 200k tracks and works fine (10i7)
Following tripleCrotchetTonyâs suggestion, I split my âvariousâ directory into many subdirectories and have pointed Roon storage to each of these smaller sub-directories separately. _AB / _CD / _EF, etc. (I found _ST to be very big, so I split it into _S and _T.) I disabled them all and have now begun to re-enable them one at a time as soon as Roon has added and identified all of the files in the previous subdirectory. You know what? It seems to be working. Thus far, everything in each sub-directory that I have re-enabled has been added and identified by Roon! (I have gotten through Non-Alpha, _AB, _CD, and _EF thus far.)
Did you clean up the library using the Settings>Library>´Clean up files that are not associatedâŚÂ´ option after renaming and resorting the folders?
If not Roon might still having all the metadata and interdependencies stored in the database trying to re-identify the very same tracks the moment they appear in renamed folders. Which means if your setup is running smoothly after resorting folders keeping the very same tracks we could for sure say it had been an issue of folder structure and the system had been pushing itself to the limits by monitoring this one folder.
But it is too early to draw conclusions. My experience is that the closer you bring the system to the limits the more you might end up in exponentially decreasing performance. The last folders you will be activating might make the difference.