Very slow importing content when database is using ROONALBUMTAG and ROONTRACKTAG

Custom PC, Threadripper 3970X 64-thread CPU, 64GB memory, Optane 905P Roon/database drive, 4x2TB RAID 0 SSD music folder, Windows 10 Pro x64 20H2, Roon 1.7 Build 170

Roon, database and music all on the same PC. True 1Gbps symmetrical fiber internet, local 10Gbps network. Internet speedtests consistently at ~900 down, ~700 up

Multiple hard-wired ethernet (no wi-fi) devices, primarily Mytek Manhattan II and Kef LS50 (USB to my PC).

Very slow creation of database

Hi there,

I did a major revamp of my music files, adding a lot of metadata (using Jaikoz), and decided to recreate my Roon database. Bad move.

My usual Roon server is a NUC i7, 8GB memory, Samsung NVMe 250GB primary drive, internal Samsung 4TB SSD, external USB 3.0 2TB SSD. It’s run well for three years. When I deleted the database and added the internal music folder, about 3.2TB, 130,000 FLAC tracks, the NUC added and identified the first 30,000 or so tracks quickly i.e. 2-3 per second. Then it progressively slowed down so that, at 50,000 tracks, it was doing 1-2 tracks per minute. It would take a month to finish.

I thought that it would be quicker to create the database on my custom PC, back up the database and then restore it on the NUC. The music files are mirrored on my PC and the NUC.

The PC is a very fast device, with a meaty processor, and the primary drive is an Intel Optane 905P, which is optimized for small non-sequential reads and writes. Although sequential rates are “only” 2GB/s or so, small random reads and writes are 2-3x faster than other, non-enterprise, SSDs. The music files are on 4x2TB SSD RAID 0 array, with very fast reads.

I started the database creation on the PC, pointed it at a local music folder of 130,000 tracks, 3.2TB, and the same issue is happening right now on the PC as on the NUC. It went very fast at first, but now, at about 65,000 tracks processed, it’s adding new ones to the database at about 10 per minute. Roon is only exercising one or two threads, disk activity on the Optane (Roon, datatabase) is less than 1%, 100kB/s or so. Disk activity on the music folder is negligible except for a quick 7-8MB/s spike once or twice a minute (when, I guess, Roon reads a new track), and internet access is also negligible. So, while Roon is pounding one thread (70-90%) most of the time, not much else is happening.

I’m wondering if the 1-2 threaded database adds are the factor slowing all of this down, or, maybe, Roon metadata lookups are throttled after a time? At this rate, assuming no more slowdowns, I’ve got 5-6 more days to go to build this database, and that’s before I add my other, 1TB, music folder. Your thoughts welcomed. Thanks!

It might be the analysis of the tracks themselves for dynamic range data. Have you tried adding more cores? You can up the count under settings>>library.

@killdozer thank you,

I turned off the audio analysis to prevent it from stealing disk read bandwidth from the database build. It didn’t seem to make a difference, nor did turning on 8, then 32 cores for audio analysis.

Based on my similar experience with two completely separate Roon servers (NUC, PC) with identical local music files, where everything slowed down in the low tens of thousands of tracks added, I suspect a systemic issue. Perhaps I shouldn’t expect more than 5-10 tracks added per minute in a very large database (no sarcasm intended).

My guess is that the database transactions become ever more demanding as the database size increases and it appears to be mostly, if not wholly, single-threaded. It’s not a flat database, as far as I can tell, but one with multiple tables and crosslinks. So, adding another track by, say, John Lennon, could easily requires writes to multiple tables, and reindexing. It may be the reindexing that’s slowing things down. Maybe a Roon person can comment and enlighten me?

Hmm. A thought: every track has tags in ROONALBUMTAG and ROONTRACKTAG. That’s the main difference between my old library and the new library. I wonder if indexing all those tags is slowing things down?

Hmm every track has added tags. Yep, my gut says that is probably your issue. Even if the initial scan wasnt slow, actually using those tags I bet will be crippling.

Read this thread,.

Hi @Alex_Osadzinski,

Yes, having these tags are likely what is causing the slow importing here, we have a ticket regarding this and I’ve just added your report to the ticket! For further details, please see my post here:

@Rugby and @noris thank your for your replies. At about the 70,000 track mark, importing ground to a halt, i.e. about 1 track per minute, so I gave up. I removed the ROONALBUMTAG and ROONTRACKTAG fields from all 130,000 tracks and am copying it all back to the NUC.

I hope that this issue gets resolved. The ROONALBUMTAG and ROONTRACKTAG fields were the perfect workaround for importing personal track ratings, and personal comments into Roon, but the performance issue has killed that avenue, at least for me.

1 Like

I can confirm that the problem was definitely the two Roon tag fields. Having removed them, my relatively puny NUC has added 37,000 tracks in just a few minutes, and is still adding over 50 tracks per second.

2 Likes

Hi @Alex_Osadzinski,

Thanks for the update here!

I spoke to the team about this one and we’re aware of this behavior and have added your report to the investigation ticket. I can’t comment on any timelines, but we have an active ticket regarding this so I have added your report to this ticket. Thanks again for the feedback here!

1 Like

Hi @noris, just wondering if this issue has been addressed before I try adding all my track ratings to ROONTRACKTAG again. Thanks!

Hello @Alex_Osadzinski ,

Yes, we have made some improvements here in our last release with regard to RoonTrackTag! Can you please let us know if it has helped in your case as well? Thanks!

Thank you. I’ll give it a try. It’s a fairly lengthy process but I’ll report back when I’m done.

1 Like

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.