Issue with Large Library Crashing After Recent Roon Updates (ref#XNI4EU)

BobG · April 21, 2024, 3:31am

Full form submission

What’s happening?

· Something else

How can we help?

· I am experiencing freezes or crashes

Describe the issue

Large Library: +2million tracks, crashing after ~2-3 hours following recent 2 Roon updates.

Lifetime subscriber since 2017, I've been running on ROCK with an Intel NUC 11 NUC11PAHi7 Panther Canyon Mini PC, i7-1165G7 w/32GB RAM that have built up to my current library size. As the library grew I become accustomed to having to reboot frequently, to overcome speed issues and in most cases to add additional tracks (including from title). At first when this problem occurred I did reinstall a back up to no luck. While I understand I have an edge case library here and likely at a point where I may have to upgrade hardware to accommodate, I'd like the technical team to evaluate a pattern I'm seeing in the log to pinpoint the issue and if upgraded hardware is needed, specifically what needs to to upgraded. This library took over 7years to build and while I have all locally well curated, it could take a month or more to rebuild from my experience.

I can share the patterns I have in the logs as well as other screen shots and observations. As I can only attach one screen shot here in this initial request, note that this is what occurs after the library loads and is working fine, then after 2-3 hours crashes. The is a definite pattern in the logs that are occurring each time which I can share

Marin_Weigel · April 21, 2024, 10:27am

Just another user, being of no immediate help to solve your current issue.

But please allow me to suggest breaking up your library into several much smaller watched folders, which you could enable/disable ad libidum to reduce complexity, corruption risk and resource constraints.

ged_hickman1 · April 21, 2024, 11:13am

@danny has said on the past that once rock/nucleus runs out memory it doesn’t degrade it just crashes. I wonder if the new performance improvements are using more memory to get that performance increase and so hitting your 32gb limit with 2M tracks?
If Roon don’t come back in a timely manner it might be worth investigating if you can buy 64gb on sale or return.
An extra $125 on top of buying 2m tracks…

BobG · April 21, 2024, 11:35pm

@ged_hickman1 thanks for your input. I agree with you that it appears the new performance improvements are creating a severe performance hit on RAM usage and is the likely cause as I can see from the logs and monitoring I’ve set up. Of course I would have no problem spending the $ to try more RAM but unfortunately the max supported by this NUC box I have all setup on is 32GB. As I said ^, I may have to upgrade the hardware but before I do, I really want to get ROON’s technical team to look a this before I spend the month or more it will take to migrate. It is not a matter of $ but time. And on the ROON side, they may find a way to improve upon what they are doing using by looking into this edge case.

BobG · April 21, 2024, 11:36pm

Thanks @Marin_Weigel. I already have my library broken up into 9 different volumes on my NAS and each artist and each album has its own folder. For those artists with more than 100 albums it is further broken down.

Robert_F · April 22, 2024, 1:15am

@BobG, how large is the internal drive on the NUC holding your Roon OS and database? A 7-year system containing 2M+ tracks also may be pushing the OS drive to its limits.

BobG · April 22, 2024, 1:37am

256GB but it is only using 60%.

ged_hickman1 · April 22, 2024, 7:18am

Every configurator I’ve looked at says it takes 64.

BobG · April 22, 2024, 9:58pm

@ged_hickman1 - My bad. In haste I looked up the model NUC from records I had at the office and pulled the wrong one. This is actually a NUC7i7BNH, not the NUC 11. The NUC 11s were for another customer project we have going. Unfortunately the RAM max is 32GB for the one I have here.

PS - Tried to edit my initial posting but not possible.

ged_hickman1 · April 22, 2024, 10:02pm

That would explain it then

BobG · April 22, 2024, 10:11pm

Here are some more observations noted with this recent release ROCK 2.0.33. When I first restart the server now, it goes through and scans all the media as it normally would in the past, but also noted it is trying to rescan my entire library for changes in metadata.

BobG · April 22, 2024, 10:45pm

Here’s a snapshot of a log showing the failure pattern. Note that it is NOT tied to these two albums (folders) shown in this example. The error occurs on random folders.

04/15 16:26:26 Trace: [clumping] processing clump with 12 tracks
04/15 16:26:26 Trace: [clumping] finished
04/15 16:26:26 Trace: [library/compute] computing 3 / 3 tracks
04/15 16:26:26 Trace: [dbperf] flush 470366 bytes, 239 ops in 1033 ms (cumulative 1185735614 bytes, 565013 ops in 5082762 ms)
04/15 16:26:26 Trace: [library] finished with 6488 dirty tracks 258 dirty albums 2196 dirty performers 2777 dirty works 4885 dirty performances 44 dirty genres 1 dirty tags 50 clumping tracks, 0 clumping auxfiles 3 compute tracks, 0 deleted tracks, 50 tracks to (re)load, 0 tracks to retain, 0 auxfiles to (re)load, 0 auxfiles to retain, and 10748 changed objects
04/15 16:26:26 Trace: [music/searchindex] [search-index]  removed in 0ms: 2 albums, 32 tracks, 15 works, 16 performers, 0 labels, 0 genres
04/15 16:26:26 Trace: [library] endmutation in 528ms
04/15 16:26:26 Trace: [metadatasvc] REQ [4584] https://metadataserver.roonlabs.net/md/4/updatemetadata?uid=82ea5ac1-5fd6-4b5c-a9e4-424625740f6f&lid=&token=5d1c146b-003f-4e8b-a266-474828b3e584&metadataid[]=610031393037313131383432393833&metadataid[]=610031393037313131383432393831&metadataid[]=610031393037313131383432393837&metadataid[]=610031393037313131383432393834&metadataid[]=610031393037313131383432393835&metadataid[]=610031393037313131383432393839&metadataid[]=610031393037313131383432393838&metadataid[]=610031393037313131383432393931&metadataid[]=610031393037313131383432393930&metadataid[]=610031393037313131383432393832&metadataid[]=610031393037313131383432393836&metadataid[]=6100313939313436313233333030353434&metadataid[]=6100313939313436313233333030353531&metadataid[]=6100313939313436313233333030353431&metadataid[]=6100313939313436313233333030353435&metadataid[]=6100313939313436313233333030353436&metadataid[]=6100313939313436313233333030353530&metadataid[]=6100313939313436313233333030353430&metadataid[]=6100313939313436313233333030353438&metadataid[]=6100313939313436313233333030353432&metadataid[]=6100313939313436313233333030353433&metadataid[]=6100313939313436313233333030353437&metadataid[]=6100313939313436313233333030353439&metadataid[]=7b004d5430303236363439363233&metadataid[]=7b004d5430303334313235353030&metadataid[]=7b004d5430303134353732323435&metadataid[]=7b004d5430303134353636373339&metadataid[]=7b004d5430303134353736323832&metadataid[]=7b004d5430303438313330383933&metadataid[]=7b004d5430303438313330383932&metadataid[]=7b004d5430303438313330383934&metadataid[]=7b004d5430303438313330383931&metadataid[]=7b004d5430303438313330383935&metadataid[]=7b004d5430303438313330383936&metadataid[]=7b004d5430303537373838313730&metadataid[]=7b004d5430303537373838313639&metadataid[]=7b004d5430303537373838313734&metadataid[]=7b004d5430303537373838313733&metadataid[]=7b004d5430303537373838313736&metadataid[]=7b004d5430303537373838313638&metadataid[]=7b004d5430303537373838313731&metadataid[]=7b004d5430303537373838313737&metadataid[]=7b004d5430303537373838313732&metadataid[]=7b004d5430303537373838313735&metadataid[]=6100313834303636343231343831343633&metadataid[]=6100313834303636343231343831343631&metadataid[]=6100313834303636343231343831343632&metadataid[]=7b004d5430303039383031323231&metadataid[]=7b004d5430303039383031323134&metadataid[]=7b004d5430303039383031323137&metadataid[]=7b004d5430303039383031323135&metadataid[]=7b004d5430303039383031323136&metadataid[]=7b004d5430303039383031323131&metadataid[]=7b004d5430303039383031323139&metadataid[]=7b004d5430303039383031323133&metadataid[]=7b004d5430303039383031323234&metadataid[]=7b004d5430303039383031323232&metadataid[]=7b004d5430303039383031323230&metadataid[]=7b004d5430303039383031323233&metadataid[]=7b004d5430303039383031323132&metadataid[]=7b004d5430303039383031323138&metadataid[]=79004d5730303032303336343736&metadataid[]=79004d5730303032313731303235&metadataid[]=79004d5730303030363434303339&metadataid[]=79004d5730303032353931313937&metadataid[]=79004d5730303033333831363036&metadataid[]=3e01b8cf46f342d7dd46a7df0d6fadfebd27&metadataid[]=79004d5730303030373938353132&tidal=max
04/15 16:26:26 Trace: [music/searchindex] [search-index] added in 24ms: 2 albums, 32 tracks, 15 works, 16 performers, 0 labels, 0 genres
04/15 16:26:26 Debug: [identification] <5356079> status: CouldNotIdentify
04/15 16:26:26 Trace: [music/searchindex] totals: 219536 albums, 2019223 tracks, 558493 works, 471615 performers, 25849 labels, 1315 genres
04/15 16:26:26 Trace: [identification] <2402607> Identifying album [Cappella - Tell Me the Way (Single)] with 5 tracks
04/15 16:26:26 Trace: [library] finished with 40 dirty tracks 1 dirty albums 1 dirty performers 1 dirty genres 0 clumping tracks, 0 clumping auxfiles 0 compute tracks, 0 deleted tracks, 40 tracks to (re)load, 0 tracks to retain, 0 auxfiles to (re)load, 0 auxfiles to retain, and 43 changed objects
04/15 16:26:26 Trace: [dbperf] flush 184581 bytes, 82 ops in 50 ms (cumulative 1185920195 bytes, 565095 ops in 5082812 ms)
04/15 16:26:26 Trace: [library] endmutation in 12ms
04/15 16:26:27 Trace: [metadatasvc] GOT [4584] (363ms) Sooloos.Msg.Metadata.UpdateMetadataResponse: 
04/15 16:26:27 Trace: [metadatasvc] REQ [4585] https://metadataserver.roonlabs.net/md/4/getmetadata?uid=82ea5ac1-5fd6-4b5c-a9e4-424625740f6f&lid=&token=5d1c146b-003f-4e8b-a266-474828b3e584&mode=2.0&metadataid[]=610031393037313131383432393833&metadataid[]=3e0124b0c39633b4a04ca1ce7684c86e73c9&metadataid[]=610031393037313131383432393831&metadataid[]=ae013dbb1a6c370d2c48841540adccb3927a&metadataid[]=610031393037313131383432393837&metadataid[]=ae014d8f3ef72009bf41ae2eb1fda09ab686&metadataid[]=610031393037313131383432393834&metadataid[]=610031393037313131383432393835&metadataid[]=ae014955c40b07351947933f3f1d74a7b1eb&metadataid[]=610031393037313131383432393839&metadataid[]=610031393037313131383432393838&metadataid[]=610031393037313131383432393931&metadataid[]=610031393037313131383432393930&metadataid[]=610031393037313131383432393832&metadataid[]=610031393037313131383432393836&metadataid[]=6100313939313436313233333030353434&metadataid[]=ae0136dd3501e3dd5b3283940a9faf6aa653&metadataid[]=3e0130a1edd679dcb84cb446d54a12a11792&metadataid[]=5e01ba30a72985f9cb47867ac48466514ca8&metadataid[]=7a004d4e30303030333437313838&metadataid[]=5e01d2bdad4cc3e7714ab5263bd4b15f7f7d&metadataid[]=5e01d1470cce93f4f3498c15b0118860812d&metadataid[]=7a004d4e30303030333431343037&metadataid[]=5e0182c11de61fff24439e3036ae678352e7&metadataid[]=5e01700d3c67e2c25d459943d857e5d6192b&metadataid[]=7a004d4e30303030333433333936&metadataid[]=5e019daf19eb58340246b909fc9563ac833c&metadataid[]=6100313939313436313233333030353531&metadataid[]=6100313939313436313233333030353431&metadataid[]=6100313939313436313233333030353435&metadataid[]=ae0171e27946c1b73133882e2a173f2001e4&metadataid[]=6100313939313436313233333030353436&metadataid[]=6100313939313436313233333030353530&metadataid[]=6100313939313436313233333030353430&metadataid[]=ae013e0ccdf057263f30bbe405e8c597a328&metadataid[]=6100313939313436313233333030353438&metadataid[]=ae0159d581aba6036637af0f2d41c6840cf6&metadataid[]=6100313939313436313233333030353432&metadataid[]=6100313939313436313233333030353433&metadataid[]=6100313939313436313233333030353437&metadataid[]=ae01904b82ea123dab478d9c44c0bff585e5&metadataid[]=6100313939313436313233333030353439&metadataid[]=7b004d5430303236363439363233&metadataid[]=ae019e8c0ffcd2b8b541a8fbe1a093e5092b&metadataid[]=79004d5730303030363434303339&metadataid[]=5e01f8db3decf36dd040a6d5f170fe556519&metadataid[]=7a004d4e30303030393437363137&metadataid[]=5e0122e52a7e9109e4418e0710cf8214cac4&metadataid[]=5e014aad7999887f5747a766257ec035e168&metadataid[]=7b004d5430303334313235353030&metadataid[]=ae0144a7ad5bb5d93d4b86820ea16b68b806&metadataid[]=7b004d5430303134353732323435&metadataid[]=ae01eab586f342b6fe4ab83f69b9d6b59e96&metadataid[]=7b004d5430303134353636373339&metadataid[]=ae01b7b9834c6120254ca354a27a24c0c70f&metadataid[]=7b004d5430303134353736323832&metadataid[]=ae01dcdbb811f5166b47b15cd9d588421259&metadataid[]=7b004d5430303438313330383933&metadataid[]=ae01443bb91fca033f48ac2f6de17cfa0579&metadataid[]=79004d5730303032353931313937&metadataid[]=5e01df55c034cbf8604a87708fd98a7e0e2a&metadataid[]=5e011708b180c3cd3948a8d0edbeb99aa8ea&metadataid[]=7a004d4e30303030313330393239&metadataid[]=5e01b76b00046d2a994d9db4e1c8240f01bd&metadataid[]=7a004d4e30303032303139373836&metadataid[]=7a004d4e30303030373335343036&metadataid[]=7a004d4e30303032313236323133&metadataid[]=5e0112324b9165e7624f96a211221d085d6b&metadataid[]=7a004d4e30303033363732373635&metadataid[]=5e01949aaafa75484f479229982f39618af8&metadataid[]=7b004d5430303438313330383932&metadataid[]=ae01ea134930f404874489c2b60e6ccf7bfe&metadataid[]=7b004d5430303438313330383934&metadataid[]=7b004d5430303438313330383931&metadataid[]=ae01239cf3613dc3a04d91cd6a0fa2a3d2dc&metadataid[]=7b004d5430303438313330383935&metadataid[]=7b004d5430303438313330383936&metadataid[]=7b004d5430303537373838313730&metadataid[]=5e01ed3f0ac3a749c4439973f3d661d2a2bd&metadataid[]=79004d5730303033333831363036&metadataid[]=7a004d4e30303031383537323633&metadataid[]=5e016cd1e38c729ef242a959d13fd823cc95&metadataid[]=7a004d4e30303030353233333938&metadataid[]=7b004d5430303537373838313639&metadataid[]=7b004d5430303537373838313734&metadataid[]=a3003030322d31363031363635&metadataid[]=7b004d5430303537373838313733&metadataid[]=7b004d5430303537373838313736&metadataid[]=7b004d5430303537373838313638&metadataid[]=7b004d5430303537373838313731&metadataid[]=7b004d5430303537373838313737&metadataid[]=7b004d5430303537373838313732&metadataid[]=7b004d5430303537373838313735&metadataid[]=6100313834303636343231343831343633&metadataid[]=3e01b8cf46f342d7dd46a7df0d6fadfebd27&metadataid[]=6100313834303636343231343831343631&metadataid[]=6100313834303636343231343831343632&metadataid[]=7b004d5430303039383031323231&metadataid[]=5e0191f8cd12d2656d4d99105ebe6c27c106&metadataid[]=5e01e0fc1b4c4de31644aca237dd5ccd9b97&tidal=max
04/15 16:26:27 Trace: [metadatasvc] REQ [4586] https://metadataserver.roonlabs.net/md/4/getmetadata?uid=82ea5ac1-5fd6-4b5c-a9e4-424625740f6f&lid=&token=5d1c146b-003f-4e8b-a266-474828b3e584&mode=2.0&metadataid[]=79004d5730303030373938353132&metadataid[]=5e01986f8af9138a4b48ad9421816b859ec0&metadataid[]=5e016411cca72f4dca4d98fe6d2f7730dd01&metadataid[]=7a004d4e30303031343131393937&metadataid[]=7b004d5430303039383031323134&metadataid[]=7b004d5430303039383031323137&metadataid[]=7b004d5430303039383031323135&metadataid[]=7b004d5430303039383031323136&metadataid[]=7b004d5430303039383031323131&metadataid[]=7b004d5430303039383031323139&metadataid[]=7b004d5430303039383031323133&metadataid[]=7b004d5430303039383031323234&metadataid[]=7b004d5430303039383031323232&metadataid[]=7b004d5430303039383031323230&metadataid[]=7b004d5430303039383031323233&metadataid[]=7b004d5430303039383031323132&metadataid[]=7b004d5430303039383031323138&metadataid[]=79004d5730303032303336343736&metadataid[]=79004d5730303032313731303235&tidal=max
04/15 16:26:27 Trace: [dbperf] flush 0 bytes, 0 ops in 88 ms (cumulative 1185920195 bytes, 565095 ops in 5082900 ms)
04/15 16:26:27 Trace: [storage] [directory] Cannot iterate dir: /roon/sys/storage/smbmounts/RoonStorage_8099d51c9e4dca8ecaa69a5afd8a5d33b4e2c118/Music_1/Cat Stevens/Tell 'Em I'm Gone
04/15 16:26:27 Trace: [storage] [directory] Cannot iterate dir: /roon/sys/storage/smbmounts/RoonStorage_7bf9fa313b1ab1b4fd61b0225d40d41f922c0aa3/Music_8/The High Llamas/Cold and Bouncy
04/15 16:26:27 Trace: [directorywatcher] Failed to enable events in FileSystemWatcher: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.IO.Enumeration.FileSystemEnumerator`1.InternalContinueOnError(ErrorInfo info, Boolean ignoreNotFound)
   at System.IO.Enumeration.FileSystemEnumerator`1.CreateDirectoryHandle(String path, Boolean ignoreNotFound)
   at System.IO.Enumeration.FileSystemEnumerator`1.Init()
   at System.IO.Enumeration.FileSystemEnumerableFactory.UserDirectories(String directory, String expression, EnumerationOptions options)
   at System.IO.Directory.InternalEnumeratePaths(String path, String searchPattern, SearchTarget searchTarget, EnumerationOptions options)
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance..ctor(FileSystemWatcher watcher, SafeFileHandle inotifyHandle, String directoryPath, Boolean includeSubdirectories, NotifyFilters notifyFilters, CancellationToken cancellationToken)
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
   at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value)
   at Roon.DirectoryWatcher.<_Start>b__30_0(Object <p0>)
04/15 16:26:27 Trace: [directorywatcher] Failed to enable events in FileSystemWatcher: System.IO.IOException: Too many open files : '/roon/sys/storage/smbmounts/RoonStorage_7bf9fa313b1ab1b4fd61b0225d40d41f922c0aa3/Music_7/Ian Wong/J5G4ZV~O'
   at System.IO.Enumeration.FileSystemEnumerator`1.CreateDirectoryHandle(String path, Boolean ignoreNotFound)
   at System.IO.Enumeration.FileSystemEnumerator`1.Init()
   at System.IO.Enumeration.FileSystemEnumerableFactory.UserDirectories(String directory, String expression, EnumerationOptions options)
   at System.IO.Directory.InternalEnumeratePaths(String path, String searchPattern, SearchTarget searchTarget, EnumerationOptions options)
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory parent, String directoryName)
   at System.IO.FileSystemWatcher.RunningInstance..ctor(FileSystemWatcher watcher, SafeFileHandle inotifyHandle, String directoryPath, Boolean includeSubdirectories, NotifyFilters notifyFilters, CancellationToken cancellationToken)
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
   at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value)
   at Roon.DirectoryWatcher.<_Start>b__30_0(Object <p0>)
04/15 16:26:28 Debug: [music/storage] processing simple delete for 67:0:Cat Stevens/Tell 'Em I'm Gone/10 Doors.flac from \Music_1
04/15 16:26:28 Debug: [music/storage]     LibraryTrack[591605554, Yusuf Islam, Yusuf, Cat Stevens - Tell 'Em I'm Gone - Doors IsAvailable=True IsDeleted=False ]
04/15 16:26:28 Debug: [music/storage] processing simple delete for 67:0:Cat Stevens/Tell 'Em I'm Gone/07 Gold Digger.flac from \Music_1
04/15 16:26:28 Debug: [music/storage]     LibraryTrack[591604786, Yusuf Islam, Yusuf, Cat Stevens - Tell 'Em I'm Gone - Gold Digger IsAvailable=True IsDeleted=False ]
04/15 16:26:28 Debug: [music/storage] processing simple delete for 67:0:Cat Stevens/Tell 'Em I'm Gone/02 Big Boss Man.flac from \Music_1
04/15 16:26:28 Debug: [music/storage]     LibraryTrack[591603506, Yusuf Islam, Yusuf, Cat Stevens - Tell 'Em I'm Gone - Big Boss Man IsAvailable=True IsDeleted=False ]
04/15 16:26:28 Debug: [music/storage] processing simple delete for 67:0:Cat Stevens/Tell 'Em I'm Gone/03 Dying to Live.flac from \Music_1
04/15 16:26:28 Debug: [music/storage]     LibraryTrack[591603762, Yusuf Islam, Yusuf, Cat Stevens - Tell 'Em I'm Gone - Dying to Live IsAvailable=True IsDeleted=False ]
04/15 16:26:28 Debug: [music/storage] processing simple delete for 67:0:Cat Stevens/Tell 'Em I'm Gone/09 Tell 'Em I'm Gone.flac from \Music_1
04/15 16:26:28 Debug: [music/storage]     LibraryTrack[591605298, Yusuf Islam, Yusuf, Cat Stevens - Tell 'Em I'm Gone - Tell 'Em I'm Gone IsAvailable=True IsDeleted=False ]
04/15 16:26:28 Debug: [music/storage] processing simple delete for 67:0:Cat Stevens/Tell 'Em I'm Gone/06 Cat & the Dog Trap.flac from \Music_1
04/15 16:26:28 Debug: [music/storage]     LibraryTrack[591604530, Yusuf Islam, Yusuf, Cat Stevens - Tell 'Em I'm Gone - Cat & the Dog Trap IsAvailable=True IsDeleted=False ]
04/15 16:26:28 Debug: [music/storage] processing simple delete for 67:0:Cat Stevens/Tell 'Em I'm Gone/04 You Are My Sunshine.flac from \Music_1
04/15 16:26:28 Debug: [music/storage]     LibraryTrack[591604018, Yusuf Islam, Yusuf, Cat Stevens - Tell 'Em I'm Gone - You Are My Sunshine IsAvailable=True IsDeleted=False ]
04/15 16:26:28 Debug: [music/storage] processing simple delete for 67:0:Cat Stevens/Tell 'Em I'm Gone/05 Editing Floor Blues.flac from \Music_1
04/15 16:26:28 Debug: [music/storage]     LibraryTrack[591604274, Yusuf Islam, Yusuf, Cat Stevens - Tell 'Em I'm Gone - Editing Floor Blues IsAvailable=True IsDeleted=False ]
04/15 16:26:28 Debug: [music/storage] processing simple delete for 67:0:Cat Stevens/Tell 'Em I'm Gone/01 I Was Raised in Babylon.flac from \Music_1
04/15 16:26:28 Debug: [music/storage]     LibraryTrack[591603250, Yusuf Islam, Yusuf, Cat Stevens - Tell 'Em I'm Gone - I Was Raised in Babylon IsAvailable=True IsDeleted=False ]
04/15 16:26:28 Debug: [music/storage] processing simple delete for 67:0:Cat Stevens/Tell 'Em I'm Gone/08 The Devil Came From Kansas.flac from \Music_1
04/15 16:26:28 Debug: [music/storage]     LibraryTrack[591605042, Yusuf Islam, Yusuf, Cat Stevens - Tell 'Em I'm Gone - The Devil Came From Kansas IsAvailable=True IsDeleted=False ]
04/15 16:26:28 Trace: [clumping] clumping 140 tracks and 0 auxfiles
04/15 16:26:28 Trace: [clumping] processing clump with 9 tracks
04/15 16:26:28 Trace: [clumping] processing clump with 9 tracks
04/15 16:26:28 Trace: [clumping] processing clump with 9 tracks
04/15 16:26:28 Trace: [clumping] processing clump with 11 tracks
04/15 16:26:28 Trace: [clumping] processing clump with 1 tracks
04/15 16:26:28 Trace: [clumping] processing clump with 12 tracks
04/15 16:26:28 Trace: [clumping] processing clump with 11 tracks
04/15 16:26:28 Trace: [clumping] processing clump with 11 tracks
04/15 16:26:28 Trace: [clumping] processing clump with 10 tracks
04/15 16:26:28 Trace: [clumping] processing clump with 10 tracks
04/15 16:26:28 Trace: [clumping] processing clump with 1 tracks
04/15 16:26:28 Trace: [clumping] processing clump with 20 tracks
04/15 16:26:28 Trace: [clumping] processing clump with 26 tracks
04/15 16:26:28 Trace: [clumping] finished
04/15 16:26:28 Trace: [library] finished with 10 dirty tracks 1 dirty albums 18 dirty performers 4 dirty works 4 dirty performances 1 dirty genres 0 clumping tracks, 0 clumping auxfiles 0 compute tracks, 10 deleted tracks, 10 tracks to (re)load, 0 tracks to retain, 0 auxfiles to (re)load, 0 auxfiles to retain, and 38 changed objects
04/15 16:26:28 Trace: [music/searchindex] [search-index]  removed in 0ms: 1 albums, 10 tracks, 0 works, 8 performers, 0 labels, 0 genres
04/15 16:26:28 Trace: [music/searchindex] [search-index] added in 0ms: 0 albums, 0 tracks, 0 works, 0 performers, 0 labels, 0 genres
04/15 16:26:28 Trace: [music/searchindex] totals: 219535 albums, 2019213 tracks, 558493 works, 471607 performers, 25849 labels, 1315 genres
04/15 16:26:28 Error: [broker/database] corruption detected: IO error: /roon/data/RoonServer/Database/Core/82ea5ac15fd64b5ca9e4424625740f6f/broker_4.db/91669164.ldb: Too many open files
04/15 16:26:28 Warn: [broker] detected corrupt database, notifying client
04/15 16:26:28 Warn: [broker] detected corrupt database, halting broker threads
04/15 16:26:28 Critical: Library.EndMutation: LevelDb.Exception: IO error: /roon/data/RoonServer/Database/Core/82ea5ac15fd64b5ca9e4424625740f6f/broker_4.db/91669164.ldb: Too many open files
   at LevelDb.Database._CheckError(IntPtr err)
   at LevelDb.Database.Write(WriteBatch batch)
   at LevelDb.Transaction.Commit(Boolean trace)
   at Sooloos.Broker.Music.MusicDatabase.Flush()
   at Sooloos.Broker.Music.Library.EndMutation()
04/15 16:26:28 Info: [loadstatus] IsDatabaseCorrupt False => True
04/15 16:26:28 Trace: [mobile] [remoteconnectivity] Port Verification started due to: load status changed, not testing port opening because the broker is not loaded and ready.  account status: LoggedIn, machine status: Licensed, load status: DatabaseCorrupt
04/15 16:26:28 Trace: [library] endmutation in 176ms
04/15 16:26:28 Debug: [music/storage] processing simple delete for 67:0:The High Llamas/Cold and Bouncy/14 Didball.flac from \Music_8
04/15 16:26:28 Debug: [music/storage]     LibraryTrack[941435442, The High Llamas - Cold and Bouncy - Didball IsAvailable=True IsDeleted=False ]
04/15 16:26:28 Debug: [music/storage] processing simple delete for 67:0:The High Llamas/Cold and Bouncy/05 Glide Time.flac from \Music_8
04/15 16:26:28 Debug: [music/storage]     LibraryTrack[941433138, The High Llamas - Cold and Bouncy - Glide Time IsAvailable=True IsDeleted=False ]
04/15 16:26:28 Debug: [music/storage] processing simple delete for 67:0:The High Llamas/Cold and Bouncy/01 Twisto Teck.flac from \Music_8
04/15 16:26:28 Debug: [music/storage]     LibraryTrack[941432114, The High Llamas - Cold and Bouncy - Twisto Teck IsAvailable=True IsDeleted=False ]
04/15 16:26:28 Debug: [music/storage] processing simple delete for 67:0:The High Llamas/Cold and Bouncy/16 Lobby Bears.flac from \Music_8
04/15 16:26:28 Debug: [music/storage]     LibraryTrack[941435954, The High Llamas - Cold and Bouncy - Lobby Bears IsAvailable=True IsDeleted=False ]
04/15 16:26:28 Debug: [music/storage] processing simple delete for 67:0:The High Llamas/Cold and Bouncy/06 Bouncy Glimmer.flac from \Music_8
04/15 16:26:28 Debug: [music/storage]     LibraryTrack[941433394, The High Llamas - Cold and Bouncy - Bouncy Glimmer IsAvailable=True IsDeleted=False ]
04/15 16:26:28 Debug: [music/storage] processing simple delete for 67:0:The High Llamas/Cold and Bouncy/08 Homespin Rerun.flac from \Music_8
04/15 16:26:28 Debug: [music/storage]     LibraryTrack[941433906, The High Llamas - Cold and Bouncy - Homespin Rerun IsAvailable=True IsDeleted=False ]
04/15 16:26:28 Debug: [music/storage] processing simple delete for 67:0:The High Llamas/Cold and Bouncy/09 Painters Paint.flac from \Music_8
04/15 16:26:28 Debug: [music/storage]     LibraryTrack[941434162, The High Llamas - Cold and Bouncy - Painters Paint IsAvailable=True IsDeleted=False ]
04/15 16:26:28 Debug: [music/storage] processing simple delete for 67:0:The High Llamas/Cold and Bouncy/12 Over the River.flac from \Music_8
04/15 16:26:28 Debug: [music/storage]     LibraryTrack[941434930, The High Llamas - Cold and Bouncy - Over the River IsAvailable=True IsDeleted=False ]
04/15 16:26:28 Debug: [music/storage] processing simple delete for 67:0:The High Llamas/Cold and Bouncy/10 Evergreen Vampo.flac from \Music_8
04/15 16:26:28 Debug: [music/storage]     LibraryTrack[941434418, The High Llamas - Cold and Bouncy - Evergreen Vampo IsAvailable=True IsDeleted=False ]
04/15 16:26:28 Debug: [music/storage] processing simple delete for 67:0:The High Llamas/Cold and Bouncy/11 Showstop Hip Hop.flac from \Music_8
04/15 16:26:28 Debug: [music/storage]     LibraryTrack[941434674, The High Llamas - Cold and Bouncy - Showstop Hip Hop IsAvailable=True IsDeleted=False ]
04/15 16:26:28 Debug: [music/storage] processing simple delete for 67:0:The High Llamas/Cold and Bouncy/13 End on Tick Tock.flac from \Music_8
04/15 16:26:28 Debug: [music/storage]     LibraryTrack[941435186, The High Llamas - Cold and Bouncy - End on Tick Tock IsAvailable=True IsDeleted=False ]
04/15 16:26:28 Debug: [music/storage] processing simple delete for 67:0:The High Llamas/Cold and Bouncy/15 Jazzed Carpenter.flac from \Music_8
04/15 16:26:28 Debug: [music/storage]     LibraryTrack[941435698, The High Llamas - Cold and Bouncy - Jazzed Carpenter IsAvailable=True IsDeleted=False ]
04/15 16:26:28 Debug: [music/storage] processing simple delete for 67:0:The High Llamas/Cold and Bouncy/04 Tilting Windmills.flac from \Music_8
04/15 16:26:28 Debug: [music/storage]     LibraryTrack[941432882, The High Llamas - Cold and Bouncy - Tilting Windmills IsAvailable=True IsDeleted=False ]
04/15 16:26:28 Debug: [music/storage] processing simple delete for 67:0:The High Llamas/Cold and Bouncy/02 The Sun Beats Down.flac from \Music_8
04/15 16:26:28 Debug: [music/storage]     LibraryTrack[941432626, The High Llamas - Cold and Bouncy - The Sun Beats Down IsAvailable=True IsDeleted=False ]
04/15 16:26:28 Debug: [music/storage] processing simple delete for 67:0:The High Llamas/Cold and Bouncy/03 Hi Ball Nova Scotia.flac from \Music_8
04/15 16:26:28 Debug: [music/storage]     LibraryTrack[941432370, The High Llamas - Cold and Bouncy - Hi Ball Nova Scotia IsAvailable=True IsDeleted=False ]
04/15 16:26:28 Debug: [music/storage] processing simple delete for 67:0:The High Llamas/Cold and Bouncy/07 Three Point Scrabble.flac from \Music_8
04/15 16:26:28 Debug: [music/storage]     LibraryTrack[941433650, The High Llamas - Cold and Bouncy - Three Point Scrabble IsAvailable=True IsDeleted=False ]
04/15 16:26:28 Trace: [clumping] clumping 143 tracks and 0 auxfiles
04/15 16:26:28 Trace: [clumping] processing clump with 29 tracks
04/15 16:26:28 Trace: [clumping] processing clump with 13 tracks
04/15 16:26:28 Trace: [clumping] processing clump with 2 tracks
04/15 16:26:28 Trace: [clumping] processing clump with 12 tracks
04/15 16:26:28 Trace: [clumping] processing clump with 1 tracks
04/15 16:26:28 Trace: [clumping] processing clump with 13 tracks
04/15 16:26:28 Trace: [clumping] processing clump with 13 tracks
04/15 16:26:28 Trace: [clumping] processing clump with 9 tracks
04/15 16:26:28 Trace: [clumping] processing clump with 8 tracks
04/15 16:26:28 Trace: [clumping] processing clump with 3 tracks
04/15 16:26:28 Trace: [clumping] processing clump with 12 tracks
04/15 16:26:28 Trace: [clumping] processing clump with 12 tracks
04/15 16:26:28 Trace: [clumping] processing clump with 16 tracks
04/15 16:26:28 Trace: [clumping] finished
04/15 16:26:28 Trace: [library] finished with 16 dirty tracks 1 dirty albums 2 dirty performers 1 dirty works 1 dirty performances 1 dirty genres 0 clumping tracks, 0 clumping auxfiles 0 compute tracks, 16 deleted tracks, 16 tracks to (re)load, 0 tracks to retain, 0 auxfiles to (re)load, 0 auxfiles to retain, and 22 changed objects
04/15 16:26:28 Trace: [music/searchindex] [search-index]  removed in 0ms: 1 albums, 16 tracks, 1 works, 0 performers, 0 labels, 0 genres
04/15 16:26:28 Trace: [music/searchindex] [search-index] added in 0ms: 0 albums, 0 tracks, 0 works, 0 performers, 0 labels, 0 genres
04/15 16:26:28 Trace: [music/searchindex] totals: 219534 albums, 2019197 tracks, 558492 works, 471607 performers, 25849 labels, 1315 genres
04/15 16:26:28 Error: [broker/database] corruption detected: IO error: /roon/data/RoonServer/Database/Core/82ea5ac15fd64b5ca9e4424625740f6f/broker_4.db/91669164.ldb: Too many open files
04/15 16:26:28 Warn: [broker] detected corrupt database, notifying client
04/15 16:26:28 Warn: [broker] detected corrupt database, halting broker threads
04/15 16:26:28 Critical: Library.EndMutation: LevelDb.Exception: IO error: /roon/data/RoonServer/Database/Core/82ea5ac15fd64b5ca9e4424625740f6f/broker_4.db/91669164.ldb: Too many open files
   at LevelDb.Database._CheckError(IntPtr err)
   at LevelDb.Database.Write(WriteBatch batch)
   at LevelDb.Transaction.Commit(Boolean trace)
   at Sooloos.Broker.Music.MusicDatabase.Flush()
   at Sooloos.Broker.Music.Library.EndMutation()

BobG · April 22, 2024, 11:05pm

Suggestion for DEV to consider for users with large libraries. Offer capability for Manual Scan of storage locations so we can disable the scan on Startup. This way we can scan location by location based on any changes made rather than trying to scan all locations at the same time. For us with large libraries, rebooting the server is often necessary due to performance issues. Process can take hours for larger libraries whereby the system performance is impacted. This also may be the cause of my problem here where it is trying to rescan the entire library with the recent update.

WiWavelength · April 23, 2024, 3:30am

Use the right tool for the job.

Any NUC7 or NUC11 for that matter is woefully underpowered for running a 2,000,000 track library. A laptop class processor is not the answer.

Get a desktop class i9 to handle that morbidly obese library.

AJ

ged_hickman1 · April 23, 2024, 5:57am

I doubt Roon will spend a lot of time investigating your use case as you are such an edge case. Probably one of a handful with such a large library and perhaps fewer trying to run it on inadequate hardware.
As @WiWavelength says, get a PC fit for purpose, a new desktop processor is twice as fast in single thread tasks, which is what you want for Roon.

Peter_Bruderer · April 23, 2024, 6:06am

If you look at your logs it is obvious. Your machine is out of memory. Add more RAM.

ged_hickman1 · April 23, 2024, 6:29am

He can’t as it’s an old nuc filled with it’s 32gb maximum.

BobG · April 23, 2024, 8:37am

Appreciate all the input on upgrading the hardware. Again, it’s not a matter of $ it’s a matter off the time required to reimport my entire library. When I first moved to Roon 7-years ago, I started off on a decked out iMac 5K at the time running 64GB. It began really slowing down and was a dog to bring back up when I had to reboot for software updates, etc, so decided to put a dedicated box to this with ROCK. Library was much smaller then but took a good month to import all with many a restart along the way to get the audio analysis unstuck. That said, at some point soon I will but will have to allocate a good couple months with my schedule to do this.
While the log shows the “Out of Memory Error”, which is what first caught my eye as well, it’s what leads up to this that’s quite troubling and possibly even taking this up to 64GB on the latest machine, will exhibit the same.
Issue is this crash started happening with the latest update to ROCK 2.0.33. The fact that it is trying to Rescan all my files as I posted above while at the same time scanning all my Volumes, is what is troubling. This is a behavior I’ve not seen in the past.
I’m willing to wait a bit more to have Roon Tech Support dig into the logs, in particular the pattern that seems to be consistent leading up to this crash and give me their assessment before I commit the time needed to migrate. Also, as Roon does not have the capability to test this ‘morbidly obese library’ as some would say, it does provide input on how to improve the overall performance as is the case on some of the latest releases.
Roon is still the best solution out there by far on the digital music side so I’m not going anywhere why this gets worked out. Plenty of LPs to listen to in the meantime.

Again, appreciate everyone’s input. Cheers!

tripleCrotchet · April 23, 2024, 9:01am

Whilst you are waiting for roon to respond have you tried disabling your volumes and then enabling them one by one in roon until the rescanning is complete?

Suedkiez · April 23, 2024, 11:10am

Are you sure? I looked up the NUC11PAHi7 online and found „max. 2x 32 GB“

Edit:

I see