Hi @Benjamin I’d really like to get this resolved as I’ve been trying to complete this project since November. Has the latest set of log files identified a way forward to fix this?
Hi @crom,
Thanks for your patience!
Unfortunately the only error indication we’re seeing on our end is similar to what you’ve mentioned above: out of memory. More specifically:
Critical: Library.EndMutation: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
One next step in testing could be to disable all your local storage, and create a new backup to attempt to migrate.
Another option could be to rename your current linux Roon Server database to RoonServer_Old, and manually copy+paste your windows Roon Server database folder into the subfolder location on your linux device.
Thanks @Benjamin I will give this a try.
@benjamin Is there any way to generate more detailed logs in order to identify what is causing the out of memory exception?
Hi @crom,
The only information available can be found by investigating your Roon Server logs:
Full form submission
What’s happening?
Something else
How can we help?
I am experiencing freezes or crashes
I am unable to complete a migration between windows and linux. I have tried on various builds and on 1388 I was able to backup on windows and restore on Linux. However, I am unable to make the linux version usable because as soon as I migrate the music director locations and scanning starts, the system crashes regularly. The scan does complete, however, but the result is an unstable system which is impossible to backup. Errors are the same as last time (memoryexception).
The linux system backs up and restores if I 'clean' the library of tracks but obviously this is not what I'm looking to achieve.
I am reaching the point where I need to consider wiping the db but before I do, I'd appreciate some help/advice on the implications of what I'm about to do and to see if there are any suggestions to avoid me having to do this.
As far as I can gather, this is not a hardware problem as I've tried completely different PCs to backup/migrate etc with the same result and my hunch (pure conjecture) is either that the errors are caused by the amount of data I'm trying to migrate...unlikely I guess, or number of files in each music share(12TB in one case...500k files or thereabouts)
I have 9 years of history (2016-2024) that I would love to be able to keep as history...not as a playlist and I have a huge amount of time invested in my library: tags, album curation, playlists etc. which I would also like to keep. These are my questions:
1) can I export/import my history (as history, not a playlist)?
2) If I migrate the DB, but leave the music directories as unavailable and then clean the library of 0.5million tracks that it can't find...then create new music directories...will the library update itself and keep my customisations and history...or will I lose the history/customisations that I've made?
3) I have tried to copy various files in the db structure to retain some things and can 'migrate' most of my UI settings across, but I can't find anyway of doing this for any of the actual data: history, playlists, etc Any pointers?
I am happy to supply logs and/or the entire db if a kindly Roon Developer wouldn't mind taking a look?
Thanks for any help on this!
Hello @crom ,
We have merged your new thread into the previous one. If you try to disable some of your storage locations prior to starting the backup (note - disable, not remove), does the backup succeed then?
If this still does not work, you may be able to copy the RoonServer/Database folder directly from one install to the next. Please note that this is not a “stable” way to copy Databases over, but this may work until we’ve had a chance to investigate the root cause of the issue.
Thanks Noris. The short answer to your first Q is: Yes. The backup and restore works perfectly under the latest build (I could not get this to complete when I tried using previous builds of RoonServer - I gave up after a while and the thread closed but have tried it a few times over the last few months and over the last week it worked). Roon under Linux loads perfectly for me because the storage locations are not found (obviously because they are in different places under linux) and the storage locations appear disabled under linux at the point of loading. What happens is that the instance loads just showing the tracks that I’ve added to the library from Qobuz substription (about 3000+). Instance works perfectly well and I can play Qobuz, view history, view playlists etc (albeit with my local library entries unavailable). The problems start when I point any of the storage locations at their new locations. I activated one (7Tb about 500k files) and during the scanning, the system crashes and restarts any number of times but does eventually finish. I left it going overnight and then tried to review the logs but the logs stop adding new log files at 20 and recycle so I lost the initial log files. Is there a way to stop the log filenames from being recycled?
If you would still like me to explicitly disable them at the point of backing up and run it again, please let me know and I will do this.
Hi Crom,
Are you still running Audiolinux for the new server? Have you tried running a standard linux distro like Ubuntu, or, even Windows 10 on the new machine, and restore your old Win 10 backup and see if it finishes re-scanning after you update the storage locations. If might be Audiolinux having an issue.
1 Like
Thanks @Rugby that’s a good thought. Under windows is fine - I can backup / restore to my heart’s content. TBH I haven’t tried on ubuntu or other distro. I’ll be home on Friday and will spin something up and try it out.
Hi @noris I am not making much progress, I’m afraid. A few things that I have eliminated:
I don’t believe it’s a hardware problem:
- Same hardware works perfectly under windows.
- I’ve tried an entirely different PC and I get exactly the same behaviour
I don’t believe it’s audiolinux specific:
- When I tried with alternative hardware I also tested with ubuntu (thanks @Rugby) and also audiolinux. Same result.
Things I have tried:
- migrating with storage locations enabled - errors as above.
- migrating with storage locations disabled - fine as described above.
- migrating with all storage locations deleted - works fine, as does backup following migration (it’s instantaneous because there’s nothing/little to backup).
- migrating with storage locations initially disabled and then enabled following migration - Enabling the storage locations causes regular crashes as described above but over a period of days the errors are worked through and the system becomes (more) stable. However, it is impossible to back up the library and Roon crashes a couple of times per night.
Bottom line is that under windows I have stable play/backup/restore abilities but that’s not possible under linux with my library.
I get 2 main types of errors in the logs. Either ‘too many files open’ or memory exception errors like below. The too many files error usually appears when files are being scanned/analysed/added to library. The memory exception error is generally triggered either after a number of ‘too many files open’ in the logs or during backup. The extract from the logs is following the trigger of a backup:
03/28 14:37:18 Info: [loadstatus] IsBackupInProgress False => True
03/28 14:37:18 Trace: [mobile] [remoteconnectivity] Port Verification started due to: load status changed, not testing port opening because the broker is not loaded and ready. account status: LoggedIn, machine status: Licensed, load status: BackupInProgress
03/28 14:37:18 Trace: [broker/backups] [74d1698f-9a62-4eaf-9b7c-60bbfc12c181] paused media thread
03/28 14:37:18 Trace: [zone Drawing room (roon ready)] Suspend
03/28 14:37:18 Info: [zone Drawing room (roon ready)] Canceling Pending Sleep
03/28 14:37:18 Trace: [leveldb] closing /var/roon/RoonServer/Database/Core/e60b9b1448154e50968d1f158fa1c5d5/transport/zone_16014e4e494c2600220f5b6f01541528013f.db temporarily
03/28 14:37:18 Trace: [broker/backups] [74d1698f-9a62-4eaf-9b7c-60bbfc12c181] paused transport thread
03/28 14:37:18 Trace: [leveldb] closing /var/roon/RoonServer/Database/Core/e60b9b1448154e50968d1f158fa1c5d5/broker_4.db temporarily
03/28 14:37:18 Info: [broker/database/vacuum] ==================================================================================================
03/28 14:37:18 Info: [broker/database/vacuum] ===[ Validating Database ]========================================================================
03/28 14:37:18 Info: [broker/database/vacuum] ==================================================================================================
03/28 14:37:18 Info: [broker/database/vacuum] Validating /var/roon/RoonServer/Database/Core/e60b9b1448154e50968d1f158fa1c5d5/broker_4.db
03/28 14:37:20 Info: [broker/database/vacuum] processed 134 playlist entries
03/28 14:37:20 Info: [broker/database/vacuum] processed 40362 profile entries (skipped 0 entries)
03/28 14:37:20 Info: [broker/database/vacuum] processed 1879 radio entries
03/28 14:37:30 Info: [stats] 23567mb Virtual, 5249mb Physical, 2766mb Managed, 348 Handles, 82 Threads
03/28 14:37:45 Info: [stats] 24763mb Virtual, 6369mb Physical, 2782mb Managed, 983 Handles, 86 Threads
03/28 14:38:00 Info: [stats] 24771mb Virtual, 6368mb Physical, 2815mb Managed, 1402 Handles, 86 Threads
03/28 14:38:15 Info: [stats] 24699mb Virtual, 6367mb Physical, 2808mb Managed, 1621 Handles, 74 Threads
03/28 14:38:25 Info: [broker/database/vacuum] processed 20885245 music entries
03/28 14:38:30 Info: [stats] 24699mb Virtual, 6368mb Physical, 2834mb Managed, 2527 Handles, 74 Threads
03/28 14:38:45 Info: [stats] 24723mb Virtual, 6368mb Physical, 2814mb Managed, 3332 Handles, 81 Threads
03/28 14:39:00 Info: [stats] 24731mb Virtual, 6366mb Physical, 2755mb Managed, 4131 Handles, 82 Threads
03/28 14:39:15 Info: [stats] 24707mb Virtual, 6366mb Physical, 2724mb Managed, 4966 Handles, 75 Threads
03/28 14:39:30 Info: [stats] 24683mb Virtual, 6366mb Physical, 2766mb Managed, 5831 Handles, 72 Threads
03/28 14:39:45 Info: [stats] 24803mb Virtual, 6367mb Physical, 2749mb Managed, 6694 Handles, 91 Threads
03/28 14:40:00 Info: [stats] 24803mb Virtual, 6367mb Physical, 2770mb Managed, 7545 Handles, 91 Threads
03/28 14:40:11 Warn: [broker] validation failed: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at LevelDb.KeySpace.GetEnumerator()+<>m__Finally1()
at LevelDb.KeySpace.GetEnumerator()+System.IDisposable.Dispose()
at LevelDb.KeySpace.GetEnumerator()+MoveNext()
at Sooloos.Broker.TinySooidDb.Vacuum(VacuumOperation op, KeySpace ks_from, KeySpace ks_to)
at Sooloos.Broker.VacuumOperation.Run()
at Sooloos.Broker.Database.Validate()
03/28 14:40:11 Error: [broker/database] corruption detected: Exception of type 'System.OutOfMemoryException' was thrown.
03/28 14:40:11 Warn: [broker] detected corrupt database, notifying client
03/28 14:40:11 Warn: [broker] detected corrupt database, halting broker threads
03/28 14:40:11 Trace: [broker/accounts] [heartbeat] now=3/28/2024 1:40:11PM nextauthrefresh=3/28/2024 2:02:43PM nextmachineallocate=3/28/2024 1:47:42PM
03/28 14:40:11 Info: [loadstatus] IsDatabaseCorrupt False => True
03/28 14:40:11 Trace: [mobile] [remoteconnectivity] Port Verification started due to: load status changed, not testing port opening because the broker is not loaded and ready. account status: LoggedIn, machine status: Licensed, load status: DatabaseCorrupt
03/28 14:40:11 Trace: [leveldb] re-opening /var/roon/RoonServer/Database/Core/e60b9b1448154e50968d1f158fa1c5d5/transport/zone_16014e4e494c2600220f5b6f01541528013f.db
03/28 14:40:11 Warn: [broker/backups] unexpected error doing backup on FileBrowser.Entry: / : /media/music/Audiolinux-backups/RoonBackups: Sooloos.SynchronizationContextThreadException: In Broker:Media
---> Sooloos.SynchronizationContextThreadException: In Broker:Transport
---> Sooloos.SynchronizationContextThreadException: In Broker:Misc
---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at LevelDb.KeySpace.GetEnumerator()+<>m__Finally1()
at LevelDb.KeySpace.GetEnumerator()+System.IDisposable.Dispose()
at LevelDb.KeySpace.GetEnumerator()+MoveNext()
at Sooloos.Broker.TinySooidDb.Vacuum(VacuumOperation op, KeySpace ks_from, KeySpace ks_to)
at Sooloos.Broker.VacuumOperation.Run()
at Sooloos.Broker.Database.Validate()
at Sooloos.Broker.BackupCompute.<>c__DisplayClass12_1.<ComputeBackupFiles>b__4()
at Sooloos.SynchronizationContextThread.<>c__DisplayClass48_0.<SendSafe>b__0()
--- End of inner exception stack trace ---
at Sooloos.SynchronizationContextThread.SendSafe(Action handler)
at Sooloos.Broker.BackupCompute.<>c__DisplayClass12_1.<ComputeBackupFiles>b__3()
at Sooloos.SynchronizationContextThread.<>c__DisplayClass48_0.<SendSafe>b__0()
--- End of inner exception stack trace ---
at Sooloos.SynchronizationContextThread.SendSafe(Action handler)
at Sooloos.Broker.BackupCompute.<>c__DisplayClass12_1.<ComputeBackupFiles>b__2()
at Sooloos.SynchronizationContextThread.<>c__DisplayClass48_0.<SendSafe>b__0()
--- End of inner exception stack trace ---
at Sooloos.SynchronizationContextThread.SendSafe(Action handler)
at Sooloos.Broker.BackupCompute.<>c__DisplayClass12_0.<ComputeBackupFiles>b__0()
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Sooloos.Broker.BackupCompute.ComputeBackupFiles(List`1 filesinmanifest, List`1 filestoupload, HashSet`1 latesthashes, Canceler canceler)
at Sooloos.Broker.BackupDestination`1.StartBackup(Entry location, Nullable`1 numberofbackupskept, Boolean auto)
03/28 14:40:11 Debug: [broker/backups] on done, auto: FalseThe reason I moved the Windows in the first place was that I lost my library about 4 times a year which we eventually tracked down to (from memory…) the mono libraries that were used at the time had limitations for large libraries. But this is no longer the case, I think?
So, my learnings so far: I if I keep the library under linux free from my local files, backups work fine and the system is stable. Clearly something is amiss with my local files…is that fair to assume? I have been working to this assumption over the last couple of days. Linux is generally far more forgiving of path lengths and characters but my issue is that investigating is really tricky as I can’t see what is triggering the errors in the logs.
Any ideas more than welcome.
Hey @crom ,
We’ve made a few performance improvements with our last release, is there by any chance any change in behavior on your end after updating?
Thanks @noris I’m travelling this week but will give new build a go, when I’m back home.
1 Like
A quick test to start with. I merely upgraded the server and control points and then tried a backup. Sadly before it even completed the snapshot it crashed out:
In the logs is the memory exception error that I’ve seen before.
I’d really like to get back to a stable system and as I said above I’m thinking of just wiping the DB but I’d like to keep my history and if there’s any way of saving db edits etc that would be appreciated - any ideas?
In other news, whilst I have been travelling, I have been cleansing my music, using Songkong. I fully appreciated the comment in the latest update blog about the madness of 50 different live bootlegs … and yes, my collection does stray into the ‘odd’ at points 
What I’m aiming for is an orderly folder/artist-based collection initially which should be id’d by music brains for at least part of it. I’ll use this to re-build part of the library. Then, there is the more difficult to catalogue stuff: remixes, dj mixes, etc etc and they are being rejected by Songkong at the moment. So, I won’t import them into the library for now and we’ll see if we get something that works.
I’ll keep backups of everything so I can backtrack to where I am currently if needs be.
Latest update here is that:
- I have backtracked to a backup made from Windows on 22nd March
- Restored this to Linux.
- The backup has all local storage areas disabled
- This runs and appears stable but backup fails with above error
- I have a chunk of my library (263k tracks) now ‘cleansed’ and have been testing activating a new storage location.
- At an individual level, the files appear to be being recognised as part of the original library (in that ‘this track is identical’ messages appear in the logs and some items of my history link up to these ‘cleansed’ tracks. big kudos to the developers for this achievement - it must be no easy task(!) to track individual tracks that have been moved and re-tagged and yet still identified correctly. Thank you!
- However, I’ve left this adding music process running for 2 nights running and I’ve looked in the morning and both times the entire PC is unresponsive (although still running). By unresponsive, I mean that typing anything in the command line elicits flashing cursor and no further response. IE the device’s resources are being fully used. Both times I’ve had no option other to hard-reset the device (power off/on) which has then triggered the adding music to restart. I have started this process again with a subset of the files to see if I can get something to complete. The log files are rotating about once every 8 minutes due to the large amount of data generated during the import process. I’m ashamed to say that I kicked off the re-import and left for work without taking a copy of the logs from last night.I’ll take a look if the same thing happens again.
I do feel that I am feeling my way around in the dark right now though…it would be great to get some more in depth assistance. I’ve seen other posts being offered ‘diagnostic mode’ and other levels of support…not sure why similar isn’t an option here?
Hi @noris, I wonder if this is something…
Related to the issues during import I reported in my previous post, I am wondering how memory is managed during the adding of tracks…Is the import managed in RAM, or written out to the db regularly? I’d assume the latter but I’m seeing a steadily increasing RAM usage during the import process and I’m wondering whether this is what caused the entire system to freeze in previous import attempts (see post above). CPU temp and usage remains relatively constant during the import (I’m at 190k out of 260k currently) but look at these data points on RAM:
- Up to about 140k tracks Roon appliance hovered around the 15%
- Up to about 160k tracks Roon appliance increased to about 21%
-
Up to about 180k tracks Roon appliance hovered around the 23%
-
About 182k tracks
~185k tracks
~191k tracks
I’m waiting with baited breath to see if I can get to 260k without running out of memory
Hi @crom,
Based on a recent diagnostic report, we’re seeing Roon get hung up on file tags that could potentially be originating from Song Kong, for example:
Info: GetOrCreateUserTag: there is an existing tag called Boxset: Collection
Info: GetOrCreateUserTag: there is an existing tag called Boxset: Kick 30
Info: GetOrCreateUserTag: there is an existing tag called Boxset: MTV Unplugged
Info: GetOrCreateUserTag: there is an existing tag called Boxset: Ikons
Info: GetOrCreateUserTag: there is an existing tag called Boxset: 100 Essential Tracks: 70s
Info: GetOrCreateUserTag: there is an existing tag called Boxset: XX Twenty Years
Info: GetOrCreateUserTag: there is an existing tag called Boxset: Sector 1
Are all your tracks recognized in Song Kong?
1 Like
Hi @Benjamin thanks for helping me with this - much appreciated.
I noticed many lines like this appearing in the logfiles generated recently. I’ve not seen similar before. I assumed (I guess incorrectly) that this was some additional tracing that had been added in the last Roon build.
I am in the process of running my entire library through Song Kong tagger to identify any corrupt files, add acoustic tags etc to files and to ensure that the directory structure was more logical. My library is extensive and has been built over many, many years - as I indicated above, the tagging is not always as consistent as it could be.
My process has been:
- Tag the files and group releases into separate folders
- Identify corrupt files
- Either fix or remove the corrupt files
- Move and rename the identified files and releases into a logical structure (group A)
- move those files etc that aren’t matched to a release into a ‘failed’ folder for a more manual approach.
For the past week (all the Roon log files generated since about 6th April) only contain those files from group A. I’ve set anything that wasn’t identified aside and not imported them into Roon.
As detailed in a recent post above I am using a previous backup of Roon where all the storage is disabled and in the log files I have been seeing many examples where Roon is identifying identical tracks between the old library and the ‘cleaned’ library.
I hope that gives more context to this. I’ll contact the SK developer to see if he is able to input.