Ubuntu 20.04 LTS, dual Ethernet, 24 GB RAM, 512 GB SSD boot, 20 TB SATA local storage
Roon v.2.0 (build 1244) Production
Networking Gear & Setup Details
N/A
Connected Audio Devices
N/A
Number of Tracks in Library
1.06 million, mostly FLAC
TIDAL and Qobuz services enabled
Description of Issue
The system had its database rebuilt manually (previous was archived and server restarted) in an attempt to re-enable ARC. Upon completion of the rebuild no backups have succeeded. Neither scheduled nor manually run backups complete successfully.
The logs show for a manual backup:
04/06 18:50:35 Warn: [broker/backups] unexpected error doing backup on FileBrowser.Entry: bak, HGST HTS721010A9E630 : /RoonBackups_Manual/RoonBackups: Sooloos.SynchronizationContextThreadException: In Broker:Media
—> Sooloos.SynchronizationContextThreadException: In Broker:Transport
—> Sooloos.SynchronizationContextThreadException: In Broker:Misc
—> System.OutOfMemoryException: Array dimensions exceeded supported range.
at System.IO.MemoryStream.set_Capacity(Int32 value)
at System.IO.MemoryStream.EnsureCapacity(Int32 value)
at System.IO.MemoryStream.WriteByte(Byte value)
at Sooloos.Broker.BackupCompute.<>c__DisplayClass12_1.b__1(String file, Boolean needs_copy)
at Sooloos.Broker.BackupCompute.<>c__DisplayClass12_1.b__4()
at Sooloos.SynchronizationContextThread.<>c__DisplayClass48_0.b__0()
— End of inner exception stack trace —
at Sooloos.SynchronizationContextThread.SendSafe(Action handler)
at Sooloos.Broker.BackupCompute.<>c__DisplayClass12_1.b__3()
at Sooloos.SynchronizationContextThread.<>c__DisplayClass48_0.b__0()
— End of inner exception stack trace —
at Sooloos.SynchronizationContextThread.SendSafe(Action handler)
at Sooloos.Broker.BackupCompute.<>c__DisplayClass12_1.b__2()
at Sooloos.SynchronizationContextThread.<>c__DisplayClass48_0.b__0()
— End of inner exception stack trace —
at Sooloos.SynchronizationContextThread.SendSafe(Action handler)
at Sooloos.Broker.BackupCompute.<>c__DisplayClass12_0.b__0()
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
— End of stack trace from previous location —
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
— End of stack trace from previous location —
at Sooloos.Broker.BackupCompute.ComputeBackupFiles(List1 filesinmanifest, List1 filestoupload, HashSet1 latesthashes, Canceler canceler) at Sooloos.Broker.BackupDestination1.StartBackup(Entry location, Nullable`1 numberofbackupskept, Boolean auto)
The error reported in the Roon app is: " Error backing up database."
The backup device is an external USB drive. The directory Roonbackups_Manual is newly created for tests and has read/write access.
For the time being backups are being performed manually using rsync.
Don’t understand what exactly you mean with ‘the system had its database rebuilt manually’, but with above 1 million tracks you manage a gigantic collection and I wonder if you simply run out of memory… 24 GB isn’t that much for such a large collection.
Have you tried to monitor the system’s RAM usage during a manual backup run?
I’d consider upgrading the machine’s RAM to 2x32GB, if possible.
During normal operation the server only uses about half its memory but your suggestion is well taken. I’ll watch RAM usage during a backup and report back here.
RAM usage does indeed increase a bit during backup. At the start the system is using 10.3 GB which remains stable during the Snapshotting phase. It then jumps to 14.3 GB when the backup halts with an error.
Thank you for reaching out and apologies for the delayed response here.
I checked with the dev team and this kind of error can occur when one part of your Roon backup is too large (in terms of file size) to be created. It is not the PC running out of memory, but rather some part of the database is too large to have a snapshot created of it for the backup.
At a library size of a million tracks, it is certainly larger than 99% of other Roon users, and this means that it can behave a bit differently than other databases. Your report is unique since this is actually the first Out of Memory of this type that I’ve seen so far. As for the next steps here, can you please:
Access your current RoonServer database and upload it to the below link so that QA can test the database for this error? The below guide goes over how to access this, though you may need to split your database into a few .zip/rar archives if it is too large for the uploader.
After uploading the current database, please ensure that you have a manual backup of the database made using rsync, connect all your desired storage locations, and then perform a Roon Settings → Library → Clean Up Library function.
Note, it is very important that you have all of your database active at this time (if you have any disabled storage folders, these should be enabled). What this action does is clean up any old database links to files that have been moved, deleted, or are no longer necessary.
After performing a clean-up library, please try to see if Backups within Roon start to work again.
A copy of the RoonServer directory and Database have been uploaded as requested.
A Library Cleanup was run as requested but prior to the Cleanup the Roon application requested an update to the Roon database. After the database upgrade an application upgrade was available. The application update was applied.
The Library Cleanup went through the following process:
“Snapshotting databases for backup…”
“Waiting for your Roon Core…”
“Uh oh, something’s not right”
[This is normal. Waiting…]
[Back to Settings > Backups screen, Spinner]
Backup status, Preparing…
Transferring… X%
[This is the first time ‘Transferring X%’ has appeared]
Backup succeeded!
Remote: Roon version 2.0 (build 1272) production *
Core: Roon version 2.0 (build 1272) production
Thanks for sending the database over and glad to hear that performing the Library Cleanup allowed the Roon backup to work!
I’ve put in a ticket for QA/devs to try to see if we can improve the behavior when performing backups on large databases, but in the meantime, I will go ahead and mark this issue as solved.
Please, do let us know if you run into any other backup issues and we can take another look, thanks!