Corruption on ROCK/Archlinux on same NUC

Hi @JempeZen,.

If you are getting recurring corruption, I would check the hardware of the ROCK, possibly perform a SSD check and a RAM test via memtest64 as it’s possible you have failing hardware. Do let us know if you require further assistance after making the OS change.

Not a hardware issue, i have make a lot of test, i have tested the ssd/memory/cpu
The problem is a roon issue and i not the only on with this problem
That’s a line i have found in the log:

Critical: Library.EndMutation: System.OverflowException: Arithmetic operation resulted in an overflow.

look at this
https://community.roonlabs.com/t/database-corrupted-at-each-restart/114901/8

1 Like

Core Machine (Operating system/System info/Roon build number)

NUC8i3 with archlinux kernel 5.7.9-arch1-1 with Roon server build 571

Network Details (Including networking gear model/manufacturer and if on WiFi/Ethernet)

Lan and Wifi nothing special, the Nuc is on ethernet

Audio Devices (Specify what device you’re using and its connection type - USB/HDMI/etc.)

Allo digione with raspberry on ethernet

Description Of Issue

There was no issue at the beginning for some month, my system was the same but with Rock, not archlinux.
After an update (i think 2 or 3 update from now) i have my first database issue.
I have loaded an backup and it’s was ok. But i could not take the last backup, after the restore there was no playlist, and no album… So i have to find an old one that was not corrupted. This happen some times so i switched to archlinux. But again my database got corrupted.
I have make a lot of memtest but i have never an error.
I have check my nvme samsung 970 evo with magician software, again no error…

I have now make some test and what i have found that the problem is comming at every restart of the roon service.

I have installed roon with the aur package.
https://aur.archlinux.org/packages/roonserver/

I restart the service with “systemctl restart roonserver.service

Please help my system is unusable, i lost every time so much change on my playlist and other things…

Here is some log with critical and warning parts:

    07/20 23:24:23 Critical: Library.EndMutation: System.OverflowException: Arithmetic operation resulted in an overflow.
  at (wrapper alloc) System.Object.AllocVector(intptr,intptr)
  at FSE.MyBinaryArrayReader.ReadBytes (System.Int32 count) [0x00000] in <bcc476b068974b54b8aeef25f4f0facc>:0 
  at FSE.BinaryReader.get_Sooid () [0x0003a] in <bcc476b068974b54b8aeef25f4f0facc>:0 
  at Messaging.FSEMessageDecoder.DecodeOneMessage (IMessage m, System.Int32& requestId, System.Boolean& isFinal) [0x002d4] in <bcc476b068974b54b8aeef25f4f0facc>:0 
  at Messaging.FSEMessageDecoder.DecodeOneMessage (IMessage m, System.Int32& requestId, System.Boolean& isFinal) [0x000c0] in <bcc476b068974b54b8aeef25f4f0facc>:0 
  at Messaging.FSEMessageDecoder.DecodeCommon (System.Int32& requestId, System.Boolean& isFinal) [0x00066] in <bcc476b068974b54b8aeef25f4f0facc>:0 
  at Messaging.FSEMessageDecoder.DecodeMessage () [0x00000] in <bcc476b068974b54b8aeef25f4f0facc>:0 
  at Messaging.FSEMessageDecodeSession.DecodeBinaryMessage (Base.ByteBuffer s) [0x0003d] in <bcc476b068974b54b8aeef25f4f0facc>:0 
  at Messaging.FSEMessageHelper.DecodeBinaryMessage (Base.ByteBuffer s) [0x00011] in <bcc476b068974b54b8aeef25f4f0facc>:0 
  at Sooloos.Broker.Music.MusicDatabase._TryGetValue[T] (System.Object traceid, System.Byte[] key, T& msg) [0x00010] in <5ed9ab2915684a1ca1d9d54b789bb67d>:0 
  at Sooloos.Broker.Music.MusicDatabase+TransactionCache`3[LOCAL,METADATA,OVERLAY].TryGetValue (System.Sooid id, LOCAL& msg, System.Boolean cache) [0x00051] in <5ed9ab2915684a1ca1d9d54b789bb67d>:0 
  at Sooloos.Broker.Music.MusicDatabase.TryGetValue (System.Sooid trackid, MusicDb.LocalTrack& msg, System.Boolean cache) [0x00000] in <5ed9ab2915684a1ca1d9d54b789bb67d>:0 
  at Sooloos.Broker.Music.Library+ComputeTrack.get_LocalTrack () [0x00028] in <5ed9ab2915684a1ca1d9d54b789bb67d>:0 
  at Sooloos.Broker.Music.Library.PerformWorkPartMunge (Sooloos.Broker.Music.Library+ComputeEnv env, System.Collections.Generic.List`1[T] album_tracks, System.Collections.Generic.IDictionary`2[TKey,TValue] load_tracklitedata) [0x0040d] in <5ed9ab2915684a1ca1d9d54b789bb67d>:0 
  at Sooloos.Broker.Music.Library.Compute (System.Collections.Generic.IList`1[T] tracks, System.Collections.Generic.IDictionary`2[TKey,TValue] load_tracklitedata) [0x001cf] in <5ed9ab2915684a1ca1d9d54b789bb67d>:0 
  at Sooloos.Broker.Music.LibraryMutationEnv.Finish () [0x00b9a] in <5ed9ab2915684a1ca1d9d54b789bb67d>:0 
  at Sooloos.Broker.Music.Library.EndMutation () [0x00068] in <5ed9ab2915684a1ca1d9d54b789bb67d>:0 
07/20 23:24:23 Trace: [library] endmutation in 5919ms
07/20 23:24:23 Info: [music] first pass through media thread is done
07/20 23:24:23 Info: [loadstatus] IsLibraryLoadFailed False => True
07/20 23:24:23 Info: [loadstatus] IsLibraryLoading True => False
07/20 23:24:23 Info: [transport] initializing
07/20 23:24:23 Info: [loadstatus] IsTransportLoading True => False
07/20 23:24:23 Info: [loadstatus] IsStartup True => False
07/20 23:24:23 Info: [roonapi] initializing
07/20 23:24:23 Info: [roonapi] listening on port 9150
07/20 23:24:25 Warn: [linuxmount/pinfo] coudn't ascertain disk block device from sys
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /proc because it is not a /dev/sd[0-9]* (mountline: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /sys because it is not a /dev/sd[0-9]* (mountline: sys /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /dev because it is not a /dev/sd[0-9]* (mountline: dev /dev devtmpfs rw,nosuid,relatime,size=3999680k,nr_inodes=999920,mode=755 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /run because it is not a /dev/sd[0-9]* (mountline: run /run tmpfs rw,nosuid,nodev,relatime,mode=755 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /sys/kernel/security because it is not a /dev/sd[0-9]* (mountline: securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /dev/shm because it is not a /dev/sd[0-9]* (mountline: tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /dev/pts because it is not a /dev/sd[0-9]* (mountline: devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /sys/fs/cgroup because it is not a /dev/sd[0-9]* (mountline: tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /sys/fs/cgroup/unified because it is not a /dev/sd[0-9]* (mountline: cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /sys/fs/cgroup/systemd because it is not a /dev/sd[0-9]* (mountline: cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /sys/fs/pstore because it is not a /dev/sd[0-9]* (mountline: pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /sys/fs/bpf because it is not a /dev/sd[0-9]* (mountline: none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /sys/fs/cgroup/blkio because it is not a /dev/sd[0-9]* (mountline: cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /sys/fs/cgroup/net_cls,net_prio because it is not a /dev/sd[0-9]* (mountline: cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /sys/fs/cgroup/hugetlb because it is not a /dev/sd[0-9]* (mountline: cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /sys/fs/cgroup/memory because it is not a /dev/sd[0-9]* (mountline: cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /sys/fs/cgroup/cpu,cpuacct because it is not a /dev/sd[0-9]* (mountline: cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /sys/fs/cgroup/freezer because it is not a /dev/sd[0-9]* (mountline: cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /sys/fs/cgroup/pids because it is not a /dev/sd[0-9]* (mountline: cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /sys/fs/cgroup/devices because it is not a /dev/sd[0-9]* (mountline: cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /sys/fs/cgroup/perf_event because it is not a /dev/sd[0-9]* (mountline: cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /sys/fs/cgroup/cpuset because it is not a /dev/sd[0-9]* (mountline: cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /sys/fs/cgroup/rdma because it is not a /dev/sd[0-9]* (mountline: cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /proc/sys/fs/binfmt_misc because it is not a /dev/sd[0-9]* (mountline: systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2623 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /dev/hugepages because it is not a /dev/sd[0-9]* (mountline: hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /dev/mqueue because it is not a /dev/sd[0-9]* (mountline: mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /sys/kernel/debug because it is not a /dev/sd[0-9]* (mountline: debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /sys/kernel/tracing because it is not a /dev/sd[0-9]* (mountline: tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /tmp because it is not a /dev/sd[0-9]* (mountline: tmpfs /tmp tmpfs rw,nosuid,nodev 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /sys/kernel/config because it is not a /dev/sd[0-9]* (mountline: configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /mnt/RoonStorage_91acd38a10e5dc49f5c5295fbbf5889313c93497 because it is not a /dev/sd[0-9]* (mountline: //172.16.42.4/Music /mnt/RoonStorage_91acd38a10e5dc49f5c5295fbbf5889313c93497 cifs rw,relatime,vers=2.1,cache=strict,username=roon,domain=WORKGROUP,uid=0,noforceuid,gid=0,noforcegid,addr=172.16.42.4,file_mode=0755,dir_mode=0755,soft,nounix,mapposix,rsize=1048576,wsize=1048576,bsize=1048576,echo_interval=60,actimeo=1 0 0)
07/20 23:24:27 Debug: [broker/filebrowser/volumeattached] skipping /run/user/1000 because it is not a /dev/sd[0-9]* (mountline: tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=801472k,mode=700,uid=1000,gid=1000 0 0)
07/20 23:24:28 Warn: [linuxmount/pinfo] coudn't ascertain disk block device from dev
07/20 23:24:28 Warn: [linuxmount/pinfo] coudn't ascertain disk block device from run
07/20 23:24:28 Warn: [linuxmount/pinfo] coudn't ascertain disk block device from securityfs
07/20 23:24:28 Warn: [linuxmount/pinfo] coudn't ascertain disk block device from devpts
07/20 23:24:28 Info: [stats] 2578mb Virtual, 479mb Physical, 220mb Managed, 0 Handles, 40 Threads
07/20 23:24:28 Warn: [linuxmount/pinfo] coudn't ascertain disk block device from cgroup2
07/20 23:24:28 Warn: [linuxmount/pinfo] coudn't ascertain disk block device from pstore



07/20 23:24:28 Warn: [linuxmount/pinfo] coudn't ascertain disk block device from systemd-1
07/20 23:24:28 Warn: [linuxmount/pinfo] coudn't ascertain disk block device from hugetlbfs
07/20 23:24:28 Warn: [linuxmount/pinfo] coudn't ascertain disk block device from mqueue
07/20 23:24:28 Warn: [linuxmount/pinfo] coudn't ascertain disk block device from debugfs
07/20 23:24:28 Warn: [linuxmount/pinfo] coudn't ascertain disk block device from tracefs
07/20 23:24:28 Warn: [linuxmount/pinfo] coudn't ascertain disk block device from configfs

I have tested to restore a lot of backup, from all my backup and i have a lot only 3 are valid… That’s not cool and this doesn’t give me trust on the system.
Why Roon don’t check the integrity of the database before a backup? How it’s possible that roon backup an corrupted database and erase the good one? Roon accept only 99 backup after that the old one are erased.
I realize that it’s possible to add/modify playlist and more on an database that is already not possible to restore. That’s a big problem!
Please give me support to solve this and edit code that prevent this situation.
Thank’s

I have found something.
My last good backup that i can restore is from 8 June. If i restore this database and restart immediately after the restore is done all is ok. But roon start to scan my new files after that and there is 1476 new files since this backup. After roon have finished to scan all files (it’s take time) i can’t restart roon, if i do there is an error and i lose the database.
But before i restart, roon don’t complain at all… I can modify and add playlist, change tag, artist/album picture without any complain from roon. But the database can not be restored… It’s corrupted.
I think there is a files somewhere that cause this issue. But how to find it?
I have a lot of different folder and i have absolutely no idea which files cause this issue.
With this new information I’m sure you will have an idea to help me, and maybe learn from this issue to make roon more robust.

Can’t help you brother, except to say I been there and will continue to post about it -

Doubtful.

1 Like

I have removed some files but no change… short after restore i can restart without any problem, but after some minutes i can’t restart roon without losing all my playlist!
Something really strange, i can see all my directory but force re-scan does nothing… And if i go to album roon say me that my folder are offline! I see Tidal suggestion but no tidale playlist. And there is no option to reset or repair… nothing! Did i have to uninstall and go from zero??? Roon is really expensive i’m really disappointed, i can’t recommend roon anymore to anyone. My trust to roon go slowly to zero…

Why not backup every night?

I backup every night it’s not the problem.
The problem is that Roon library has ‘latent corruption’ issue so the backup are corrupted! And there is no warning nothing, slowly the good backup (the old one) are deleted…
That’s a big issue!!!

1 Like

I guess it depends on how many backups you keep and when the corruption first happened. It would be a huge problem finding the last good backup if it happened a while ago. I keep 20 backups. I’ve only used one.

I slowly understand that i have lost all my playlist and tunning on the library.
I have a valid backup but maybe it’s not really valid, this backup can be restored but the futur backup will be corrupted…

Maybe you need to totally delete Roon and start over with a clean install and restore from your last good backup.

Only 20 backups? :smirk:

My schedule -

Really doesn’t address Roon backing up corrupted database. It just means, in the worse case, I have so many more corrupted backups.

OTOH, my new scheme gives a year of quarterly backups among others, so I’m probably safe from a reoccurrence of having to start my Roon library from scratch.

Still, doesn’t excuse Roon’s short comings in this regard.

If he can find a good backup, that is.

1 Like

If not, he may just need to start over with a fresh install.

Aye, and that’s the pain of it.

1 Like

I have backup from my backup but this is not the problem. I’m on the way to copy my old backup from backup and i will see how many week/month i have lost. I use borgbackup, i will change my script to keep the old backup longer…

It’s a big roon issue!
No one can trust the current library with this issue. You take a long week-end to play good music on an cold winter week, tuning the picture from artist, making, changing some playlist. And some day later, an update is coming, roonserver is restarted and the problem appear. Your system is broken… Ok no problem you have some backup that roon make for you, but the backup are corrupted!? You don’t have time this day to resolve this, you have to go for some days. When you finally have the time you slowly understand that roon have an issue, and maybe during the last day roon have deleted the last good backup.
You have lost week/month/year of you library tuning!

Some important update
If you can restore a backup this will not say that the db has no ‘latent corruption’…
In my case, i have a backup that i have restored a couple of time from my first db problem. There was no sign of any problem, i could modify/change what i want and roon never give a sign of any issue. But… This backup was corrupted but still can be restored, but all auto backup from this time was totally corrupted and could not be restored. And if roonserver have to be restarted this will fail!
After the restore is completed roon start to analyse new file and from this time any new backup will fail and any restart will fail!

This the main reason i have lost so much. There is absolutely no sign that there is a problem in background!

I have finaly found a backup that don’t suffer from this ‘latent corruption’ ■■■■■■ issue. I have take this backup from a double backup system that i have.
This backup is from 31 may!!! So i have lost a lot! That’s not cool.

Please i wait to have an official support for this issue.
I will periodically maintain this post updated so google can index this…
I hope there will be soon a patch released from the roon team.
I can provide 2 type of db, one with a ‘latent corruption’ (still can be restored but the backup after will be full corrupted and an other (the day after) totally corrupted and this one in not possible to restore.

On the waiting time, if the support don’t give any sign of interest i will make some buzz.
But i really like roon so for i trust that the roon team will take this seriously

Thank you

Hi @JempeZen,

I’ve consolidated your two threads so we can have all the information in one place.

What is the size of your database on the ROCK? How many tracks/albums do you have?

If the issue only occurs after you add more files to the library, I wonder if your Core might be running out of resources to process a library that is too large.

Ok i understand but you still go to the hardward issue and i know that’s not the case.
Archlinux have the same problem only while the backup i have used was corrupted but still possible to restore.
No there is no hardward issue read my post before please.

I have 56000 track and 4800 album, it’s not so much
8 giga memory and a good i3@3ghz

[root@rooneserver Database]# ls -lah
total 28K
drwxr-xr-x 7 root root 4.0K Jul 21 20:03 .
drwxr-xr-x 7 root root 4.0K Jul 21 20:36 …
drwxr-xr-x 2 root root 4.0K Jul 21 20:03 Cache
drwxr-xr-x 3 root root 4.0K Jul 21 20:03 Core
drwxr-xr-x 2 root root 4.0K Jul 21 20:03 Cx
drwxr-xr-x 3 root root 4.0K Jul 21 20:37 Orbit
drwxr-xr-x 6 root root 4.0K Jul 21 20:01 Registry
[root@rooneserver Database]# pwd
/var/roon/RoonServer/Database
[root@rooneserver Database]# du -csh ./
6.3G ./
6.3G total

HI @JempeZen,

You are correct, that is not that many tracks.

Do you have the full log set from this trace? Can you please send me a full log package so I can ask QA to review it?