Though not happening on a wide scale bit rot may be a problem with larger collections, affecting track after track over the years. The larger the collection, the larger the risk.
I thought about using ReFS or one of its linux silblings as a first line of defense. Any thoughts on this?
These advanced file systems are not really plug and play, as their intended use is for enterprise with professional sysadmins.
You’ll have to understand their specific capabilities, the way to configure them (as to achieve the desired gains), and also how to diagnose and repair them when something fails.
So, use the OS you’re familiar with, and able to manage.
If Windows : ReFS is the only choice.
With Linux, ZFS has good reputation but is not fully integrated in the kernel (license incompatibility), and BTRFS is officially supported but some people are not satisfied with its behavior.
On my side I use BTRFS on a pair of SSD (for redundancy, necessary for recovering when the checksum on one side detects a corrupted bloc). It works well, but I had some failures at the mounting of the file system (no data loss, but fs not mounted, apparently due to a timing problem between the systemd services)
See A Roon on (Synology) NAS Primer - under my post (the first post), under Durability and preventing bit-rot
Thank you, very helpful input. I think I will go with Asustore 2-Bay with BTRFS/Raid1 for the NAS and keep the core on a seperate machine. That should do for recognizing and fixing bitrot. I have two backups seperate from this. These are on extFAT and NTFS, so I do not know the condition they are in (nor do I know the status of may live data).
But better late than never. Mostly listening to qobuz these days, anyway. Again, thank you. Great insights.
I have rather mundane solution for bit rot on my “NAS”, which is an old (but capable) laptop with a USB external enclosure: every few months, I run a task that hashes all files and compares each hash to its previous value. If they differ, it flags bit rot. It doesn’t correct it of course, but I can backup the file again.
If you have doubt about the quality of your backups, you may test the music files (if they are contained in flac) with the command line utility : flac -t file1.flac file2.flac …
That works only for FLAC, so it’s rather limited, even for a music library, and I don’t know what kind of errors “flac -t” detects.
the flac stream contains an MD5 checksum of the PCM (uncompressed) data.
Understood, but corruption can happen somewhere else in the file, i.e. in metadata. I looks to me the MD5 hash is more for verifying that the compression is truly lossless than a way to detect corruption. Also, they probably chose MD5 because it’s fast, but it uses only 128 bits. I am using SHA512, to reduce the chances of collisions.
The advantage of the flac MD5 is that it is native and already in place whenever someone uses flac files.
Your add-on SHA512 is more robust and covers everything, but I’m not sure Joachim has done something similar on the two backups he already has. But if he uses flac, he does have checksum to test, and the possibility to detect a corrupted music (and recover from the other backup).
Sounds interesting. Can you provide details, i.e. how to? How long does it take per say 1.000 tracks?
I’ve set up a task in Windows’ task scheduler that runs a PowerShell script. It’s not very fast, it takes a little under 5 days for 7.5TB of data. That’s all my data, not only music. That comes to about 15 hours per TB. It does’t interfere with anything.
But now that you mentioned it, I should probably look into using ReFS. It should be transparent to Roon or any other app that accesses the share.
I did some quick research on this. ReFS is not supported anymore on Windows client, ZFS is not on Windows, while BtrFS requires a driver. I guess I’ll stick with NTFS for now and keep using my solution.
You need to run Windows 10/11 pro workstation to run ReFS. While the “official” license is expensive, there are many vendors who offer a valid serial for little money.
I had access to a license via my microsoft partner account, so it was easy to try. Works nicely on external drives, too.
I just ordered two Asus Lockerstor boxes for a client that will use BtrFS. Will post here on setup and experiences with these. Use case there is secure and tamper free storage of large amounts of legally relevant documents.
Features that drove me to Asus were BtrFS, Snapshots, handling of archive drives and handling of Backups to internal Harddisk (online only for Backup).