For those of you suffering from bit rot, might I suggest an immediate appointment with your doctor. 
2 Likes
Already did that and he referred me to a shrink.
Had one session with him but now heās in the loony bin.
1 Like
Give it another year and youāll have forgotten just about everything! Well, thatās my experience. 
Nonetheless, ā¦
Why is this a nightmare? It seems to me that those who value their data already follow this rule, even if theyāve not come across the term.
Keeping your original data plus a minimum of two backup copies is straightforward, with many free and proprietary solutions available to achieve this goal. Possibly the greatest obstacle is the cost of cloud storage for multi-TB collections, but this is unecessary, too, when using the tried and test off-site storage method, AKA storing a copy with a friend or relative.
What I would caution against is relying on cloud storage, e.g., Dropbox, iCloud etc. You may have a copy of your current data, but any unnoticed deletion, corruption, malware, or edits / changes will overwrite the cloud copy.
Incidentally, I have two copies if the original UK release of the Joshua Tree, and both have pinholes in the substrate. Fortunately, I was able to rip both to FLAC a decade or so ago to reconstruct the entire album. Same issue with Van Morrisonās Enlightenment ā both show signs of bronzing.
Iām glad you moved your data and donāt need me to do it for you, but I still donāt get your point. The fact that same data can be transferred to new media over time proves media itself doesnāt matter. Itās just a necessary evil if you want. That is most evident with the cloud; I donāt care what they store my data on, how many copies they keep and how they deal with rot, as long as I can get it back when I need it. Hopefully never.
No, and interestingly magnetic tape is still used for longevity and to save on maintenance and electricity costs.
Oh yeah, tape is still big for archival. And the performance of the latest LTO tape is impressive: 18TB capacity and 400MBps transfer speed for uncompressed data. That beats HDDs and rivals SATA SSDs.
Interesting!
I used to use DAT tapes for backup long ago.
Any systems/drives that are affordable for a home user?
Preferably with huge capacity.
I have a friend who runs a high end tape business and the state of the art is pretty impressive with the latest LTO and massive robot solutions, at a very high price point.
Usually Banks as customers, but very niche
CD pressing plants in the UK had problems in the mid to late 1980s. Only EMI Swindon seemed unscathed. But PDO UK produced CDs that now are notorious for bronzing and becoming unreadable, and Nimbus UK did the same for discs becoming sticky on the label side. Major pressing plants in Japan, West Germany, and the US were much better.
AJ
3 Likes
LTO tape still suffers from data rot. The tape is designed for a data retention period of 15 to 30 years provided it is kept in clean, cool and dry conditions. If not kept properly, data retention can be as short as 6 months 
Moisture, dust and heat all contribute to reduce the data retention time.
Years ago, I had a consumer tape drive (Travan - probably TR3 from memory - not DAT) for backup purposes. It was fine to start with, but wear on the drive mechanism and/or moisture from the home environment meant that within a year I had stopped using it because it just was not reliable enough.
I seem to remember that the drive cost about Ā£300 at the time but the problem was that the tape cartridges were about Ā£65 each at the time so I only had a couple. However, at the time it was about the only way to backup my 1GByte disk drive 
Thanks!
I had a similar system with DAT tapes somewhere in the nineties.
But I saw the prices for LTO drives. A bunch of drives will be cheaper. 
It makes sense only if you have a lot of data - as in, youāre a cloud storage provider that offers archival/offline tier. AI says average price per TB for HDD is $14. A 10-pack of 6TB LTO-7 tapes on Amazon is $500, so it comes down to $8.34/TB. Or a 20-pack of 12TB LTO-8 tapes for about $1000, which comes down to $4.50/TB.
Microsoft ReFS file system, Linux Bftrs file system, ZFS file system can be configured to such that data stored on disks cannot be lost due bit rot events that may happen over time. Bftrs is offered eg. by Synology at least on their higher value NAS devices.
2 Likes
I donāt know about the others (I donāt use Linux), but I looked into ReFS some time ago and it doesnāt seem to me that you canāt lose your data without additional precautions:
- It requires Storage Spaces for automatic correction of bit rot. If you install it on a simple drive, it will just detect it and return an error.
- If I understand it correctly, detection happens only when data is accessed, so if you donāt read your data, you donāt know it got corrupted.
- For media files, which can get really large, it may take quite a bit to compute a file-level checksum at every access.
- It uses file-level āchecksumsā. To me, that doesnāt seem enough to detect corruption of large files.
I implemented my own bit rot detection on my āNASā (laptop + external USB enclosure). I have a scheduled task that runs every few months and computes SHA512 hashes of files and compares them with the hashes from the previous scan. If there are differences in files that havenāt changed, it reports bit rot.
What are your conclusions? How common is it? I assume you see a point in this regime, hence the implementation.
It shouldnāt happen really. Iāve seen it only once, in an ISO file that was a few tens of GB in size, and I was surprised. I checked the original and copied it again. Iām doing it because I work with data and I know it can happen, not because itās common. (And because Iām a bit paranoid - from working with data.)
1 Like
By all accounts a rotting bit is very rare as Marian wrote, unless one deals with huge amounts of data. And if a bit does rot, it isnāt necessarily a bit one really cares about apart from paranoid reasons (which I do understand, though). Even if a rare event happens: In music files, one bit in one particular sample isnāt necessarily audible.
I have had files on computers at home and at work for 30 years, I have never encountered a file that was broken due to bit rot. I am sure the same is true for nearly everyone. Personally, I do understand people who care about it for purely sanitary reasons, but I donāt and I donāt take special precautions except normal back-ups.
4 Likes
I totally agree. There were a few considerations that made me do it though.
- While a bit flip in an audio track will most probably go unnoticed, it may not be that harmless in compressed video (e.g. DVD and BD rips). I was surprised when corruption was flagged that one time, but I wasnāt surprised where it occurred: in a BD ISO.
- Regarding the amount of data, it has gone up significantly for me over the years. Pictures got bigger, music got bigger (because of switching to lossless, not because of the hi-res nonsense - I still buy CD format when I can), I started backing up my movies, and I keep adding to it. Just last week I moved from 8TB to 16TB and transferred all my data to new HDDs.
- Since I double my storage when I do that, it will likely result in keeping the disks longer and longer, which increases the possibility of corruption.
- Finally, itās a matter of principle for me. Iāve said on multiple occasions that digital, when managed properly, is permanent. If I let bits flip without knowing, itās not going to be the same data over time, and Iām not only talking about my lifetime here. I just want to deliver on that promise. Cloud made it a lot easier, but Iām not looking forward to recovering 8TB from the internet.
- Oh, and of course itās been a fun project for a software engineer.
3 Likes
Yeah, I can understand all of that. Itās great that you have the energy 
Just saying that if someone doesnāt, they probably will be fine, too.
As David Letterman used to say, āWe all know how painful that can be.ā
2 Likes