I am a nightly cloud provider backup of music files and Roon backup data to a S3 cloud backup provider (Wasabi - https://wasabi.com/) . I have about 80,000 tracks in my database and the daily backup takes about half an hour (because of the checksumming of nearly 40,000 files in the backup directory). I use rclone (https://rclone.org/) and it comes highly recommended from me.
Some cloud providers (such as Backblaze B2) charge for API calls, since the checksumming does create a load on the cloud servers themselves.
Since I only do Roon Backup twice monthly, I thought it would be nice to write a script that “skips” the backup/checksumming of the dataset if the “root” file has not been changed.
I wanted to create a thread here for anyone who may have a similar use case to follow along, and perhaps speed up the solution.
I use Wasabi S3 but my preferred backup software is Duplicacy. This uses lock-free deduplication and it the fastest backup software I’ve used with cloud storage. It’s a great choice because metadata changes have minimal impact on the backup; only the changed chunk is uploaded. I prefer using the CLI version with a BASH script.
I also have a bash script that runs nightly. If I can query the storage provider and check for the existence of the b_YYYYMMDD file, I think I can possibly skip the backup of the entire backup, which is usually complete but takes a long time to checksum and verify.