I am a nightly cloud provider backup of music files and Roon backup data to a S3 cloud backup provider (Wasabi - https://wasabi.com/) . I have about 80,000 tracks in my database and the daily backup takes about half an hour (because of the checksumming of nearly 40,000 files in the backup directory). I use rclone (https://rclone.org/) and it comes highly recommended from me.
Some cloud providers (such as Backblaze B2) charge for API calls, since the checksumming does create a load on the cloud servers themselves.
Since I only do Roon Backup twice monthly, I thought it would be nice to write a script that “skips” the backup/checksumming of the dataset if the “root” file has not been changed.
I wanted to create a thread here for anyone who may have a similar use case to follow along, and perhaps speed up the solution.
I use Wasabi S3 but my preferred backup software is Duplicacy. This uses lock-free deduplication and it the fastest backup software I’ve used with cloud storage. It’s a great choice because metadata changes have minimal impact on the backup; only the changed chunk is uploaded. I prefer using the CLI version with a BASH script.
interesting. Rclone I have been using for several years now. it supports a command line interactive - so I can go :
rclone lsl Wasabi:[bucketName]/RoonBackups/[UniqueIDofRoonServer]/xx
9705901 2019-12-26 00:05:06.849148177 b_20191226050000_2ac7e979-18c2-46d1-8a38-438e8be01863
9719073 2020-01-11 00:06:32.357069751 b_20200111050123_4f574597-bfd8-4632-b731-a053188357bf
I also have a bash script that runs nightly. If I can query the storage provider and check for the existence of the b_YYYYMMDD file, I think I can possibly skip the backup of the entire backup, which is usually complete but takes a long time to checksum and verify.