Optimizing (speeding up) rclone backup to S3 cloud provider

I am a nightly cloud provider backup of music files and Roon backup data to a S3 cloud backup provider (Wasabi - https://wasabi.com/) . I have about 80,000 tracks in my database and the daily backup takes about half an hour (because of the checksumming of nearly 40,000 files in the backup directory). I use rclone (https://rclone.org/) and it comes highly recommended from me.

Some cloud providers (such as Backblaze B2) charge for API calls, since the checksumming does create a load on the cloud servers themselves.

Since I only do Roon Backup twice monthly, I thought it would be nice to write a script that “skips” the backup/checksumming of the dataset if the “root” file has not been changed.

I wanted to create a thread here for anyone who may have a similar use case to follow along, and perhaps speed up the solution.

I use Wasabi S3 but my preferred backup software is Duplicacy. This uses lock-free deduplication and it the fastest backup software I’ve used with cloud storage. It’s a great choice because metadata changes have minimal impact on the backup; only the changed chunk is uploaded. I prefer using the CLI version with a BASH script.

1 Like

interesting. Rclone I have been using for several years now. it supports a command line interactive - so I can go :

rclone lsl Wasabi:[bucketName]/RoonBackups/[UniqueIDofRoonServer]/xx

9705901 2019-12-26 00:05:06.849148177 b_20191226050000_2ac7e979-18c2-46d1-8a38-438e8be01863
9719073 2020-01-11 00:06:32.357069751 b_20200111050123_4f574597-bfd8-4632-b731-a053188357bf


I also have a bash script that runs nightly. If I can query the storage provider and check for the existence of the b_YYYYMMDD file, I think I can possibly skip the backup of the entire backup, which is usually complete but takes a long time to checksum and verify.