You could try the following. I mount my ROCK SSD to a convenient location, in my case I have a share that is primarily for backup mounts to my DS416, but I mount my ROCK SSD there as well.
The following rsync job is set up in my Task Scheduler, which I run manually, never got around to defining an automatic trigger:
Youāre a scholar and a gentleman. Very much appreciated.
youāre welcome, happy to help.
Hey, I have a question for the esteemed folks on this threadā¦
Iād ideally like to treat my music folder on my Synology 918+ as master, and have it mirrored to two different ROCKsā local storage (so every night after Iāve added new folders to my NASās music folder, the next morning theyāre on my ROCKās local storage and library. Each one is visible on my local network at 192.168.1.100 and 192.168.10.100. Ideally Iād be able to use Active Backup to copy from the NAS to the ROCKās USB-attached storage, but I canāt figure out how to go in that direction since I canāt install the Active Backup Agent on the ROCK (obviously). Is there a way to achieve this? Am I doing something crazy by treating NAS as master? (Donāt worry, I also keep quarterly offsite backups of airgapped hard drives just to be darn sure because I donāt trust NAS for backups)
Oh my good gosh. This took me a while to figure out (mounting my remote ROCK cifs shares, what exactly happened based on directory names, etc) but this was pretty rookie stuff and now that Iām through it itās just amazing. Master on NAS, automagically copies immediately to my home ROCK and my remote ROCK. Dang. Ninja style stuff. Thank you @ACvitus !
EDIT: thereās one issue. Iām trying to back up my NAS-based FLAC masters to two locations: 1 is a local ROCK, whose internal storage is mounted as a CIFS share, and the other is a remote ROCK, whose internal storage is mounted as a CIFS share over a site-to-site OpenVPN tunnel. The problem is that CloudSync can only add 1 WebDAV sync to the same localhost:5005, and you canāt add two backup tasks to the same WebDAV CloudSync task that share the same local folder. And you obviously canāt run WebDAV on ROCK. I suppose I could daisy chain the two and copy from one ROCK mount to the other over the tunnel, but that intuitively sounds fragile. Perhaps I should use a different approach - I do have a Synology in each location, and I could mount the remote ROCKās storage on the remote synology, and then run WebDAV on that Synologyā¦ Iām not very deep in this stuff, so not sure if that would work.
This can be done. It took me forever to figure this out and I finally got it thanks to some other poster here.
rsync -av --delete --exclude ā@eaDirā /volume1/Library/ /volume1/ROCK/Library
Initially, I recommend setting it up to email you and running manually to make sure everything is working OK. For example, I had to fix up some folder names with special characters. Once youāre satisfied it works as desired, Schedule the task to run as frequently as you please. I have it running on 1 minute intervals! Itās practically real time.
Um, youāre amazing, thank you. Wish I could sticky this. Others will need this. Amazing.
I did eventually figure out that the single quotes in the script you pasted need to be replaced with standard single quotes, because I kept getting @eaDir files. I think this forum software replaces single quotes with āfancy beginning and end quotesā for visual display. But thatās on me, not you, and I figured it out which made me feel clever.
I wish there was a way to āterminate after 15 minutesā or pause or something like that so I could test better before putting the whole several GB running. But, like you, Iāll do it manual with full logging and email alerts for everything for a while. This is AWESOME. Again, thank you. I sound like a little kid, but this is so much better than running cloud sync (whcih I hear is being deprecated anyways?).
This is a great thread. Whenever I set up any kind of sync, I use the exclusion list below. I suggest you use something similar for your rsync task. In addition to creating @eaDir folders, Synology can create #snapshot and #recycle folders depending on how you have your shares configured. .DS_Store folders can show up if you ever mount shares on a Mac. Youāre never going to regret excluding these but you might regret not doing it 
@eaDir
#snapshot
#recycle
.DS_Store
As always, additive, on point, and gentlemanly. Had a bunch of .DS_Store files, glad to be rid of them in my copies. Cleaning things up left & right. Gracias.
Paranthetically: Feel like a bit of stud when I have stuff like this running in the background, which is traversing a CIFS mount of a ROCK-attached USB disk over an OpenVPN tunnel.
rsync -av --delete --exclude ā@eaDirā --exclude '#snapshot' --exclude '#recycle' --exclude '.DS_Store' /volume1/music/ /volume1/MyMounts/HomeROCK/1_44_1-42218_SPK_DD5661978392C_b66cf5ac-27e3-4c3d-8e42-645be011x20b-p1/music
Old me would not have believed I could do this.
Ok, been running for a while - and I have a question for those who know more about this stuff than me (@EvilGnome6 @gTunes and other smart folks here). Does rsync send the whole file even though itās already on the destination?
Current status:
-av --delete --exclude options even though the destination files were identical already (and seemed to be by my own inspection of timestamp, filesize, etc).Should I just let it be, accept that itāll take a few days to a week for the initial upload and not worry about it? Or should I try to understand rsync options like --partial-dir and -c? I thought if Iād pre-seeded 100% of the content Iād be in good shape. But house B is > 1 hour away, so Iām not going to drive back and forth to try to pre-seed it differently.
Thanks!
Did you use the same command line options for both scenarios? I havenāt used rsync in a while but looking at the man page, -a expands to -rlptgoD. Most of that looks right but I donāt know if you want āpā for either scenario - thatās āpermissionsā. āgā is group and that probably doesnāt make sense either. āoā is owner". āDā is devices.
I donāt think this is your perf issue but I donāt know if you want to be trying to clone any permission stuff across systems - that tends to only help if you know exactly what youāre doing.
Regarding perf - the first and most obvious question is āWhat do you expect it to be doing and are you sure itās not doing exactly that?ā
If youād specified -c, Iām fairly certain it would have to pull file contents from remote to local in order to compute checksums. Without that option, I think it just compares file sizes and modification times. Did whatever you do to seed your remote data set preserve modification times? Should be easy to check.
For some of this kind of stuff, I use syncThing. I donāt think itās right for your NAS ā ROCK scenario but I would use it for the remote stuff. Or Iād use Synologyās Drive Server/Drive ShareSync. Both of those would give you remote sync without the need for a site to site VPN (each has a redirector). syncThing has a steep learning curve but once youāre in, youāre in. I use it to do full private cloud type stuff - sync our āDocumentsā directories on computers to our āHomeā directories on Synology. Iāve tried using Synology Drive on user machines in the past and canāt stand it - has bugs and is a resource hog. Once youāve got syncThing up, a lot of problems look like syncThing problems.
Okā¦commercial over. In case you do want to jump ship from rsync for the remote scenario, thatās what Iād suggest you look at. Short of that, though, check to see if it different mod times are causing your issue because if mod times donāt match, and youāre in āpushā mode, I assume itāll just overwrite the file (and you are in push mode, right?)
Before I got rsync working directly on my NAS, I was synchronizing my NAS and ROCK using Robocopy on a Windows box. Other than the issues I mentioned with special characters in folder names, no other files were re-copied. As @gTunes mentioned, it may have to do with the -a switch. I honestly donāt know. This is my first foray into rsync.
Seriously? Nice! Would not have guessed that based on your posts!
Cool. None of us are experts, which I kind of love. Permission to muck about and fail. Worst case scenario it takes a week I suppose, and after that it should be pretty straightforward for the smaller adds/edits I can envision. Not sure I should chase another solution before I get this one working, and I know it works even if initial is inefficient.
Thanks all!
EDIT: on thinking about it, I bet it would have worked if Iād first seeded the remote drive locally using rsync instead of just network file copy; so now it doesnāt really look like itās copying, but it is because it needs to. You live and learn.
Also, I discovered something clever if youāre ever in this situation where youāre not sure whatās going on with rsync over a slow connection / big job on Synology Scheduled Tasks. If you want to āsee where it is working as it is traversing the file structureā, you can go in DSM to Resource Monitor / Connections / Accessed Files and youāll be able to see the most recently touched file by rsync (which runs as a process under the DSM Desktop Service, assuming youāre running it through Task Scheduler). If you go to that directory in the target file system, youāll actually see files populate / get a sense of how fast theyāre going and some reassurance that something is happening. I donāt know exactly in what order the file system is traversed, so this isnāt anything like an ability to estimate % completion, but it does give you a sense of progress, eg that something is occurring. Which is frankly what I needed.
Questionā¦ and I think the answer is going to be ātry it and report backā, but I figured Iād see if anyone had any ideas.
Currently my way of getting files from my NAS music master to my 2 coresā USB-attached SSDs is the task scheduler rsync jobs listed above. One is local, and I have it run hourly. The other goes across a VPN tunnel to another house. Both are set up via remote CIFS mounts on the local Synology, so the remote houseās job looks like:
rsync -av --delete --exclude '@eaDir' --exclude '#snapshot' --exclude '#recycle' --exclude '.DS_Store' /volume1/music/ /volume1/homes/admin/RemoteRock/1_44_1-42218_SSK_DD5641988763D_c57f2da1-6301-49bc-6299-3841eef904e3-p1/music
(Again, RemoteRock is a CIFS mount of a NUC in another state that has a local IP address through an OpenVPN tunnel)
Theyāre both on different companiesā cable plans, and have guaranteed upload speeds in the range of 20Mbps. I currently get more like 7Mbps (measured at the Unifi console, and itās much less in reality - it took 9 days for my 600Gb music share to traverse the connection (I was stupid and didnāt start the copy locally, but once it was going I was lazy and decided to see how it did). Iām sure thereās a bunch of overhead, and Iām sure thereās a bottleneck here or there. Mostly itās fine. It works, which is amazing. Iām never adding more than 3-4 albums a day, and that is infrequent. So I should probably leave well enough alone.
Butā¦ I also have a crappy little Synology ds220j that sits in my Remote House that I use as a Hyper Backup Vault for important stuff on my primary home NAS. So I have backup rotation etc in case my house burns down or a guest goes into my primary home server closet and takes my DS918+ and drop kicks it.
But what Iām wondering is whether I should mount the remote rock via smb on the remote synology and then use the syntax more like rsync -av /volume1/music/ rsyncuser@192.168.10.232::/volume1/mounts/secondhomerock/1_44_1-42218_SSK_DD5641988763D_c57f2da1-6301-49bc-6299-3841eef904e3-p1/music instead, relying on the fact that the second home synology can act as an rsync daemon.
In other words, is there an advantage in speed / reliability to allowing the CIFS mount to be a local one and have rsync traverse the tunnel, or should I keep it like it is where itās the CIFS mount that is traversing the tunnel and rsync is doing all its work within the local synology? (Sorry if Iām not explaining exactly with right words, hope this is clear)
Also, note that Iām not encrypting anything over rsync because Iām relying on my site-to-site VPN as my source of security. I rotate that password periodically and try to assume that that canāt be a worse vulnerability than a frontal attack on either homeās Unifi IPS detection, firewall, ARC port etc).
Thanks for any thoughts. Never thought Iād be trying to figure stuff like this out for myself.
I suspect that rsync isnāt well suited for transferring data over WAN connections (lots of acknowledgements over a high latency connection will slow down transfer rates). If you have a second Synology at the Remote House, I might try running ShareSync to sync the music over to the second NAS and rsync it over to your second ROCK from there.
So ShareSync looks cool, and I can see in the docs that I can actually get sharesync to work to a mount on the remote NAS so I donāt have to do a two-step process. But if thatās a good method for remote, why wouldnāt it be a good method for my local? In other words, if Iām copying localNAS->RemoteROCK, why shouldnāt I use the same method for localNAS->localROCK? Is that impossible to use that tool from one NAS to a mount on itself?
Iām not a great administrator, so having a single approach for both would be far better for me and all my many abilities to remember (and not remember) how to do things.
EDIT: got it, my guess was right - ShareSync is pretty great at keeping folders in sync across two Synology NASās, but canāt do sync to a local drive. Tried to see if I could fool it by getting it to connect to localhost:5001, but no such luck. Drawing board. Perhaps rsync locally and sharesync to remoteNAS with destination of CIFS-mounted folder on remoteROCK. Having just sent 600Gb over this connection with rsync, and having a connection that works, I may leave well enough alone for now (Iām not sure how Cox & Xfinity are going to love me after this one; data caps arenāt yet in effect where I live but Iām not sure I want to really get involved). On my list of stuff to do.
localhost:5005 has worked for me using Synologyās Cloud Sync, to designate the NAS as the ācloudā
Update. I think Iāve reached the acceptance phase.
Iām not done with this yet, Iām just doing the local initial sync of the two Synology drives via ShareSync Drive. After that gotta then bring the drive out to the remote house, then set up rsync out there. But this feels like a plan. Iāve learned a bunch.
The advantage of this is that it actually applies to much more than just music; thereās also a lot of scanned family history docs etc. that are too big to pay to keep in Google Drive and which I canāt bear the thought of being lost forever in a fire. And Iām too lazy to figure out in addition to all this how to deal with Glacier backup or whatever. So for now this works, I think.
