Hey all. I’m back again with another installment in my (hopefully) popular bash scripting series. In fact this is the penultimate installment for the current time frame.
For this, ‘one before last’ scripting blog I have decided to take a look at one of the most powerful commands, the “rsync” command.
Rsync, the number one choice for your syncing needs
As the name implies (these binary writers can certainly be helpful) this command line tool is used to sync files and folders in directory x with the files in directory y. This can include over a network share, using an ssh connection, using destructive or non-destructive syncing as well as controlling the output.
As you’d image, the command format is similar to most bash commands
rsync [arguments/options] [source location] [destination location]
General rsync Arguments
So you like the look of what you can do with rsync? Open up your Terminal.app and type in ‘man rsync’. After scrolling through a lot of explanation, you’ll see a long list of different arguments and options you can pass through it, each with their own uses and context requirements. In light of this, I have recommended a few options for most uses:
- “rsync –v”– Show verbose mode, e.g. show more information on the transfer.
- “rsync –r” – Recursive, e.g. to copy not just the items in the directory, but the folders below it.
- “rsync –n” – Dry Run, e.g. run through the sync without actually copying anything. This can be used to present an estimated amount of data that will be changed/moved.
- “rsync –z” – Compress, e.g. compress the files that need to be transferred on the fly.
- “rsync –a” – Use archive mode, e.g. run the rsync with a whole bunch of common arguments and options in use. I’ll leave it to you to research each option, but is effectively runs “rsync –rlptgoD”.
That should give you a nice bunch of items to try and test and see if any take your fancy. But please do remember, run testing on a completely unrequired test system, using test data, in case the worst should happen.
But what about those other amazing things it can do?
Ah, you mean those features I mentioned above? OK fine, I’ll give you a brief run down on those here:
- Syncing to a network share – Simply ensure that the network share is mounted (see my previous blog on mount) and set the volume as the destination (e.g. /Volumes/MNH/Destination_Directory).
- Syncing across an SSH connection – “rsync –e ssh [source directory] [SSH user]@[SSH server address]:[destination directory]” – One word of warning, this will request the password of the SSH account to initiate the connection. If you require password-less SSH connection, then you need to investigate the creation and use of SSH key pairs. This may be a topic I cover at a later date.
- Use destructive syncing – “rsync –del” – This will delete any items on the destination that are not present on the source. WARNING: This will result in data ‘loss’ as that is its intention. Please test this thoroughly before rolling out anywhere. The ‘del’ command has a number of different options.
a. “rsync –delete-before” – This will delete the ‘extra’ files before the transfer.
b. “rsync –delete-during” – This will delete the ‘extra’ files during the transfer.
c. “rsync –delete-after” – This will delete the ‘extra’ files after the transfer (this is what we would recommend if you will carry out the delete command). - Use non-destructive syncing – This is the default for using rsync and so does not require any extra arguments/options.
- Controlling the output – This is simply a cause of using either “rsync –v” to show a verbose output or “rsync –q” to suppress any non-error messages.
Summary
There you have it. That’ll give you some food for thought in regards to syncing methods. As mentioned previously, there is a massive list of additional options you can use. I would heavily recommend consulting these to see if there may be another option that is helpful for your site.
Next week will be the last post in this series that I hope has been helpful. For the last blog I’m planning to provide some example scripts combining many of these post into helpful scripts to give you some more ideas.
Any hints, tips or opinions? Let us know in the comments below and I’ll try to respond to as many as I can.