rsync is the perfect utility for keeping two local or remote directories synchronized with each other. It works efficiently by only transferring the differences between the files in two directories. In this guide, we’ll show you the various commands you can use with rsync to transfer, backup, and synchronize your files.
rsync has become a native utility found on Linux systems. Every distribution supports it, and it’s even been ported to Windows, MacOS, and BSD. The usefulness of rsync cannot be overstated.
Although keeping two directories in sync with each other seems like a straightforward task, the amount of options you can pass to rsync is staggering. It’s touted as the ideal backup and file transfer tool by many, and for good reason. With a little knowledge of how it works, you’ll soon be using it for all of your file transfer tasks.
Local File Transfers With rsync
The most basic rsync command that you’ll want to familiarize yourself with uses the following syntax:
rsync -av /path/to/directory1/ /path/to/directory2/
With this command, rsync will synchronize the contents of
directory2. It will do exactly the same thing as the
cp (copy) command, except it checks files for differences and only transfers the changes, making the total size of the transfer smaller.
Let’s talk about the options passed to rsync. We included
-av in our command because they’re two of the most common options to have in an rsync command. So common, in fact, it’s likely you’ll be using them almost every time you run rsync. Here’s what they do:
-a= Archive mode. This does a few things, basically wrapping up the most common options into a single flag. It makes the transfer recursive and tells rsync to transfer symbolic links, device files, and special files. It also preserves the permissions, owner, group, and modified time of each file. In other words, the files are kept identical (right down to the permission settings and mtime) from the source directory to the destination directory.
-v= Increase verbosity. Without this option, rsync won’t produce any output unless it encounters some kind of error. When you have this option enabled, rsync will display which file it’s currently transferring and gives you the opportunity to see a list of everything that’s been transferred at the end.
The other thing to note about our command syntax above is the trailing slash on directory names. Without the trailing slashes, rsync would create
directory1 inside of
directory2. This may be fine in some situations, but our goal here is to keep the file structure of the two directories exactly the same.
More rsync Options
Let’s take a look at some other rsync options you’re likely to need.
--delete= Delete extraneous files from the destination directory. With this option, rsync will synchronize two directories and if the destination directory contains a file that isn’t present in the source directory, it will delete it.
rsync -av --delete /path/to/directory1/ /path/to/directory2/
some_file.txt was transferred from the source directory to the destination like in our previous example, but not before
some_other_file.txt was deleted from the destination directory, since it isn’t part of the source directory’s contents.
-n= Dry run. Running an rsync command with the
--deleteoption can be a little risky. You may end up deleting something that you didn’t realize was only present in the destination directory. The
-noption allows you to execute your rsync command without actually making any changes. This gives you a chance to examine the output and make sure you’re not about to make any undesirable changes.
rsync -avn --delete /path/to/directory1/ /path/to/directory2/
This time, adding
-n to our rsync command allows us to see that our
some_file.txt file is going to be transferred and the
some_other_file.txt file will be deleted. rsync reminds us that this is just a dry run in the final line of the output. Apart from the “DRY RUN” text, this is the exact output rsync would give us if we took away the
Excluding Files From Transfer
If you have files or directories that you want rsync to skip over, there are a couple of options for telling rsync what to exclude.
--exclude= This option allows you to specify what to exclude from within the rsync command.
--exclude-from= With this option, you can tell rsync to refer to a file that contains a list of all the files to exclude.
To exclude a specific file with
rsync -av --exclude 'some_file.txt' /path/to/directory1/ /path/to/directory2/
If you need to exclude more than one file, just use additional
rsync -av --exclude 'some_file.txt' --exclude 'some_other_file.txt' /path/to/directory1/ /path/to/directory2/
Note that you can also specify patterns by using wildcards:
rsync -av --exclude '*.txt' /path/to/directory1/ /path/to/directory2/
The other option is to create a plaintext file where you list each file you wish to exclude on a separate line. Refer to this file with the
rsync -av --exclude-from='exclude-list.txt' /path/to/directory1/ /path/to/directory2/
rsync detects changes between files by inspecting the modified time and size of a file. That’s how it’s able to quickly parse tons of files and know if there are changes that need to be transferred. Although this should suffice for nearly every possible scenario, you can also make rsync perform a checksum on each file to ensure that your files are 1:1 exact copies.
I personally find this option useful when I want to check for data degradation on one of my backups. If silent data corruption occurs to a file, checksuming your files with rsync should detect the problem.
-c= Transfer files based on checksum differences, not size and modified time.
Requiring rsync to checksum a bunch of files is both CPU intensive and time consuming, so don’t expect a fast rsync if you’re using this on a fair amount of files.
rsync -avnc /path/to/directory1/ /path/to/directory2/
If you’re using the
-c option, please be sure to pair it with
-n (dry run) as well. If rsync detects any differences, this will allow you an opportunity to check the condition of both files to see why the checksums don’t match. If one is corrupted, you’ll know which version to keep better than rsync will.
Remote File Transfers With rsync
rsync can use SSH to synchronize a local directory with a remote directory (or vice versa). The command syntax is the same as we’ve seen above, except that we need to include another option.
-e ssh= Use SSH as remote shell.
rsync -avn --delete -e ssh /path/to/directory1/ user@hostname:/path/to/directory2/
This command will synchronize
directory1 on our local system with
directory2 on a remote system, using SSH to authenticate and subsequently the SSH tunnel to transfer all of the data. Note that we’re still using the
-n (dry run) flag in this example. It’s always best to see what changes we’re making first.
Side note: If you’re making a script that uses rsync over SSH, it’s a good idea to configure RSA keys for logging into SSH without a password. Otherwise, your script will stop to prompt you for a password each time.
Remote rsync Options
There are a couple of options that are useful specifically when working with remote file transfers.
-z= Use compression. With this option, rsync will compress data before sending it over the network. It knows not to compress files that already use their own compression, like JPEG files. This option will put some additional stress on your CPU, but could save you a lot of bandwidth and time if you’re doing a big transfer.
-P= This flag combines two common options,
--progress. The former allows a transfer to be interrupted and resumed where it left off, and the latter will show you the current progress of whichever file is being transferred.
rsync -avP --delete -e ssh /path/to/directory1/ user@hostname:/path/to/directory2/
That’s about it for everyday rsync options. The vast majority of your rsync needs should be satisfied by basing your commands on the examples we’ve shown you. Still, we’ve barely scraped the surface of rsync. Check out the man page for rsync if you’re interested in further customization.