rsync File Sync

rsync cheatsheet — sync files locally and remotely over SSH. rsync -avz src/ dest/, rsync --delete, rsync --exclude, rsync --dry-run. Full file sync reference.

7 min read

What it is

A versatile command-line utility for efficiently synchronizing files and directories between locations, often used for backups, mirroring, and remote data transfer.

Installation

Linux:

sudo apt update && sudo apt install rsync
# or
sudo yum install rsync

macOS: rsync is typically pre-installed. If not:

brew install rsync

Windows: rsync is not natively available. You can use:

  • WSL (Windows Subsystem for Linux): Install a Linux distribution from the Microsoft Store and then use the Linux installation commands.
  • Cygwin: Install Cygwin and select the rsync package during installation.
  • Third-party GUI tools: Many graphical tools leverage rsync in the background.

Core Concepts

  • Delta Transfer Algorithm: rsync’s core efficiency comes from this algorithm. It compares files at the source and destination and only transfers the differences (deltas) rather than the entire file, making it very fast for repeated syncs.
  • Permissions and Metadata: rsync can preserve file permissions, ownership, timestamps, symbolic links, and other metadata.
  • Remote Shell: By default, rsync uses SSH for secure remote transfers.

Commands / Usage

Basic Synchronization

Local to Local:

rsync -avh /home/user/documents/ /media/backup/documents/
  • -a: Archive mode (preserves permissions, timestamps, symbolic links, ownership, etc.; equivalent to -rlptgoD).
  • -v: Verbose output (shows files being transferred).
  • -h: Human-readable numbers (e.g., 1K, 2M).
  • Copies /home/user/documents/ to /media/backup/documents/, including its contents. Note the trailing slash on the source directory means "copy the contents of this directory." Without it, it would copy the directory itself into the destination.

Local to Remote (via SSH):

rsync -avz -e ssh /home/user/project/ user@remote.server.com:/var/www/html/
  • -z: Compress file data during the transfer.
  • -e ssh: Explicitly specifies using SSH as the remote shell.
  • Copies /home/user/project/ to the html directory on remote.server.com as user user.

Remote to Local (via SSH):

rsync -avz -e ssh user@remote.server.com:/var/log/ /home/user/logs/
  • Copies files from /var/log/ on remote.server.com to /home/user/logs/ locally.

Excluding Files and Directories

Exclude a specific file:

rsync -av --exclude 'temp.log' /home/user/data/ /media/backup/data/
  • --exclude 'pattern': Skips files or directories matching the pattern.

Exclude multiple files:

rsync -av --exclude 'cache/*' --exclude '*.tmp' /home/user/app/ /media/backup/app/
  • --exclude 'cache/*' skips the cache directory and its contents.
  • --exclude '*.tmp' skips all files ending in .tmp.

Exclude from a file:

rsync -av --exclude-from '/home/user/rsync_exclude.txt' /home/user/project/ /media/backup/project/
  • --exclude-from 'file': Reads exclusion patterns from the specified file, one pattern per line.

Include specific files (after excluding others):

rsync -av --exclude '*' --include 'important.conf' /home/user/config/ /media/backup/config/
  • --include 'pattern': Includes files matching the pattern. When used with --exclude '*', it effectively copies only the specified files. Order matters: include rules are checked before exclude rules.

Deleting Files

Delete files at destination that are no longer at source:

rsync -av --delete /home/user/website/ /media/backup/website/
  • --delete: Removes files from the destination directory that don’t exist in the source directory. Use with caution!

Dry run (see what would be deleted):

rsync -av --delete --dry-run /home/user/website/ /media/backup/website/
# or shorthand
rsync -avn --delete /home/user/website/ /media/backup/website/
  • --dry-run or -n: Performs a trial run without making any changes. Shows what files would be transferred, updated, or deleted.

Bandwidth Limiting

Limit bandwidth to 1MB/s:

rsync -av --bwlimit=1000 /home/user/large_files/ /media/backup/large_files/
  • --bwlimit=KBPS: Limits the I/O bandwidth in Kilobytes per second. 1000 is approximately 1MB/s.

Transferring Specific Files/Types

Copy only files modified in the last 24 hours:

rsync -av --update /home/user/data/ /media/backup/data/
# or shorthand
rsync -avu /home/user/data/ /media/backup/data/
  • --update or -u: Skips files that are newer on the destination.

Copy only files larger than 100MB:

rsync -av --max-size='100M' /home/user/downloads/ /media/backup/downloads/
  • --max-size='SIZE': Do not transfer files larger than the specified size.

Copy only files smaller than 1KB:

rsync -av --min-size='1K' /home/user/logs/ /media/backup/logs/
  • --min-size='SIZE': Do not transfer files smaller than the specified size.

Preserving Attributes

Do not preserve permissions:

rsync -av --no-p /home/user/data/ /media/backup/data/
  • --no-p: Does not preserve permissions. Useful if the destination has different user/group contexts.

Preserve modification times only:

rsync -av --times /home/user/data/ /media/backup/data/
  • --times or -t: Preserves modification times.

Preserve owner and group:

rsync -av --owner --group /home/user/data/ /media/backup/data/
  • --owner: Preserves the file owner.
  • --group: Preserves the file group. Requires root privileges on the destination to change ownership.

Progress and Verbosity

Show progress per file:

rsync -av --progress /home/user/large_file.zip /media/backup/
  • --progress: Shows a progress bar for each file being transferred.

Show total progress:

rsync -av --info=progress2 /home/user/large_directory/ /media/backup/large_directory/
  • --info=progress2: Shows overall transfer progress instead of per-file progress.

Quiet mode (suppress most output):

rsync -azq /home/user/data/ /media/backup/data/
  • -q: Quiet mode. Suppresses informational messages, only showing errors.

Remote Shell Options

Use a non-standard SSH port:

rsync -av -e 'ssh -p 2222' /home/user/local_dir/ user@remote.server.com:/remote/dir/
  • -e 'ssh -p PORT': Specifies the command to run for the remote shell, including options like the port number.

Use a specific SSH key:

rsync -av -e 'ssh -i /path/to/your/id_rsa' /home/user/local_dir/ user@remote.server.com:/remote/dir/
  • -e 'ssh -i KEY_PATH': Specifies the identity file (private key) to use for SSH authentication.

Common Patterns

Mirror a directory (delete extraneous files):

# First, do a dry run to see what will happen
rsync -av --delete --dry-run /home/user/website/ /var/www/mirror/
# If the output looks correct, run it for real
rsync -av --delete /home/user/website/ /var/www/mirror/
  • This ensures the destination is an exact replica of the source.

Backup with timestamped directories:

TIMESTAMP=$(date +%Y-%m-%d_%H-%M-%S)
rsync -av --delete /home/user/data/ /mnt/backup/data/$TIMESTAMP/
  • Creates a new backup directory for each run, preserving historical snapshots. The --delete flag here would delete files within the current backup directory that are no longer in the source, which might not be desired for historical backups. For true historical backups, omit --delete.

Syncing a local directory to a remote server and then back to another local location:

# Sync local to remote
rsync -avz /home/user/project/ user@remote.server.com:/srv/project/
# Sync remote back to a different local backup location
rsync -avz -e ssh user@remote.server.com:/srv/project/ /home/user/backups/project_remote/

Using rsync over SSH without a password (key-based auth): Ensure your SSH key is set up (ssh-copy-id user@remote.server.com). Then you can run:

rsync -avz /home/user/data/ user@remote.server.com:/backup/data/
  • This is essential for scripting and automation.

Syncing only changed files efficiently:

# Initial sync
rsync -avz /home/user/photos/ user@remote.server.com:/media/photos/
# Subsequent syncs will only transfer differences
rsync -avz /home/user/photos/ user@remote.server.com:/media/photos/
  • The -a flag combined with rsync’s delta algorithm makes this very efficient.

Gotchas

  • Trailing Slashes: The presence or absence of a trailing slash (/) on the source directory dramatically changes behavior:
    • rsync source/ destination/: Copies the contents of source into destination.
    • rsync source destination/: Copies the directory source itself into destination, resulting in destination/source/.
  • --delete is Dangerous: Always use --dry-run (-n) with --delete first. A mistake here can lead to irreversible data loss. Ensure your source path is correct!
  • Permissions on Remote: When syncing to a remote server, ensure the remote user has write permissions in the destination directory. If you need to preserve ownership (-o, -g), you’ll likely need sudo on the destination or be the root user.
  • SSH Port and Key Paths: When using -e 'ssh ...', ensure the command is correctly quoted, especially if it contains spaces or special characters. Double-check the SSH port and key file path.
  • Exclusion Order: Include/exclude rules are processed in the order they appear on the command line. Wildcards (*) need careful handling. Rules like --exclude '*' followed by --include 'important.txt' work because the include rule is processed after the exclude rule.
  • Symbolic Links: By default (-a), rsync copies symbolic links as symbolic links. If you want to copy the target of the link instead, use the --copy-links (-k) option.
  • Bandwidth Limiting: --bwlimit is specified in kilobytes per second. 1000 = 1MB/s.