Rsync by examples - A fast and versatile file copying tool.

Rsync by examples - A fast and versatile file copying tool.

Rsync is a fast and versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon.
Famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.

Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file’s data does not need to be updated.

Some of the additional features of rsync are:

  • support for copying links, devices, owners, groups, and permissions
  • exclude and exclude-from options similar to GNU tar
  • a CVS exclude mode for ignoring the same files that CVS would ignore
  • can use any transparent remote shell, including ssh or rsh
  • does not require super-user privileges
  • pipelining of file transfers to minimize latency costs
  • support for anonymous or authenticated rsync daemons (ideal for mirroring)

Here is the Rsync syntax:
rsync options source destination

If no destination arg is given it will list the source files instead of copying.

Copy/Sync Files and Directory Locally

rsync -av /home/user/project /home/user/backup
This will copy all files and folders from /home/user/project to /home/user/backup/project

-a : archive mode, rsync preserves all file permissions, ownership, and modification times. (this is an alias of -rlptgoD)
-v: verbose

rsync -av /home/user/project/ /home/user/backup

Notice the slash after project/

This will copy all files and folders from /home/user/project to the root of /home/user/backup/

rsync Dry-run - test your command behavior

rsync -avn /home/user/project/ /home/user/backup | more

If you are unsure of your command option you can add the following flags:

-n: dry-run. It just tells you what the command would have done without doing anything

Copy/Sync Files and Directory from or to a remote server

rsync -avz --partial --progress --bwlimit=500 user@ /var/www/ 

Rsync is using ssh as default remote shell, so you don't have to specify it

-z: compression. Compress all data (default at level 6). You can change the compression level using --compress-level=[0-9] instead of -z
--partial: is used to recover a transfer that fails without resending all the parts of files already sent. This can be handy when sending large files over the network.
--progress: display the transfer progress

-P: can be use as a short land for --progress --partial

--bwlimit: limit socket I/O bandwidth. Expressed in KByte/sec.

Rsync multiple thread file sync

Rsync is still too slow? try it with multiple threads to transfer multiple files at the same time.

find folderToSync -type d | xargs -n1 -P4 -I {}  rsync -avzP {} user@remoteHost:~/destinationFolder/must/exist/

Note: For this tips to be really effective you will need to have multi thread CPU on both side.