'Rsync calculate file count before transfer?

When transferring lots of small files in many folders with rsync, it tends to stay around ir-chk=1000/xxxxx, where xxxxx keeps counting up as it discovers new files, and the amount to check stays around 1000 until its on its last few folders.

How can I have it check for the entire file count before copying?

The command I was using to copy was:

rsync -av --progress source dest
bashrsync


Solution 1:[1]

rsync -av --progress --dry-run --stats source dest
  • Option --dry-run doesn't transfer any files but will show how many bytes will be transferred.
  • Option --stats shows a summary.

a sample output:

...
tests/
tests/__init__.py
tests/test_config.py

Number of files: 5,033 (reg: 2,798, dir: 2,086, link: 149)
Number of created files: 5,032 (reg: 2,798, dir: 2,085, link: 149)
Number of deleted files: 0
Number of regular files transferred: 2,798
Total file size: 26,035,530 bytes
Total transferred file size: 26,032,322 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.004 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 158,821
Total bytes received: 17,284

sent 158,821 bytes  received 17,284 bytes  117,403.33 bytes/sec
total size is 26,035,530  speedup is 147.84 (DRY RUN)

Get the number of files will be transferred

rsync -av --progress --dry-run --stats source dest | 
  fgrep 'Number of files' | 
  cut -d' ' -f4 | 
  tr -d ,

Solution 2:[2]

Since version 3.0.0, you can use this option to explicitly turn off incremental recursion:

--no-i-r

Details from the rsync man page:

Beginning with rsync 3.0.0, the recursive algorithm used is now an incremental scan that uses much less memory than before and begins the transfer after the scanning of the first few directories have been completed. This incremental scan only affects our recursion algorithm, and does not change a non-recursive transfer. It is also only possible when both ends of the transfer are at least version 3.0.0.

Some options require rsync to know the full file list, so these options disable the incremental recursion mode. These include: --delete-before, --delete-after, --prune-empty-dirs, and --delay-updates. Because of this, the default delete mode when you specify --delete is now --delete-during when both ends of the connection are at least 3.0.0 (use --del or --delete-during to request this improved deletion mode explicitly). See also the --delete-delay option that is a better choice than using --delete-after.

Incremental recursion can be disabled using the --no-inc-recursive option or its shorter --no-i-r alias.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1
Solution 2 asmoore82

Related Questions