rsync sends everything, not just changes

[peterzay]

Repeated runs of rsync from PC to USB key send everything, not just the changes.

This is reproducible.

ProblemType: Bug
DistroRelease: Ubuntu 19.04
Package: rsync 3.1.3-6
ProcVersionSignature: Ubuntu 5.0.0-29.31-generic 5.0.21
Uname: Linux 5.0.0-29-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.20.10-0ubuntu27.1
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Fri Sep 27 09:18:19 2019
InstallationDate: Installed on 2019-06-10 (109 days ago)
InstallationMedia: Ubuntu 19.04 "Disco Dingo" - Release amd64 (20190416)
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: rsync
UpgradeStatus: No upgrade log present (probably fresh install)

[Rafael David Tinoco (rafaeldtinoco)]

Hello peterzay,

IF you are facing something like:

https://unix.stackexchange.com/questions/85993/why-rsync-attempts-to-copy-file-that-is-already-up-to-date

Then I would recommend reading the 1st answer to that question, which explains why and when rsync would try to re-sync everything again.

If not,

Could you please provide a better description of the problem, and, perhaps, a way on how to reproduce, step by step, the issue you are facing ?

[peterzay (peterzay)]

The following

rsync -v -a -r -W -p -o -g -t --delete --stats
--exclude=".*"
--exclude="/snap*"
~
/media/$USER/USB-128GB-3/
>bkp_log

copies all contents of home folder each time it is run.

The expectation is that only changed files will be copied and replaced on the target.

This is reproducible.

[Andreas Hasenack (ahasenack)]

What's the filesystem on the target? If the target is vfat, for example, setting ownership and permissions won't work, and -W tells rsync to transfer the whole file.

[peterzay (peterzay)]

I frequently rebuild USB keys.

If memory serves me correctly, the USB device was LUKS encrypted ext4.

[Andreas Hasenack (ahasenack)]

When using vfat and running rsync twice, this is the diff in the stats output:
--- log1.vfat 2019-10-08 11:43:04.381667860 -0300
+++ log2.vfat 2019-10-08 11:43:17.609728451 -0300
@@ -1,5 +1,5 @@
 Number of files: 78 (reg: 77, dir: 1)
-Number of created files: 77 (reg: 77)
+Number of created files: 0
 Number of deleted files: 0
 Number of regular files transferred: 77
 Total file size: 3,990,531 bytes
@@ -12,6 +12,6 @@
 Total bytes sent: 3,996,271
 Total bytes received: 8,830

-sent 3,996,271 bytes received 8,830 bytes 534,013.47 bytes/sec
+sent 3,996,271 bytes received 8,830 bytes 2,670,067.33 bytes/sec
 total size is 3,990,531 speedup is 1.00
 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1196) [sender=3.1.2]

As expected, the files are transferred again, because rsync couldn't set the permissions or ownership. These concepts don't exist in vfat.

When using ext4, the same rsync command, the diff between the two runs shows a different story:
--- log1.ext4 2019-10-08 11:49:01.687411326 -0300
+++ log2.ext4 2019-10-08 11:49:11.795461543 -0300
@@ -1,16 +1,16 @@
 Number of files: 78 (reg: 77, dir: 1)
-Number of created files: 77 (reg: 77)
-Number of deleted files: 1 (dir: 1)
-Number of regular files transferred: 77
+Number of created files: 0
+Number of deleted files: 0
+Number of regular files transferred: 0
 Total file size: 3,990,531 bytes
-Total transferred file size: 3,990,531 bytes
-Literal data: 3,990,531 bytes
+Total transferred file size: 0 bytes
+Literal data: 0 bytes
 Matched data: 0 bytes
 File list size: 0
 File list generation time: 0.001 seconds
 File list transfer time: 0.000 seconds
-Total bytes sent: 3,996,271
-Total bytes received: 1,504
+Total bytes sent: 1,598
+Total bytes received: 19

-sent 3,996,271 bytes received 1,504 bytes 7,995,550.00 bytes/sec
-total size is 3,990,531 speedup is 1.00
+sent 1,598 bytes received 19 bytes 3,234.00 bytes/sec
+total size is 3,990,531 speedup is 2,467.86

The rsync command I used was:
rsync -v -a -r -W -p -o -g -t --delete --stats /testdata/ /media/andreas/dataext4/

/testdata was populated with a copy of all files in /usr/bin that start with "b*". I don't think the --exclusion parameters you used play a factor here.

Can you please verify the target filesystem in your test case, and if rsync reported errors? And there are other scenarios, like outlined in comment #2.

I guess, to summarize, we need a testcase. I showed you two cases with pendrives (vfat, and ext4), where what you describe does NOT happen.

[peterzay (peterzay)]

I have run 2 test scenarios.

A trivial folder containing 3 text files as source.

A real world case of ~/ as source.

Both cases work as expected on an LUKS encrypted ext4 USB key.

I will do some more testing on fat32 and report back.

[peterzay (peterzay)]

On a prior OS (16.04 or 18.04, cannot remember which) the following

rsync -v -a -r -W -p -o -g -t --delete --stats ~/Pictures /media/$USER/USB-128GB-1/ >bkp_log

would only send differences on subsequent runs with a fat32 USB key.

Your test above seems to indicate that subsequent runs on 19.04 send everything for vfat.

Therefore, my initial test on 19.04 must have been a fat32 USB key and not a LUKS/ext4 USB key.

Please close this case if you are satisfied with these explanations. Thanks.

[Bryce Harrington (bryce)]

Thanks for following up, I'm closing the issue as requested.