rsync -u --inplace --partial -a can't resume transfer

[Nahuel Greco]

Suppose you have a file in hostA:

  hostA$ ls -l /tmp/files
   -rw-rw-r-- 2 root root 563016 Jan 10 15:01 test.txt

You download it from the hostB using:

  hostB$ rsync -u --inplace --partial -a hostA::files/* .

If the transfer is aborted, hostB will get only a partial file:

  hostB$ ls -l /tmp/files
   -rw-rw-r-- 2 root root 2024 Jan 11 18:00 test.txt

BUT the ctime/mtime of hostB/test.txt now is NEWER than hostA/test.txt(and mtime == ctime). So, if you run the same rsync -u command again:

  hostB$ rsync -u --inplace --partial -a hostA::files/* .

Rsync will SKIP THE FILE, because hostB/test.txt is "newer" than hostA/test.txt. So you CAN'T resume using rsync -u command, and you will think there are no differences.

To avoid this bug, rsync must create the file with ctime=mtime=0. And if the file already exists before transfer, rsync -u must not change his current ctime/mtime. Ctime/mtime must be updated ONLY after the transfer was successfully completed.

Note this is really need because there are scenarios where checksum comparison can't be used, only comparison by time. For example, to avoid deleting changes made in hostB to test.txt. Also I need to use --inplace.

[Nahuel Greco (ngreco)]

A reproducible test:

~$ cd /tmp/
/tmp$ mkdir a b
/tmp$ cd a
/tmp/a$ head -c 100000 /dev/urandom > test
/tmp/a$ ls -l test
-rw-r--r-- 1 nahuel nahuel 100000 Jan 27 18:27 test
/tmp/a$ cd ../b
/tmp/b$ timeout 3 rsync -u --inplace --partial --bwlimit=2k --progress -va ../a/test .
sending incremental file list
test
         32,768 32% 0.00kB/s 0:00:00
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(632) [sender=3.1.1]
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at io.c(504) [generator=3.1.1]
/tmp/b$ ls -l test
-rw------- 1 nahuel nahuel 0 Jan 27 18:28 test
/tmp/b$ timeout 3 rsync -u --inplace --partial --bwlimit=2k --progress -va ../a/test .
sending incremental file list

sent 59 bytes received 12 bytes 142.00 bytes/sec
total size is 100,000 speedup is 1,408.45
/tmp/b$ ls -l test
-rw------- 1 nahuel nahuel 0 Jan 27 18:28 test
nahuel@serge:/tmp/b$

[Nahuel Greco (ngreco)]

rsync version: rsync version 3.1.1 protocol version 31
ubuntu version: 3.1.1-3ubuntu1

[Nahuel Greco (ngreco)]

Researching more, I see POSIX has NO way to disable mtime updating while calling write()'s, so there is no way to atomically leave a partial file with an mtime=0 mark while using --inplace.

"rsync --update --partial" (no --inplace) can do it because it transfers first to a temporal file, then updates his mtime (to 0 if unsuccessful, or to the original file mtime if transfer completed), and then rename()'s the file.

So now I think this can't be solved, but a warning should be placed in the rsync manpage about this "-u --partial" unexpected behavior. Take this as a documentation bug.

[Paride Legovini (paride)]

Hello Nahuel,

According to your reasoning I assume the "root" issue should still affect the versions of rsync shipped with the newer Ubuntu releases.

I agree it would be nice for the behavior you describe to be documented, but I don't think it's worth patching the Ubuntu package for it. I recognize we have a papercut here, but I don't think the fix for it belongs to Ubuntu; it should instead be driven (and thus also validated) upstream. For this reason I'm marking this bug as a Won't Fix. Should you disagree with my assessment please comment back and set the bug status back to New, we'll look at it again. Thanks!

[Nahuel Greco (ngreco)]

Hello Paride, 4 years have passed from my initial report, I forgot completely about this bug :)

Fine, I will post this issue to upstream.

[Nahuel Greco (ngreco)]

Reported to upstream at: https://github.com/WayneD/rsync/issues/236

[Paride Legovini (paride)]

Thanks Nahuel! We're doing some retriage and cleanup of old bugs, and I stumbled on this one :-)

I linked your upstream bug report to this bug, Launchpad will keep its status in sync. You may have noticed that I didn't set the status of this bug to Wontfix but to Opinion ("Doesn't fit with the project, but can be discussed"), I think it fits better.