rsync fails after installing level 3.2.1

[Cliff Carson]

After installation of rsync.3.2.1 my attempt to sync fails with chroot error. The rsyncd.log indicates that rsync can't find the directories I want to sync.

2020/07/18 10:01:20 [3689] rsyncd version 3.2.1 starting, listening on port 873
2020/07/18 11:00:01 [5786] name lookup failed for 192.168.1.159: Name or service not known
2020/07/18 11:00:01 [5786] connect from UNKNOWN (192.168.1.159)
2020/07/18 11:00:01 [5786] rsync allowed access on module Bin_dir from UNKNOWN (192.168.1.159)
2020/07/18 11:00:01 [5786] rsync: [Receiver] chroot /home/cliff/Bin failed: No such file or directory (2)
2020/07/18 11:00:01 [5794] name lookup failed for 192.168.1.159: Name or service not known
2020/07/18 11:00:01 [5794] connect from UNKNOWN (192.168.1.159)
2020/07/18 11:00:01 [5794] rsync allowed access on module Bin_dir from UNKNOWN (192.168.1.159)
2020/07/18 11:00:01 [5794] rsync: [Receiver] chroot /home/cliff/Bin failed: No such file or directory (2)
2020/07/18 11:00:01 [5795] name lookup failed for 192.168.1.159: Name or service not known
2020/07/18 11:00:01 [5795] connect from UNKNOWN (192.168.1.159)
2020/07/18 11:00:01 [5795] rsync allowed access on module Drawings_dir from UNKNOWN (192.168.1.159)
2020/07/18 11:00:01 [5795] rsync: [Receiver] chroot /home/cliff/Drawings failed: No such file or directory (2)

Dropping back to Ubuntu 20.04 and rsync works again. The rsync update was install 7/18.

ProblemType: Bug
DistroRelease: Ubuntu 20.10
Package: rsync 3.2.1-1ubuntu2
ProcVersionSignature: Ubuntu 5.4.0-26.30-generic 5.4.30
Uname: Linux 5.4.0-26-generic x86_64
ApportVersion: 2.20.11-0ubuntu42
Architecture: amd64
CasperMD5CheckResult: skip
CurrentDesktop: ubuntu:GNOME
Date: Thu Jul 23 10:28:17 2020
InstallationDate: Installed on 2020-07-06 (16 days ago)
InstallationMedia: Ubuntu 20.10 "Groovy Gorilla" - Alpha amd64 (20200609)
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: rsync
UpgradeStatus: No upgrade log present (probably fresh install)
mtime.conffile..etc.default.rsync: 2020-07-06T11:31:56.141217

[Christian Ehrhardt  (paelzer)]

Hi Cliff and thank you for the report.

Hmm, neither the later 3.2.2 nor the yet unreleased 3.2.3 mention any fix in that regard.
https://download.samba.org/pub/rsync/NEWS#3.2.2

I guess we'd be down to bisecting for the offending change or something like it.
As-is I'm not seeing the issue when rsyncing two directories on groovy.

I have not went into lengths copying the example config file at /usr/share/doc/rsync/examples/rsyncd.conf to modify and set it up.

Do you happen to have steps to configure a clean new Groovy install to trigger the issue?

[Cliff Carson (ccarson1)]

Christian, had this sync setup over many releases of Ubuntu without issue. Have the syncd running on 20.10 and the client on 20.04. After the install of 20.10 sync ran OK until the 3.2.1 level was installed. Added the current syncd.conf (haven't made changes to this file for several releases of Ubuntu).

[Christian Ehrhardt  (paelzer)]

Thanks, with that config (or a subset thereof) how do you run your server (if it is anything more than sshd or such) and your client (the actual rsync call)?

[Cliff Carson (ccarson1)]

Don't understand the server question. The syncd is enabled and started at boot with the rsyncd.conf in /etc, rsyncd.srct in /etc and rsync in /etc/default. The client start the rsync process with a cron job with the attached script.

[Christian Ehrhardt  (paelzer)]

Thanks Cliff!
Note: the server question you wondered would have been the details how exactly (commands&config( you do "syncd is enabled and started at boot with the rsyncd.conf". Because in that might lay the details we need.

I was setting up a server on my own aligned to your config.
The following makes it easy for everyone to try to re-create as it is just copy&paste to get it working - it will work on a fresh install of any Ubuntu with rsync installed.

Summary:
Host 20.10 - Rsync 3.2.1
Client 20.04 - Rsync 3.1.3

#1 Setup on the Host to prep, config and debug-start rsyncd as demon:

# Something to sync
mkdir /mnt/testrsync
touch /mnt/testrsync/testfile
mkdir /mnt/testrsync/testdir
touch /mnt/testrsync/testdir/testfile2

# Config
cat << EOF > /etc/rsyncd.conf
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock

[test_dir]
path = /mnt/testrsync
comment = Test Directory
uid = ubuntu
gid = ubuntu
read only = no
list = yes
hosts allow = 10.253.194.0/24
auth users = ubuntu
secrets file = /etc/rsyncd.scrt
strict modes = false
EOF
cat << EOF > /etc/rsyncd.scrt
ubuntu:ubuntu
EOF

# Start with max debug in local console
rsync -vv --daemon --config /etc/rsyncd.conf --no-detach

#2 Then on the client try to connect

mkdir /mnt/client
export RSYNC_PASSWORD=ubuntu
rsync -vurptC rsync://ubuntu@10.253.194.39/test_dir /mnt/client
receiving incremental file list
./
testfile
testdir/
testdir/testfile2

sent 77 bytes received 218 bytes 590.00 bytes/sec
total size is 0 speedup is 0.00

Works just fine, at the same time the server says:

$ rsync -vv --daemon --config /etc/rsyncd.conf --no-detach --log-file=/dev/stdout
2020/07/28 05:55:52 [7940] rsyncd version 3.2.1 starting, listening on port 873
2020/07/28 05:56:21 [7942] connect from f.lxd (10.253.194.30)
2020/07/28 05:56:21 [7942] rsync allowed access on module test_dir from f.lxd (10.253.194.30)

In my case (just like in yours) it recognizes the configured module "test_dir" in my case "Bin_dir" in yours. From there it finds all the content and syncs it.

Your error suggests a chroot issue that I can't reproduce even with that setup close to yours.
I added "use chroot = yes" to the daemon config but it behaves the same.

Might on your upgrade anything else have changed that influences this - like under which user the daemon runs?

[Christian Ehrhardt  (paelzer)]

@Cliff
- if you run the above simplified case does it work for you?
- if it does, could you try making it closer and closer to your setup until you've identified what
  makes it break?

[Cliff Carson (ccarson1)]

Couldn't figure out how to make your example run on my 20.10 (or even 20.04) server systems. Using your example took apart my server/client and made a simple case (using the /mnt/synctest directories) which runs when the server is either 20.04 or 20.10. Will start looking at what to add to make it break.

[Cliff Carson (ccarson1)]

Stumped to what to do next. Have two test cases, one works, one fails with chroot error. Here is the rsyncd.conf that workds;

cat << EOF > /etc/rsyncd.conf
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock

[test_dir]
path = /mnt/testrsync
comment = Test Directory
uid = cliff
gid = cliff
read only = no
list = yes
hosts allow = 192.168.1.0/24
auth users = cliff
secrets file = /etc/rsyncd.scrt
strict modes = false
EOF
cat << EOF > /etc/rsyncd.scrt
cliff:ubuntu
EOF

The one that fails;

cat << EOF > /etc/rsyncd.conf
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock

[Bin_dir]
path = /home/cliff/Bin
comment = Common Bin Directory
uid = cliff
gid = cliff
read only = no
list = yes
hosts allow = 192.168.1.0/24
auth users = cliff
secrets file = /etc/rsyncd.scrt
strict modes = false
EOF
cat << EOF > /etc/rsyncd.scrt
cliff:ubuntu
EOF

Tried to change the failure.

Change the path = %HOME%/Bin - no change
Added use chroot = false - the failure change to "chdir failed"

Suggestion on what to try next?

[Cliff Carson (ccarson1)]

This doesn't seem right to me. Changed the path = Bin and the call to rsync to use the -R option which I think is to use relative path. Still fails on chroot but the rsyncd.log has the following;

2020/07/29 08:52:17 [10277] rsync allowed access on module Bin_dir from UNKNOWN (192.168.1.159)
2020/07/29 08:52:17 [10277] rsync: [Receiver] chroot /Bin failed: No such file or directory (2)

The error message is correct there is no /Bin directory but the path statement indicates Bin as a relative path (ensure the test script running from /home/cliff and there is a Bin directory there.

[Christian Ehrhardt  (paelzer)]

Maybe permission differences of the path to /home/cliff/Bin vs /mnt/testrsync

Did you start the daemon like I did as root or under a different user?

[Cliff Carson (ccarson1)]

No differences between /home/cliff/Bin and /mnt/testrsync (0755) both owned by cliff:cliff. In building my test cast wanted to duplicated the target directories as much as I could.

No I use the systemctl start/stop rsync so the answer is the daemon is running under root.

[Paride Legovini (paride)]

Hi Cliff,

This is odd, but if I understand correctly you have a setup that triggers the problem and a slightly different setup that does not, on the same system. This means we are in a good position already. I'd follow Christian's suggestion and make them even more and more similar until you can spot what actually triggers the problem.

E.g.: Can you move /mnt/testrsync to /home? To /home/cliff? Can you rename it to Bin?

Sounds silly, but there *has* to be something that makes it work with /mnt/testrsync but not with /home/cliff/Bin.

[Cliff Carson (ccarson1)]

Have a /etc/rsyncd.conf that moves data from path = /mnt/testrsync directory on my desktop to my laptop and it works. Changing the /etc/rsyncd.conf to path = /home/cliff/Bin fails with a chroot failure (unable to find the path). Copied the /mnt to my home directory (cp -r /mnt ./) so the only difference was the mnt was not root owner. Changed the /etc/rsyncd.conf path = /home/cliff/mnt/testrsync and it fails with a chroot error.

[Cliff Carson (ccarson1)]

Appears to me is that failure criteria is if the directory being rsync'ed is off of my home it will fail. Hard to believe I'm the only one seeing this. Again this 20.10 system was functioning normally until 7/18 when 3.2.1 was installed.

[Paride Legovini (paride)]

OK, so the directory being in /home is the culprit, I could reproduce the issue and found where the problem is. At some point rsync added

  ProtectHome=on

option to its service file, /lib/systemd/system/rsync.service. This has been reverted in upstream git at [1] as it's too restrictive. The fix landed in rsync v3.2.2 which is already in Debian; Ubuntu will pick it up with the next sync. In the meantime you can workaround the problem by manually disabling ProtectHome.

Thanks again for reporting the issue and for providing all the details.

[1] https://github.com/WayneD/rsync/commit/ce12142c459788b611da5f5d525e0486822b043a

[Cliff Carson (ccarson1)]

My resync is working fine again. Thanks for your attention to this bug-report.

[Christian Ehrhardt  (paelzer)]

This won't auto-sync as there is Delta.

This MP is up to merge the latest version from Debian.
=> https://code.launchpad.net/~paelzer/ubuntu/+source/rsync/+git/rsync/+merge/388554

[Sebastien Bacher (seb128)]

@Christian, sounds like a fix probably worth SRUing to the LTS?

[Christian Ehrhardt  (paelzer)]

Hi Seb, as far as I understood Cliff it only started to occur with version 3.2.1 in Groovy.
And after we found the root cause that is now confirmed, 3.2.1 has:
ProtectSystem=full
ProtectHome=on
PrivateDevices=on
NoNewPrivileges=on
But the version in Focal doesn't have that and thereby is fine without a fix.

[Sebastien Bacher (seb128)]

Oh, right, sorry for overlooking that and thanks for the details!

[Launchpad Janitor (janitor)]

This bug was fixed in the package rsync - 3.2.2-2ubuntu1

---------------
rsync (3.2.2-2ubuntu1) groovy; urgency=medium

  * Merge with Debian unstable (LP: #1888685). Remaining changes:
    - d/t/control, d/t/upstream-tests: Make autopkgtests cross-test-friendly

rsync (3.2.2-2) unstable; urgency=medium

  * d/p/fix_bwlimit_0.patch: New upstream patch to fix bwlimit=0
    (closes: #964587)

rsync (3.2.2-1) unstable; urgency=medium

  * New upstream version 3.2.2
  * d/rsync.install: Install rsync-no-vanished (closes: #353697)

 -- Christian Ehrhardt <email address hidden> Mon, 03 Aug 2020 11:22:30 +0200