rsync: --xattrs is extremely slow

[Mikko Rantalainen]

Ubuntu 16.04 LTS repositories contain only rsync version 3.1.1:

$ apt policy rsync
rsync:
  Installed: 3.1.1-3ubuntu1.2
  Candidate: 3.1.1-3ubuntu1.2
  Version table:
 *** 3.1.1-3ubuntu1.2 500
        500 http://fi.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
        100 /var/lib/dpkg/status
     3.1.1-3ubuntu1 500
        500 http://fi.archive.ubuntu.com/ubuntu xenial/main amd64 Packages

However, this version has a bug where flag --xattrs causes very very high CPU usage if source contains lots of unique extended attributes. I'm trying to sync 13 TB (roughly 6.5 M files) with extended attributes and this bug seems to make this slower and slower as the time goes by. The system has been copying the stuff for three weeks (!) now. It's getting slower every day but still seems to make progress so I try to run it to completion. First couple of terabytes suggested that this sync operation should have taken around 4 days. The source directory is used by glusterfs which creates checksums as extended attributes and practically every file has different checksum.

Please, update rsync package on Ubuntu 16.04 to 3.1.2-2 or later.

See also: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=799143

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: rsync 3.1.1-3ubuntu1.2
ProcVersionSignature: Ubuntu 4.15.0-51.55~16.04.1-lowlatency 4.15.18
Uname: Linux 4.15.0-51-lowlatency x86_64
NonfreeKernelModules: nvidia_uvm nvidia_drm nvidia_modeset nvidia
ApportVersion: 2.20.1-0ubuntu2.18
Architecture: amd64
CurrentDesktop: MATE
Date: Tue Jul 2 09:07:49 2019
InstallationDate: Installed on 2015-02-23 (1589 days ago)
InstallationMedia: Ubuntu 14.04.2 LTS "Trusty Tahr" - Release amd64 (20150218.1)
SourcePackage: rsync
UpgradeStatus: Upgraded to xenial on 2016-06-10 (1116 days ago)

[Mikko Rantalainen (mira)]

Note that this bug is NOT a duplicate of https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1524703

[Bryce Harrington (bryce)]

This issue was fixed in Debian with the attached patch from upstream.

SRU process prohibits version jumps so would reject directly updating the rsync package on Ubuntu 16.04 to 3.2.2-2.

However, if you can define a test case to reliably reproduce the bug, then it may be possible to get the specific patch accepted. Would you be willing to help draft a set of steps to reproduce the bug - preferably something that can be performed within a few minutes or an hour, as opposed to a few weeks (else it'll be too hard for folks to test)?

You can see other steps required for filing an SRU at https://wiki.ubuntu.com/StableReleaseUpdates. Anything you can do towards making this bug report fulfill those requirements will make it more likely for this to get attention.

One of the potential problems in this case is that the patch is rather large so may be harder to review. SRU's tend to go faster when the patch is just a few lines.