rsync 3.1.0 and 3.0.9 incompatibility

[Simon Green]

When running rsync on a directory of a few hundred files from rsync 3.0.9-4 on Ubuntu 13.04 Raring to rsync 3.1.0-2 on Ubuntu 14.04 Trusty rsync errors with:

rsync error: error in rsync protocol data stream (code 12) at io.c(441) [sender=3.0.9]

Here is the full command I'm trying to run:

rsync --delay-updates -FF --compress --timeout=10 --archive --rsh 'ssh -o StrictHostKeyChecking=no' --out-format='<<CHANGED>>%i %n%L' templates/dotfiles/ wirehive@172.2.0.1.128:/home/wirehive/

If I repeatedly run this after maybe 4 times it will succeed.

The same sounding bug is reported here:
https://bugzilla.redhat.com/show_bug.cgi?id=1014769
and here:
https://bugzilla.samba.org/show_bug.cgi?id=10332

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Nick Craig-Wood (nick-craig-wood)]

I can confirm this problem.

It seems to be due to a miscompilation of rsync and the zlib library

https://bugzilla.samba.org/show_bug.cgi?id=10372

Trying the transfer without -z / --compress works fine.

The bug seems to show itself when transferring big files ( 1.5GB in my case)

I think this should be fixed urgently!

[Steve (lp-z)]

This also seems to be a dup of:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=741628#50

[Steve (lp-z)]

and I'm also getting this same error between rsync 3.1.0 and rsync 3.1.0, so it's not just an incompatibility between the two versions:

inflate returned -3 (0 bytes)
rsync error: error in rsync protocol data stream (code 12) at token.c(557) [receiver=3.1.0]
rsync: [generator] write error: Broken pipe (32)
rsync error: error in socket IO (code 10) at io.c(837) [generator=3.1.0]

Both sides have version: 3.1.0-2ubuntu0.1

A workaround, reported in the other bugs above (and I've also confirmed from my own use) is to avoid using the "-z" option.

[Jürgen (j-w-ott)]

rsync should ignore -z until the problem with zlib is fixed

[Charles Peters II (cp)]

Attached is a patch which seems to fix the bug for Ubuntu Trusty or 14.04.

Please test the package from my PPA and confirm it works OK.
https://launchpad.net/~cp/+archive/ubuntu/bug-fixes/+packages

The easy way to install it:
sudo add-apt-repository ppa:cp/bug-fixes
sudo apt-get update
sudo apt-get upgrade

Changelog

rsync (3.1.0-2ubuntu0.2) trusty; urgency=medium

  * Use included zlib as transfers failed when using -z with the separate zlib.
    (LP: #11300367)
    - debian/rules: Add -Izlib to CPPFLAGS and change .configure to use
      -with-included-zlib=yes
    - debian/control: Remove Build-Depends zlib1g-dev

My testing shows transfers with the -z option are working, but the real test will be when my backups run late tonight. ;)

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Use rsync's internal zlib" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Robie Basak (racb)]

Thank you for the patch!

Unfortunately using rsync's internal zlib violates policy section 4.13 (see in particular footnote 31 for rationale): http://people.canonical.com/~cjwatson/ubuntu-policy/policy.html/ch-source.html#s-embeddedfiles

For example: if a security vulnerability were found in zlib, then the security team expects to be able to fix it once in the archive. If we used rsync's internal zlib, then it would escape being fixed, and this would be dangerous.

However, from the changelog I see the Debian does the same thing:

  * revert to using included zlib as there have been numerous reports of failed
    transfers when using -z with the separate zlib.

So perhaps we need to make an exception in this case.

To update Trusty, we need to follow the SRU procedure documented here: https://wiki.ubuntu.com/StableReleaseUpdates#Procedure and submit the patch for sponsoring as described in the link from there. Please could you see how far you can get with this?

[Nick Craig-Wood (nick-craig-wood)]

I tried the patch and I can confirm it worked once. However it didn't fail every time before so it probably needs a bit more testing.

[Steve (lp-z)]

Just wanted to note that the patch in #6 has been working fine for me. Any luck on getting it patched into Trusty?

[Anakan (planetaz)]

Simply move compression from rsync to ssh: -a -e "ssh -C"

[Gabriel Staples (ercaguy)]

Work-around: just upgrade your rsync versions to ensure they are *exactly the same* on both the sending and receiving PCs. See my answer here: https://serverfault.com/questions/883487/unable-to-rsync-due-to-broken-pipe/988794#988794.