rsync hangs on select(5, [], [4], [], {60, 0}
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
rsync |
Unknown
|
Unknown
|
|||
rsync (Ubuntu) |
Fix Released
|
Medium
|
Unassigned | ||
Bionic |
Fix Released
|
Medium
|
Miriam España Acebal | ||
Focal |
Fix Released
|
Medium
|
Miriam España Acebal |
Bug Description
[Impact]
What the user suffering from this bug experiences is that the big amount of informative messages related to the copy process with the three spawned processes(sender, receiver and generator) exhaust the I/O buffer and the sync gets stuck, either because there are too many files to synchronise and/or because too many detail messages (levels of verbose mode) have been requested in the output.
The fix, that comes from upstream and is applied there since version 3.2.0., increments the size of the receiver's I/O buffer.
[Test Plan]
This test plan is for Focal, but it's the same for Bionic.
0.Preparing the test environment:
#Preparing the container
lxc launch images:ubuntu/focal rsync-iobuffer-
lxc shell rsync-iobuffer-
apt update -y
apt upgrade -y
#Installing necessary tools
apt install rsync
#Get test cases from comments #16 and #19 on this LP bug: As test case #16 covers both aspects (a lot of files and upper verbosity) and test #19 uses a huge tarball (120 Mb), I'm removing from this SRU the #19 scenario (but, please, feel to reach me it if you consider it necessary and I'll provide the steps and bad/good scenarios).
cd /tmp/
#16
Paste the contents of https:/
#!/bin/bash
mkdir source_dir
pushd source_dir
dd if=/dev/zero of=source bs=600K count=1
for i in `seq 1 11500`;
do
cp -v source file_$i;
done
rm source
for i in `seq 1 10`;
do
dd if=/dev/zero of=file_large_$i bs=200M count=1
done
popd
echo "Created 11500 files with size 600K and 10 files with size 200M, try the following command:"
echo "rsync -avvvz --delete source_dir target_dir"
in a new file script_comment16.sh
chmod +x script_comment16.sh
./script_
1. Bad cases (without and with using strace):
# Scenario from comment 16
$ rsync -avvvz --delete source_dir target_dir
sending incremental file list
[sender] make_file(
send_file_list done
[sender] pushing local filters for /root/source_dir/
[sender] make_file(
[sender] make_file(
[sender] make_file(
[sender] make_file(
[sender] make_file(
[sender] make_file(
[...]
sending file_sum
false_alarms=0 hash_hits=0 matches=0
sender finished source_
send_files(903, source_
send_files mapped source_
calling match_sums source_
source_
It hangs here, where using strace we can see:
$ strace rsync -avvvz --delete source_dir target_dir
source_
read(3, "\0\0\0\
read(3, "\0\0\0\
read(3, "\0\0\0\
select(6, [5], [4], [5], {tv_sec=60, tv_usec=0}) = 1 (in [5], left {tv_sec=59, tv_usec=999996})
read(5, "\0\0\0\
select(5, [], [4], [], {tv_sec=60, tv_usec=0}) = 0 (Timeout)
select(5, [], [4], [], {tv_sec=60, tv_usec=0}) = 0 (Timeout)
select(5, [], [4], [], {tv_sec=60, tv_usec=0}
1. Good cases:
# Scenario from comment 16
$ rsync -avvvz --delete source_dir target_dir
sending incremental file list
[sender] make_file(
send_file_list done
[sender] pushing local filters for /tmp/source_dir/
[sender] make_file(
[sender] make_file(
[sender] make_file(
[sender] make_file(
[sender] make_file(
[sender] make_file(
[sender] make_file(
[sender] make_file(
[sender] make_file(
[sender] make_file(
[...]
generate_files finished
sent 9,555,678 bytes received 3,599,560 bytes 124,694.20 bytes/sec
total size is 9,162,752,000 speedup is 696.51
[Where problems could occur]
Perhaps the buffer size may not be sufficient for an operation involving a very huge amount of data, as reported upstream here ( https:/
[Other Info]
The fix has been picked from upstream here https:/
[Original Report]
-------
In the last few months my home directory backup stopped completing. I've been able to reproduce the problem on a single subdirectory although I had to add the --debug=all flag to reproduce it on that smaller directory. Specifically, this command never completes:
rsync --debug=all -avz /tmp/html2 /tmp/rsynctest/
The html2 directory is a copy of gnuradio-
When I strace the command, I see this:
write(1, "sender finished /tmp/html2/
) = 58
write(1, "send_files(338, /tmp/html2/
) = 59
open("html2/
fstat64(3, {st_mode=
write(1, "html2/
) = 37
read(3, "\211PNG\
select(6, [5], [4], [5], {60, 0}) = 2 (in [5], out [4], left {59, 999996})
read(5, "\0\0\0\
write(4, "r\311\
select(5, [], [4], [], {60, 0}) = 1 (out [4], left {59, 999997})
write(4, "\7\320\
select(5, [], [4], [], {60, 0}
The select command times out over and over. I get the same behavior when trying to back up my entire home directory but I don't need the --debug=all flag in that case.
lsb_release -rd
Description: Ubuntu 14.04.3 LTS
Release: 14.04
apt-cache policy rsync
rsync:
Installed: 3.1.0-2ubuntu0.1
Candidate: 3.1.0-2ubuntu0.1
Version table:
*** 3.1.0-2ubuntu0.1 0
500 http://
500 http://
100 /var/lib/
3.1.0-2 0
500 http://
ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: rsync 3.1.0-2ubuntu0.1
ProcVersionSign
Uname: Linux 3.13.0-74-generic i686
NonfreeKernelMo
ApportVersion: 2.14.1-0ubuntu3.19
Architecture: i386
CurrentDesktop: KDE
Date: Wed Dec 23 09:44:17 2015
EcryptfsInUse: Yes
InstallationDate: Installed on 2010-09-18 (1922 days ago)
InstallationMedia: Ubuntu 10.10 "Maverick Meerkat" - Beta i386 (20100901.1)
SourcePackage: rsync
UpgradeStatus: Upgraded to trusty on 2014-12-27 (361 days ago)
Related branches
- Andreas Hasenack: Approve
- Canonical Server: Pending requested
-
Diff: 69 lines (+55/-0)2 files modifieddebian/changelog (+11/-0)
debian/patches/avoid-deadlock-huge-amounts-verbose-messages.patch (+44/-0)
- Andreas Hasenack: Approve
- Canonical Server: Pending requested
-
Diff: 78 lines (+56/-0)3 files modifieddebian/changelog (+11/-0)
debian/patches/avoid-deadlock-huge-amounts-verbose-messages.patch (+44/-0)
debian/patches/series (+1/-0)
Changed in rsync (Ubuntu Bionic): | |
assignee: | nobody → Miriam España Acebal (mirespace) |
Changed in rsync (Ubuntu Focal): | |
assignee: | nobody → Miriam España Acebal (mirespace) |
description: | updated |
Changed in rsync (Ubuntu Focal): | |
status: | New → In Progress |
description: | updated |
description: | updated |
description: | updated |
Changed in rsync (Ubuntu Bionic): | |
status: | New → In Progress |
The same problem happens on a 15.04 system:
lsb_release -rd us.archive. ubuntu. com/ubuntu/ vivid/main amd64 Packages dpkg/status
Description: Ubuntu 15.04
Release: 15.04
stl@stl6:/tmp$ apt-cache policy rsync
rsync:
Installed: 3.1.1-3
Candidate: 3.1.1-3
Version table:
*** 3.1.1-3 0
500 http://
100 /var/lib/