apt-get update very slow when ulimit -n is big

Bug #1332440 reported by tom916 on 2014-06-20
64
This bug affects 12 people
Affects Status Importance Assigned to Milestone
apt (Ubuntu)
Undecided
Unassigned
Trusty
Undecided
Julian Andres Klode

Bug Description

[Impact]
apt is slow at spawning subprocesses, as it calls fcntl() on each file descriptor, causing significant slowdown. On my container:

0.71user 1.26system 0:01.98elapsed 99%CPU (0avgtext+0avgdata 34164maxresident)k

0.14user 0.05system 0:00.20elapsed 99%CPU (0avgtext+0avgdata 34428maxresident)k

- a 10 times difference.

[Test case]
(1) Run strace -f -c apt-cache search test >/dev/null, count getrlimit() and fcntl() calls
(2) Upgrade libapt-pkg4.12 to proposed
(3) Run (1) again, make sure it's significantly lower.

[Regression potential]
Isolated fix in closing "unwanted" file descriptors. Has been in use for years now, but if there were a regression it would probably be an fd leak.

[Original bug report]
I use strace to see why apt-get update is too slow,
finally I found apt-get loop 3~n to '(? , F_SETFD, FD_CLOEXEC) = 0'

ubuntu 14.04

root@dev2:/tmp/apt-1.0.1ubuntu2.1# apt-cache show apt
Package: apt
Priority: important
Section: admin
Installed-Size: 3576
Maintainer: Ubuntu Developers <email address hidden>
Original-Maintainer: APT Development Team <email address hidden>
Architecture: amd64
Version: 1.0.1ubuntu2.1
Replaces: manpages-it (<< 2.80-4~), manpages-pl (<< 20060617-3~), openjdk-6-jdk (<< 6b24-1.11-0ubuntu1~), sun-java5-jdk (>> 0), sun-java6-jdk (>> 0)
Depends: libapt-pkg4.12 (>= 0.9.16), libc6 (>= 2.15), libgcc1 (>= 1:4.1.1), libstdc++6 (>= 4.6), ubuntu-keyring, gnupg
Suggests: aptitude | synaptic | wajig, dpkg-dev (>= 1.17.2), apt-doc, python-apt
Conflicts: python-apt (<< 0.7.93.2~)
Breaks: manpages-it (<< 2.80-4~), manpages-pl (<< 20060617-3~), openjdk-6-jdk (<< 6b24-1.11-0ubuntu1~), sun-java5-jdk (>> 0), sun-java6-jdk (>> 0)
Filename: pool/main/a/apt/apt_1.0.1ubuntu2.1_amd64.deb
Size: 952374

[pid 20525] stat("/var/lib/apt/lists/mirrors.163.com_ubuntu_dists_trusty-updates_universe_i18n_Translation-en%5fUS", 0x7fffaabe5b30) = -1 ENOENT (No such file or directory)
[pid 20525] stat("/usr/lib/apt/methods/bzip2", {st_mode=S_IFREG|0755, st_size=23056, ...}) = 0
[pid 20525] pipe([9, 11]) = 0
[pid 20525] pipe([13, 14]) = 0
[pid 20525] fcntl(9, F_SETFD, FD_CLOEXEC) = 0
[pid 20525] fcntl(11, F_SETFD, FD_CLOEXEC) = 0
[pid 20525] fcntl(13, F_SETFD, FD_CLOEXEC) = 0
[pid 20525] fcntl(14, F_SETFD, FD_CLOEXEC) = 0
[pid 20525] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f99f1ea9a50) = 20538
Process 20538 attached
[pid 20525] fcntl(9, F_GETFL) = 0 (flags O_RDONLY)
[pid 20525] fcntl(9, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
[pid 20538] rt_sigaction(SIGPIPE, {SIG_DFL, [PIPE], SA_RESTORER|SA_RESTART, 0x7f99f0e72ff0}, <unfinished ...>
[pid 20525] fcntl(14, F_GETFL <unfinished ...>
[pid 20538] <... rt_sigaction resumed> {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f99f0e72ff0}, 8) = 0
[pid 20525] <... fcntl resumed> ) = 0x1 (flags O_WRONLY)
[pid 20538] rt_sigaction(SIGQUIT, {SIG_DFL, [QUIT], SA_RESTORER|SA_RESTART, 0x7f99f0e72ff0}, <unfinished ...>
[pid 20525] fcntl(14, F_SETFL, O_WRONLY|O_NONBLOCK <unfinished ...>
[pid 20538] <... rt_sigaction resumed> {SIG_DFL, [], 0}, 8) = 0
[pid 20525] <... fcntl resumed> ) = 0
[pid 20538] rt_sigaction(SIGINT, {SIG_DFL, [INT], SA_RESTORER|SA_RESTART, 0x7f99f0e72ff0}, <unfinished ...>
[pid 20525] close(11 <unfinished ...>
[pid 20538] <... rt_sigaction resumed> {SIG_DFL, [], 0}, 8) = 0
[pid 20525] <... close resumed> ) = 0
[pid 20538] rt_sigaction(SIGWINCH, {SIG_DFL, [WINCH], SA_RESTORER|SA_RESTART, 0x7f99f0e72ff0}, <unfinished ...>
[pid 20525] close(13 <unfinished ...>
[pid 20538] <... rt_sigaction resumed> {0x409140, [WINCH], SA_RESTORER|SA_RESTART, 0x7f99f0e72ff0}, 8) = 0
[pid 20525] <... close resumed> ) = 0
[pid 20538] rt_sigaction(SIGCONT, {SIG_DFL, [CONT], SA_RESTORER|SA_RESTART, 0x7f99f0e72ff0}, <unfinished ...>
[pid 20525] select(10, [9], NULL, NULL, NULL <unfinished ...>
[pid 20538] <... rt_sigaction resumed> {SIG_DFL, [], 0}, 8) = 0
[pid 20538] rt_sigaction(SIGTSTP, {SIG_DFL, [TSTP], SA_RESTORER|SA_RESTART, 0x7f99f0e72ff0}, {SIG_DFL, [], 0}, 8) = 0
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(3, F_SETFD, FD_CLOEXEC) = 0
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(4, F_SETFD, FD_CLOEXEC) = 0
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(5, F_SETFD, FD_CLOEXEC) = 0
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(6, F_SETFD, FD_CLOEXEC) = 0
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(7, F_SETFD, FD_CLOEXEC) = 0
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(8, F_SETFD, FD_CLOEXEC) = 0
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(9, F_SETFD, FD_CLOEXEC) = 0
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(10, F_SETFD, FD_CLOEXEC) = 0
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(11, F_SETFD, FD_CLOEXEC) = 0
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(12, F_SETFD, FD_CLOEXEC) = 0
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(13, F_SETFD, FD_CLOEXEC) = 0
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(14, F_SETFD, FD_CLOEXEC) = 0
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(15, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(16, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(17, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(18, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(19, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(20, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(21, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(22, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(23, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(24, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(25, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(26, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(27, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(28, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(29, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(30, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(31, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(32, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(33, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(34, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(35, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(36, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(37, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(38, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(39, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(40, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(41, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(42, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(43, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(44, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(45, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(46, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(47, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(48, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(49, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(50, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(51, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(52, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(53, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(54, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(55, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(56, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(57, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(58, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(59, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(60, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(61, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(62, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(63, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(64, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(65, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(66, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(67, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(68, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(69, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(70, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(71, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(72, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(73, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(74, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(75, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(76, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(77, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(78, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(79, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(80, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(81, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(82, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(83, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(84, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(85, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(86, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(87, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(88, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(89, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(90, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(91, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(92, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(93, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(94, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(95, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(96, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(97, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(98, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(99, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] dup2(11, 1) = 1
[pid 20538] dup2(13, 0) = 0
[pid 20538] fcntl(1, F_SETFD, 0) = 0
[pid 20538] fcntl(0, F_SETFD, 0) = 0

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: apt 1.0.1ubuntu2
ProcVersionSignature: Ubuntu 3.13.0-24.46-generic 3.13.9
Uname: Linux 3.13.0-24-generic x86_64
ApportVersion: 2.14.1-0ubuntu3
Architecture: amd64
Date: Fri Jun 20 15:36:00 2014
InstallationDate: Installed on 2011-10-08 (985 days ago)
InstallationMedia: Ubuntu-Server 11.04 "Natty Narwhal" - Release amd64 (20110426)
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: apt
UpgradeStatus: Upgraded to trusty on 2014-04-29 (51 days ago)

tom916 (tom916) wrote :

Thanks for taking the time to report this bug and helping to make Ubuntu better. We appreciate the difficulties you are facing, but this appears to be a "regular" (non-security) bug. I have unmarked it as a security issue since this bug does not show evidence of allowing attackers to cross privilege boundaries nor directly cause loss of data/privacy. Please feel free to report any other bugs you may find.

information type: Private Security → Public
Seth Arnold (seth-arnold) wrote :

Setting 100,000 file descriptors to close-on-exec while re-checking the limits each iteration takes 0.092 seconds on my laptop.

Just how large did you set your ulimits?

tom916 (tom916) wrote :

@ seth-arnold

I set ulimit 2560000 on my machine,here is the test.

My question is Why in new version of apt-get,it's must check file descriptors?

root@dev2:~# ulimit -n 2560000
root@dev2:~# time apt-get update > /dev/null

real 0m32.587s
user 0m19.211s
sys 0m11.127s
root@dev2:~# ulimit -n 1000
root@dev2:~# time apt-get update > /dev/null

real 0m3.357s
user 0m2.624s
sys 0m0.339s

Ian Babrou (bobrik) wrote :

Building docker image with ulimit -n 500k takes literally forever in virtualbox. Iterating 500k fds that are not even open on every fork in apt (and there are many of them) is not okay.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in apt (Ubuntu):
status: New → Confirmed
TimK. (f-ubsntu-o) wrote :

Same problem with Docker containers, apt-get update take forever or just fails.

Jon Skarpeteig (lesspublic) wrote :

Running apt-get update takes ~80 seconds for me running:

* Virtualbox 4.3.12
* Ubuntu 14.04 LTS 64 bit desktop (3.13.0-30-generic #55-Ubuntu SMP)
* Docker version 1.0.1, build 990021a
* Official 'ubuntu:14.04' docker image

In the VirtualBox machine it takes < 10s

Running a similar setup in VMware does not display this behavior:

* VMware ESXi 5.5
* Ubuntu 14.04 LTS 64 bit server (3.13.0-30-generic #55-Ubuntu SMP)
* Docker version 1.0.1, build 990021a

apt-get executed in 9.87 seconds
* Official 'ubuntu:14.04' docker image

Jason (jasoncwatt) wrote :

I can also confirm this running docker with the library/ubuntu:14.04 image on a virtualbox host running ubuntu 14.04

running apt-get update times out or takes an extremely long time.

Host Virtualbox:

$ ulimit -n
1024

docker container:

root@1518e024f53a:/# ulimit -n
524288

root@1518e024f53a:/# time apt-get update

real 0m47.489s
user 0m24.408s
sys 0m21.516s

root@1518e024f53a:/# ulimit -n 100000
root@1518e024f53a:/# time apt-get update

real 0m11.165s
user 0m6.598s
sys 0m4.359s

root@1518e024f53a:/# ulimit -n 1024
root@1518e024f53a:/# time apt-get update

real 0m3.109s
user 0m2.543s
sys 0m0.374s

Ian Babrou (bobrik) wrote :

Looks like it was fixed in some update. Can anyone confirm that?

Scrivs (michael-a-endsley) wrote :

If it's been fixed, I don't know what in. This still appears to be an issue for me. If I've failed to apply some update that fixed it, I'd love to know what it was.

Ian Babrou (bobrik) wrote :
Download full text (16.7 KiB)

Yep, it's not fixed:

~ λ docker run --rm ubuntu:14.04 bash -c 'ulimit -n 500000 && apt-get install strace && strace -c -f apt-get update'
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  strace
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 113 kB of archives.
After this operation, 504 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu/ trusty/main strace amd64 4.8-1ubuntu5 [113 kB]
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (This frontend requires a controlling tty.)
debconf: falling back to frontend: Teletype
dpkg-preconfigure: unable to re-open stdin:
Fetched 113 kB in 0s (179 kB/s)
Selecting previously unselected package strace.
(Reading database ... 11527 files and directories currently installed.)
Preparing to unpack .../strace_4.8-1ubuntu5_amd64.deb ...
Unpacking strace (4.8-1ubuntu5) ...
Setting up strace (4.8-1ubuntu5) ...
Process 47 attached
Process 48 attached
Process 49 attached
Ign http://archive.ubuntu.com trusty InRelease
Ign http://archive.ubuntu.com trusty-updates InRelease
Ign http://archive.ubuntu.com trusty-security InRelease
Hit http://archive.ubuntu.com trusty Release.gpg
Get:1 http://archive.ubuntu.com trusty-updates Release.gpg [933 B]
Get:2 http://archive.ubuntu.com trusty-security Release.gpg [933 B]
Process 50 attached
Process 51 attached
Hit http://archive.ubuntu.com trusty Release
Get:3 http://archive.ubuntu.com trusty-updates Release [63.5 kB]
Process 52 attached
Process 53 attached
Get:4 http://archive.ubuntu.com trusty-security Release [63.5 kB]
Process 54 attached
Get:5 http://archive.ubuntu.com trusty/main Sources [1335 kB]
Process 55 attached
Process 56 attached
^CGet:6 http://archive.ubuntu.com trusty/restricted Sources [5335 B]
Get:7 http://archive.ubuntu.com trusty/universe Sources [7926 kB]
Get:8 http://archive.ubuntu.com trusty/main amd64 Packages [1743 kB]
Get:9 http://archive.ubuntu.com trusty/restricted amd64 Packages [16.0 kB]
Get:10 http://archive.ubuntu.com trusty/universe amd64 Packages [7589 kB]
Get:11 http://archive.ubuntu.com trusty-updates/main Sources [248 kB]
Get:12 http://archive.ubuntu.com trusty-updates/restricted Sources [2310 B]
Get:13 http://archive.ubuntu.com trusty-updates/universe Sources [141 kB]
Get:14 http://archive.ubuntu.com trusty-updates/main amd64 Packages [636 kB]
Get:15 http://archive.ubuntu.com trusty-updates/restricted amd64 Packages [15.1 kB]
Get:16 http://archive.ubuntu.com trusty-updates/universe amd64 Packages [356 kB]
Get:17 http://archive.ubuntu.com trusty-security/main Sources [97.7 kB]
Get:18 http://archive.ubuntu.com trusty-security/restricted Sources [1874 B]
Get:19 http://archive.ubuntu.com trusty-security/universe Sources [23.3 kB]
Get:20 http://archive.ubuntu.com trusty-security/main amd64 Packages [329 kB]
Get:21 http://archive.ubuntu.com trusty-security/restricted amd64 Packages [14.8 kB]
Get:22 http://archive.ubuntu.com trusty-security/universe amd...

Jon Skarpeteig (jskarpet) wrote :

When running Ubuntu 14.04 inside Docker I've seen this issue with CentOS as well:

CentOS 7.2
3.10.0-327.4.4.el7.x86_64

I tested a few kernels with Ubuntu 14.04 as host OS and I found:

apt-get update extremely slow on kernels:

3.13.0-76
3.13.0-65
3.13.0-30
3.13.0-29

apt-get update fast on kernel:

3.13.0-45

So my workaround for now is to install and run kernel 3.13.0-45

My working setup is;

# docker --version
Docker version 1.9.1, build a34a1d5
# uname -a
Linux lys-jenkins 3.13.0-45-generic #74-Ubuntu SMP Tue Jan 13 19:36:28 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

I'm still seeing this as a problem (I think) on Ubuntu 15.10 Wily, kernel 4.2.0-25-generic.

Apologies, I retract that -- looks like the issue was related to MTU and PMTU discovery.. It just happened that the machines with the mtu issues were also the ones with huge ulimits configured.

Changed in apt (Ubuntu):
status: Confirmed → Fix Released
Changed in apt (Ubuntu Trusty):
status: New → Triaged
assignee: nobody → Julian Andres Klode (juliank)
Julian Andres Klode (juliank) wrote :

This was fixed in 1.0.10 or something, and I just uploaded a fixed version to trusty (1.0.1ubuntu2.18).

description: updated
Changed in apt (Ubuntu Trusty):
status: Triaged → In Progress
description: updated

Hello tom916, or anyone else affected,

Accepted apt into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.0.1ubuntu2.18 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-trusty to verification-done-trusty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-trusty. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in apt (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-trusty
Julian Andres Klode (juliank) wrote :
Download full text (5.0 KiB)

Verified. (BTW, I aborted the initial run as it simply took too long so actual numbers would have been much higher)

root@t:~# apt-cache policy libapt-pkg4.12
libapt-pkg4.12:
  Installed: 1.0.1ubuntu2.17
  Candidate: 1.0.1ubuntu2.18
  Version table:
     1.0.1ubuntu2.18 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-proposed/main amd64 Packages
 *** 1.0.1ubuntu2.17 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
        100 /var/lib/dpkg/status
     1.0.1ubuntu2 0
        500 http://archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
root@t:~# strace -f -c apt-cache search test >/dev/null
Process 2338 attached
^CProcess 2337 detached
Process 2338 detached
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
 50.32 0.827909 3 329234 getrlimit
 49.55 0.815234 2 329234 329227 fcntl
  0.03 0.000510 10 50 mmap
  0.02 0.000407 10 40 open
  0.01 0.000225 10 23 mprotect
  0.01 0.000212 6 34 read
  0.01 0.000209 5 40 close
  0.01 0.000202 7 29 fstat
  0.01 0.000167 13 13 13 access
  0.00 0.000044 6 8 2 stat
  0.00 0.000027 14 2 getdents
  0.00 0.000018 9 2 munmap
  0.00 0.000018 6 3 brk
  0.00 0.000016 16 1 execve
  0.00 0.000008 8 1 pipe
  0.00 0.000007 7 1 openat
  0.00 0.000004 4 1 1 lseek
  0.00 0.000004 4 1 clone
  0.00 0.000003 3 1 1 ioctl
  0.00 0.000003 3 1 arch_prctl
  0.00 0.000000 0 6 rt_sigaction
------ ----------- ----------- --------- --------- ----------------
100.00 1.645227 658725 329244 total
root@t:~# apt install libapt-pkg4.12
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libfreetype6 os-prober
Use 'apt-get autoremove' to remove them.
The following packages will be upgraded:
  libapt-pkg4.12
1 upgraded, 0 newly installed, 0 to remove and 18 not upgraded.
Need to get 638 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu/ trusty-proposed/main libapt-pkg4.12 amd64 1.0.1ubuntu2.18 [638 kB]
Fetched 638 kB in 0s (687 kB/s)
(Reading database ... 25096 files and directories currently installed.)
Preparing to unpack .../libapt-pkg4.12_1.0.1ubuntu2.18_amd64.deb ...
Unpacking libapt-pkg4.12:amd64 (1.0.1ubuntu2.18) over (1.0.1ubuntu2.17) ...
Setting up libapt-pkg4.12:amd64 (1.0.1ubuntu2.18) ...
Processing triggers fo...

Read more...

tags: added: verification-done verification-done-trusty
removed: verification-needed verification-needed-trusty
Julian Andres Klode (juliank) wrote :

Time for apt-cache policy apt also decreased from 1.86 seconds to 0.04 seconds :)

Robie Basak (racb) wrote :

There are some dep8 regressions reported in http://people.canonical.com/~ubuntu-archive/pending-sru.html. I've requested some retries. If they're still failing, please could you take a look?

Julian Andres Klode (juliank) wrote :

I have verified that none of these failures are regressions; they all appear in previous apt uploads/run against other triggers too.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apt - 1.0.1ubuntu2.18

---------------
apt (1.0.1ubuntu2.18) trusty; urgency=medium

  * ExecFork: Use /proc/self/fd to determine which files to close
    (Closes: #764204) (LP: #1332440).

 -- Julian Andres Klode <email address hidden> Mon, 09 Apr 2018 15:32:09 +0200

Changed in apt (Ubuntu Trusty):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for apt has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers