apt-get update very slow when ulimit -n is big

[tom916]

[Impact]
apt is slow at spawning subprocesses, as it calls fcntl() on each file descriptor, causing significant slowdown. On my container:

0.71user 1.26system 0:01.98elapsed 99%CPU (0avgtext+0avgdata 34164maxresident)k

0.14user 0.05system 0:00.20elapsed 99%CPU (0avgtext+0avgdata 34428maxresident)k

- a 10 times difference.

[Test case]
(1) Run strace -f -c apt-cache search test >/dev/null, count getrlimit() and fcntl() calls
(2) Upgrade libapt-pkg4.12 to proposed
(3) Run (1) again, make sure it's significantly lower.

[Regression potential]
Isolated fix in closing "unwanted" file descriptors. Has been in use for years now, but if there were a regression it would probably be an fd leak.

[Original bug report]
I use strace to see why apt-get update is too slow,
finally I found apt-get loop 3~n to '(? , F_SETFD, FD_CLOEXEC) = 0'

ubuntu 14.04

root@dev2:/tmp/apt-1.0.1ubuntu2.1# apt-cache show apt
Package: apt
Priority: important
Section: admin
Installed-Size: 3576
Maintainer: Ubuntu Developers <email address hidden>
Original-Maintainer: APT Development Team <email address hidden>
Architecture: amd64
Version: 1.0.1ubuntu2.1
Replaces: manpages-it (<< 2.80-4~), manpages-pl (<< 20060617-3~), openjdk-6-jdk (<< 6b24-1.11-0ubuntu1~), sun-java5-jdk (>> 0), sun-java6-jdk (>> 0)
Depends: libapt-pkg4.12 (>= 0.9.16), libc6 (>= 2.15), libgcc1 (>= 1:4.1.1), libstdc++6 (>= 4.6), ubuntu-keyring, gnupg
Suggests: aptitude | synaptic | wajig, dpkg-dev (>= 1.17.2), apt-doc, python-apt
Conflicts: python-apt (<< 0.7.93.2~)
Breaks: manpages-it (<< 2.80-4~), manpages-pl (<< 20060617-3~), openjdk-6-jdk (<< 6b24-1.11-0ubuntu1~), sun-java5-jdk (>> 0), sun-java6-jdk (>> 0)
Filename: pool/main/a/apt/apt_1.0.1ubuntu2.1_amd64.deb
Size: 952374

[pid 20525] stat("/var/lib/apt/lists/mirrors.163.com_ubuntu_dists_trusty-updates_universe_i18n_Translation-en%5fUS", 0x7fffaabe5b30) = -1 ENOENT (No such file or directory)
[pid 20525] stat("/usr/lib/apt/methods/bzip2", {st_mode=S_IFREG|0755, st_size=23056, ...}) = 0
[pid 20525] pipe([9, 11]) = 0
[pid 20525] pipe([13, 14]) = 0
[pid 20525] fcntl(9, F_SETFD, FD_CLOEXEC) = 0
[pid 20525] fcntl(11, F_SETFD, FD_CLOEXEC) = 0
[pid 20525] fcntl(13, F_SETFD, FD_CLOEXEC) = 0
[pid 20525] fcntl(14, F_SETFD, FD_CLOEXEC) = 0
[pid 20525] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f99f1ea9a50) = 20538
Process 20538 attached
[pid 20525] fcntl(9, F_GETFL) = 0 (flags O_RDONLY)
[pid 20525] fcntl(9, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
[pid 20538] rt_sigaction(SIGPIPE, {SIG_DFL, [PIPE], SA_RESTORER|SA_RESTART, 0x7f99f0e72ff0}, <unfinished ...>
[pid 20525] fcntl(14, F_GETFL <unfinished ...>
[pid 20538] <... rt_sigaction resumed> {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f99f0e72ff0}, 8) = 0
[pid 20525] <... fcntl resumed> ) = 0x1 (flags O_WRONLY)
[pid 20538] rt_sigaction(SIGQUIT, {SIG_DFL, [QUIT], SA_RESTORER|SA_RESTART, 0x7f99f0e72ff0}, <unfinished ...>
[pid 20525] fcntl(14, F_SETFL, O_WRONLY|O_NONBLOCK <unfinished ...>
[pid 20538] <... rt_sigaction resumed> {SIG_DFL, [], 0}, 8) = 0
[pid 20525] <... fcntl resumed> ) = 0
[pid 20538] rt_sigaction(SIGINT, {SIG_DFL, [INT], SA_RESTORER|SA_RESTART, 0x7f99f0e72ff0}, <unfinished ...>
[pid 20525] close(11 <unfinished ...>
[pid 20538] <... rt_sigaction resumed> {SIG_DFL, [], 0}, 8) = 0
[pid 20525] <... close resumed> ) = 0
[pid 20538] rt_sigaction(SIGWINCH, {SIG_DFL, [WINCH], SA_RESTORER|SA_RESTART, 0x7f99f0e72ff0}, <unfinished ...>
[pid 20525] close(13 <unfinished ...>
[pid 20538] <... rt_sigaction resumed> {0x409140, [WINCH], SA_RESTORER|SA_RESTART, 0x7f99f0e72ff0}, 8) = 0
[pid 20525] <... close resumed> ) = 0
[pid 20538] rt_sigaction(SIGCONT, {SIG_DFL, [CONT], SA_RESTORER|SA_RESTART, 0x7f99f0e72ff0}, <unfinished ...>
[pid 20525] select(10, [9], NULL, NULL, NULL <unfinished ...>
[pid 20538] <... rt_sigaction resumed> {SIG_DFL, [], 0}, 8) = 0
[pid 20538] rt_sigaction(SIGTSTP, {SIG_DFL, [TSTP], SA_RESTORER|SA_RESTART, 0x7f99f0e72ff0}, {SIG_DFL, [], 0}, 8) = 0
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(3, F_SETFD, FD_CLOEXEC) = 0
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(4, F_SETFD, FD_CLOEXEC) = 0
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(5, F_SETFD, FD_CLOEXEC) = 0
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(6, F_SETFD, FD_CLOEXEC) = 0
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(7, F_SETFD, FD_CLOEXEC) = 0
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(8, F_SETFD, FD_CLOEXEC) = 0
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(9, F_SETFD, FD_CLOEXEC) = 0
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(10, F_SETFD, FD_CLOEXEC) = 0
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(11, F_SETFD, FD_CLOEXEC) = 0
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(12, F_SETFD, FD_CLOEXEC) = 0
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(13, F_SETFD, FD_CLOEXEC) = 0
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(14, F_SETFD, FD_CLOEXEC) = 0
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(15, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(16, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(17, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(18, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(19, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(20, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(21, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(22, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(23, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(24, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(25, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(26, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(27, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(28, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(29, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(30, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(31, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(32, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(33, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(34, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(35, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(36, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(37, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(38, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(39, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(40, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(41, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(42, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(43, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(44, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(45, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(46, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(47, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(48, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(49, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(50, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(51, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(52, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(53, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(54, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(55, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(56, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(57, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(58, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(59, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(60, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(61, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(62, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(63, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(64, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(65, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(66, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(67, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(68, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(69, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(70, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(71, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(72, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(73, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(74, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(75, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(76, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(77, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(78, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(79, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(80, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(81, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(82, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(83, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(84, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(85, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(86, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(87, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(88, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(89, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(90, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(91, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(92, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(93, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(94, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(95, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(96, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(97, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(98, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] fcntl(99, F_SETFD, FD_CLOEXEC) = -1 EBADF (Bad file descriptor)
[pid 20538] getrlimit(RLIMIT_NOFILE, {rlim_cur=100, rlim_max=100}) = 0
[pid 20538] dup2(11, 1) = 1
[pid 20538] dup2(13, 0) = 0
[pid 20538] fcntl(1, F_SETFD, 0) = 0
[pid 20538] fcntl(0, F_SETFD, 0) = 0

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: apt 1.0.1ubuntu2
ProcVersionSignature: Ubuntu 3.13.0-24.46-generic 3.13.9
Uname: Linux 3.13.0-24-generic x86_64
ApportVersion: 2.14.1-0ubuntu3
Architecture: amd64
Date: Fri Jun 20 15:36:00 2014
InstallationDate: Installed on 2011-10-08 (985 days ago)
InstallationMedia: Ubuntu-Server 11.04 "Natty Narwhal" - Release amd64 (20110426)
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: apt
UpgradeStatus: Upgraded to trusty on 2014-04-29 (51 days ago)

[Seth Arnold (seth-arnold)]

Thanks for taking the time to report this bug and helping to make Ubuntu better. We appreciate the difficulties you are facing, but this appears to be a "regular" (non-security) bug. I have unmarked it as a security issue since this bug does not show evidence of allowing attackers to cross privilege boundaries nor directly cause loss of data/privacy. Please feel free to report any other bugs you may find.

[Seth Arnold (seth-arnold)]

Setting 100,000 file descriptors to close-on-exec while re-checking the limits each iteration takes 0.092 seconds on my laptop.

Just how large did you set your ulimits?

[tom916 (tom916)]

@ seth-arnold

I set ulimit 2560000 on my machine,here is the test.

My question is Why in new version of apt-get,it's must check file descriptors?

root@dev2:~# ulimit -n 2560000
root@dev2:~# time apt-get update > /dev/null

real 0m32.587s
user 0m19.211s
sys 0m11.127s
root@dev2:~# ulimit -n 1000
root@dev2:~# time apt-get update > /dev/null

real 0m3.357s
user 0m2.624s
sys 0m0.339s

[Ian Babrou (bobrik)]

Building docker image with ulimit -n 500k takes literally forever in virtualbox. Iterating 500k fds that are not even open on every fork in apt (and there are many of them) is not okay.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Tim K. (tkubnt)]

Same problem with Docker containers, apt-get update take forever or just fails.

[Jon Skarpeteig (lesspublic)]

Running apt-get update takes ~80 seconds for me running:

* Virtualbox 4.3.12
* Ubuntu 14.04 LTS 64 bit desktop (3.13.0-30-generic #55-Ubuntu SMP)
* Docker version 1.0.1, build 990021a
* Official 'ubuntu:14.04' docker image

In the VirtualBox machine it takes < 10s

Running a similar setup in VMware does not display this behavior:

* VMware ESXi 5.5
* Ubuntu 14.04 LTS 64 bit server (3.13.0-30-generic #55-Ubuntu SMP)
* Docker version 1.0.1, build 990021a

apt-get executed in 9.87 seconds
* Official 'ubuntu:14.04' docker image

[Jason (jasoncwatt)]

I can also confirm this running docker with the library/ubuntu:14.04 image on a virtualbox host running ubuntu 14.04

running apt-get update times out or takes an extremely long time.

Host Virtualbox:

$ ulimit -n
1024

docker container:

root@1518e024f53a:/# ulimit -n
524288

root@1518e024f53a:/# time apt-get update

real 0m47.489s
user 0m24.408s
sys 0m21.516s

root@1518e024f53a:/# ulimit -n 100000
root@1518e024f53a:/# time apt-get update

real 0m11.165s
user 0m6.598s
sys 0m4.359s

root@1518e024f53a:/# ulimit -n 1024
root@1518e024f53a:/# time apt-get update

real 0m3.109s
user 0m2.543s
sys 0m0.374s

[Ian Babrou (bobrik)]

Looks like it was fixed in some update. Can anyone confirm that?

[Scrivs (michael-a-endsley)]

If it's been fixed, I don't know what in. This still appears to be an issue for me. If I've failed to apply some update that fixed it, I'd love to know what it was.

[Ian Babrou (bobrik)]

Yep, it's not fixed:

~ λ docker run --rm ubuntu:14.04 bash -c 'ulimit -n 500000 && apt-get install strace && strace -c -f apt-get update'
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  strace
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 113 kB of archives.
After this operation, 504 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu/ trusty/main strace amd64 4.8-1ubuntu5 [113 kB]
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (This frontend requires a controlling tty.)
debconf: falling back to frontend: Teletype
dpkg-preconfigure: unable to re-open stdin:
Fetched 113 kB in 0s (179 kB/s)
Selecting previously unselected package strace.
(Reading database ... 11527 files and directories currently installed.)
Preparing to unpack .../strace_4.8-1ubuntu5_amd64.deb ...
Unpacking strace (4.8-1ubuntu5) ...
Setting up strace (4.8-1ubuntu5) ...
Process 47 attached
Process 48 attached
Process 49 attached
Ign http://archive.ubuntu.com trusty InRelease
Ign http://archive.ubuntu.com trusty-updates InRelease
Ign http://archive.ubuntu.com trusty-security InRelease
Hit http://archive.ubuntu.com trusty Release.gpg
Get:1 http://archive.ubuntu.com trusty-updates Release.gpg [933 B]
Get:2 http://archive.ubuntu.com trusty-security Release.gpg [933 B]
Process 50 attached
Process 51 attached
Hit http://archive.ubuntu.com trusty Release
Get:3 http://archive.ubuntu.com trusty-updates Release [63.5 kB]
Process 52 attached
Process 53 attached
Get:4 http://archive.ubuntu.com trusty-security Release [63.5 kB]
Process 54 attached
Get:5 http://archive.ubuntu.com trusty/main Sources [1335 kB]
Process 55 attached
Process 56 attached
^CGet:6 http://archive.ubuntu.com trusty/restricted Sources [5335 B]
Get:7 http://archive.ubuntu.com trusty/universe Sources [7926 kB]
Get:8 http://archive.ubuntu.com trusty/main amd64 Packages [1743 kB]
Get:9 http://archive.ubuntu.com trusty/restricted amd64 Packages [16.0 kB]
Get:10 http://archive.ubuntu.com trusty/universe amd64 Packages [7589 kB]
Get:11 http://archive.ubuntu.com trusty-updates/main Sources [248 kB]
Get:12 http://archive.ubuntu.com trusty-updates/restricted Sources [2310 B]
Get:13 http://archive.ubuntu.com trusty-updates/universe Sources [141 kB]
Get:14 http://archive.ubuntu.com trusty-updates/main amd64 Packages [636 kB]
Get:15 http://archive.ubuntu.com trusty-updates/restricted amd64 Packages [15.1 kB]
Get:16 http://archive.ubuntu.com trusty-updates/universe amd64 Packages [356 kB]
Get:17 http://archive.ubuntu.com trusty-security/main Sources [97.7 kB]
Get:18 http://archive.ubuntu.com trusty-security/restricted Sources [1874 B]
Get:19 http://archive.ubuntu.com trusty-security/universe Sources [23.3 kB]
Get:20 http://archive.ubuntu.com trusty-security/main amd64 Packages [329 kB]
Get:21 http://archive.ubuntu.com trusty-security/restricted amd64 Packages [14.8 kB]
Get:22 http://archive.ubuntu.com trusty-security/universe amd...

[Jon Skarpeteig (jskarpet)]

When running Ubuntu 14.04 inside Docker I've seen this issue with CentOS as well:

CentOS 7.2
3.10.0-327.4.4.el7.x86_64

I tested a few kernels with Ubuntu 14.04 as host OS and I found:

apt-get update extremely slow on kernels:

3.13.0-76
3.13.0-65
3.13.0-30
3.13.0-29

apt-get update fast on kernel:

3.13.0-45

So my workaround for now is to install and run kernel 3.13.0-45

My working setup is;

# docker --version
Docker version 1.9.1, build a34a1d5
# uname -a
Linux lys-jenkins 3.13.0-45-generic #74-Ubuntu SMP Tue Jan 13 19:36:28 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[Toby Corkindale (tjc-wintrmute)]

I'm still seeing this as a problem (I think) on Ubuntu 15.10 Wily, kernel 4.2.0-25-generic.

[Toby Corkindale (tjc-wintrmute)]

Apologies, I retract that -- looks like the issue was related to MTU and PMTU discovery.. It just happened that the machines with the mtu issues were also the ones with huge ulimits configured.

[Julian Andres Klode (juliank)]

This was fixed in 1.0.10 or something, and I just uploaded a fixed version to trusty (1.0.1ubuntu2.18).

[Łukasz Zemczak (sil2100)]

Hello tom916, or anyone else affected,

Accepted apt into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/1.0.1ubuntu2.18 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-trusty to verification-done-trusty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-trusty. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Julian Andres Klode (juliank)]

Verified. (BTW, I aborted the initial run as it simply took too long so actual numbers would have been much higher)

root@t:~# apt-cache policy libapt-pkg4.12
libapt-pkg4.12:
  Installed: 1.0.1ubuntu2.17
  Candidate: 1.0.1ubuntu2.18
  Version table:
     1.0.1ubuntu2.18 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-proposed/main amd64 Packages
 *** 1.0.1ubuntu2.17 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
        100 /var/lib/dpkg/status
     1.0.1ubuntu2 0
        500 http://archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
root@t:~# strace -f -c apt-cache search test >/dev/null
Process 2338 attached
^CProcess 2337 detached
Process 2338 detached
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
 50.32 0.827909 3 329234 getrlimit
 49.55 0.815234 2 329234 329227 fcntl
  0.03 0.000510 10 50 mmap
  0.02 0.000407 10 40 open
  0.01 0.000225 10 23 mprotect
  0.01 0.000212 6 34 read
  0.01 0.000209 5 40 close
  0.01 0.000202 7 29 fstat
  0.01 0.000167 13 13 13 access
  0.00 0.000044 6 8 2 stat
  0.00 0.000027 14 2 getdents
  0.00 0.000018 9 2 munmap
  0.00 0.000018 6 3 brk
  0.00 0.000016 16 1 execve
  0.00 0.000008 8 1 pipe
  0.00 0.000007 7 1 openat
  0.00 0.000004 4 1 1 lseek
  0.00 0.000004 4 1 clone
  0.00 0.000003 3 1 1 ioctl
  0.00 0.000003 3 1 arch_prctl
  0.00 0.000000 0 6 rt_sigaction
------ ----------- ----------- --------- --------- ----------------
100.00 1.645227 658725 329244 total
root@t:~# apt install libapt-pkg4.12
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libfreetype6 os-prober
Use 'apt-get autoremove' to remove them.
The following packages will be upgraded:
  libapt-pkg4.12
1 upgraded, 0 newly installed, 0 to remove and 18 not upgraded.
Need to get 638 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu/ trusty-proposed/main libapt-pkg4.12 amd64 1.0.1ubuntu2.18 [638 kB]
Fetched 638 kB in 0s (687 kB/s)
(Reading database ... 25096 files and directories currently installed.)
Preparing to unpack .../libapt-pkg4.12_1.0.1ubuntu2.18_amd64.deb ...
Unpacking libapt-pkg4.12:amd64 (1.0.1ubuntu2.18) over (1.0.1ubuntu2.17) ...
Setting up libapt-pkg4.12:amd64 (1.0.1ubuntu2.18) ...
Processing triggers fo...

[Julian Andres Klode (juliank)]

Time for apt-cache policy apt also decreased from 1.86 seconds to 0.04 seconds :)

[Robie Basak (racb)]

There are some dep8 regressions reported in http://people.canonical.com/~ubuntu-archive/pending-sru.html. I've requested some retries. If they're still failing, please could you take a look?

[Julian Andres Klode (juliank)]

I have verified that none of these failures are regressions; they all appear in previous apt uploads/run against other triggers too.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 1.0.1ubuntu2.18

---------------
apt (1.0.1ubuntu2.18) trusty; urgency=medium

  * ExecFork: Use /proc/self/fd to determine which files to close
    (Closes: #764204) (LP: #1332440).

 -- Julian Andres Klode <email address hidden> Mon, 09 Apr 2018 15:32:09 +0200

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for apt has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.