apt-ftparchive fails on scanning large repositories

[Adam Conrad]

See, for instance, the build log at:

http://people.canonical.com/~ubuntu-archive/livefs-build-logs/precise/ubuntu-server-omap4/20120419/livecd-20120419-armhf.out

grep for "Failed to create"

This is also reproducible locally with a sufficiently large collection of debs (over 1015ish, it seems like)

[Michael Vogt (mvo)]

To reproduce do:
$ mkdir pool; cd pool
$ apt-get download 2vcard
$ for i in $(seq 1200); do cp 2vcard*.deb homer_clone_$i.deb; done
$ apt-ftparchive packages . > /dev/null

E: Failed to create pipes - pipe (24: Too many open files)

[Michael Vogt (mvo)]

Strace shows that the pipe is not closed when the compressor child exists:

$ strace -e trace=open,dup,close,pipe apt-ftparchive package. > /dev/null
open("./homer_clone_444.deb", O_RDONLY) = 5
pipe([261, 262]) = 0
close(262) = 0
--- SIGCHLD (Child exited) @ 0 (0) ---

It seems like "AutoClose" in FileFd is set to false for compressed images which leads to "FileFd::Close()" not
closing the internal iFD which then leads to this leak.

[Michael Vogt (mvo)]

I think this is the right approach as AFAIK the old code did close Fds set via FileFd.Fd()

[Michael Vogt (mvo)]

I uploaded a minimal fix to precise-proposed now.

[Martin Pitt (pitti)]

Hello Adam, or anyone else affected,

Accepted apt into precise-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Martin Pitt (pitti)]

Adam confirmed on IRC that the new apt fixes the bug.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 0.8.16~exp12ubuntu9

---------------
apt (0.8.16~exp12ubuntu9) precise-proposed; urgency=low

  * apt-inst/contrib/extracttar.cc:
    - ensure that in StartGzip the InFd is set to "AutoClose" to ensure
      that the pipe is closed when InFd is closed. This fixes a Fd leak
      (LP: #985452)
 -- Michael Vogt <email address hidden> Thu, 19 Apr 2012 11:38:43 +0200