apt-ftparchive fails on scanning large repositories

Bug #985452 reported by Adam Conrad
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
apt (Ubuntu)
Fix Released
Critical
Michael Vogt
Precise
Fix Released
Critical
Michael Vogt

Bug Description

See, for instance, the build log at:

http://people.canonical.com/~ubuntu-archive/livefs-build-logs/precise/ubuntu-server-omap4/20120419/livecd-20120419-armhf.out

grep for "Failed to create"

This is also reproducible locally with a sufficiently large collection of debs (over 1015ish, it seems like)

Related branches

Adam Conrad (adconrad)
Changed in apt (Ubuntu):
milestone: none → ubuntu-12.04
assignee: nobody → Michael Vogt (mvo)
importance: Undecided → Critical
Revision history for this message
Michael Vogt (mvo) wrote :

To reproduce do:
$ mkdir pool; cd pool
$ apt-get download 2vcard
$ for i in $(seq 1200); do cp 2vcard*.deb homer_clone_$i.deb; done
$ apt-ftparchive packages . > /dev/null

E: Failed to create pipes - pipe (24: Too many open files)

Revision history for this message
Michael Vogt (mvo) wrote :

Strace shows that the pipe is not closed when the compressor child exists:

$ strace -e trace=open,dup,close,pipe apt-ftparchive package. > /dev/null
open("./homer_clone_444.deb", O_RDONLY) = 5
pipe([261, 262]) = 0
close(262) = 0
--- SIGCHLD (Child exited) @ 0 (0) ---

It seems like "AutoClose" in FileFd is set to false for compressed images which leads to "FileFd::Close()" not
closing the internal iFD which then leads to this leak.

Revision history for this message
Michael Vogt (mvo) wrote :

I think this is the right approach as AFAIK the old code did close Fds set via FileFd.Fd()

Revision history for this message
Michael Vogt (mvo) wrote :
Michael Vogt (mvo)
Changed in apt (Ubuntu):
status: New → In Progress
Revision history for this message
Michael Vogt (mvo) wrote :

I uploaded a minimal fix to precise-proposed now.

Changed in apt (Ubuntu):
milestone: ubuntu-12.04 → precise-updates
tags: added: patch
Revision history for this message
Martin Pitt (pitti) wrote : Please test proposed package

Hello Adam, or anyone else affected,

Accepted apt into precise-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in apt (Ubuntu Precise):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Martin Pitt (pitti) wrote :

Adam confirmed on IRC that the new apt fixes the bug.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apt - 0.8.16~exp12ubuntu9

---------------
apt (0.8.16~exp12ubuntu9) precise-proposed; urgency=low

  * apt-inst/contrib/extracttar.cc:
    - ensure that in StartGzip the InFd is set to "AutoClose" to ensure
      that the pipe is closed when InFd is closed. This fixes a Fd leak
      (LP: #985452)
 -- Michael Vogt <email address hidden> Thu, 19 Apr 2012 11:38:43 +0200

Changed in apt (Ubuntu Precise):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers