too many open files when using a heap of parts of plugin autotools

Bug #1537705 reported by Alexander Sack on 2016-01-25
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Snapcraft
Status tracked in Trunk
1.x
High
Kyle Fazzari
Trunk
High
Alexander Sack

Bug Description

having a project with lots of autotools part snapcraft ends up running out of fd:

E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-security_restricted_binary-i386_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-security_main_binary-i386_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-security_multiverse_binary-amd64_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-security_universe_binary-amd64_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-security_restricted_binary-amd64_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-security_main_binary-amd64_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_universe_i18n_Translation-en - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_restricted_i18n_Translation-en - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_multiverse_i18n_Translation-en - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_main_i18n_Translation-en - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_multiverse_binary-i386_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_universe_binary-i386_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_restricted_binary-i386_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_main_binary-i386_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_multiverse_binary-amd64_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_universe_binary-amd64_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_restricted_binary-amd64_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_main_binary-amd64_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_universe_i18n_Translation-en - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_restricted_i18n_Translation-en - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_multiverse_i18n_Translation-en - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_main_i18n_Translation-en - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_multiverse_binary-i386_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_universe_binary-i386_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_restricted_binary-i386_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_main_binary-i386_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_multiverse_binary-amd64_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_universe_binary-amd64_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_restricted_binary-amd64_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_main_binary-amd64_Packages - open (24: Too many open files)

guess some FD leak somewhere.

Alexander Sack (asac) wrote :
Alexander Sack (asac) wrote :

testcase attached... shows it quite nicely :)

Alexander Sack (asac) wrote :

seems we dont close the apt cache and autotools accumulates quite some packagers in new list...

made patch that closes cache and that also makes sure we dont process duplicates (which speeds building the testcase up quitea lot)...

diff --git a/snapcraft/repo.py b/snapcraft/repo.py
index ea1e443..1af7903 100644
--- a/snapcraft/repo.py
+++ b/snapcraft/repo.py
@@ -58,11 +58,16 @@ _GEOIP_SERVER = "http://geoip.ubuntu.com/lookup"

 def install_build_packages(packages):
+ seen = set()
+ seen_add = seen.add
+ unique_packages = [ x for x in packages if not (x in seen or seen_add(x))]
     new_packages = []
- for pkg in packages:
+ for pkg in unique_packages:
         try:
- if not apt.Cache()[pkg].installed:
+ aptc = apt.Cache()
+ if not aptc[pkg].installed:
                 new_packages.append(pkg)
+ aptc.close()
         except KeyError:
             logger.error('Could not find all the "build-packages" required '
                          'in snapcraft.yaml')

Changed in snapcraft:
assignee: nobody → Alexander Sack (asac)
importance: Undecided → High
status: New → Fix Committed
Changed in snapcraft:
milestone: none → 2.1
Changed in snapcraft:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers