too many open files when using a heap of parts of plugin autotools

[Alexander Sack]

having a project with lots of autotools part snapcraft ends up running out of fd:

E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-security_restricted_binary-i386_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-security_main_binary-i386_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-security_multiverse_binary-amd64_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-security_universe_binary-amd64_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-security_restricted_binary-amd64_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-security_main_binary-amd64_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_universe_i18n_Translation-en - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_restricted_i18n_Translation-en - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_multiverse_i18n_Translation-en - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_main_i18n_Translation-en - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_multiverse_binary-i386_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_universe_binary-i386_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_restricted_binary-i386_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_main_binary-i386_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_multiverse_binary-amd64_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_universe_binary-amd64_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_restricted_binary-amd64_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-updates_main_binary-amd64_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_universe_i18n_Translation-en - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_restricted_i18n_Translation-en - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_multiverse_i18n_Translation-en - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_main_i18n_Translation-en - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_multiverse_binary-i386_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_universe_binary-i386_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_restricted_binary-i386_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_main_binary-i386_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_multiverse_binary-amd64_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_universe_binary-amd64_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_restricted_binary-amd64_Packages - open (24: Too many open files), E:Could not open file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_main_binary-amd64_Packages - open (24: Too many open files)

guess some FD leak somewhere.

[Alexander Sack (asac)]

testcase attached... shows it quite nicely :)

[Alexander Sack (asac)]

seems we dont close the apt cache and autotools accumulates quite some packagers in new list...

made patch that closes cache and that also makes sure we dont process duplicates (which speeds building the testcase up quitea lot)...

diff --git a/snapcraft/repo.py b/snapcraft/repo.py
index ea1e443..1af7903 100644
--- a/snapcraft/repo.py
+++ b/snapcraft/repo.py
@@ -58,11 +58,16 @@ _GEOIP_SERVER = "http://geoip.ubuntu.com/lookup"

 def install_build_packages(packages):
+ seen = set()
+ seen_add = seen.add
+ unique_packages = [ x for x in packages if not (x in seen or seen_add(x))]
     new_packages = []
- for pkg in packages:
+ for pkg in unique_packages:
         try:
- if not apt.Cache()[pkg].installed:
+ aptc = apt.Cache()
+ if not aptc[pkg].installed:
                 new_packages.append(pkg)
+ aptc.close()
         except KeyError:
             logger.error('Could not find all the "build-packages" required '
                          'in snapcraft.yaml')

[Alexander Sack (asac)]

submitted PR: https://github.com/ubuntu-core/snapcraft/pull/257