Launchpad itself

buildd-manager dispatches all downloads at once and can hit EMFILE

Bug #1584744 reported by Matthias Klose
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
Critical
Colin Watson

Bug Description

seen with
https://launchpad.net/ubuntu/+source/gcc-6-cross-ports/8ubuntu1

the build itself succeeds.

Related branches

lp:~cjwatson/launchpad/buildmaster-twisted-agent
William Grant: Approve (code)
Celso Providelo (community): Approve
Diff: 363 lines (+189/-20)
5 files modified
daemons/buildd-manager.tac (+13/-4)
lib/lp/buildmaster/interactor.py (+112/-10)
lib/lp/buildmaster/tests/mock_slaves.py (+3/-3)
lib/lp/buildmaster/tests/test_interactor.py (+53/-3)
lib/lp/services/config/schema-lazr.conf (+8/-0)
lp:~cjwatson/launchpad/fix-buildmaster-twisted-agent
Colin Watson (community): Approve
Diff: 141 lines (+41/-9)
5 files modified
lib/lp/buildmaster/interactor.py (+14/-5)
lib/lp/buildmaster/manager.py (+2/-1)
lib/lp/buildmaster/model/buildfarmjobbehaviour.py (+4/-3)
lib/lp/buildmaster/tests/mock_slaves.py (+5/-0)
lib/lp/buildmaster/tests/test_interactor.py (+16/-0)
Revision history for this message
Colin Watson (cjwatson) wrote :

buildd-manager is getting EMFILE, caused by it just firing off all the downloads at once. The exact behaviour will depend on what else is going on in the build farm at the time.

We could just bump the resource limit, but I think the proper fix is probably to migrate to twisted.web.client.Agent, which uses a connection pool. We would still have a problem if the number of builders multiplied by HTTPConnectionPool.maxPersistentPerHost (2 by default) approached or exceeded the resource limit, but we're quite some way off that being a problem.

Compare https://stackoverflow.com/questions/28162301/twisted-getpage-exceptions-oserror-errno-24-too-many-open-files, where Glyph says we shouldn't be using the interface we're currently using.

Changed in launchpad:
assignee: nobody → Colin Watson (cjwatson)
importance: Undecided → Critical
status: New → Triaged
summary: - launchpad doesn't accept builds producing a large number of files
+ buildd-manager dispatches all downloads at once and can hit EMFILE
tags: added: soyuz-build
Colin Watson (cjwatson)
Changed in launchpad:
status: Triaged → In Progress
Revision history for this message
Launchpad QA Bot (lpqabot) wrote :

Fixed in stable r18068 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/18068>.

tags: added: qa-needstesting
Changed in launchpad:
status: In Progress → Fix Committed
Colin Watson (cjwatson)
tags: added: qa-bad
removed: qa-needstesting
Changed in launchpad:
status: Fix Committed → In Progress
Revision history for this message
Launchpad QA Bot (lpqabot) wrote :

Fixed in stable r18071 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/18071>.

tags: added: qa-needstesting
removed: qa-bad
Changed in launchpad:
status: In Progress → Fix Committed
Revision history for this message
Colin Watson (cjwatson) wrote :

This seems to be behaving itself on dogfood now.

tags: added: qa-ok
removed: qa-needstesting
Colin Watson (cjwatson)
Changed in launchpad:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.