buildd-manager dispatches all downloads at once and can hit EMFILE

Bug #1584744 reported by Matthias Klose on 2016-05-23
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Critical
Colin Watson

Bug Description

Related branches

Colin Watson (cjwatson) wrote :

buildd-manager is getting EMFILE, caused by it just firing off all the downloads at once. The exact behaviour will depend on what else is going on in the build farm at the time.

We could just bump the resource limit, but I think the proper fix is probably to migrate to twisted.web.client.Agent, which uses a connection pool. We would still have a problem if the number of builders multiplied by HTTPConnectionPool.maxPersistentPerHost (2 by default) approached or exceeded the resource limit, but we're quite some way off that being a problem.

Compare https://stackoverflow.com/questions/28162301/twisted-getpage-exceptions-oserror-errno-24-too-many-open-files, where Glyph says we shouldn't be using the interface we're currently using.

Changed in launchpad:
assignee: nobody → Colin Watson (cjwatson)
importance: Undecided → Critical
status: New → Triaged
summary: - launchpad doesn't accept builds producing a large number of files
+ buildd-manager dispatches all downloads at once and can hit EMFILE
tags: added: soyuz-build
Colin Watson (cjwatson) on 2016-05-23
Changed in launchpad:
status: Triaged → In Progress
Launchpad QA Bot (lpqabot) wrote :
tags: added: qa-needstesting
Changed in launchpad:
status: In Progress → Fix Committed
Colin Watson (cjwatson) on 2016-05-26
tags: added: qa-bad
removed: qa-needstesting
Changed in launchpad:
status: Fix Committed → In Progress
Launchpad QA Bot (lpqabot) wrote :
tags: added: qa-needstesting
removed: qa-bad
Changed in launchpad:
status: In Progress → Fix Committed
Colin Watson (cjwatson) wrote :

This seems to be behaving itself on dogfood now.

tags: added: qa-ok
removed: qa-needstesting
Colin Watson (cjwatson) on 2016-06-02
Changed in launchpad:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers