livefs builds fail without logs (possibly due to multiple large artifacts)

Bug #1795877 reported by Francis Ginther on 2018-10-03
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
launchpad-buildd
High
Colin Watson

Bug Description

CPC is seeing livefs build failures without any artifacts or logs to indicate what the problem is. Examples:

 * https://launchpad.net/~cloudware/+livefs/ubuntu/cosmic/cpc/+build/146547
 * https://launchpad.net/~cloudware/+livefs/ubuntu/cosmic/cpc/+build/146649

One of these was believed to be just a network glitch at the wrong time, but we've seen this in other builds as well (will try to get a better list).

These livefs builds create a number of cloud images and filesystems as artifacts which total multiple GBs.

The problem we are currently seeing only occurs on cosmic builds. The suspicion is that the recent addition of the lxd snap to all images has increased image size across the entire set and pushed the time it takes to copy artifacts past a threshold that causes them to get terminated.

Related branches

Colin Watson (cjwatson) wrote :

We think this should probably be fixed by deferring gatherResults calls to a thread rather than running them in the reactor thread (which relies on it being possible to gather all the files from the backend before buildd-manager gets bored of the lack of status responses and times out).

affects: launchpad → launchpad-buildd
Changed in launchpad-buildd:
importance: Undecided → High
status: New → Triaged
Colin Watson (cjwatson) on 2018-10-19
Changed in launchpad-buildd:
status: Triaged → Fix Committed
assignee: nobody → Colin Watson (cjwatson)
Colin Watson (cjwatson) on 2018-10-26
Changed in launchpad-buildd:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers