Move live filesystem building into Launchpad

[Colin Watson]

(This has been a long-standing to-do entry, and we discussed it at e.g. the last release engineering sprint, but it doesn't look as though anyone ever filed a bug about it.)

We currently build live filesystem ("livefs") images for Ubuntu using dedicated machines running BuildLiveCD from the livecd-rootfs package as an SSH trigger. This architecture is unable to support scaling horizontally in cases when we need to build several images in parallel, and furthermore the dedicated machines sit idle most of the time. This build farm, such as it is, has identical security and range-of-architectures requirements to the current devirtualised build farm in Launchpad; it would benefit both sides if they were merged.

The plan we agreed on at the release engineering sprint in July was roughly as follows:

 * Implement build cancellation so that we don't end up in a situation where several parallel livefs build jobs block urgent package builds (done, bug 173018);
 * Add a livefs build manager to launchpad-buildd;
 * Make DB/interface/model changes to Launchpad to represent at most one livefs per distroseries/architecture/flavour, so that we can keep appropriate librarian references and deal with GC;
 * Add a livefs build behaviour to Launchpad;
 * Make some improvements to longpoll so that we can efficiently wait for a livefs build to complete (I'm fuzzy on the details of this; perhaps William can flesh this out)
 * Change ubuntu-cdimage to request livefs builds from LP rather than from the dedicated machines
 * Perhaps change the build log mirroring system on snakefruit to mirror logs from Launchpad, depending on how we implement the Launchpad side of this (alternatively, we might be able to keep the logs approximately forever in Launchpad)

[Colin Watson (cjwatson)]

We deployed launchpad-buildd 120 to all builders quite a while ago.

[Colin Watson (cjwatson)]

Local testing revealed a small bug in the buildd code: we need to avoid trying to store output files that are symlinks, because trying to fetch them may fail due to removing a symlink target before the symlink itself, and in any case they will result in duplicate files in the librarian rather than anything more sensible. The things we were trying to do by buildd-side symlinks would be better handled in the client.

[Launchpad QA Bot (lpqabot)]

r17003 in stable (http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/17003) is part of this bug's fix.

[Colin Watson (cjwatson)]

I can't QA the duplicate-build-files branch fully until the other pieces land, but I've tested it locally, and I've also confirmed that normal package uploads still work fine on dogfood.

[Colin Watson (cjwatson)]

launchpad-buildd (122) hardy; urgency=medium

  * Drop the status_dict XML-RPC method, now that the master uses the
    new-style dict-flavoured status method.
  * Don't add symlinks to the results of livefs builds (LP: #1247461).
  * Cope with builds that return multiple files with identical content.
  * If a build is aborted between subprocesses, pretend that it was
    terminated by a signal.

 -- Colin Watson <email address hidden> Tue, 13 May 2014 16:46:52 +0100

[Colin Watson (cjwatson)]

revno: 1401
fixes bug: https://launchpad.net/bugs/1247461
committer: Colin Watson <email address hidden>
branch nick: ubuntu-cdimage
timestamp: Wed 2014-05-28 11:18:44 +0200
message:
  Support livefs building via Launchpad (configurable, relies on not-yet-merged Launchpad changes).