Move live filesystem building into Launchpad
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Launchpad itself |
Fix Released
|
High
|
Colin Watson | ||
Ubuntu CD Images |
Fix Released
|
High
|
Colin Watson | ||
launchpad-buildd |
Fix Released
|
High
|
Colin Watson |
Bug Description
(This has been a long-standing to-do entry, and we discussed it at e.g. the last release engineering sprint, but it doesn't look as though anyone ever filed a bug about it.)
We currently build live filesystem ("livefs") images for Ubuntu using dedicated machines running BuildLiveCD from the livecd-rootfs package as an SSH trigger. This architecture is unable to support scaling horizontally in cases when we need to build several images in parallel, and furthermore the dedicated machines sit idle most of the time. This build farm, such as it is, has identical security and range-of-
The plan we agreed on at the release engineering sprint in July was roughly as follows:
* Implement build cancellation so that we don't end up in a situation where several parallel livefs build jobs block urgent package builds (done, bug 173018);
* Add a livefs build manager to launchpad-buildd;
* Make DB/interface/model changes to Launchpad to represent at most one livefs per distroseries/
* Add a livefs build behaviour to Launchpad;
* Make some improvements to longpoll so that we can efficiently wait for a livefs build to complete (I'm fuzzy on the details of this; perhaps William can flesh this out)
* Change ubuntu-cdimage to request livefs builds from LP rather than from the dedicated machines
* Perhaps change the build log mirroring system on snakefruit to mirror logs from Launchpad, depending on how we implement the Launchpad side of this (alternatively, we might be able to keep the logs approximately forever in Launchpad)
Related branches
- William Grant (community): Approve (code)
-
Diff: 504 lines (+415/-4)9 files modifiedMakefile (+3/-2)
buildd-slave.tac (+2/-0)
buildlivefs (+167/-0)
debian/changelog (+2/-0)
debian/rules (+1/-1)
debian/upgrade-config (+14/-1)
lpbuildd/livefs.py (+105/-0)
lpbuildd/tests/test_livefs.py (+118/-0)
template-buildd-slave.conf (+3/-0)
- William Grant (community): Approve (db)
- Stuart Bishop: Pending (db) requested
-
Diff: 200 lines (+143/-0)2 files modifieddatabase/schema/patch-2209-56-0.sql (+127/-0)
database/schema/security.cfg (+16/-0)
- William Grant (community): Approve (code)
-
Diff: 4048 lines (+3484/-37)39 files modifiedlib/lp/_schema_circular_imports.py (+15/-1)
lib/lp/app/browser/launchpad.py (+3/-2)
lib/lp/app/browser/tests/test_webservice.py (+7/-1)
lib/lp/archiveuploader/livefsupload.py (+58/-0)
lib/lp/archiveuploader/tests/test_livefsupload.py (+67/-0)
lib/lp/archiveuploader/uploadprocessor.py (+28/-1)
lib/lp/buildmaster/browser/builder.py (+9/-1)
lib/lp/buildmaster/enums.py (+7/-1)
lib/lp/buildmaster/interfaces/buildfarmjobbehaviour.py (+4/-1)
lib/lp/buildmaster/model/buildfarmjobbehaviour.py (+12/-6)
lib/lp/buildmaster/tests/test_buildfarmjobbehaviour.py (+35/-1)
lib/lp/code/model/tests/test_recipebuilder.py (+18/-1)
lib/lp/registry/browser/person.py (+25/-1)
lib/lp/registry/interfaces/person.py (+2/-2)
lib/lp/registry/model/person.py (+1/-1)
lib/lp/security.py (+55/-1)
lib/lp/services/helpers.py (+5/-1)
lib/lp/soyuz/adapters/archivedependencies.py (+10/-7)
lib/lp/soyuz/browser/archive.py (+1/-1)
lib/lp/soyuz/browser/build.py (+1/-1)
lib/lp/soyuz/browser/configure.zcml (+24/-1)
lib/lp/soyuz/browser/livefs.py (+26/-0)
lib/lp/soyuz/browser/livefsbuild.py (+15/-0)
lib/lp/soyuz/browser/tests/test_livefs.py (+29/-0)
lib/lp/soyuz/configure.zcml (+67/-1)
lib/lp/soyuz/emailtemplates/livefsbuild-notification.txt (+11/-0)
lib/lp/soyuz/interfaces/binarypackagebuild.py (+3/-3)
lib/lp/soyuz/interfaces/livefs.py (+288/-0)
lib/lp/soyuz/interfaces/livefsbuild.py (+199/-0)
lib/lp/soyuz/interfaces/webservice.py (+9/-1)
lib/lp/soyuz/mail/livefsbuild.py (+89/-0)
lib/lp/soyuz/model/livefs.py (+292/-0)
lib/lp/soyuz/model/livefsbuild.py (+383/-0)
lib/lp/soyuz/model/livefsbuildbehaviour.py (+159/-0)
lib/lp/soyuz/tests/test_binarypackagebuildbehaviour.py (+14/-0)
lib/lp/soyuz/tests/test_livefs.py (+649/-0)
lib/lp/soyuz/tests/test_livefsbuild.py (+497/-0)
lib/lp/soyuz/tests/test_livefsbuildbehaviour.py (+289/-0)
lib/lp/testing/factory.py (+78/-0)
- William Grant (community): Approve (code)
-
Diff: 145 lines (+69/-4)2 files modifiedlib/lp/scripts/garbo.py (+25/-1)
lib/lp/scripts/tests/test_garbo.py (+44/-3)
- William Grant (community): Approve (code)
-
Diff: 70 lines (+24/-3)3 files modifieddebian/changelog (+1/-0)
lpbuildd/slave.py (+2/-2)
lpbuildd/tests/test_buildd_slave.py (+21/-1)
- William Grant (community): Approve (code)
-
Diff: 150 lines (+26/-27)4 files modifiedlib/lp/buildmaster/interactor.py (+6/-7)
lib/lp/buildmaster/model/buildfarmjobbehaviour.py (+2/-2)
lib/lp/buildmaster/tests/mock_slaves.py (+4/-4)
lib/lp/buildmaster/tests/test_interactor.py (+14/-14)
- William Grant (community): Approve (code)
-
Diff: 1806 lines (+1646/-10)10 files modifiedlib/lp/app/browser/configure.zcml (+13/-1)
lib/lp/app/browser/tales.py (+13/-1)
lib/lp/soyuz/browser/configure.zcml (+82/-0)
lib/lp/soyuz/browser/livefs.py (+335/-1)
lib/lp/soyuz/browser/livefsbuild.py (+160/-2)
lib/lp/soyuz/browser/tests/test_livefs.py (+411/-5)
lib/lp/soyuz/browser/tests/test_livefsbuild.py (+261/-0)
lib/lp/soyuz/templates/livefs-index.pt (+109/-0)
lib/lp/soyuz/templates/livefs-new.pt (+45/-0)
lib/lp/soyuz/templates/livefsbuild-index.pt (+217/-0)
- William Grant (community): Approve (code)
-
Diff: 182 lines (+82/-5)5 files modifiedlib/lp/registry/personmerge.py (+23/-4)
lib/lp/registry/tests/test_personmerge.py (+39/-1)
lib/lp/soyuz/interfaces/livefs.py (+3/-0)
lib/lp/soyuz/model/livefs.py (+4/-0)
lib/lp/soyuz/tests/test_livefs.py (+13/-0)
tags: | added: soyuz-build |
Changed in launchpad: | |
status: | New → Triaged |
importance: | Undecided → High |
Changed in ubuntu-cdimage: | |
status: | New → Triaged |
importance: | Undecided → High |
Changed in launchpad-buildd: | |
status: | New → Triaged |
importance: | Undecided → High |
Changed in launchpad-buildd: | |
assignee: | nobody → Colin Watson (cjwatson) |
status: | Triaged → In Progress |
Changed in launchpad-buildd: | |
status: | In Progress → Fix Committed |
Changed in launchpad: | |
status: | Triaged → In Progress |
assignee: | nobody → Colin Watson (cjwatson) |
Changed in launchpad: | |
status: | In Progress → Fix Committed |
status: | Fix Committed → In Progress |
Changed in launchpad-buildd: | |
status: | In Progress → Fix Committed |
Changed in launchpad-buildd: | |
status: | In Progress → Fix Committed |
Changed in ubuntu-cdimage: | |
status: | Triaged → In Progress |
assignee: | nobody → Colin Watson (cjwatson) |
Changed in launchpad: | |
status: | In Progress → Fix Released |
We deployed launchpad-buildd 120 to all builders quite a while ago.