cannot specify specific chroot for a build

Bug #671190 reported by LaMont Jones on 2010-11-04
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
High
Colin Watson

Bug Description

When we bootstrap a package (due to circular dependencies), we eventually wind up putting all of the affected architecture's buildds on manual while we upload a polluted chroot with the build-deps preinstalled and launch the build of the package we're bootstrapping, then restoring the pristine chroot. This leaves us with an exposure to other buildd admins "helping" by putting buildds back on manual, or PPA builders showing up (coming back from their other uses) and being on auto.

The switching of many builders to manual and back is currently done using SQL directly, which is a badness. The exposure to buildds magically arriving on auto is treated as ignorable, since we have no current solution to the issue.

There should be a button (or api entry point) that allows me to put all buildds of an architecture on manual. And having done so, there should be a way for a launchpad-buildd-admin to tell the system "yes, I know the builders are all on manual. Start $THIS build on $THAT builder as though it were on auto and next in the queue".

Diagnosis
=========

* Allowing a specific chroot to be used for a build would short-circuit all the shenanigans and allow any builder to be used. OTOH it would also allow weaker auditability so probably needs to be a admin (that is ducky) function.

Related branches

LaMont Jones (lamont) on 2010-11-04
tags: added: canonical-losa-lp
Jelmer Vernooij (jelmer) on 2010-11-05
Changed in soyuz:
status: New → Confirmed
Julian Edwards (julian-edwards) wrote :

The builders are exposed on the API and you can set "manual" there for any as you see fit. You can also bump the priority of a particular build to force it to be next in the queue using the API.

I don't think there's anything else we need to do for you to be able to do this, is there?

Changed in soyuz:
status: Confirmed → Incomplete
Francis J. Lacoste (flacoste) wrote :

The problem is that when doing a bootstrap build we seem to use a custom chroot that must not be used by any other build.

There is a race when enablement returns a builders, it comes back automatically in automatic and could get a build dispatched using the custom chroot.

(See https://wiki.canonical.com/InformationInfrastructure/OSA/BootstrappingPackages for the process.)

Maybe what is needed is a way to specify per builder the chroot to use. That way, to do bootstrapping we would need a way to specify a custom chroot for a specific builder and a way to dispatch a build to that builder. The rest of the build farm could continue it's normal activity at that time.

Francis J. Lacoste (flacoste) wrote :

Escalated via the ISO/LP call.

Changed in launchpad:
importance: Undecided → High
Julian Edwards (julian-edwards) wrote :

Chroots are currently picked on the basis of the build job.

The right way of fixing this is some way of extending this so that we can specify that a particular build needs a particular chroot. Specifying chroots per builder won't help much because you still need all the other hoop-jumping.

Changed in launchpad:
status: Incomplete → Triaged
tags: added: buildfarm soyuz-build
Francis J. Lacoste (flacoste) wrote :

Of course, that's even a lot better! LaMont, do you agree?

summary: - Need a better single-threading control options for buildds
+ cannot prevent all builds on a given arch (e.g. while bootstrapping a
+ circular dependency)
summary: - cannot prevent all builds on a given arch (e.g. while bootstrapping a
- circular dependency)
+ cannot specify specific chroot for a build
description: updated
Robert Collins (lifeless) wrote :

So this probably needs something like:
 - a X- header in the dsc (to avoid race conditions)
 - -> mirrored into the datamodel (in SPPH ?)
 - which is permitted to refer to a chroot
 - do chroots need to be only provided by archive admins?
   * if so (because they can attack the buildd), then from a preset list
   * if not, then allow them to be attached to e.g. an Archive by anyone with edit

Colin Watson (cjwatson) wrote :

The permissions on anything to do with buildd management should have something to do with the launchpad-buildd-admins celebrity; e.g. the existing AdminByBuilddAdmin helper class in security.py.

Colin Watson (cjwatson) wrote :

Nowadays, we don't generally use a custom chroot for bootstrapping; I don't recall any instance in the last two years when this has been necessary. Rather, the chroots for the current development series have a manually-maintained bootstrap archive in their sources.list, which is normally kept empty but can be populated with a handful of packages if need be (or indeed an entire stage-N archive when bootstrapping a new architecture). For more extreme cases we can use devirtualised PPAs of other series or a different selection of pockets to dig ourselves out of holes; the need for that is even rarer.

This is almost good enough, and it's certainly more auditable since packages fetched from the bootstrap archive show up in build logs as such, but it's still not ideal because in theory the bootstrap archive can affect any devel build that happens to be scheduled. However, the modern approach suggests a simpler solution to this problem. How about we add a column to BinaryPackageBuild that's only writeable by launchpad-buildd-admins and that contains an extra sources.list line? This could then be set over the API, either directly (for already-pending builds) or by way of a keyword argument to the retry method, and BinaryPackageBuildBehaviour would include that in the sources.list lines it sends to the builder when dispatching the build. That seems relatively simple and almost elegant to me, and doesn't open any security problems that aren't already present (since buildd-admins can upload modified chroots anyway).

Colin Watson (cjwatson) wrote :

Indeed, this could look very like Archive.external_dependencies, and just be BinaryPackageBuild.external_dependencies with the same semantics except that it wouldn't need %(series)s substitution.

Colin Watson (cjwatson) wrote :

Actually only archive admins can upload modified chroots, not buildd admins. We can perhaps debate the right security setting for this.

Launchpad QA Bot (lpqabot) wrote :
Changed in launchpad:
assignee: nobody → Colin Watson (cjwatson)
tags: added: qa-needstesting
Changed in launchpad:
status: Triaged → In Progress
Colin Watson (cjwatson) wrote :

2016-01-05 21:41:31,594 INFO 2209-72-0 applied just now in 0.0 seconds

tags: added: qa-ok
removed: qa-needstesting
Launchpad QA Bot (lpqabot) wrote :
tags: added: qa-needstesting
removed: qa-ok
Changed in launchpad:
status: In Progress → Fix Committed
Colin Watson (cjwatson) wrote :

This can still build things normally and so is safe to deploy, but I missed the necessary change to get_sources_list_for_building so it doesn't actually take effect. Whoops.

tags: added: qa-ok
removed: qa-needstesting
Colin Watson (cjwatson) on 2016-01-07
Changed in launchpad:
status: Fix Committed → In Progress
Launchpad QA Bot (lpqabot) wrote :
tags: added: qa-needstesting
removed: qa-ok
Changed in launchpad:
status: In Progress → Fix Committed
Colin Watson (cjwatson) on 2016-01-08
tags: added: qa-ok
removed: qa-needstesting
Colin Watson (cjwatson) on 2016-01-12
Changed in launchpad:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers