Capomastro lacks the ability to build images when their code tree changes

[Caio Begotti]

When using Capomastro for an OEM project, it would be good to be able to automatically start builds when the code changes. It would be desirable to simply build all incoming merge proposals on the gerrit source tree used for OEM phone projects. These could then be used to automatically +1 merge proposals.

[Daniel Manrique (roadmr)]

Some thought would need to be given to locking in this scenario. If build time is longer than "poll for new merges" time, then we risk overlapping builds which can have adverse effects, mainly on performance.

[Daniel Manrique (roadmr)]

As noted in https://bugs.launchpad.net/capomastro/+bug/1386719:

"
On the topic of change-based builds (and which interestingly also affects "only build if source has changed" periodic builds), is that capomastro has no concept or knowledge of revision control utilities, and thus has no idea that a build is for a specific revision in a tree. So this functionality would have to be implemented in the project-specific build scripts. Consecutive revnos like bzr make it easy to figure out if the tree has changed, with git it may be harder because there's no consecutive revision ID, but I'm sure we can figure something out. But again, that probably belongs in https://launchpad.net/bygmester and its per-project scripts.
"

To update the above, Capomastro now has the "scheduled builds" functionality and that means what we have a periodic task scheduler (celery beat). So we could design a solution that periodically runs $SOMETHING that is included in each job type's branch (under https://code.launchpad.net/bygmester) and depending on whether that $SOMETHING returns true or false, launch a build for that dependency.

A brain dump of things needed for this:

Some sort of dependency locking is needed. If a dependency is already building, subsequent builds should either be discarded (silently? or loudly with exceptions and errors?) or queued so it runs after any previously-scheduled ones (can celery do this?).

We also need to define (and if needed, fix) the "build a single dependency" behavior. I've argued that rebuilding a dependency should still result in the project showing a build with all dependencies linked, though perhaps only the rebuilt one will be up-to-date, while the rest will be links to old artifacts. This is needed to avoid surprises if builds are triggered by some magic.

Also "runs $SOMETHING" needs to be defined better. Capomastro doesn't run anything itself, it delegates everything to Jenkins, which again, tries to run everything inside an LXC. I wonder if spawning a full LXC environment to run what essentially amounts to "bzr revno" is overkill, but it would be architecturally consistent.

Related to the above, we should figure out where/how to store revision data and how to decide if a rebuild is needed. For instance, if we store bzr revnos, they're easy to compare because they're sequential, unlike git commit identifiers. On the other hand, we could simply say "if it changed, then launch a build", which works for both bzr and git, and as long as e.g. someone doesn't overwrite a branch such that revnos or commit ids change, should be consistent. It still needs storing "last revno" somewhere, and bygmester jobs are designed to be mostly stateless, so we'd also need to decide where/how to store this.