Template generation jobs are not aggregated

[Henning Eggers]

A translation template build job is created for every commit of the branch. This results in many identical builds happening for the same branch. This is starting to block a lot of builders for other builds.

Instead, commits should be collected into one job while another job is in progress so that no more than one build occurs for a branch at one point in time.

Julian considers this critical.

[Данило Шеган (danilo)]

We are actually not generating a huge amount of jobs it seems. It's just that they get retried because of a build-manager bug. Thus, lowering priority on this one.

I guess the nice way to avoid any problems (including concurrency problems) is to limit number of jobs per branch to two pending jobs (or other small finite number: two might still be picked up by different builders at roughly the same time and then new changes in between won't be seen). It will get an odd unneeded job in, but we'll make sure that we have processed all the changes, and won't do too much of the duplicate work.

[William Grant (wgrant)]

Doesn't a template build just care about the current state of the tree? If so, it's completely pointless having more than one pending build.

[Данило Шеган (danilo)]

Exactly — for the first part. :)

But, there is a potential window of time between build-manager picking up a job (and build job getting the current state of the tree) and new revision coming into the tree (for which we might need a new job). If these happen concurrently, we may end up not running the job for the latest revision. Granted, this window will usually be negative (i.e. a DB commit indicating that the build is in progress is going to be very quick, and the time until a builder actually gets the branch tip will be a minute or two), but it's not unseen for a DB commit to stall for a few minutes because of DB locks being held by other processes.

Limiting the number of jobs to two would just reduce the chances of this happening. Without using DB locking on our side, we can't do much to avoid the above problem (well, using number_of_builders + 1 as the limit would always avoid the problem, but we should strive for a lower limit).