should track and then disable failing jobs

[Martin Pool]

In the context of <https://code.launchpad.net/~mbp/launchpad/690021-rlimit/+merge/43733>:

Launchpad should notice when a job repeatedly fails, and eventually disable it, and log what's happening.

There may already be code for this in soyuz or branch imports? It should be put into a standard place.

Detecting failed jobs should include detecting when the process running them simply disappeared because of eg hitting a resource limit or crashing without a Python exception.

[Julian Edwards (julian-edwards)]

On Wednesday 22 December 2010 01:43:48 you wrote:>
> There may already be code for this in soyuz or branch imports? It
> should be put into a standard place.

The buildd-manager has rudimentary detection by counting failures and trying
to work out whether the job or the builder (job runner in effect) is failing
more.

We could generalise this into a mixin implementing IHasFailures or similar.

[Robert Collins (lifeless)]

+1 on generalisation. I wouldn't use a mixin though. The general
pattern is called 'Circuit Breaker', see 'Release it', Michael Nygard.