All builds on android-build report failure because they can't be run through LAVA (because LAVA is down)

[Zach Pfeffer]

Build report failed if LAVA is down. They should actually report something else like BUILT and then if LAVA is up TESTED.

[Paul Sokolovsky (pfalcon)]

> They should actually report something else like BUILT and then if LAVA is up TESTED.

That's actually not that easy to do, because Jenkins doesn't have custom build status support ;-(.

But for a situation like this (LAVA on (prolonged) scheduled downtime), we should have a build config option with semantics "don't treat LAVA submission failure as fatal to the build". Let it be as false by default, but still have control to cope with such situations.

[Paul Sokolovsky (pfalcon)]

Hopefully LAVA_SUBMIT_FATAL=0/1 is not too confusing name for the option.

[Fathi Boudra (fboudra)]

On 10 November 2011 15:47, Paul Sokolovsky <email address hidden> wrote:
> Hopefully LAVA_SUBMIT_FATAL=0/1 is not too confusing name for the
> option.

LAVA being down can happen again and LAVA_SUBMIT_FATAL doesn't seem a
long term solution.
When LAVA is down, you don't want to report build failure. We need
another state to Jenkins.
Should we open another bug to track the long term solution?

[Paul Sokolovsky (pfalcon)]

Ok, LAVA_SUBMIT_FATAL=0 was implemented, tested and documented at
https://wiki.linaro.org/Platform/Android/LinaroAndroidBuildService#Build_variables_for_LAVA_integration

By this time LAVA is back up, so I don't propagate it to all daily builds, but in case LAVA downtimes will repeat, we have the solution ready.

[Paul Sokolovsky (pfalcon)]

On Thu, 10 Nov 2011 17:40:56 -0000
Fathi Boudra <email address hidden> wrote:

> On 10 November 2011 15:47, Paul Sokolovsky
> <email address hidden> wrote:
> > Hopefully LAVA_SUBMIT_FATAL=0/1 is not too confusing name for the
> > option.
>
> LAVA being down can happen again and LAVA_SUBMIT_FATAL doesn't seem a
> long term solution.
> When LAVA is down, you don't want to report build failure. We need
> another state to Jenkins.
> Should we open another bug to track the long term solution?

I don't have specific grounded opinion on that, should be escalated to
Danilo I guess.

But otherwise (i.e. IMHO): first of all, this tracker is not intended to
be a bug tracker, but rather a request ticket tracker, sometimes it's
called "fast track" or "fast pipe" - if Android team has specific
request/issue, they post it here, and Infrastructure team tries to
resolve exactly that issue fast and low-profile (if possible), without
piling up tasks.

In this specific case the issue was that builds fail due to
infrastructural issue, and we're on the track to fight those, so there
was proposed and implemented solution to make possible to avoid
such failures. The immediate matter of this ticket is thus resolved,
and IMHO, nothing in *this ticket* calls for queuing long-term tasks.

That doesn't mean that we wouldn't benefit from custom build statuses
in Jenkins - but it's separate issue from this one. And we already have
queue of "nice to have" issues for Jenkins, and barely had any
progress with them during last 6 months, so I'm not sure it's productive
to make that queue longer. Again, it's simply "nice to have" feature
(one among infinite number of other nice-to-have's), not something which
causes us big problems (so far, may need to revisit later).

--
Best Regards,
Paul

Linaro.org | Open source software for ARM SoCs
Follow Linaro: http://www.facebook.com/pages/Linaro
http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog