Failure to boot test kernel image not reflected in CI status

[Gary S. Robertson]

A recent bug in a CI script caused filesystem images and hwpack images generated for kernel testing to be built without device trees. The actual Jenkins build process completed without error because the instructions to build the device tree were missing, and all the included build operations completed successfully even though the resulting image contained no device tree.

When these defective images were passed to LAVA for post-build testing the test kernel images naturally failed to boot. LAVA responded appropriately and eventually logged the failure after repeated retries... however the failure status was not propagated back to reflect in the status of the Jenkins CI build/test operation. The result was that the cumulative status reported for the Jenkins CI builds showed a 'green light' and gave no indication that there were errors in the build or in the post-build tests.

See LAVA job #120948 for a log of the problem.

[Milo Casagrande (milo)]

I'm not sure I follow here: do we have a mechanism in LAVA that tells Jenkins CI that something failed on LAVA side? Or is this a request/wishlit to have something similar?

[Gary S. Robertson (gary-robertson)]

I was under the impression that there was some linkage
between build scripts and post-build test results
which was supposed to be propagated back into
the Jenkins project status indicators,
such that failures in post-build LAVA tests
would be reflected somehow in those statuses.

Based upon your response I take it this is not the case...
and if that isn't part of the design intent then
the implementation isn't broken after all.

That said, we have an instance here where the build process
was broken badly enough to produce an un-bootable kernel,
and only the post-build tests could tell us that.
The current CI / LAVA setup failed to produce any emails
or other status indicators to alert anyone of this
catastrophic regression.

So consider the bug report just a flag-raising about this
vulnerability in our current system.
If some coupling of post-build test statuses to the
build status indicators and alerts is feasible and sane,
then I guess this should be a wish-list item.
If there isn't a reasonable way to achieve this coupling
then I guess this is an 'invalid - won't fix' scenario
and we all just have to be aware of the discontinuity.

On Tue, Apr 22, 2014 at 2:30 AM, Milo Casagrande
<email address hidden>wrote:

> I'm not sure I follow here: do we have a mechanism in LAVA that tells
> Jenkins CI that something failed on LAVA side? Or is this a
> request/wishlit to have something similar?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1309703
>
> Title:
> Failure to boot test kernel image not reflected in CI status
>
> Status in Linaro Continuous Integration:
> New
>
> Bug description:
> A recent bug in a CI script caused filesystem images and hwpack images
> generated for kernel testing to be built without device trees. The
> actual Jenkins build process completed without error because the
> instructions to build the device tree were missing, and all the
> included build operations completed successfully even though the
> resulting image contained no device tree.
>
> When these defective images were passed to LAVA for post-build testing
> the test kernel images naturally failed to boot. LAVA responded
> appropriately and eventually logged the failure after repeated
> retries... however the failure status was not propagated back to
> reflect in the status of the Jenkins CI build/test operation. The
> result was that the cumulative status reported for the Jenkins CI
> builds showed a 'green light' and gave no indication that there were
> errors in the build or in the post-build tests.
>
> See LAVA job #120948 for a log of the problem.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/linaro-ci/+bug/1309703/+subscriptions
>

[Milosz Wasilewski (mwasilew)]

@Gary
I'm not sure there is a way to alter the build status after it's completed in Jenkins. The best you can have is to get some testing results and have green/red indication in this area. AFAIK that is not implemented with post-build-lava and I didn't hear any plans to do so.

As a workaround you can probably use qa-reports.linaro.org (just launched). It shows both: build in jenkins and lava test statuses in one place. The prerequisite is to have post-build-lava configured for your build and your jenkins project added to qa-reports. If you're interested, let me know. The project is still in early stages but I'm happy to help with all requests coming :)

[Gary S. Robertson (gary-robertson)]

Thanks for your suggestion - I'll mention it to Mike Holmes and
it might be a good direction for the future.

In retrospect I understand that linking post-build test results
back into the Jenkins project status is probably not feasible due to
the complexity and open nature of the LAVA test capabilities...
i.e. defining which results should alter that status and how they
should alter it.

Meanwhile Mike addressed the lack of notification with a filter on the LAVA
results
that generates emails to subscribers when LAVA gets such a
catastrophic failure... so we should be covered if something like this
happens again.

On Fri, Apr 25, 2014 at 4:37 AM, Milosz Wasilewski <
<email address hidden>> wrote:

> @Gary
> I'm not sure there is a way to alter the build status after it's completed
> in Jenkins. The best you can have is to get some testing results and have
> green/red indication in this area. AFAIK that is not implemented with
> post-build-lava and I didn't hear any plans to do so.
>
> As a workaround you can probably use qa-reports.linaro.org (just
> launched). It shows both: build in jenkins and lava test statuses in one
> place. The prerequisite is to have post-build-lava configured for your
> build and your jenkins project added to qa-reports. If you're
> interested, let me know. The project is still in early stages but I'm
> happy to help with all requests coming :)
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1309703
>
> Title:
> Failure to boot test kernel image not reflected in CI status
>
> Status in Linaro Continuous Integration:
> New
>
> Bug description:
> A recent bug in a CI script caused filesystem images and hwpack images
> generated for kernel testing to be built without device trees. The
> actual Jenkins build process completed without error because the
> instructions to build the device tree were missing, and all the
> included build operations completed successfully even though the
> resulting image contained no device tree.
>
> When these defective images were passed to LAVA for post-build testing
> the test kernel images naturally failed to boot. LAVA responded
> appropriately and eventually logged the failure after repeated
> retries... however the failure status was not propagated back to
> reflect in the status of the Jenkins CI build/test operation. The
> result was that the cumulative status reported for the Jenkins CI
> builds showed a 'green light' and gave no indication that there were
> errors in the build or in the post-build tests.
>
> See LAVA job #120948 for a log of the problem.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/linaro-ci/+bug/1309703/+subscriptions
>

[Chase Qi (chase-qi)]

Migrated to bugzilla: https://bugs.linaro.org/show_bug.cgi?id=520