Checkbox needs an option to ONLY report overall PASS or FAIL

[Jeff Lane ]

This is a request from Open Compute who are planning to standardize their entire certification program on Plainbox/Checkbox-NG

They want a way to have it ONLY report an overall PASS or FAIL

Personally, I think that's a bit much for checkbox to handle on it's own, and I really think it would be better to see the full result set. SO, what I was thinking was this:

How about adding an overall PASS/FAIL to the result summary?

In the section with all the hardware and stuff there could be a new field called something like "RESULT:" and that could either be PASS or FAIL depending on the outcome of the tests.

Of cousre, this would require all the tests to exit with a PASS criteria. This too may be a good thing though, as it could force some fixing to tests to avoid our continued use of "FailPass" results (where a test fails but actually passes).

Obviously there may still be situations where a test may fail and actually pass after examination, but for OCP, they want things to be a bit more cut and dry with regards to certification.

[Daniel Manrique (roadmr)]

Hello!

Can you clarify the meaning of "overall PASS/FAIL"? Something like:

if any([result == FAIL for result in testrun.results]):
     print("overall: FAIL")
else:
    print("overall: PASS")

[Ara Pulido (ara)]

Marking as Incomplete as per Daneil's question

[Jeff Lane  (bladernr)]

Yes something like that. The general idea is this:

Tester installs the suite and runs the tests.
Suite is executed with some option like --ultimate-result-only or whatever
Suite completes
All tests pass
suite returns Result: PASS

OR

Some tests pass
suite returns Result: FAIL

The request was to have an All - or - None test suite so that either all tests pass or the system fails. Something I'm not entirely against, I am really not a fan of the FaillPass/Soft Fail/programmed to fail tests we sometimes employ (though we have perfectly valid reasons for doing so)

This, coupled with a completely user-free experience (e.g. NO user interaction at all) would be ideal. The ideal workflow (and this would be awesome for server self-test customers as well) would be:

User installs system
User installs test suite
User runs command: opencompute-certification that runs checkbox-ng with a pre-defined whitelist and does not provide any further prompts, no welcome, no suite selection, no test selection, it immediately starts testing.
Test Completes
Test outputs PASS or FAIL to console (OR in the OCP case, they would like to be able to use a submission add-in to automatically submit either PASS or FAIL to OCP servers somewhere).

This could be grest for cert as well:

User installs system via MAAS
MAAS installs canonical-certification-server
User logs in and runs: canonical-certification-server --hwid a00D000000ABCDEF
tests run automatically with no further input
tests complete and results are automatically sent to C3

and if --hwid is not used, then user results are tarballed for the user and placed in ~ such as:

~/certification-test-results-$DATE.tar.gz

[Zygmunt Krynicki (zyga)]

I think this should be handled by the exit code of the command that was used to start the test. It also affects plainbox and it's a pain for using plainbox to do self-testing on the checkbox project.

[Jeff Lane  (bladernr)]

Actually, now that I've heard just a bit about the new job fields that mark required vs optional test cases, that could affect this positively such that:

PASS = All jobs marked required are PASSed, Optional (Greylist) jobs could be failed.
FAIL = 1 or more jobs marked Required are FAILed.

[Jeff Lane  (bladernr)]

Sylvain, unless you guys see value in this, I'm demoting it to wishlist because OCP is on hold for now so it's not as big a need as before for us.

[Jeff Lane  (bladernr)]

How is this used? Is there a launcher bit that we need to use, or is
there an argument passed to checkbox-cli?

On Fri, Sep 30, 2016 at 5:12 AM, Pierre Equoy
<email address hidden> wrote:
> ** Changed in: checkbox-ng
> Status: Fix Committed => Fix Released
>
> ** Changed in: plainbox
> Status: Fix Committed => Fix Released
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1375328
>
> Title:
> Checkbox needs an option to ONLY report overall PASS or FAIL
>
> Status in Next Generation Checkbox (CLI):
> Fix Released
> Status in PlainBox (Toolkit):
> Fix Released
>
> Bug description:
> This is a request from Open Compute who are planning to standardize
> their entire certification program on Plainbox/Checkbox-NG
>
> They want a way to have it ONLY report an overall PASS or FAIL
>
> Personally, I think that's a bit much for checkbox to handle on it's
> own, and I really think it would be better to see the full result set.
> SO, what I was thinking was this:
>
> How about adding an overall PASS/FAIL to the result summary?
>
> In the section with all the hardware and stuff there could be a new
> field called something like "RESULT:" and that could either be PASS or
> FAIL depending on the outcome of the tests.
>
> Of cousre, this would require all the tests to exit with a PASS
> criteria. This too may be a good thing though, as it could force some
> fixing to tests to avoid our continued use of "FailPass" results
> (where a test fails but actually passes).
>
> Obviously there may still be situations where a test may fail and
> actually pass after examination, but for OCP, they want things to be a
> bit more cut and dry with regards to certification.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/checkbox-ng/+bug/1375328/+subscriptions
>
> Launchpad-Notification-Type: bug
> Launchpad-Bug: product=checkbox-ng; milestone=0.29; status=Fix Released; importance=Wishlist; <email address hidden>;
> Launchpad-Bug: product=plainbox; milestone=0.31; status=Fix Released; importance=Wishlist; <email address hidden>;
> Launchpad-Bug-Information-Type: Public
> Launchpad-Bug-Private: no
> Launchpad-Bug-Security-Vulnerability: no
> Launchpad-Bug-Commenters: apulido bladernr roadmr zyga
> Launchpad-Bug-Reporter: Jeff Lane (bladernr)
> Launchpad-Bug-Modifier: Pierre Equoy (pierre-equoy)
> Launchpad-Message-Rationale: Subscriber
> Launchpad-Message-For: bladernr

--
"Entropy isn't what it used to be."

Jeff Lane -
Server Certification Lead, Warrior Poet, Biker, Lover of Pie
Phone: [phone]
Ubuntu Ham: W4KDH Freenode IRC: bladernr or bladernr_
gpg: 1024D/3A14B2DD 8C88 B076 0DD7 B404 1417 C466 4ABD 3635 3A14 B2DD

[Sylvain Pineau (sylvain-pineau)]

A launcher option, more precisely a new exporter to configure. for example we used this launcher for a diag tool:

[launcher]
app_id = 2015.com.canonical.qa.foo:checkbox
launcher_version = 1
stock_reports = submission_files

[test plan]
unit = 2015.com.canonical.qa.foo::bar
forced = yes

[test selection]
forced = yes

[ui]
type = silent

[exporter:global2]
unit = 2013.com.canonical.plainbox::global

[transport:fi]
type = file
path = /writable/foo-result

[report:result]
exporter = global2
transport = fi
forced = yes