Comment 3 for bug 1375328

Yes something like that. The general idea is this:

Tester installs the suite and runs the tests.
Suite is executed with some option like --ultimate-result-only or whatever
Suite completes
All tests pass
suite returns Result: PASS

OR

Some tests pass
suite returns Result: FAIL

The request was to have an All - or - None test suite so that either all tests pass or the system fails. Something I'm not entirely against, I am really not a fan of the FaillPass/Soft Fail/programmed to fail tests we sometimes employ (though we have perfectly valid reasons for doing so)

This, coupled with a completely user-free experience (e.g. NO user interaction at all) would be ideal. The ideal workflow (and this would be awesome for server self-test customers as well) would be:

User installs system
User installs test suite
User runs command: opencompute-certification that runs checkbox-ng with a pre-defined whitelist and does not provide any further prompts, no welcome, no suite selection, no test selection, it immediately starts testing.
Test Completes
Test outputs PASS or FAIL to console (OR in the OCP case, they would like to be able to use a submission add-in to automatically submit either PASS or FAIL to OCP servers somewhere).

This could be grest for cert as well:

User installs system via MAAS
MAAS installs canonical-certification-server
User logs in and runs: canonical-certification-server --hwid a00D000000ABCDEF
tests run automatically with no further input
tests complete and results are automatically sent to C3

and if --hwid is not used, then user results are tarballed for the user and placed in ~ such as:

~/certification-test-results-$DATE.tar.gz