plainbox: some tests run in wrong order

[Daniel Manrique]

Tests should be run in whitelist order unless that order doesn't make sense.

In the autotesting.whitelist, we want fwts_test to run first, and then, regardless of the result (even if fail), fwts_result.log should run to attach the whole log to the submission for later perusal. Thus, fwts_result.log can't really depends on fwts_test.

This is expressed in the whitelist order:

__miscellanea__
miscellanea/submission-resources
miscellanea/fwts_test
miscellanea/fwts_results.log
__networking__
networking/detect

However, fwts_results.log is failing because it's running *before* fwts_test and thus is unable to find the file it wants to attach:

__miscellanea__
miscellanea/submission-resources
miscellanea/fwts_test
miscellanea/fwts_results.log
__networking__
networking/detect

A possible workaround would be adding a *third* job to parse the results log for failures, then both fwts_results.log and (say) fwts_analyze can depend on fwts_test. Then we modify fwts_test to always exit with true, regardless of fwts exit code. This is a kludge however :(

[Zygmunt Krynicki (zyga)]

Is this a dupe of the whitelist ordering bug or something new?

[Daniel Manrique (roadmr)]

It may be a duplicate of https://bugs.launchpad.net/checkbox/+bug/1213893. That one says "whitelist ordering of jobs is not preserved by clients". If "plainbox sru" is considered a "client" then it would be a dupe.

[Zygmunt Krynicki (zyga)]

Yeah, it's a dupe.