canonical-certification-server doesn't run resource jobs, produces incomplete, invalid submission.xml

[Daniel Manrique]

A report came in of a submission.xml that was unparsable. On analyzing the faulty submission I found that it lacked some mandatory resources, specifically uname, dpkg and lsb which are mandatory for a valid submission.

Further digging revealed that the canonical-certification-server utility "eats" resources and doesn't run them.

To reproduce this, run canonical-certification-server (if running from source it helps to point it to a whitelist with -w, if from package, just deselect all but cpu/topology).

The resources aren't shown so you can't explicitly select/deselect them, so the logical behavior is for them to actually run.

Complete the run and then look at the list of executed tests and you won't see any resources. Contrast that with running e.g. plainbox run -w some.whitelist, where the resources do run and show up.

Further, look at the submission.xml, and notice that at the end in the summary section, it lacks the architecture, kernel-release, distribution and distroseries elements that are required for a valid submission. Architecture comes from dpkg, distribution and distroseries from lsb, and kernel-release from uname.

Looking at the source code I see in checkbox_ng/checkbox_ng/commands/cli.py a section in the run method where the user is shown the list of jobs to make a selection.

A breakpoint on line 563 reveals that, up to this point, manager.state.run_list contains the list of all the tests we'd like to run, as identified initially by the given whitelist.

After running the tree and looking at its tree.selection, the resources have disappeared, so when we do self._update_desired_job_list on line 570 it dutifully updates the run_list removing the resources.

Looking now at the SelectableJobTreeNode, the add_job method just returns when it sees a "resource" job, thus it never adds it to the tree's selection.

To test this theory I changed the "return" on line 154 to "pass". When running things, I see the resources in the selection list (which is not entirely OK UI-wise, but I just leave them alone), and when completing the test run the resources have been executed and the xml contains all the needed data.

I'll come up with a fix in a moment, just wanted to describe the problem for future reference.