Empty submission result on certification website when running canonical-certification-client

[Yung Shen]

Summary:

canonical-certification-client generated submission.xml seems to be a valid xml,but after submitting to certification website it shows empty results.

Steps to reproduce:

1) Run to canonical-certification-client
2) Select just all informational tests and the vertical scrolling touchpad test
3) Start the run
4) When the run is finished, save the XML report instead of submitting directly (I have to do this because the submit button is failing for me due to bug 1318538)
5) Use checkbox-certification-submit to submit the generated XML report (attached) to C3

Expected results:

 * The submission shows correctly in C3 with results and device information

Actual results:

 * The submission is empty in C3:

   https://certification.canonical.com/hardware/201211-12071/submission/98651

Other information:

If I use the checkbox-certification-client (legacy) with the same selection of tests and I submit, it shows normal results on web site.

I have attached both XML files (the one generated using Checkbox-NG, and the one generated using Legacy)

related pacakges versions:

ii canonical-certification-client 0.1~dev+bzr2992+pkg14~ubuntu14.04.1 all Client Certification Launcher
ii checkbox 0.17.10+bzr2819+pkg76+legacy1~ubuntu14.04.1 amd64 System testing application
ii checkbox-certification 0.20+bzr648+201402141719~ubuntu14.04.1 all Checkbox Certification Tests
ii checkbox-certification-client 0.20+bzr648+201402141719~ubuntu14.04.1 all Client Certification
ii checkbox-certification-tools 0.20+bzr648+201402141719~ubuntu14.04.1 all Checkbox Certification Tools
ii checkbox-gui 0.17.9~dev+bzr2992+pkg2~ubuntu14.04.1 amd64 QML based interface for system testing based on PlainBox.
ii checkbox-ng 0.4~dev+bzr2996+pkg7~ubuntu14.04.1 all PlainBox based test runner
ii checkbox-ng-service 0.4~dev+bzr2996+pkg7~ubuntu14.04.1 all CheckBox D-Bus service
ii checkbox-qt 0.17.10+bzr2819+pkg76+legacy1~ubuntu14.04.1 amd64 QT4 interface for checkbox
ii plainbox-provider-certification-client 0.1~dev+bzr2992+pkg14~ubuntu14.04.1 all Client Certification
ii plainbox-provider-checkbox 0.5~dev+bzr2992+pkg4~ubuntu14.04.1 amd64 CheckBox provider for PlainBox
ii plainbox-provider-resource-generic 0.4~dev+bzr2992+pkg5~ubuntu14.04.1 amd64 CheckBox generic resource jobs provider
ii plainbox-secure-policy 0.6~dev+bzr2996+pkg3~ubuntu14.04.1 all policykit policy required to use plainbox (secure version)
ii python3-checkbox 0.17.10+bzr2819+pkg76+legacy1~ubuntu14.04.1 all CheckBox python3 library
ii python3-checkbox-ng 0.4~dev+bzr2996+pkg7~ubuntu14.04.1 all PlainBox based test runner (Python 3 library)
ii python3-checkbox-support 0.3~dev+bzr2996+pkg2~ubuntu14.04.1 all collection of Python modules used by PlainBox providers
ii python3-plainbox 0.6~dev+bzr2996+pkg3~ubuntu14.04.1 all toolkit for software and hardware testing (python3 module)

[Yung Shen (kaxing)]

attaching submissions from checkbox-ng and checkbox(legacy).

[Yung Shen (kaxing)]

Try to use checkbox-ng with client-selftest.whitelist from checkbox(legacy),
selected the same tests, but the xml generated by checkbox-ng still empty on website.

[Kevin McDermott (bigkevmcd)]

I've checked the integration code between certification and checkbox, and the checkbox parser we rely on is returning an empty list of messages, and no errors.

[Zygmunt Krynicki (zyga)]

I can confirm that. I've fixed bugs 1319729 and 1312808 so now we can see some progress on the actual issue at hand.

[Daniel Manrique (roadmr)]

OK, I looked at both of Yung's submissions.

 I bet the legacy one produces the expected json output when run through the parser, while the checkbox-ng one doesn't. This is because the summary section of the xml file lacks a few items that are needed for the parser to consider this a complete, valid submission and actually process it. This in turn is caused by not having run some of the mandatory resources. The mandatory resources are:

 cpuinfo
 dmi
 dpkg
 lsb
 package
 requirements
 uname

This is the summary from a valid submission:

<summary>
<architecture value="amd64"/>
<client name="checkbox-certification-client" version="0.19~ubuntu14.04.1"/>
<contactable value="False"/>
<date_created value="2014-05-15T06:51:01"/>
<distribution value="Ubuntu"/>
<distroseries value="14.04"/>
<kernel-release value="3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC
<live_cd value="False"/>
<private value="False"/>
<system_id value="d00f84de8a555815fa1c4660280da308"/>
</summary>

This is the invalid submission's summary:

<summary>
    <client name="checkbox-gui" version="0.6.0"/>
    <date_created value="2014-05-15T06:37:08"/>
    <private value="False"/>
    <contactable value="False"/>
    <live_cd value="False"/>
    <system_id value=""/>
</summary>

Now, in Yung's case, the whitelists he's using have the resources listed explicitly in the whitelist, but some of the mandatory ones are not required by any job, which means bug 1302388 is the cause of his problem. A possible solution would be to include miscellanea/submission_resources in those whitelists; we had kept the explicit list of resources for compatibility with checkbox legacy but that's no longer needed so we can potentially prune this.

I remember there was another bug affecting resource running from checkbox-gui, other than that possibility I'd say this bug is a duplicate of byg 1302388, and we can mitigate it with the workaround I mentioned.

[Zygmunt Krynicki (zyga)]

I'm closing this as invalid now (it's a dupe of the bug daniel mentioned) and I'm linking it to bug 1320253 to keep issues like that easy to solve next.