Certification-status tags not working if poweroff/reboot related tests are run

[Kristin Chuang]

Title: Certification-status tags not working if poweroff/reboot related tests are run

Summary: Certification-status tags not working if poweroff/reboot related tests are run

Steps:
1. Make sure a .pxu matching the following criteria exist:

(a) contains any one of the following test cases:
  power-management/reboot
  power-management/poweroff
  stress/poweroff
  stress/reboot
(b) the above cases are tagged "certification-status=blocker" in the .pxu
(c) has some other cases with "certification-status=blocker" tags in this .pxu

2. Run plainbox with the above .pxu
3. Check the result submission.html & submission.xlsx files

Expected results: All cases' certification statuses in submission.html & submission.xlsx should be correct

Actual results: All cases' certification statuses in submission.html shows "unspecified", all cases' certification statuses in submission.xlsx is empty

Additional information:
Plainbox versions:

u@u-HP-x360-310-G2-PC:~$ dpkg -l | grep plainbox
ii plainbox 0.24~ppa~ubuntu14.04.1 all toolkit for software and hardware integration testing
ii plainbox-glmark2-es2-meta 0.20.1-2 amd64 Meta-package to selectively install glmark2-es2
ii plainbox-insecure-policy 0.24~ppa~ubuntu14.04.1 all policykit policy required to use plainbox (insecure version)
ii plainbox-provider-certification-client 0.20.1-2 all Client Certification provider for Plainbox
ii plainbox-provider-checkbox 0.23~ppa1~ubuntu14.04.1 amd64 CheckBox provider for PlainBox
ii plainbox-provider-oem 0.27 all plainbox oem provider
ii plainbox-provider-oem-stella 0.27 all plainbox oem provider for stella
ii plainbox-provider-resource-generic 0.21~ppa~ubuntu14.04.1 amd64 CheckBox generic resource jobs provider
ii python3-plainbox 0.24~ppa~ubuntu14.04.1 all toolkit for software and hardware testing (python3 module)

Image: stella-kenting151027-trusty-amd64-20151029-0

CPU: Intel(R) Pentium(R) CPU N3700 @ 1.60GHz (4x)

GPU: 00:02.0 VGA compatible controller: Intel Corporation Device 22b1 (rev 21)

Manufacturer: Hewlett-Packard

ProductName: HP x360 310 G2 PC

BiosVersion: F.40

[Kristin Chuang (kristinchuang)]

Automatically attached

[Kristin Chuang (kristinchuang)]

Some investigation shows that this issue only occurs if poweroff/reboot related cases are run.

When using the same .pxu:

[Case A]
Select and run several cases (include selection of poweroff/reboot tests):
=> All selected cases' certification-status will be "unspecified" in report outputs. (Not OK)

[Case B]
Select and run several cases (exclude selection of poweroff/reboot tests):
=> All selected cases' certification-status will reflect the correct status defined in .pxu (OK)

[Case C]
Select several cases and include selection of poweroff/reboot tests, but *manually skip* the poweroff/reboot tests when prompted to run it:
=> All selected cases' certification-status will reflect the correct status defined in .pxu (OK)

Therefore suspect that something happened during the pm_test poweroff and reboot that reset the certification-status tag.