Checkbox only reports the first resource found for a job requirement

[Marc Tardif]

The impact of this bug is that when test results are reported, partial device information might be linked to the results. This fix makes sure that all related devices are properly linked to the test results.

Checkbox uses job requirements in order to link resources to results. For example, the following requirement should like a result to all the network devices found on the system:

  device.category == 'NETWORK'

So, if there are two network devices, such as an ethernet and a wireless controller, then these should be linked to the result. However, the checkbox.resource module seems to bail as soon as the first resource is found.

Note that the requirements assessment is still correct, whether to run a job or not. However, the details of the requirements are simply missing information.

[Daniel Manrique (roadmr)]

Hello, the following information should complete this bug as a valid SRU report, to hopefully upload this fix to Lucid.

Solution:

Checbox revision 786 adds code to ensure that all related devices are properly linked to the test results.

TEST CASE:

We need to be tricky when running this test case, as checkbox 0.9.1 doesn't log detailed information about each test that we can access after a test run.

1- Launch System Testing on a system with Ubuntu 10.04 and two network controllers (eth0 and eth1 for instance). A virtual machine with two interfaces works fine for this test.
2- On the test selection screen, press "Deselect All", then expand "Network tests" and select only "network_test". Press Next.
3- When the test runs, confirm that it detected both network controllers. DO NOT PRESS NEXT.
4- Here's the tricky part: close checkbox by pressing the X on the top left corner. This forces it to log an exception for the execution interruption, and in this case it *does* log the information we need.

Now look at the log file (~/.cache/checkbox/checkbox.log), near the end there's a log line starting with ERROR.

Without the fix:

Keyboard interrupt while running event handler None.ManualTest.prompt_manual() for event type 'prompt-manua
l' with args (<checkbox_gtk.gtk_interface.GTKInterface object at 0xb7701eec>, {'status': 'uninitiated', 'description': u'Detecting your netw
ork controller(s):\n\n$output\n\nIs this correct?', 'plugin': 'manual', 'requires': ["device.category == 'NETWORK'"], 'command': 'network_te
st', 'suite': 'network', 'type': 'test', 'resources': [{'category': 'NETWORK', 'subproduct_id': '30', 'product_id': '4110', 'bus': 'pci', 'v
endor_id': '32902', 'driver': 'e1000', 'path': '/devices/pci0000:00/0000:00:03.0', 'subvendor_id': '32902'}], 'name': 'network_test'}) {}.

With the fix:

Keyboard interrupt while running event handler None.ManualTest.prompt_manual() for event type 'prompt-manual' with args (<checkbox_gtk.gtk_interface.GTKInterface object at 0xb76b9eec>, {'status': 'uninitiated', 'description': u'Detecting your network controller(s):\n\n$output\n\nIs this correct?', 'plugin': 'manual', 'requires': ["device.category == 'NETWORK'"], 'command': 'network_test', 'suite': 'network', 'type': 'test', 'resources': [{'category': 'NETWORK', 'subproduct_id': '30', 'product_id': '4110', 'bus': 'pci', 'vendor_id': '32902', 'driver': 'e1000', 'path': '/devices/pci0000:00/0000:00:03.0', 'subvendor_id': '32902'}, {'category': 'NETWORK', 'subproduct_id': '30', 'product_id': '4110', 'bus': 'pci', 'vendor_id': '32902', 'driver': 'e1000', 'path': '/devices/pci0000:00/0000:00:08.0', 'subvendor_id': '32902'}], 'name': 'network_test'}) {}.

Notice how the resources array contains data for the two network cards with the fixed code, whereas the un-fixed version only shows the first card.

Regression potential:

The logic changes in themselves are quite harmless. They will potentially return larger sets of data, so if other parts of checkbox aren't ready to deal with this data, that might entail failures where there were previously none. However this hasn't happened in real-world usage of this code, and no subsequent bug fixes are related to r...

[Ara Pulido (ara)]

This bug is awaiting verification that the checkbox version in lucid-proposed solves the problem. Please test checkbox and update this bug with the results. If the problem is solved, change the tag 'verification-needed' to 'verification-done'.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Martin Pitt (pitti)]

Hello Marc, or anyone else affected,

Accepted checkbox into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Chad A Davis (chadadavis)]

Verified fix from checkbox 0.9.2 on Lucid

[Launchpad Janitor (janitor)]

This bug was fixed in the package checkbox - 0.9.2

---------------
checkbox (0.9.2) lucid-proposed; urgency=low

  New upstream release (LP: #567568):
  * Added referer when sending submissions to Launchpad (LP: #550973)
  * Added suggests to checkbox package in debian/control file (LP: #352740)
  * Fixed udev_resource script to be more resilient (LP: #556824)
  * Fixed cdimage_resource script to read casper.log (LP: #558728)
  * Fixed reporting all resources found for a job (LP: #560948)
  * Fixed stalling when using kdesudo to start backend (LP: #557443)
  * Fixed starting the appropriate default browser on UNR (LP: #563050)
  * Fixed opening the report with the gconf preferred browser (LP: #562580)
  * Fixed suspend_test to use relative time for wakealarm (LP: #349768)
  * Fixed backend not getting terminated upon closing (LP: #553328)
 -- Daniel Manrique <email address hidden> Wed, 22 Jun 2011 14:18:08 -0400