bluetooth/detect shouldn't run on a system with no Bluetooth device

[Nicola Heald]

Running the latest oneiric, checkbox and checkbox-gtk from ppa:checkbox-dev checkbox_0.12.8+bzr1082+201109290705~oneiric1 inside a Virtualbox virtual machine.

I get to the bluetooth tests, and get a dialog box saying "bluetooth/detect crashed" I have no bluetooth hardware attached.

I try to submit a bug report but the process freezes and I can't carry on with the test run.

[Impact]
The bluetooth/detect test will run (and fail) on systems with no Bluetooth hardware.
On systems with apport enabled, this will pop up a crash report window, needlessly worrying the user.
On all systems, this will mistakenly report to Ubuntu Friendly that bluetooth hardware was present but failed, which penalizes the user's score for optional hardware that is not actually present. This means that a computer without bluetooth hardware is unable to get a perfect 5-star score in UF, which is misrepresentative of a system that may work perfectly but have no Bluetooth support.

[Development fix]
Checkbox revision 1114 changes the test definition so it requires an actual bluetooth device to run. Otherwise the test is marked as skipped due to unmet requirements, which is the correct behavior.

[Stable fix]
This is fixed in http://bazaar.launchpad.net/~checkbox-dev/checkbox/trunk/revision/1114. Also, the linked branch (https://code.launchpad.net/~roadmr/ubuntu/oneiric/checkbox/0.12.8.1) contains this and other fixes and merges cleanly against Ubuntu checkbox 0.12.8.

[Test Case]
- Start "system testing" (checkbox-gtk version 0.12.8) on an 11.10 system with NO bluetooth hardware (check that output of hcitool dev shows no devices). As originally reported, a virtual machine may fulfill this requirement.
- Press "Next" and provide password when prompted.
- Press "Deselect all", then select "Bluetooth tests" and press Next.

Expected result:
- Bluetooth test doesn't run
- Test gets marked as skipped / requirements not met in the final report.

Actual result:
- Bluetooth test attempts to run, crashes, if apport is enabled the user will be requested to file a bug report.
- Test appears as failed in the final report.

[Regression potential]This actually reduces the pool of systems on which this test will run, and the added code simply checks that a bluetooth resource (essentially a string match against a text file) is present. Thus the potential for regression is almost nil. Maybe if a system has a bluetooth device that is not presented as such it will be untested now, but such a scenario is unlikely.

[Nicola Heald (notnownikki)]

If i say that I don't want to submit a bug report, the run continues to the bluetooth file transfer test, even though I don't have the hardware.

[Ara Pulido (ara)]

Can you attach checkbox.log and submission.xml, please?

Both need to be under .cache/checkbox

[Nicola Heald (notnownikki)]

checkbox log from after the test is reported as crashing

[Nicola Heald (notnownikki)]

There is no submission.xml in .cache/checkbox.

There is no output from hcitool dev | tail -n+2 | awk '{print $2}';

[Nicola Heald (notnownikki)]

Just been asked to run:

hcitool dev | tail -n+2 | awk '{print $2}'; hcitool dev | tail -n+2 | awk '{print $2}';

No output from that command.

$? gives 0

[Brendan Donegan (brendan-donegan)]

I can't reproduce this - bluetooth/detect does *fail* on my VirtualBox, but that's to be somewhat expected. I'll take a look through the logs and see if I notice anything.

[Brendan Donegan (brendan-donegan)]

Mike, can you also attach subunit.log?

[Brendan Donegan (brendan-donegan)]

Ok, so bluetooth/detect (and all Bluetooth tests) should really require: device.category == 'BLUETOOTH', but it doesn't. Given that the test only impacts on portable systems and the chances of finding a modern portable system *without* Bluetooth are slim to none, this isn't very urgent - but we should get the fix in for P anyway.

[Ara Pulido (ara)]

Let's fix it in the first Oneiric sru for checkbox

[Ara Pulido (ara)]

Moving again to 0.13
Finally we are going to ignore errors in /detect

[Brendan Donegan (brendan-donegan)]

I've stopped working on this one for now. It isn't urgent at the moment.

[Martin Pitt (pitti)]

Is this also fixed in precise's 0.13? If no, please fix it there first, otherwise please close the task. Thanks!

[Brendan Donegan (brendan-donegan)]

Hi Martin,

Yes, this was fixed in https://code.launchpad.net/~brendan-donegan/checkbox/bug862322_bt_detect_requires_bt in Precise.

[Daniel Manrique (roadmr)]

As per Brendan's comment, and since the revision implementing this change is integrated into 0.13, I'll mark the Precise task Fix Released.

Apologies for the mixup, and thanks for reviewing these changes!

[Daniel Manrique (roadmr)]

Oh, I'm really messing things up today. I shouldn't have marked the Oneiric task Fix Released, not until the SRU has been published.

My sincerest apologies for this mess. I'll put the task back to New.

[Chris Halse Rogers (raof)]

Marking as fix released in Precise.

[Chris Halse Rogers (raof)]

Hello Mike, or anyone else affected,

Accepted checkbox into oneiric-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Daniel Manrique (roadmr)]

OK, I ran the test case with both 0.12.8 and 0.12.9 and in both cases on a system with no Bluetooth, the tests are marked as failed, meaning that checkbox is trying to run them even if no bluetooth hardware is present.

The cause for this is that 0.12.9 doesn't contain the required code fix, even though the changelog says it does.

This was an oversight on my part. Apologies for this. I'm linking a branch that can be merged against lp:ubuntu/oneiric/checkbox to fix this by adding the necessary bits of code.

https://code.launchpad.net/~roadmr/ubuntu/oneiric/checkbox/fix-662322-in-0.12.9

I'm marking this verification-failed and await instructions on how to proceed.

Thanks!

[Martin Pitt (pitti)]

Hello Mike, or anyone else affected,

Accepted checkbox into oneiric-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Daniel Manrique (roadmr)]

Hi,

I ran this on a Dell Inspiron 1545 with no Bluetooth adapter.

I deselected all tests in checkbox, and reselected only the bluetooth category.

Checkbox 0.12.8 marks the bluetooth/detect test as FAILED (old, wrong behavior).

I then upgraded checkbox and checkbox-gtk to 0.12.10 and reran the bluetooth tests. This time they appear in grey in the report, saying "Job requirements not met", meaning the jobs were skipped altogether and don't count as a failure, which is the desired behavior.

I'm adding verification-done to this bug.

Thanks!

[Launchpad Janitor (janitor)]

This bug was fixed in the package checkbox - 0.12.10

---------------
checkbox (0.12.10) oneiric-proposed; urgency=low

   [Brendan Donegan]
   * Have the bluetooth/detect tests require a device with the category
     BLUETOOTH to run, thus preventing the test from failing on systems with
     no Bluetooth device (LP: #862322)
   * Remove fix for bug 877752 on connect_wireless, as it was not working as
     intended.
 -- Daniel Manrique <email address hidden> Tue, 07 Feb 2012 15:30:36 -0500