Based on the comment that this was observed on servers, I tried this on a server installation of Ubuntu, though the problem should still occur with a client installation.

First, with checkbox 0.13.7 installed:

$ dpkg --list 'checkbox*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version        Description
+++-==============-==============-============================================
ii  checkbox       0.13.7         System testing application
ii  checkbox-cli   0.13.7         Command line interface for checkbox
un  checkbox-gtk   <none>         (no description available)

I had to change the test command to use checkbox-cli instead of checkbox-qt. Otherwise the way it works is exactly the same. I did the test run, the fwts_wakealarm test did NOT cause the system to suspend. The xml report shows the test as passed:

<question name="power-management/fwts_wakealarm">
<answer type="multiple_choice">pass</answer>
<answer_choices>
<value type="str">fail</value>
<value type="str">pass</value>
<value type="str">uninitiated</value>
<value type="str">unresolved</value>
<value type="str">unsupported</value>
<value type="str">untested</value>
</answer_choices>
<comment></comment>
</question>

but then the fwts_wakealarm.log says it didn't actually do anything:

00012 wakealarm       Test ACPI Wakealarm.
00013 wakealarm       -----------------------------------------------------------------------------
00014 wakealarm       Must be run as root or sudo to be able to read system information.
00015 wakealarm       Aborted test, insufficient privilege.
00016 wakealarm       =============================================================================
00017 wakealarm       0 passed, 0 failed, 0 warnings, 4 aborted, 0 skipped, 0 info only.
00018 wakealarm       =============================================================================


OK, then I remove the .cache/checkbox directory, upgrade to checkbox 0.13.8 from proposed:

$ dpkg --list 'checkbox*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version        Description
+++-==============-==============-============================================
ii  checkbox       0.13.8         System testing application
ii  checkbox-cli   0.13.8         Command line interface for checkbox
un  checkbox-gtk   <none>         (no description available)

I ran the tests using the given testing command. I noticed the system did *not* suspend, though it *did* take longer than the previous try. The test is now running as root, and the proof is that the fwts-wakealarm.log file is getting created in /. This is not entirely desirable but the behavior is at least better than it was previously (i.e. non-running test).

Here's the result from the fwts-wakealarm.log:

00012 wakealarm       Test ACPI Wakealarm.
00013 wakealarm       -----------------------------------------------------------------------------
00014 wakealarm       Test 1 of 4: Check existance of /sys/class/rtc/rtc0/wakealarm.
00015 wakealarm       PASSED: Test 1, /sys/class/rtc/rtc0/wakealarm found.
00016 wakealarm       
00017 wakealarm       Test 2 of 4: Trigger wakealarm for 1 seconds in the future.
00018 wakealarm       Trigger wakealarm for 1 seconds in the future.
00019 wakealarm       PASSED: Test 2, RTC wakealarm was triggered successfully.
00020 wakealarm       
00021 wakealarm       Test 3 of 4: Check if wakealarm is fired.
00022 wakealarm       PASSED: Test 3, RTC wakealarm triggered and fired successfully.
00023 wakealarm       
00024 wakealarm       Test 4 of 4: Multiple wakealarm firing tests.
00025 wakealarm       Trigger wakealarm for 1 seconds in the future.
00026 wakealarm       Trigger wakealarm for 2 seconds in the future.
00027 wakealarm       Trigger wakealarm for 3 seconds in the future.
00028 wakealarm       Trigger wakealarm for 4 seconds in the future.
00029 wakealarm       PASSED: Test 4, RTC wakealarm triggered and fired successfully.
00030 wakealarm       
00031 wakealarm       =============================================================================
00032 wakealarm       4 passed, 0 failed, 0 warnings, 0 aborted, 0 skipped, 0 info only.
00033 wakealarm       =============================================================================



So it's now passing because it was able to run.

The test case mentions the system should suspend, but since we're just testing the wakealarm functionality, it's not really necessary to suspend, just to ensure that the wakealarm is set and triggers correctly.

I think this is good enough, I'll set the bug to verification-done since at least the test is now running.