Newly-added fwts test is interactive, making it unsuitable for use on servers

[Rod Smith]

As per IRC discussion, at least one specific fwts test added to a recent development version of checkbox is interactive, which causes it to hang when run on a server (via canonical-certification-server). Below is the IRC discussion with details and version information.

<rodsmith> roadmr: The fwts suite is hanging when using the ppa:checkbox-dev/ppa version. It's OK when using ppa:hardware-certification/public. Any insights?
<zyga> rodsmith: the former is more recent, the later has older software
<zyga> rodsmith: regression in fwts?
<roadmr> rodsmith: sure, let's look at what may be happening
<roadmr> rodsmith: so when you say fwts suite, you mean a specific fwts test is hanging? if so, which one is it?
<rodsmith> roadmr, zyga: The fwts package is the same in both cases (14.07.00-0ubuntu~t).
<rodsmith> roadmr, zyga: canonical-certification-server hangs when running the fwts test with the checkbox dev ppa, though.
<roadmr> rodsmith: ok, that's one thing out of the way. So can you be more specific about the hang? what's the last thing you see on-screen?
<rodsmith> roadmr: Sorry; accidentally hit Ctrl+C with "screen" running, so it's gone. Give me a minute to re-create it....
<roadmr> rodsmith: thanks!
<rodsmith> roadmr: https://pastebin.canonical.com/114917/
<rodsmith> roadmr: It sits like that for at least 2.5 hours.
<roadmr> rodsmith: ok, is it doing that right now? don't stop it if it is
<roadmr> rodsmith: see if ~/.cache/plainbox/sessions/ has some session directories in it
<roadmr> rodsmith: if so, please go into the most recent one, and see if there's an fwts_results.log somewhere in there, that'll tell us what fwts is doing right now
<rodsmith> roadmr: https://pastebin.canonical.com/114918/
<roadmr> rodsmith: ahh... I see
<roadmr> rodsmith: we recently added some tests to the list of fwts tests we want, my guess is some of them are interactive and so cause the hang
<roadmr> rodsmith: this is a server, I guess, right?
<rodsmith> roadmr: Essentially. It's an Orange Box.
<roadmr> rodsmith: ok, let me see what's going on with those new tests

$ dpkg -l canonical-certification-server checkbox-ng
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-=================================
ii canonical-cert 0.5+bzr3167+ all Server Certification Launcher
ii checkbox-ng 0.8+bzr3167+ all PlainBox based test runner

[Rod Smith (rodsmith)]

Oops; mis-cut the critical final comment from roadmr:

<roadmr> rodsmith: ok, got it, ac_adapter is interactive, it wants you to press enter and do some actions with the power adapter

[Daniel Manrique (roadmr)]

OK, so these tests were added as a request for CDTS. The problem is that we have no provision to run an interactive session with fwts, so there's no way to run these tests as requested :/

The issue, more precisely, is that fwts_test, the wrapper, expects all fwts tests to be non-interactive, so it completely ignores stdin and just dumps stdout to a file for later analysis. There are a handful of tests in the list that require interaction from the user and use more advanced tty control, and so break the assumptions of our wrapper and are unrunnable from within it.

ac_adapter
battery
brightness
hotkey
lid
power_button

battery, brightness, hotkey, lid, and power_button are covered by other checkbox tests, so removing them does not reduce our coverage significantly.

I'm a bit unsure about ac_adapter, but I think it's also present in other checkbox tests.

[Daniel Manrique (roadmr)]

Those tests were added as part of CDTS, so adding it as affected to keep track of the fact that we're removing those tests from the script.