checkbox-ng ignores UI type declaration in launcher

[Jeff Lane ]

Reading the docs at read the docs says in the launcher tutorial:

type

Type of UI to use. This has to be set to interactive, silent, converged, or converged-silent. Default value: interactive, which runs the Checkbox command line version. Note: the converged and converged-silent UI types will launch the QML interface and requires checkbox-converged to be installed on your system. Note: using silent or converged-silent UI types requires forcing test selection and test plan selection.

One would presume "silent" would cause the UI to produce NO output. Thus I set it up like so:

[ui]
type = silent
output = hide

but the type = seems to be ignored by checkbox-ng as it displays a lot of output during testing.

bladernr@galactica:~$ ./testlauncher
Preparing...
============[ Bootstrap com.canonical.certification::package (1/3) ]============
=============[ Bootstrap com.canonical.certification::fwts (2/3) ]==============
WARNING:plainbox.ctrl:Ignoring firmware/fwts_{name} with missing template parameter name
WARNING:plainbox.ctrl:Ignoring firmware/fwts_{name}.log with missing template parameter name
============[ Bootstrap com.canonical.certification::device (3/3) ]=============
Reports will be saved to: /home/bladernr/.local/share/checkbox-ng
--------------[ Running job 1 / 56. Estimated time left: unknown ]--------------
------------------[ Attach PCI configuration space hex dump ]-------------------
ID: com.canonical.certification::lspci_standard_config_json
Category: com.canonical.plainbox::info
(Command output hidden)
Outcome: job passed
--------------[ Running job 2 / 56. Estimated time left: unknown ]--------------
-------------[ Collect information about installed snap packages ]--------------
ID: com.canonical.certification::snap
Category: com.canonical.plainbox::uncategorised
(Command output hidden)
Outcome: job passed
--------------[ Running job 3 / 56. Estimated time left: unknown ]--------------
---------------------------------[ udev_json ]----------------------------------
ID: com.canonical.certification::udev_json
Category: com.canonical.plainbox::info
(Command output hidden)
Outcome: job passed
<SNIP>

bladernr@galactica:~$ dpkg -l |grep checkbox|awk '{print $2"\t\t"$3}'
checkbox 1.2.4-0ubuntu1
checkbox-converged 1.7.0+git201710231600+pkg14~ubuntu16.04.1
checkbox-gui 1.2.4-0ubuntu1
checkbox-ng 0.35.0+git201712221005+pkg83~ubuntu16.04.1
plainbox-provider-checkbox 0.41.0+git201712201300+pkg72~ubuntu16.04.1
python3-checkbox-ng 0.35.0+git201712221005+pkg83~ubuntu16.04.1
python3-checkbox-support 0.36.0+git201712211115+pkg69~ubuntu16.04.1

[Maciej Kisielewski (kissiel)]

@Jeff
I understand that you suggest that 'silent' type should produce only the return code (and reports on the filesystem if any).

[Maciej Kisielewski (kissiel)]

'silent' in Checkbox means no urwid interactive screen. More like 'no interaction' than 'no output'. I guess it's better to introduce a new value that will introduce truly silent run; too many launchers are already configured to run as 'silent'.

I think 'None' could be such value.