canonical-certification-submit broken (exec checkbox not found)

[Jeff Lane ]

A customer reported issues with canonical-certification-submit not working.

I did some testing, and the following occurs:

First from the dev ppa:
bladernr@xenial-dev:~$ dpkg -l |grep checkbox
ii  checkbox-ng                      0.34.0+git201707180907+pkg73~ubuntu16.04.1 all          PlainBox based test runner
ii  python3-checkbox-ng              0.34.0+git201707180907+pkg73~ubuntu16.04.1 all          PlainBox based test runner (Python 3 library)
bladernr@xenial-dev:~$ canonical-certification-submit -h
/usr/bin/canonical-certification-submit: 21: exec: checkbox: not found

and now on the stable PPA:
bladernr@xenial-dev:~$ dpkg -l |grep checkbox
ii checkbox-ng 0.34.0~ppa~ubuntu16.04.1 all PlainBox based test runner
ii python3-checkbox-ng 0.34.0~ppa~ubuntu16.04.1 all PlainBox based test runner (Python 3 library)
bladernr@xenial-dev:~$ canonical-certification-submit -h
/usr/bin/canonical-certification-submit: 21: exec: checkbox: not found

This happened since the last update of checkbox, it was working fine before that.

Because of this, customers are unable to submit results from testing (and I am not able to do so for them either).

[Jeff Lane  (bladernr)]

Downrevved to the very old 0.23 and got this:

bladernr@xenial-dev:~$ dpkg -l |grep checkbox
ii checkbox-ng 0.23-2 all PlainBox based test runner
ii python3-checkbox-ng 0.23-2 all PlainBox based test runner (Python 3 library)
ii python3-checkbox-support 0.35.0~ppa~ubuntu16.04.1 all collection of Python modules used by PlainBox providers
bladernr@xenial-dev:~$ checkbox --help
Traceback (most recent call last):
  File "/usr/bin/checkbox", line 9, in <module>
    load_entry_point('checkbox-ng==0.23', 'console_scripts', 'checkbox')()
  File "/usr/lib/python3/dist-packages/checkbox_ng/main.py", line 40, in main
    raise SystemExit(CheckboxTool().main(argv))
  File "/usr/lib/python3/dist-packages/plainbox/impl/clitools.py", line 310, in main
    self._parser = self.construct_parser(early_ns)
  File "/usr/lib/python3/dist-packages/plainbox/impl/clitools.py", line 469, in construct_parser
    self.add_subcommands(subparsers, early_ns)
  File "/usr/lib/python3/dist-packages/checkbox_ng/tools.py", line 96, in add_subcommands
    from checkbox_ng.commands.launcher import LauncherCommand
  File "/usr/lib/python3/dist-packages/checkbox_ng/commands/launcher.py", line 32, in <module>
    from checkbox_ng.commands.newcli import CliInvocation2
  File "/usr/lib/python3/dist-packages/checkbox_ng/commands/newcli.py", line 52, in <module>
    from plainbox.vendor.textland import get_display
ImportError: No module named 'plainbox.vendor.textland'

so different error but seems to work?

then installed back to the latest stable and got this:

bladernr@xenial-dev:~$ dpkg -l |grep checkbox
ii checkbox-ng 0.34.0~ppa~ubuntu16.04.1 all PlainBox based test runner
ii python3-checkbox-ng 0.34.0~ppa~ubuntu16.04.1 all PlainBox based test runner (Python 3 library)
ii python3-checkbox-support 0.35.0~ppa~ubuntu16.04.1 all collection of Python modules used by PlainBox providers
bladernr@xenial-dev:~$ checkbox --help
The program 'checkbox' is currently not installed. You can install it by typing:
sudo apt install checkbox-ng
bladernr@xenial-dev:~$ sudo apt install checkbox-ng
Reading package lists... Done
Building dependency tree
Reading state information... Done
checkbox-ng is already the newest version (0.34.0~ppa~ubuntu16.04.1).
The following packages were automatically installed and are no longer required:
  gir1.2-gudev-1.0 libatasmart4 libgudev-1.0-0 libudisks2-0 libxslt1.1 python3-bs4 python3-checkbox-support python3-html5lib python3-lxml python3-pyparsing udisks2
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 172 not upgraded.

[Paul Larson (pwlars)]

I'm not too up on this change, but I do recall from the latest release announcement that checkbox-submit had been replaced by 'checkbox-cli submit'. Does this work for you?

[Jeff Lane  (bladernr)]

I am able to submit using checkbox-cli directly:
bladernr@xenial-dev:~$ checkbox-cli submit a00D000000M60SE ~/Downloads/submission_119832.xml
Successfully sent, submission status at https://certification.canonical.com/submissions/status/31990

however modifying the canonical-certification-submit launcher to use cli fails because --url is not supported:

bladernr@xenial-dev:~$ canonical-certification-submit a00D000000M60SE ~/Downloads/submission_119832.xml
usage: checkbox-cli [-h] [-v] [--debug]
                    {check-config,launcher,list,run,startprovider,submit,list-bootstrapped}
                    ...
checkbox-cli: error: unrecognized arguments: --url=https://certification.canonical.com/submissions/submit/

which means there's no method now to submit to a different endpoint

but modifying the launcher further to remove --url works:

bladernr@xenial-dev:~$ canonical-certification-submit -s a00D000000M60SE ~/Downloads/submission_119832.xml
Successfully sent, submission status at https://certification.staging.canonical.com/submissions/status/29691

bladernr@xenial-dev:~$ grep checkbox-cli /usr/bin/canonical-certification-submit -a1
# Canonical certifiation website.
exec checkbox-cli submit \
    "$@"

[Jeff Lane  (bladernr)]

Ok, so just to clear up confusion:

1: CURRENTLY (and historically) the client and server submit launchers (canonical-certification-submit) have called 'checkbox' like so:

exec checkbox submit --url=https://certification.canonical.com/submissions/submit/ "$@"

Note, trunk for BOTH providers use this for c-c-submit.

2: As of the most recent release, 'checkbox' no longer exists, so canonical-certification-submit no longer works.

3: All I'm asking is if the proper way NOW is to call checkbox-cli submit (which does not honor --url).

I already have an updated packaging branch for server to fix this, but I want to know if that's the correct way forward before I push a merge that has to be reverted down the road.

Sorry for the rambling commentary earlier as I was digging into the cause.

[Jeff Lane  (bladernr)]

Because i'm about to leave for 10 days on vacation, I went ahead and pushed a fix for this that works around the issue.

If this is the proper way going forward, then great, if it's not, it can be reverted later.

[Jeff Lane  (bladernr)]

And yes, after chatting with Paul, that seems to be the fix (the wrapper was just not updated to call the new checkbox-cli method).