manifest provider is not loaded by checkbox-ng

[Po-Hsu Lin]

When testing the Thunderbolt test cases with this branch:
https://code.launchpad.net/~cypressyew/checkbox/thunderbolt-testing

checkbox-cli would crash with after it's about to enter the job selection section

Test steps (tested on the TB system from our lab):
1. branch https://code.launchpad.net/~cypressyew/checkbox/thunderbolt-testing
2. run checkbox-cli in venv
3. select client-cert whitelist to start

It will fail with the following error message:

========================[ Selecting Jobs For Execution ]========================
CRITICAL plainbox.crashes: Executable 'checkbox-launcher' invoked with Namespace(color=None, command=<checkbox_ng.commands.launcher.LauncherCommand object at 0x7f9fb4db9dd8>, debug_console=False, debug_interrupt=False, dont_suppress_output=False, dry_run=False, exclude_pattern_list=[], include_pattern_list=[], launcher='/tmp/working/bin/checkbox-cli', log_level=None, non_interactive=False, pdb=False, secure_id=None, test_plan=None, trace=[], url=None, whitelist=[]) has crashed
Traceback (most recent call last):
  File "/home/u/thunderbolt-testing/plainbox/plainbox/impl/depmgr.py", line 275, in _visit
    next_job = self._job_map[job_id]
KeyError: '2013.com.canonical.plainbox::manifest'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/u/thunderbolt-testing/plainbox/plainbox/impl/session/state.py", line 850, in update_desired_job_list
    self._job_list, self._desired_job_list)
  File "/home/u/thunderbolt-testing/plainbox/plainbox/impl/depmgr.py", line 214, in resolve_dependencies
    return cls(job_list)._solve(visit_list)
  File "/home/u/thunderbolt-testing/plainbox/plainbox/impl/depmgr.py", line 247, in _solve
    self._visit(job)
  File "/home/u/thunderbolt-testing/plainbox/plainbox/impl/depmgr.py", line 284, in _visit
    self._visit(next_job, trail)
  File "/home/u/thunderbolt-testing/plainbox/plainbox/impl/depmgr.py", line 277, in _visit
    raise DependencyMissingError(job, job_id, dep_type)
plainbox.impl.depmgr.DependencyMissingError: missing dependency: '2013.com.canonical.plainbox::manifest' (resource)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/u/thunderbolt-testing/plainbox/plainbox/impl/clitools.py", line 560, in dispatch_and_catch_exceptions
    return self.dispatch_command(ns)
  File "/home/u/thunderbolt-testing/plainbox/plainbox/impl/clitools.py", line 556, in dispatch_command
    return ns.command.invoked(ns)
  File "/home/u/thunderbolt-testing/checkbox-ng/checkbox_ng/commands/launcher.py", line 90, in invoked
    self.provider_loader, lambda: self.config, ns, launcher
  File "/home/u/thunderbolt-testing/checkbox-ng/checkbox_ng/commands/newcli.py", line 150, in run
    return self.do_normal_sequence()
  File "/home/u/thunderbolt-testing/checkbox-ng/checkbox_ng/commands/newcli.py", line 198, in do_normal_sequence
    self.interactively_pick_jobs_to_run()
  File "/home/u/thunderbolt-testing/checkbox-ng/checkbox_ng/commands/newcli.py", line 316, in interactively_pick_jobs_to_run
    self.manager.state.job_list, self._qualifier_list))
  File "/home/u/thunderbolt-testing/plainbox/plainbox/impl/commands/inv_run.py", line 908, in _update_desired_job_list
    problem_list = self.state.update_desired_job_list(desired_job_list)
  File "/home/u/thunderbolt-testing/plainbox/plainbox/impl/session/state.py", line 854, in update_desired_job_list
    self._desired_job_list.remove(exc.affected_job)
ValueError: list.remove(x): x not in list
Traceback (most recent call last):
  File "/home/u/thunderbolt-testing/plainbox/plainbox/impl/depmgr.py", line 275, in _visit
    next_job = self._job_map[job_id]
KeyError: '2013.com.canonical.plainbox::manifest'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/u/thunderbolt-testing/plainbox/plainbox/impl/session/state.py", line 850, in update_desired_job_list
    self._job_list, self._desired_job_list)
  File "/home/u/thunderbolt-testing/plainbox/plainbox/impl/depmgr.py", line 214, in resolve_dependencies
    return cls(job_list)._solve(visit_list)
  File "/home/u/thunderbolt-testing/plainbox/plainbox/impl/depmgr.py", line 247, in _solve
    self._visit(job)
  File "/home/u/thunderbolt-testing/plainbox/plainbox/impl/depmgr.py", line 284, in _visit
    self._visit(next_job, trail)
  File "/home/u/thunderbolt-testing/plainbox/plainbox/impl/depmgr.py", line 277, in _visit
    raise DependencyMissingError(job, job_id, dep_type)
plainbox.impl.depmgr.DependencyMissingError: missing dependency: '2013.com.canonical.plainbox::manifest' (resource)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/working/bin/checkbox-launcher", line 9, in <module>
    load_entry_point('checkbox-ng==0.19.dev0', 'console_scripts', 'checkbox-launcher')()
  File "/home/u/thunderbolt-testing/checkbox-ng/checkbox_ng/main.py", line 62, in launcher
    raise SystemExit(CheckboxLauncherTool().main(argv))
  File "/home/u/thunderbolt-testing/plainbox/plainbox/impl/clitools.py", line 324, in main
    return self.dispatch_and_catch_exceptions(ns)
  File "/home/u/thunderbolt-testing/plainbox/plainbox/impl/clitools.py", line 560, in dispatch_and_catch_exceptions
    return self.dispatch_command(ns)
  File "/home/u/thunderbolt-testing/plainbox/plainbox/impl/clitools.py", line 556, in dispatch_command
    return ns.command.invoked(ns)
  File "/home/u/thunderbolt-testing/checkbox-ng/checkbox_ng/commands/launcher.py", line 90, in invoked
    self.provider_loader, lambda: self.config, ns, launcher
  File "/home/u/thunderbolt-testing/checkbox-ng/checkbox_ng/commands/newcli.py", line 150, in run
    return self.do_normal_sequence()
  File "/home/u/thunderbolt-testing/checkbox-ng/checkbox_ng/commands/newcli.py", line 198, in do_normal_sequence
    self.interactively_pick_jobs_to_run()
  File "/home/u/thunderbolt-testing/checkbox-ng/checkbox_ng/commands/newcli.py", line 316, in interactively_pick_jobs_to_run
    self.manager.state.job_list, self._qualifier_list))
  File "/home/u/thunderbolt-testing/plainbox/plainbox/impl/commands/inv_run.py", line 908, in _update_desired_job_list
    problem_list = self.state.update_desired_job_list(desired_job_list)
  File "/home/u/thunderbolt-testing/plainbox/plainbox/impl/session/state.py", line 854, in update_desired_job_list
    self._desired_job_list.remove(exc.affected_job)
ValueError: list.remove(x): x not in list

[Po-Hsu Lin (cypressyew)]

This is the ~/.cache/plainbox directory, error log could be found in it.

[Po-Hsu Lin (cypressyew)]

the script "movie" for this, please cat it.

[Po-Hsu Lin (cypressyew)]

It seems that we will need to add
collect-manifest
and
has_thunderbolt
into the whitelist
or we will need to run the collect-manifest job manually first

[Zygmunt Krynicki (zyga)]

Adding a debugger stop like this:

(thunderbolt-testing)zyga@fx:~/checkbox.bzr/thunderbolt-testing$ bzr diff
=== modified file 'checkbox-ng/checkbox_ng/commands/newcli.py'
--- checkbox-ng/checkbox_ng/commands/newcli.py 2015-03-31 07:26:46 +0000
+++ checkbox-ng/checkbox_ng/commands/newcli.py 2015-04-14 14:08:19 +0000
@@ -312,6 +312,7 @@

     def interactively_pick_jobs_to_run(self):
         print(self.C.header(_("Selecting Jobs For Execution")))
+ import pdb; pdb.set_trace()
         self._update_desired_job_list(select_jobs(
             self.manager.state.job_list, self._qualifier_list))
         if self.launcher.skip_test_selection or not self.is_interactive:

Yields this information:

(Pdb) self.manager.default_device_context.provider_list
[<Provider1 name:'2014.com.ubuntu:ubuntu-touch'>, <Provider1 name:'2013.com.canonical.certification:certification-client'>, <Provider1 name:'2013.com.canonical.certification:certification-server'>, <Provider1 name:'2015.com.canonical.certification:qml-tests'>, <Provider1 name:'2013.com.canonical.certification:checkbox'>, <Provider1 name:'2013.com.canonical.certification:plainbox-resources'>, <Provider1 name:'2013.com.canonical.certification:piglit'>]

It seems that the manifest provider is not there for some reason.

[Zygmunt Krynicki (zyga)]

Adding another breakpoint I see this:

=== modified file 'checkbox-ng/checkbox_ng/commands/newcli.py'
--- checkbox-ng/checkbox_ng/commands/newcli.py 2015-03-31 07:26:46 +0000
+++ checkbox-ng/checkbox_ng/commands/newcli.py 2015-04-14 14:14:02 +0000
@@ -78,6 +78,7 @@
     def __init__(self, provider_loader, config_loader, ns, launcher,
                  display=None):
         super().__init__(provider_loader, config_loader, ns, ns.color)
+ import pdb; pdb.set_trace()
         if display is None:
             display = get_display()
         self._launcher = launcher

> /home/zyga/checkbox.bzr/thunderbolt-testing/checkbox-ng/checkbox_ng/commands/newcli.py(82)__init__()
-> if display is None:
(Pdb) provider_loader
<bound method CheckboxLauncherTool._load_providers of <checkbox_ng.tools.CheckboxLauncherTool object at 0x7fa9bfc1aba8>>

Which leads me to tools.py which simply loads all providers by calling _load_providers().

So there are two bugs here:
 - depmgr should not crash in this case
 - plainbox should have better API than all_providers() (which loads all except for categories and manifest provider)

I'll patch each bug separately. I'll report them and link them here.