Climate Manager crashes if a plugin crash during initialization

[Pablo Andres Fuente]

If any of the plugins loaded by Climate Manager crashes, the whole manager crashes. It would be better if a warning is shown telling which plugin fail and why, and load the rest of plugins that actually works.

This is an example of the actual output when a plugin crash during initialization:

2014-03-18 11:56:54.604 15875 ERROR stevedore.extension [-] Could not load 'physical.host.plugin': no such option in group physical:host: climate_username
2014-03-18 11:56:54.604 15875 ERROR stevedore.extension [-] no such option in group physical:host: climate_username
2014-03-18 11:56:54.604 15875 TRACE stevedore.extension Traceback (most recent call last):
2014-03-18 11:56:54.604 15875 TRACE stevedore.extension File "/opt/stack/stevedore/stevedore/extension.py", line 162, in _load_plugins
2014-03-18 11:56:54.604 15875 TRACE stevedore.extension verify_requirements,
2014-03-18 11:56:54.604 15875 TRACE stevedore.extension File "/opt/stack/stevedore/stevedore/enabled.py", line 66, in _load_one_plugin
2014-03-18 11:56:54.604 15875 TRACE stevedore.extension verify_requirements,
2014-03-18 11:56:54.604 15875 TRACE stevedore.extension File "/opt/stack/stevedore/stevedore/extension.py", line 179, in _load_one_plugin
2014-03-18 11:56:54.604 15875 TRACE stevedore.extension obj = plugin(*invoke_args, **invoke_kwds)
2014-03-18 11:56:54.604 15875 TRACE stevedore.extension File "/opt/stack/climate/climate/plugins/oshosts/host_plugin.py", line 64, in __init__
2014-03-18 11:56:54.604 15875 TRACE stevedore.extension self.username = config.climate_username
2014-03-18 11:56:54.604 15875 TRACE stevedore.extension File "/opt/stack/oslo.config/oslo/config/cfg.py", line 2226, in __getattr__
2014-03-18 11:56:54.604 15875 TRACE stevedore.extension return self._conf._get(name, self._group)
2014-03-18 11:56:54.604 15875 TRACE stevedore.extension File "/opt/stack/oslo.config/oslo/config/cfg.py", line 1964, in _get
2014-03-18 11:56:54.604 15875 TRACE stevedore.extension value = self._do_get(name, group, namespace)
2014-03-18 11:56:54.604 15875 TRACE stevedore.extension File "/opt/stack/oslo.config/oslo/config/cfg.py", line 1982, in _do_get
2014-03-18 11:56:54.604 15875 TRACE stevedore.extension info = self._get_opt_info(name, group)
2014-03-18 11:56:54.604 15875 TRACE stevedore.extension File "/opt/stack/oslo.config/oslo/config/cfg.py", line 2106, in _get_opt_info
2014-03-18 11:56:54.604 15875 TRACE stevedore.extension raise NoSuchOptError(opt_name, group)
2014-03-18 11:56:54.604 15875 TRACE stevedore.extension NoSuchOptError: no such option in group physical:host: climate_username
2014-03-18 11:56:54.604 15875 TRACE stevedore.extension
2014-03-18 11:56:54.606 15875 CRITICAL climate [-] PluginConfigurationError: Plugin Configuration error : Not all requested plugins are loaded.
2014-03-18 11:56:54.606 15875 TRACE climate Traceback (most recent call last):
2014-03-18 11:56:54.606 15875 TRACE climate File "/usr/local/bin/climate-manager", line 10, in <module>
2014-03-18 11:56:54.606 15875 TRACE climate sys.exit(main())
2014-03-18 11:56:54.606 15875 TRACE climate File "/opt/stack/climate/climate/cmd/manager.py", line 36, in main
2014-03-18 11:56:54.606 15875 TRACE climate manager_service.ManagerService()
2014-03-18 11:56:54.606 15875 TRACE climate File "/opt/stack/climate/climate/manager/service.py", line 58, in __init__
2014-03-18 11:56:54.606 15875 TRACE climate self.plugins = self._get_plugins()
2014-03-18 11:56:54.606 15875 TRACE climate File "/opt/stack/climate/climate/manager/service.py", line 87, in _get_plugins
2014-03-18 11:56:54.606 15875 TRACE climate raise exceptions.PluginConfigurationError(error=msg)
2014-03-18 11:56:54.606 15875 TRACE climate PluginConfigurationError: Plugin Configuration error : Not all requested plugins are loaded.
2014-03-18 11:56:54.606 15875 TRACE climate

[Dina Belova (dbelova)]

Won't you mind to work on this bug?

Also there should be some kind of mechanism to reject requests coming to manager, if they refer to failed plugin.

[Pablo Andres Fuente (pablo-a-fuente)]

Regarding to "Also there should be some kind of mechanism to reject requests coming to manager, if they refer to failed plugin." At this moment in the code, if no plugin is registered for a particular resource type, some exceptions are raised: https://github.com/stackforge/climate/blob/master/climate/manager/service.py#L362
Are those exceptions enough or you prefer a new exception that tells explicitly that the resource type is rejected because the plugin fail during the initialization?

[Dina Belova (dbelova)]

Well, currently if there are any problems exception is raised and no manager started at all. We need simply warning + running manager + some exception when there ir request to the not loaded plugins.

Exceptions you refer to are some kind of trick to load extra methods from physical plugin. So that's not really that stuff you need here.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/82181

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/82181
Committed: https://git.openstack.org/cgit/stackforge/climate/commit/?id=ba4c18921c2799a68be127ed50388406e129a32e
Submitter: Jenkins
Branch: master

commit ba4c18921c2799a68be127ed50388406e129a32e
Author: Pablo Andres Fuente <email address hidden>
Date: Wed Mar 19 15:12:26 2014 -0300

    Climate Manager now starts even if a plugin fails

    The initialization of plugins loaded by Climate Manager was deferred in
    order to catch any exception that the plugins could raise during it
    initialization. Doing this, Climate Manager starts correctly loading
    only the plugins that could be correctly initialized. A warning is
    logged when this situation happens.
    When a lease is about to be created, the resource_type of any
    reservation that could contain, is checked against all the registered
    plugins. If none of them handles the resource_type, an
    UnsupportedResourceType exception is raised.

    Change-Id: I356a39d432dc2cdaf349d817d8b341c531ece5b0
    Closes-Bug: #1294175