Juniper Openstack

16.04.2 - SMLite openstack provision fails as neutron unable to load NeutronPluginContrailCoreV2 plugin (with ceilometer)

Bug #1695741 reported by Sudheendra Rao on 2017-06-04

This bug affects 3 people

	Status	Importance	Assigned to	Milestone
Juniper Openstack	Status tracked in Trunk
R4.0	Fix Committed	High	Dheeraj Gautam	Juniper Openstack r4.0.1.0 "r4.0.1.0"
Trunk	Fix Committed	High	Dheeraj Gautam	Juniper Openstack r4.1.0.0-fcs "r4.1"

Bug Description

16.04.2 newton openstack provision fails with SMLite as it fails to load neutron plugin NeutronPluginContrailCoreV2 on build19 and provision get stuck at openstack_started.

Used minimal kickstart file as per suggestion from Dheeraj while reimaging the target nodes.

Setup is in problem state.

Topology:
SMLite: nodei21
contrail-controller: nodei21, nodei22, nodei23
contrail-analytics: nodei21, nodei22, nodei23
contrail-analyticsdb: nodei21, nodei22, nodei23
contrail-lb: nodei24
openstack: nodei24
contrail-compute: nodei25, nodei26

neutron log:

2017-06-04 19:17:53.013 30924 INFO neutron.common.config [-] Logging enabled!
2017-06-04 19:17:53.013 30924 INFO neutron.common.config [-] /usr/bin/neutron-server version 9.2.0
2017-06-04 19:17:53.029 30924 INFO neutron.manager [-] Loading core plugin: neutron_plugin_contrail.plugins.opencontrail.contrail_plugin.NeutronPluginContrailCoreV2
2017-06-04 19:17:53.031 30924 WARNING stevedore.named [-] Could not load neutron_plugin_contrail.plugins.opencontrail.contrail_plugin.NeutronPluginContrailCoreV2
2017-06-04 19:17:53.031 30924 ERROR neutron.common.utils [-] Error loading class by alias
2017-06-04 19:17:53.031 30924 ERROR neutron.common.utils Traceback (most recent call last):
2017-06-04 19:17:53.031 30924 ERROR neutron.common.utils File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 400, in load_class_by_alias_or_classname
2017-06-04 19:17:53.031 30924 ERROR neutron.common.utils mgr = driver.DriverManager(namespace, name)
2017-06-04 19:17:53.031 30924 ERROR neutron.common.utils File "/usr/lib/python2.7/dist-packages/stevedore/driver.py", line 46, in __init__
2017-06-04 19:17:53.031 30924 ERROR neutron.common.utils verify_requirements=verify_requirements,
2017-06-04 19:17:53.031 30924 ERROR neutron.common.utils File "/usr/lib/python2.7/dist-packages/stevedore/named.py", line 77, in __init__
2017-06-04 19:17:53.031 30924 ERROR neutron.common.utils self._init_plugins(extensions)
2017-06-04 19:17:53.031 30924 ERROR neutron.common.utils File "/usr/lib/python2.7/dist-packages/stevedore/driver.py", line 98, in _init_plugins
2017-06-04 19:17:53.031 30924 ERROR neutron.common.utils (self.namespace, name))
2017-06-04 19:17:53.031 30924 ERROR neutron.common.utils NoMatches: No 'neutron.core_plugins' driver found, looking for 'neutron_plugin_contrail.plugins.opencontrail.contrail_plugin.NeutronPluginContrailCoreV2'
2017-06-04 19:17:53.031 30924 ERROR neutron.common.utils
2017-06-04 19:17:53.032 30924 ERROR neutron.common.utils [-] Error loading class by class name
2017-06-04 19:17:53.032 30924 ERROR neutron.common.utils Traceback (most recent call last):
2017-06-04 19:17:53.032 30924 ERROR neutron.common.utils File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 406, in load_class_by_alias_or_classname
2017-06-04 19:17:53.032 30924 ERROR neutron.common.utils class_to_load = importutils.import_class(name)
2017-06-04 19:17:53.032 30924 ERROR neutron.common.utils File "/usr/lib/python2.7/dist-packages/oslo_utils/importutils.py", line 30, in import_class
2017-06-04 19:17:53.032 30924 ERROR neutron.common.utils __import__(mod_str)
2017-06-04 19:17:53.032 30924 ERROR neutron.common.utils ImportError: No module named neutron_plugin_contrail.plugins.opencontrail.contrail_plugin
2017-06-04 19:17:53.032 30924 ERROR neutron.common.utils
2017-06-04 19:17:53.033 30924 ERROR neutron.service [-] Unrecoverable error: please check log for details.
2017-06-04 19:17:53.033 30924 ERROR neutron.service Traceback (most recent call last):
2017-06-04 19:17:53.033 30924 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/service.py", line 87, in serve_wsgi
2017-06-04 19:17:53.033 30924 ERROR neutron.service service.start()
2017-06-04 19:17:53.033 30924 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/service.py", line 63, in start
2017-06-04 19:17:53.033 30924 ERROR neutron.service self.wsgi_app = _run_wsgi(self.app_name)
2017-06-04 19:17:53.033 30924 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/service.py", line 289, in _run_wsgi
2017-06-04 19:17:53.033 30924 ERROR neutron.service app = config.load_paste_app(app_name)
2017-06-04 19:17:53.033 30924 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/common/config.py", line 125, in load_paste_app
2017-06-04 19:17:53.033 30924 ERROR neutron.service app = loader.load_app(app_name)
2017-06-04 19:17:53.033 30924 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/oslo_service/wsgi.py", line 353, in load_app
2017-06-04 19:17:53.033 30924 ERROR neutron.service return deploy.loadapp("config:%s" % self.config_path, name=name)
2017-06-04 19:17:53.033 30924 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2017-06-04 19:17:53.033 30924 ERROR neutron.service return loadobj(APP, uri, name=name, **kw)
2017-06-04 19:17:53.033 30924 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
2017-06-04 19:17:53.033 30924 ERROR neutron.service return context.create()
2017-06-04 19:17:53.033 30924 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2017-06-04 19:17:53.033 30924 ERROR neutron.service return self.object_type.invoke(self)
2017-06-04 19:17:53.033 30924 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2017-06-04 19:17:53.033 30924 ERROR neutron.service **context.local_conf)
2017-06-04 19:17:53.033 30924 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2017-06-04 19:17:53.033 30924 ERROR neutron.service val = callable(*args, **kw)
2017-06-04 19:17:53.033 30924 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/paste/urlmap.py", line 28, in urlmap_factory
2017-06-04 19:17:53.033 30924 ERROR neutron.service app = loader.get_app(app_name, global_conf=global_conf)
2017-06-04 19:17:53.033 30924 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2017-06-04 19:17:53.033 30924 ERROR neutron.service name=name, global_conf=global_conf).create()
2017-06-04 19:17:53.033 30924 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2017-06-04 19:17:53.033 30924 ERROR neutron.service return self.object_type.invoke(self)
2017-06-04 19:17:53.033 30924 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2017-06-04 19:17:53.033 30924 ERROR neutron.service **context.local_conf)
2017-06-04 19:17:53.033 30924 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2017-06-04 19:17:53.033 30924 ERROR neutron.service val = callable(*args, **kw)
2017-06-04 19:17:53.033 30924 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/auth.py", line 71, in pipeline_factory
2017-06-04 19:17:53.033 30924 ERROR neutron.service app = loader.get_app(pipeline[-1])
2017-06-04 19:17:53.033 30924 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2017-06-04 19:17:53.033 30924 ERROR neutron.service name=name, global_conf=global_conf).create()
2017-06-04 19:17:53.033 30924 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2017-06-04 19:17:53.033 30924 ERROR neutron.service return self.object_type.invoke(self)
2017-06-04 19:17:53.033 30924 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 146, in invoke
2017-06-04 19:17:53.033 30924 ERROR neutron.service return fix_call(context.object, context.global_conf, **context.local_conf)
2017-06-04 19:17:53.033 30924 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2017-06-04 19:17:53.033 30924 ERROR neutron.service val = callable(*args, **kw)
2017-06-04 19:17:53.033 30924 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/api/v2/router.py", line 72, in factory
2017-06-04 19:17:53.033 30924 ERROR neutron.service return cls(**local_config)
2017-06-04 19:17:53.033 30924 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/api/v2/router.py", line 76, in __init__
2017-06-04 19:17:53.033 30924 ERROR neutron.service plugin = manager.NeutronManager.get_plugin()
2017-06-04 19:17:53.033 30924 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/manager.py", line 244, in get_plugin
2017-06-04 19:17:53.033 30924 ERROR neutron.service return weakref.proxy(cls.get_instance().plugin)
2017-06-04 19:17:53.033 30924 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/manager.py", line 238, in get_instance
2017-06-04 19:17:53.033 30924 ERROR neutron.service cls._create_instance()
2017-06-04 19:17:53.033 30924 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 274, in inner
2017-06-04 19:17:53.033 30924 ERROR neutron.service return f(*args, **kwargs)
2017-06-04 19:17:53.033 30924 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/manager.py", line 224, in _create_instance
2017-06-04 19:17:53.033 30924 ERROR neutron.service cls._instance = cls()
2017-06-04 19:17:53.033 30924 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/manager.py", line 126, in __init__
2017-06-04 19:17:53.033 30924 ERROR neutron.service plugin_provider)
2017-06-04 19:17:53.033 30924 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/manager.py", line 159, in _get_plugin_instance
2017-06-04 19:17:53.033 30924 ERROR neutron.service plugin_class = self.load_class_for_provider(namespace, plugin_provider)
2017-06-04 19:17:53.033 30924 ERROR neutron.service File "/usr/lib/python2.7/dist-packages/neutron/manager.py", line 156, in load_class_for_provider
2017-06-04 19:17:53.033 30924 ERROR neutron.service raise ImportError(_("Plugin '%s' not found.") % plugin_provider)
2017-06-04 19:17:53.033 30924 ERROR neutron.service ImportError: Plugin 'neutron_plugin_contrail.plugins.opencontrail.contrail_plugin.NeutronPluginContrailCoreV2' not found.
2017-06-04 19:17:53.033 30924 ERROR neutron.service
2017-06-04 19:17:53.036 30924 CRITICAL neutron [-] ImportError: Plugin 'neutron_plugin_contrail.plugins.opencontrail.contrail_plugin.NeutronPluginContrailCoreV2' not found.

Tags:

Revision history for this message

Dheeraj Gautam (dgautam) wrote on 2017-06-04:

Looked at the setup. Ceilometer is enabled for this and its needed packages are not available,

root@nodei24:~# /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install python-pymongo
Reading package lists...
Building dependency tree...
Reading state information...
Package python-pymongo is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'python-pymongo' has no installation candidate
root@nodei24:~#

After disabling the ceilometer and re-provisioning, it went past by openstack provisioning

Revision history for this message

Sudheendra Rao (sudheendra-k) wrote on 2017-06-04:

provision completed after disabling ceilometer, hence removing the blocker.
Also this needs to be release noted, added releasenote tag.

tags:

added: releasenote
removed: blocker

Jeba Paulaiyan (jebap) on 2017-06-05

Changed in juniperopenstack:
importance:	Critical → High
milestone:	none → r4.1.0.0-fcs

Dheeraj Gautam (dgautam) on 2017-06-27

summary:

16.04.2 - SMLite openstack provision fails as neutron unable to load
- NeutronPluginContrailCoreV2 plugin
+ NeutronPluginContrailCoreV2 plugin (with ceilometer)

Revision history for this message

Dheeraj Gautam (dgautam) wrote on 2017-07-05:

Ceilometer requires mongodb and it used to come as part of contrail-database role previously. Now with newer roles and containers doesn't have mongodb installed, so provisioning fails for that. once mongodb installation is taken care of, we would revisit this bug.

Revision history for this message

Jeba Paulaiyan (jebap) wrote on 2017-07-17:

comment #3 does not justify why the bug should be moved to "Won't Fix" for R4.0

Revision history for this message

Dheeraj Gautam (dgautam) wrote on 2017-07-17:

Waiting for mongodb changes (needed for ceilometer provisioning), so marking it incomplete.

Revision history for this message

Megh Bhatt (meghb) wrote on 2017-07-25:

This seems to be ceilometer dependency and not mongodb dependency. So the mongodb changes will not get python-pymongo package. python-pymongo is mongodb python client that ceilometer package needs and hence needs to be added to the cache and package configs.

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2017-08-04: [Review update] R4.0

Review in progress for https://review.opencontrail.org/34326
Submitter: Dheeraj Gautam (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2017-08-07: [Review update] master

Review in progress for https://review.opencontrail.org/34360
Submitter: Dheeraj Gautam (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2017-08-08: A change has been merged

#10

Reviewed: https://review.opencontrail.org/34326
Committed: http://github.com/Juniper/contrail-puppet/commit/9a0aee0a046a133363b41e8365525afc1da9db71
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 9a0aee0a046a133363b41e8365525afc1da9db71
Author: Dheeraj Gautam <email address hidden>
Date: Fri Aug 4 15:28:05 2017 -0700

fix sm ceilometer provisioning

Partial-Bug: #1695741

Pending changes:
1. newton testing
2. compute provisioning for ceilometer

PATCH 2:
fix merge conflict

Conflicts:
contrail/environment/modules/contrail/manifests/profile/openstack_controller.pp

Change-Id: Ia6a0cb2958e5146ae1a43613b8e376e89091e46f

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2017-08-08:

#11

Reviewed: https://review.opencontrail.org/34360
Committed: http://github.com/Juniper/contrail-puppet/commit/9c6719e30539372244b3e766311555635b66f425
Submitter: Zuul (<email address hidden>)
Branch: master

commit 9c6719e30539372244b3e766311555635b66f425
Author: Dheeraj Gautam <email address hidden>
Date: Fri Aug 4 15:28:05 2017 -0700

fix sm ceilometer provisioning

Partial-Bug: #1695741

Pending changes:
1. newton testing
2. compute provisioning for ceilometer

PATCH 2:
fix merge conflict

Conflicts:
contrail/environment/modules/contrail/manifests/profile/openstack_controller.pp

Change-Id: Ia6a0cb2958e5146ae1a43613b8e376e89091e46f

Revision history for this message

Dheeraj Gautam (dgautam) wrote on 2017-08-09:

#12

Marking this as completed as provisioning completes after above changes

Report a bug

This report contains Public information

Everyone can see this information.

Duplicates of this bug

Bug #1708400

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.