SMR3.1build17 kilo:ceilometer processes are dead

Bug #1610506 reported by sundarkh
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.1
Fix Committed
High
Nitish Krishna Kaveri
Trunk
Fix Committed
High
Nitish Krishna Kaveri

Bug Description

SMR3.1build17:ceilometer processes are dead

== Ceilometer services ==
ceilometer-api: active
ceilometer-agent-central: dead
ceilometer-agent-compute: dead
ceilometer-collector: dead

related bug https://bugs.launchpad.net/juniperopenstack/+bug/1605827 SM:R3.1 : neutron-server and ceilometer -agent multiple respawns slow provision Edit

2016-08-05 21:53:19.080 7052 TRACE ceilometer File "/usr/bin/ceilometer-agent-central", line 10, in <module>
2016-08-05 21:53:19.080 7052 TRACE ceilometer sys.exit(main_central())
2016-08-05 21:53:19.080 7052 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/cmd/polling.py", line 89, in main_central
2016-08-05 21:53:19.080 7052 TRACE ceilometer os_service.launch(manager.AgentManager(['central'])).wait()
2016-08-05 21:53:19.080 7052 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/openstack/common/service.py", line 187, in wait
2016-08-05 21:53:19.080 7052 TRACE ceilometer status, signo = self._wait_for_exit_or_signal(ready_callback)
2016-08-05 21:53:19.080 7052 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/openstack/common/service.py", line 165, in _wait_for_exit_or_signal
2016-08-05 21:53:19.080 7052 TRACE ceilometer CONF.log_opt_values(LOG, logging.DEBUG)
2016-08-05 21:53:19.080 7052 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2195, in log_opt_values
2016-08-05 21:53:19.080 7052 TRACE ceilometer _sanitize(opt, getattr(group_attr, opt_name)))
2016-08-05 21:53:19.080 7052 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2516, in __getattr__
2016-08-05 21:53:19.080 7052 TRACE ceilometer return self._conf._get(name, self._group)
2016-08-05 21:53:19.080 7052 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2238, in _get
2016-08-05 21:53:19.080 7052 TRACE ceilometer value = self._do_get(name, group, namespace)
2016-08-05 21:53:19.080 7052 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2275, in _do_get
2016-08-05 21:53:19.080 7052 TRACE ceilometer return convert(opt._get_from_namespace(namespace, group_name))
2016-08-05 21:53:19.080 7052 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2270, in convert
2016-08-05 21:53:19.080 7052 TRACE ceilometer self._substitute(value, group, namespace), opt)
2016-08-05 21:53:19.080 7052 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2321, in _substitute
2016-08-05 21:53:19.080 7052 TRACE ceilometer self.StrSubWrapper(self, group=group, namespace=namespace))
2016-08-05 21:53:19.080 7052 TRACE ceilometer File "/usr/lib/python2.7/string.py", line 205, in safe_substitute
2016-08-05 21:53:19.080 7052 TRACE ceilometer return self.pattern.sub(convert, self.template)
2016-08-05 21:53:19.080 7052 TRACE ceilometer File "/usr/lib/python2.7/string.py", line 190, in convert
2016-08-05 21:53:19.080 7052 TRACE ceilometer return '%s' % (mapping[named],)
2016-08-05 21:53:19.080 7052 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2596, in __getitem__
2016-08-05 21:53:19.080 7052 TRACE ceilometer value = self.conf._get(key, namespace=self.namespace)
2016-08-05 21:53:19.080 7052 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2238, in _get
2016-08-05 21:53:19.080 7052 TRACE ceilometer value = self._do_get(name, group, namespace)
2016-08-05 21:53:19.080 7052 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2256, in _do_get
2016-08-05 21:53:19.080 7052 TRACE ceilometer info = self._get_opt_info(name, group)
2016-08-05 21:53:19.080 7052 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2384, in _get_opt_info
2016-08-05 21:53:19.080 7052 TRACE ceilometer raise NoSuchOptError(opt_name, group)
2016-08-05 21:53:19.080 7052 TRACE ceilometer NoSuchOptError: no such option: openstack
2016-08-05 21:53:19.080 7052 TRACE ceilometer

2016-08-05 21:53:33.658 9889 CRITICAL ceilometer [-] NoSuchOptError: no such option: openstack
2016-08-05 21:53:33.658 9889 TRACE ceilometer Traceback (most recent call last):
2016-08-05 21:53:33.658 9889 TRACE ceilometer File "/usr/bin/ceilometer-agent-compute", line 10, in <module>
2016-08-05 21:53:33.658 9889 TRACE ceilometer sys.exit(main_compute())
2016-08-05 21:53:33.658 9889 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/cmd/polling.py", line 83, in main_compute
2016-08-05 21:53:33.658 9889 TRACE ceilometer os_service.launch(manager.AgentManager(['compute'])).wait()
2016-08-05 21:53:33.658 9889 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/openstack/common/service.py", line 187, in wait
2016-08-05 21:53:33.658 9889 TRACE ceilometer status, signo = self._wait_for_exit_or_signal(ready_callback)
2016-08-05 21:53:33.658 9889 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/openstack/common/service.py", line 165, in _wait_for_exit_or_signal
2016-08-05 21:53:33.658 9889 TRACE ceilometer CONF.log_opt_values(LOG, logging.DEBUG)
2016-08-05 21:53:33.658 9889 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2195, in log_opt_values
2016-08-05 21:53:33.658 9889 TRACE ceilometer _sanitize(opt, getattr(group_attr, opt_name)))
2016-08-05 21:53:33.658 9889 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2516, in __getattr__
2016-08-05 21:53:33.658 9889 TRACE ceilometer return self._conf._get(name, self._group)
2016-08-05 21:53:33.658 9889 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2238, in _get
2016-08-05 21:53:33.658 9889 TRACE ceilometer value = self._do_get(name, group, namespace)
2016-08-05 21:53:33.658 9889 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2275, in _do_get
2016-08-05 21:53:33.658 9889 TRACE ceilometer return convert(opt._get_from_namespace(namespace, group_name))
2016-08-05 21:53:33.658 9889 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2270, in convert
2016-08-05 21:53:33.658 9889 TRACE ceilometer self._substitute(value, group, namespace), opt)
2016-08-05 21:53:33.658 9889 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2321, in _substitute
2016-08-05 21:53:33.658 9889 TRACE ceilometer self.StrSubWrapper(self, group=group, namespace=namespace))
2016-08-05 21:53:33.658 9889 TRACE ceilometer File "/usr/lib/python2.7/string.py", line 205, in safe_substitute
2016-08-05 21:53:33.658 9889 TRACE ceilometer return self.pattern.sub(convert, self.template)
2016-08-05 21:53:33.658 9889 TRACE ceilometer File "/usr/lib/python2.7/string.py", line 190, in convert
2016-08-05 21:53:33.658 9889 TRACE ceilometer return '%s' % (mapping[named],)
2016-08-05 21:53:33.658 9889 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2596, in __getitem__
2016-08-05 21:53:33.658 9889 TRACE ceilometer value = self.conf._get(key, namespace=self.namespace)
2016-08-05 21:53:33.658 9889 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2238, in _get
2016-08-05 21:53:33.658 9889 TRACE ceilometer value = self._do_get(name, group, namespace)
2016-08-05 21:53:33.658 9889 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2256, in _do_get
2016-08-05 21:53:33.658 9889 TRACE ceilometer info = self._get_opt_info(name, group)
2016-08-05 21:53:33.658 9889 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2384, in _get_opt_info
2016-08-05 21:53:33.658 9889 TRACE ceilometer raise NoSuchOptError(opt_name, group)
2016-08-05 21:53:33.658 9889 TRACE ceilometer NoSuchOptError: no such option: openstack
2016-08-05 21:53:33.658 9889 TRACE ceilometer
~

2016-08-05 21:53:14.530 6694 CRITICAL ceilometer [-] NoSuchOptError: no such option: openstack
2016-08-05 21:53:14.530 6694 TRACE ceilometer Traceback (most recent call last):
2016-08-05 21:53:14.530 6694 TRACE ceilometer File "/usr/bin/ceilometer-collector", line 10, in <module>
2016-08-05 21:53:14.530 6694 TRACE ceilometer sys.exit(main())
2016-08-05 21:53:14.530 6694 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/cmd/collector.py", line 25, in main
2016-08-05 21:53:14.530 6694 TRACE ceilometer workers=service.get_workers('collector')).wait()
2016-08-05 21:53:14.530 6694 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/openstack/common/service.py", line 187, in wait
2016-08-05 21:53:14.530 6694 TRACE ceilometer status, signo = self._wait_for_exit_or_signal(ready_callback)
2016-08-05 21:53:14.530 6694 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/openstack/common/service.py", line 165, in _wait_for_exit_or_signal
2016-08-05 21:53:14.530 6694 TRACE ceilometer CONF.log_opt_values(LOG, logging.DEBUG)
2016-08-05 21:53:14.530 6694 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2195, in log_opt_values
2016-08-05 21:53:14.530 6694 TRACE ceilometer _sanitize(opt, getattr(group_attr, opt_name)))
2016-08-05 21:53:14.530 6694 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2516, in __getattr__
2016-08-05 21:53:14.530 6694 TRACE ceilometer return self._conf._get(name, self._group)
2016-08-05 21:53:14.530 6694 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2238, in _get
2016-08-05 21:53:14.530 6694 TRACE ceilometer value = self._do_get(name, group, namespace)
2016-08-05 21:53:14.530 6694 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2275, in _do_get
2016-08-05 21:53:14.530 6694 TRACE ceilometer return convert(opt._get_from_namespace(namespace, group_name))
2016-08-05 21:53:14.530 6694 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2270, in convert
2016-08-05 21:53:14.530 6694 TRACE ceilometer self._substitute(value, group, namespace), opt)
2016-08-05 21:53:14.530 6694 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2321, in _substitute
2016-08-05 21:53:14.530 6694 TRACE ceilometer self.StrSubWrapper(self, group=group, namespace=namespace))
2016-08-05 21:53:14.530 6694 TRACE ceilometer File "/usr/lib/python2.7/string.py", line 205, in safe_substitute
2016-08-05 21:53:14.530 6694 TRACE ceilometer return self.pattern.sub(convert, self.template)
2016-08-05 21:53:14.530 6694 TRACE ceilometer File "/usr/lib/python2.7/string.py", line 190, in convert
2016-08-05 21:53:14.530 6694 TRACE ceilometer return '%s' % (mapping[named],)
2016-08-05 21:53:14.530 6694 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2596, in __getitem__
2016-08-05 21:53:14.530 6694 TRACE ceilometer value = self.conf._get(key, namespace=self.namespace)
2016-08-05 21:53:14.530 6694 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2238, in _get
2016-08-05 21:53:14.530 6694 TRACE ceilometer value = self._do_get(name, group, namespace)
2016-08-05 21:53:14.530 6694 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2256, in _do_get
2016-08-05 21:53:14.530 6694 TRACE ceilometer info = self._get_opt_info(name, group)
2016-08-05 21:53:14.530 6694 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2384, in _get_opt_info
2016-08-05 21:53:14.530 6694 TRACE ceilometer raise NoSuchOptError(opt_name, group)
2016-08-05 21:53:14.530 6694 TRACE ceilometer NoSuchOptError: no such option: openstack
2016-08-05 21:53:14.530 6694 TRACE ceilometer
~

tags: added: blocker
Abhay Joshi (abhayj)
Changed in juniperopenstack:
assignee: Abhay Joshi (abhayj) → Nitish Krishna Kaveri (nitishk)
Revision history for this message
Nitish Krishna Kaveri (nitishk) wrote :

Connection is set wrongly in ceilometer.conf:

connection = mongodb://ceilometer:$openstack::keystone::admin_password@10.204.217.61:27017,10.204.217.98:27017,10.204.217.73:27017/ceilometer?replicaSet=rs-ceilometer

This should be set to correct keystone admin password from hieradata

Revision history for this message
Nitish Krishna Kaveri (nitishk) wrote :

Another error seen:

DriverLoadFailure: Failed to load transport driver "ceilometer.openstack.common.rpc.impl_kombu": No 'oslo.messaging.drivers' driver found, looking for 'ceilometer.openstack.common.rpc.impl_kombu'
2016-08-08 10:23:25.447 27900 ERROR ceilometer

Fab doesn't set rpc_backend, we are setting to ceilometer.openstack.common.rpc.impl_kombu

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.1

Review in progress for https://review.opencontrail.org/23065
Submitter: Nitish Krishna Kaveri (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/23067
Submitter: Nitish Krishna Kaveri (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/23067
Committed: http://github.org/Juniper/contrail-puppet/commit/7412141975e92a273eaa9cf8cc92cfec8f57b6ec
Submitter: Zuul
Branch: R3.1

commit 7412141975e92a273eaa9cf8cc92cfec8f57b6ec
Author: nitishkrishna <email address hidden>
Date: Mon Aug 8 16:19:45 2016 -0700

Closes-Bug: #1610506 - Ceilometer misconfigured due to param name mismatch

The param name used is ceilometer::mongo::password but should be ceilometer::mongo
One parameter for primary db had typo
Also rpc_backend needs to be removed from ceilometer.conf

Change-Id: I477537412ff248e7abd020640c7559ef3478c5cd

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/23065
Committed: http://github.org/Juniper/contrail-server-manager/commit/1dd85944ac59b059ba0fda4703ca15983f773465
Submitter: Zuul
Branch: R3.1

commit 1dd85944ac59b059ba0fda4703ca15983f773465
Author: nitishkrishna <email address hidden>
Date: Mon Aug 8 15:53:27 2016 -0700

Related-Bug: #1610506 - Wrong ceilometer mongo password passed from hiera

Name mismatch in hiera and puppet modules

Change-Id: I82dce47576e406cecd354f9405ccab9e465ac01c

Revision history for this message
sundarkh (sundar-kh) wrote :

verified after the patch, that ceilometer processes are active

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/23177
Submitter: Nitish Krishna Kaveri (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/23179
Submitter: Nitish Krishna Kaveri (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/23179
Committed: http://github.org/Juniper/contrail-puppet/commit/3a36214635d0f80abf5c12668cc6e64c6eb28ac0
Submitter: Zuul
Branch: master

commit 3a36214635d0f80abf5c12668cc6e64c6eb28ac0
Author: nitishkrishna <email address hidden>
Date: Mon Aug 8 16:19:45 2016 -0700

Closes-Bug: #1610506 - Ceilometer misconfigured due to param name mismatch

The param name used is ceilometer::mongo::password but should be ceilometer::mongo
One parameter for primary db had typo
Also rpc_backend needs to be removed from ceilometer.conf

Change-Id: I477537412ff248e7abd020640c7559ef3478c5cd

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/23177
Committed: http://github.org/Juniper/contrail-server-manager/commit/12ef59c0ad7b015a027932b8a22e0df0e13bfa3d
Submitter: Zuul
Branch: master

commit 12ef59c0ad7b015a027932b8a22e0df0e13bfa3d
Author: nitishkrishna <email address hidden>
Date: Mon Aug 8 15:53:27 2016 -0700

Related-Bug: #1610506 - Wrong ceilometer mongo password passed from hiera

Name mismatch in hiera and puppet modules

Change-Id: I82dce47576e406cecd354f9405ccab9e465ac01c

Revision history for this message
sundarkh (sundar-kh) wrote :
Download full text (6.6 KiB)

Processs are active, but the ceilometer log complains of driver failure build 19

2016-08-11 11:03:42.287 349 DEBUG ceilometer.hardware.pollsters.generic [-] Hardware snmp meter definition file: /usr/lib/python2.7/dist-packages/ceilometer/hardware/pollsters/data/snmp.yaml setup_meters_config /usr/lib/python2.7/dist-packages/ceilometer/hardware/pollsters/generic.py:232
2016-08-11 11:03:42.309 349 INFO ceilometer.hardware.pollsters.generic [-] Meter Definitions: {'metric': [{'snmp_inspector': {'oid': '1.3.6.1.4.1.2021.10.1.3.1', 'type': 'lambda x: float(str(x))', 'matching_type': 'type_exact'}, 'type': 'gauge', 'name': 'hardware.cpu.load.1min', 'unit': 'process'}, {'snmp_inspector': {'oid': '1.3.6.1.4.1.2021.10.1.3.2', 'type': 'lambda x: float(str(x))', 'matching_type': 'type_exact'}, 'type': 'gauge', 'name': 'hardware.cpu.load.5min', 'unit': 'process'}, {'snmp_inspector': {'oid': '1.3.6.1.4.1.2021.10.1.3.3', 'type': 'lambda x: float(str(x))', 'matching_type': 'type_exact'}, 'type': 'gauge', 'name': 'hardware.cpu.load.15min', 'unit': 'process'}, {'snmp_inspector': {'post_op': '_post_op_disk', 'oid': '1.3.6.1.4.1.2021.9.1.6', 'type': 'int', 'matching_type': 'type_prefix', 'metadata': {'device': {'oid': '1.3.6.1.4.1.2021.9.1.3', 'type': 'str'}, 'path': {'oid': '1.3.6.1.4.1.2021.9.1.2', 'type': 'str'}}}, 'type': 'gauge', 'name': 'hardware.disk.size.total', 'unit': 'KB'}, {'snmp_inspector': {'post_op': '_post_op_disk', 'oid': '1.3.6.1.4.1.2021.9.1.8', 'type': 'int', 'matching_type': 'type_prefix', 'metadata': {'device': {'oid': '1.3.6.1.4.1.2021.9.1.3', 'type': 'str'}, 'path': {'oid': '1.3.6.1.4.1.2021.9.1.2', 'type': 'str'}}}, 'type': 'gauge', 'name': 'hardware.disk.size.used', 'unit': 'KB'}, {'snmp_inspector': {'oid': '1.3.6.1.4.1.2021.4.5.0', 'type': 'int', 'matching_type': 'type_exact'}, 'type': 'gauge', 'name': 'hardware.memory.total', 'unit': 'KB'}, {'snmp_inspector': {'oid': '1.3.6.1.4.1.2021.4.6.0', 'type': 'int', 'matching_type': 'type_exact', 'post_op': '_post_op_memory_avail_to_used'}, 'type': 'gauge', 'name': 'hardware.memory.used', 'unit': 'KB'}, {'snmp_inspector': {'oid': '1.3.6.1.4.1.2021.4.3.0', 'type': 'int', 'matching_type': 'type_exact'}, 'type': 'gauge', 'name': 'hardware.memory.swap.total', 'unit': 'KB'}, {'snmp_inspector': {'oid': '1.3.6.1.4.1.2021.4.4.0', 'type': 'int', 'matching_type': 'type_exact'}, 'type': 'gauge', 'name': 'hardware.memory.swap.avail', 'unit': 'KB'}, {'snmp_inspector': {'oid': '1.3.6.1.4.1.2021.4.14.0', 'type': 'int', 'matching_type': 'type_exact'}, 'type': 'gauge', 'name': 'hardware.memory.buffer', 'unit': 'KB'}, {'snmp_inspector': {'oid': '1.3.6.1.4.1.2021.4.15.0', 'type': 'int', 'matching_type': 'type_exact'}, 'type': 'gauge', 'name': 'hardware.memory.cached', 'unit': 'KB'}, {'snmp_inspector': {'post_op': '_post_op_net', 'oid': '1.3.6.1.2.1.2.2.1.10', 'type': 'int', 'matching_type': 'type_prefix', 'metadata': {'mac': {'oid': '1.3.6.1.2.1.2.2.1.6', 'type': "lambda x: x.prettyPrint().replace('0x', '')"}, 'speed': {'oid': '1.3.6.1.2.1.2.2.1.5', 'type': 'lambda x: int(x) / 8'}, 'name': {'oid': '1.3.6.1.2.1.2.2.1.2', 'type': 'str'}}}, 'type': 'cumulative', 'name': 'hardware.network.incoming.bytes', 'unit': '...

Read more...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.