Juniper Openstack

SM:R3.0.3 build69 : ceilometer-api process is dead -could not connect to 10.204.217.77:27017: [Errno 111] ECONNREFUSED

Bug #1628416 reported by sundarkh
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.0
Fix Committed
High
Nitish Krishna Kaveri
Juniper Openstack r3.0.4.0
R3.0.2.x
Won't Fix
High
Nitish Krishna Kaveri
Juniper Openstack r3.0.2.1
R3.0.3.x
Fix Committed
High
Nitish Krishna Kaveri
Juniper Openstack r3.0.3.1
R3.1
Fix Committed
High
Nitish Krishna Kaveri
Juniper Openstack r3.1.1.0
Trunk
Fix Committed
High
Nitish Krishna Kaveri
Juniper Openstack r3.2.0.0-fcs "r3.2.0.0"

Bug Description

SM:R3.0.3 build69 : ceilometer-api process is dead

SM Provisioned target node (Single node setup)

== Ceilometer services ==
ceilometer-api: dead
ceilometer-agent-central: active
ceilometer-agent-compute: active
ceilometer-collector: active

Ceilometer Api Log
---------------------

2016-09-27 21:19:30.117 8249 CRITICAL ceilometer [-] ConnectionFailure: could not connect to 10.204.217.77:27017: [Errno 111] ECONNREFUSED
2016-09-27 21:19:30.117 8249 TRACE ceilometer Traceback (most recent call last):
2016-09-27 21:19:30.117 8249 TRACE ceilometer File "/usr/bin/ceilometer-api", line 10, in <module>
2016-09-27 21:19:30.117 8249 TRACE ceilometer sys.exit(main())
2016-09-27 21:19:30.117 8249 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/cmd/api.py", line 23, in main
2016-09-27 21:19:30.117 8249 TRACE ceilometer srv = app.build_server()
2016-09-27 21:19:30.117 8249 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/api/app.py", line 157, in build_server
2016-09-27 21:19:30.117 8249 TRACE ceilometer app = load_app()
2016-09-27 21:19:30.117 8249 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/api/app.py", line 153, in load_app
2016-09-27 21:19:30.117 8249 TRACE ceilometer return deploy.loadapp("config:" + cfg_file)
2016-09-27 21:19:30.117 8249 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2016-09-27 21:19:30.117 8249 TRACE ceilometer return loadobj(APP, uri, name=name, **kw)
2016-09-27 21:19:30.117 8249 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
2016-09-27 21:19:30.117 8249 TRACE ceilometer return context.create()
2016-09-27 21:19:30.117 8249 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-09-27 21:19:30.117 8249 TRACE ceilometer return self.object_type.invoke(self)
2016-09-27 21:19:30.117 8249 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 203, in invoke
2016-09-27 21:19:30.117 8249 TRACE ceilometer app = context.app_context.create()
2016-09-27 21:19:30.117 8249 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-09-27 21:19:30.117 8249 TRACE ceilometer return self.object_type.invoke(self)
2016-09-27 21:19:30.117 8249 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 146, in invoke
2016-09-27 21:19:30.117 8249 TRACE ceilometer return fix_call(context.object, context.global_conf, **context.local_conf)
2016-09-27 21:19:30.117 8249 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2016-09-27 21:19:30.117 8249 TRACE ceilometer val = callable(*args, **kw)
2016-09-27 21:19:30.117 8249 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/api/app.py", line 181, in app_factory
2016-09-27 21:19:30.117 8249 TRACE ceilometer return VersionSelectorApplication()
2016-09-27 21:19:30.117 8249 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/api/app.py", line 104, in __init__
2016-09-27 21:19:30.117 8249 TRACE ceilometer self.v2 = setup_app(pecan_config=pc)
2016-09-27 21:19:30.117 8249 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/api/app.py", line 68, in setup_app
2016-09-27 21:19:30.117 8249 TRACE ceilometer storage.get_connection_from_config(cfg.CONF, 'metering'),
2016-09-27 21:19:30.117 8249 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/storage/__init__.py", line 86, in get_connection_from_config
2016-09-27 21:19:30.117 8249 TRACE ceilometer return get_connection(url, namespace)
2016-09-27 21:19:30.117 8249 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/storage/__init__.py", line 98, in get_connection
2016-09-27 21:19:30.117 8249 TRACE ceilometer return mgr.driver(url)
2016-09-27 21:19:30.117 8249 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/storage/impl_mongodb.py", line 395, in __init__
2016-09-27 21:19:30.117 8249 TRACE ceilometer self.conn = self.CONNECTION_POOL.connect(url)
2016-09-27 21:19:30.117 8249 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/storage/mongo/utils.py", line 176, in connect
2016-09-27 21:19:30.117 8249 TRACE ceilometer client = self._mongo_connect(url)
2016-09-27 21:19:30.117 8249 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/storage/mongo/utils.py", line 190, in _mongo_connect
2016-09-27 21:19:30.117 8249 TRACE ceilometer pymongo.MongoClient(url, safe=True))
2016-09-27 21:19:30.117 8249 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 352, in __init__
2016-09-27 21:19:30.117 8249 TRACE ceilometer raise ConnectionFailure(str(e))
2016-09-27 21:19:30.117 8249 TRACE ceilometer ConnectionFailure: could not connect to 10.204.217.77:27017: [Errno 111] ECONNREFUSED
2016-09-27 21:19:30.117 8249 TRACE ceilometer
~

root@nodeg37:~# netstat -nap | grep mongodb
unix 2 [ ACC ] STREAM LISTENING 4326296 1628/mongod /tmp/mongodb-27017.sock
root@nodeg37:~#

Sanity Test case test_sample_floating_ip_transmit_packets
 fails with Traceback (most recent call last):
  File "tcutils/wrappers.py", line 73, in wrapper
    result = function(self, *args, **kwargs)
  File "scripts/ceilometer_tests/test_ceilometer.py", line 104, in test_sample_floating_ip_transmit_packets
    result = ceilometer_client.sample_list(self.cclient,m)
  File "/contrail-test/fixtures/ceilometer/ceilometer_client.py", line 207, in sample_list
    q=query, limit=limit)
  File "/usr/lib/python2.7/dist-packages/ceilometerclient/v2/samples.py", line 44, in list
    return self._list(options.build_url(path, q, params))
  File "/usr/lib/python2.7/dist-packages/ceilometerclient/common/base.py", line 58, in _list
    resp, body = self.api.json_request('GET', url)
  File "/usr/lib/python2.7/dist-packages/ceilometerclient/common/http.py", line 191, in json_request
    resp, body_iter = self._http_request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/ceilometerclient/common/http.py", line 160, in _http_request
    raise exc.CommunicationError(message=message)
CommunicationError: Error communicating with http://10.204.217.77:8777 [Errno 111] Connection refused

Note :

1) Issue is similar to 1610506 which reported in 3.1 (Where all ceilometer processes were dead)
2) ceilometer HA is not supported for HA in R3.0 - 1612774

sundarkh (sundar-kh)
summary: - SM:R3.0.3 build69 : ceilometer-api process is dead
+ SM:R3.0.3 build69 : ceilometer-api process is dead -could not connect to
+ 10.204.217.77:27017: [Errno 111] ECONNREFUSED
Revision history for this message
Nitish Krishna Kaveri (nitishk) wrote :

DriverLoadFailure: Failed to load transport driver "ceilometer.openstack.common.rpc.impl_kombu": No 'oslo.messaging.drivers' driver found, looking for 'ceilometer.openstack.common.rpc.impl_kombu'
2016-08-08 10:23:25.447 27900 ERROR ceilometer

Fab doesn't set rpc_backend, we are setting to ceilometer.openstack.common.rpc.impl_kombu

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/24573
Submitter: Nitish Krishna Kaveri (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.1

Review in progress for https://review.opencontrail.org/24625
Submitter: Nitish Krishna Kaveri (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/24626
Submitter: Nitish Krishna Kaveri (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/24626
Committed: http://github.org/Juniper/contrail-puppet/commit/28162a67021f8de74397e3bcad6c5b360eb5fe85
Submitter: Zuul
Branch: master

commit 28162a67021f8de74397e3bcad6c5b360eb5fe85
Author: nitishkrishna <email address hidden>
Date: Mon Oct 3 15:17:54 2016 -0700

Closes-Bug: #1628416 - Ceilometer compute puts back rpc_backend

With move to kilo ceilometer manifests we have separate compute ceilometer provisioning.
This also adds rpc_backend which needs to be removed (we use rabbit)
This is already done for openstack nodes

Change-Id: I39604ed733a268af65ac7d5bbbf6fcfa2602e9d4

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/24625
Committed: http://github.org/Juniper/contrail-puppet/commit/e398e0c64fae4180edfb07ee706d58c8f673fc42
Submitter: Zuul
Branch: R3.1

commit e398e0c64fae4180edfb07ee706d58c8f673fc42
Author: nitishkrishna <email address hidden>
Date: Mon Oct 3 15:17:54 2016 -0700

Closes-Bug: #1628416 - Ceilometer compute puts back rpc_backend

With move to kilo ceilometer manifests we have separate compute ceilometer provisioning.
This also adds rpc_backend which needs to be removed (we use rabbit)
This is already done for openstack nodes

Change-Id: I39604ed733a268af65ac7d5bbbf6fcfa2602e9d4

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0.3.x

Review in progress for https://review.opencontrail.org/25065
Submitter: Nitish Krishna Kaveri (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/25065
Committed: http://github.org/Juniper/contrail-puppet/commit/81b86e8de0cdc92890dfb25fdf5c4b090dfea2d9
Submitter: Zuul
Branch: R3.0.3.x

commit 81b86e8de0cdc92890dfb25fdf5c4b090dfea2d9
Author: nitishkrishna <email address hidden>
Date: Thu Sep 29 11:54:57 2016 -0700

Closes-Bug: #1628416 - Mongo uses wrong ip in single node/single intf

Also ceilometer config has rpc_backend which causes processes to crash

Change-Id: I014ed98f31d3fa3cddca21f73239b6d1013cfca8

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/24573
Submitter: Nitish Krishna Kaveri (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/24573
Committed: http://github.org/Juniper/contrail-puppet/commit/c285330dada920e9f7db3c8677619d1af7f01eff
Submitter: Zuul
Branch: R3.0

commit c285330dada920e9f7db3c8677619d1af7f01eff
Author: nitishkrishna <email address hidden>
Date: Thu Sep 29 11:54:57 2016 -0700

Closes-Bug: #1628416 - Mongo uses wrong ip in single node/single intf

Also ceilometer config has rpc_backend which causes processes to crash

Patch:
Added changes to openstack manifests which were using the old param name

Change-Id: I014ed98f31d3fa3cddca21f73239b6d1013cfca8

Revision history for this message
sundarkh (sundar-kh) wrote :
Download full text (4.8 KiB)

Seen with R3.0.2.x build 12

2016-11-15 11:00:17.383 8576 CRITICAL ceilometer [-] ConnectionFailure: could not connect to 10.204.217.112:27017: [Errno 111] ECONNREFUSED
2016-11-15 11:00:17.383 8576 TRACE ceilometer Traceback (most recent call last):
2016-11-15 11:00:17.383 8576 TRACE ceilometer File "/usr/bin/ceilometer-api", line 10, in <module>
2016-11-15 11:00:17.383 8576 TRACE ceilometer sys.exit(main())
2016-11-15 11:00:17.383 8576 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/cmd/api.py", line 23, in main
2016-11-15 11:00:17.383 8576 TRACE ceilometer srv = app.build_server()
2016-11-15 11:00:17.383 8576 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/api/app.py", line 157, in build_server
2016-11-15 11:00:17.383 8576 TRACE ceilometer app = load_app()
2016-11-15 11:00:17.383 8576 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/api/app.py", line 153, in load_app
2016-11-15 11:00:17.383 8576 TRACE ceilometer return deploy.loadapp("config:" + cfg_file)
2016-11-15 11:00:17.383 8576 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2016-11-15 11:00:17.383 8576 TRACE ceilometer return loadobj(APP, uri, name=name, **kw)
2016-11-15 11:00:17.383 8576 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
2016-11-15 11:00:17.383 8576 TRACE ceilometer return context.create()
2016-11-15 11:00:17.383 8576 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-11-15 11:00:17.383 8576 TRACE ceilometer return self.object_type.invoke(self)
2016-11-15 11:00:17.383 8576 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 203, in invoke
2016-11-15 11:00:17.383 8576 TRACE ceilometer app = context.app_context.create()
2016-11-15 11:00:17.383 8576 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-11-15 11:00:17.383 8576 TRACE ceilometer return self.object_type.invoke(self)
2016-11-15 11:00:17.383 8576 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 146, in invoke
2016-11-15 11:00:17.383 8576 TRACE ceilometer return fix_call(context.object, context.global_conf, **context.local_conf)
2016-11-15 11:00:17.383 8576 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/paste/deploy/util.py", line 55, in fix_call
2016-11-15 11:00:17.383 8576 TRACE ceilometer val = callable(*args, **kw)
2016-11-15 11:00:17.383 8576 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/api/app.py", line 181, in app_factory
2016-11-15 11:00:17.383 8576 TRACE ceilometer return VersionSelectorApplication()
2016-11-15 11:00:17.383 8576 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/api/app.py", line 104, in __init__
2016-11-15 11:00:17.383 8576 TRACE ceilometer self.v2 = setup_app(pecan_config=pc)
2016-11-15 11:00:17.383 8576 TRACE ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/api/app.py", line 68, in setup_app
2016-11-...

Read more...

Revision history for this message
Nitish Krishna Kaveri (nitishk) wrote :

Won't fix in R3.0.2.x branch. Already fixed in higher branches and in base 3.0 branch.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.