Ubuntu
ceilometer package

Gnocchi plugin fails to load when there is issue with Keystone connection

Bug #1904850 reported by Gautam Prasad
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
ceilometer (Ubuntu)
Confirmed
Undecided
Unassigned

Bug Description

When openstack services are restarted and Keystone is not fully up or is up but failed to respond the connection request, then Gnocchi publisher plugin fails to load.

I am not sure if the Gnocchi publisher can be refactored to allow plugin to load initially and initialize/retry for keystone connection later when it becomes available.. ?

Openstack release: ussuri.

2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base [-] Unable to load publisher gnocchi://: keystoneauth1.exceptions.connection.ConnectTimeout: Request to https://<host>:5000/v3/auth/tokens timed out
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base Traceback (most recent call last):
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 421, in _make_request
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base six.raise_from(e, None)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "<string>", line 3, in raise_from
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base httplib_response = conn.getresponse()
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib64/python3.6/http/client.py", line 1346, in getresponse
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base response.begin()
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib64/python3.6/http/client.py", line 307, in begin
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base version, status, reason = self._read_status()
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib64/python3.6/http/client.py", line 268, in _read_status
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib64/python3.6/socket.py", line 586, in readinto
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base return self._sock.recv_into(b)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 328, in recv_into
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base return self.recv_into(*args, **kwargs)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 326, in recv_into
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base raise timeout("The read operation timed out")
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base socket.timeout: The read operation timed out
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base During handling of the above exception, another exception occurred:
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base Traceback (most recent call last):
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base timeout=timeout
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 400, in increment
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base raise six.reraise(type(error), error, _stacktrace)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/urllib3/packages/six.py", line 703, in reraise

....

2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base Traceback (most recent call last):
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/ceilometer/pipeline/base.py", line 124, in __init__
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base self.publishers.append(publisher_manager.get(p))
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/ceilometer/pipeline/base.py", line 179, in get
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base 'ceilometer.%s.publisher' % self._purpose)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/ceilometer/publisher/__init__.py", line 36, in get_publisher
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base return loaded_driver.driver(conf, parse_result)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/ceilometer/publisher/gnocchi.py", line 249, in __init__
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base conf, request_timeout=timeout)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/ceilometer/gnocchi_client.py", line 39, in get_gnocchiclient
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base region_name=region_name)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1225, in get_endpoint
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base return auth.get_endpoint(self, **kwargs)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base allow_version_hack=allow_version_hack, **kwargs)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base service_catalog = self.get_access(session).service_catalog
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base self.auth_ref = self.get_auth_ref(session)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/v3/base.py", line 184, in get_auth_ref
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base authenticated=False, log=False, **rkwargs)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1131, in post
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base return self.request(url, 'POST', **kwargs)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 913, in request
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base resp = send(**kwargs)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1011, in _send_request
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base raise exceptions.ConnectTimeout(msg)
2020-11-16 21:10:06.147 19463 ERROR ceilometer.pipeline.base keystoneauth1.exceptions.connection.ConnectTimeout: Request to https://<host>:5000/v3/auth/tokens timed out

Gautam Prasad (gautpras)
summary: - Gnocchi plugin fails to load when there is issue with Keystrone
+ Gnocchi plugin fails to load when there is issue with Keystone
connection
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ceilometer (Ubuntu):
status: New → Confirmed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.