On a deploy with a clustered keystone, one keystone unit is ready, the other 2 are waiting. The charm requested data from keystone resulting in a http 500 error.
keystone:
charm: cs:keystone-292
series: bionic
os: ubuntu
charm-origin: jujucharms
charm-name: keystone
charm-rev: 292
exposed: false
application-status:
current: waiting
message: 'Incomplete relations: database'
since: 28 Feb 2019 04:58:47Z
relations:
cluster:
- keystone
ha:
- hacluster-keystone
identity-credentials:
- ceilometer
- openstack-service-checks
- prometheus-openstack-exporter
identity-service:
- aodh
- ceph-radosgw
- cinder
- designate
- glance
- gnocchi
- heat
- neutron-api
- nova-cloud-controller
- openstack-dashboard
juju-info:
- filebeat
- telegraf
nrpe-external-master:
- nrpe-container
shared-db:
- mysql
2019-02-28 05:35:23 ERROR juju-log Hook error:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-openstack-service-checks-0/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 73, in main
bus.dispatch(restricted=restricted_mode)
File "/var/lib/juju/agents/unit-openstack-service-checks-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch
_invoke(other_handlers)
File "/var/lib/juju/agents/unit-openstack-service-checks-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke
handler.invoke()
File "/var/lib/juju/agents/unit-openstack-service-checks-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke
self._action(*args)
File "/var/lib/juju/agents/unit-openstack-service-checks-0/charm/reactive/service_checks.py", line 184, in render_config
render_checks()
File "/var/lib/juju/agents/unit-openstack-service-checks-0/charm/reactive/service_checks.py", line 162, in render_checks
endpoint_checks = create_endpoint_checks()
File "/var/lib/juju/agents/unit-openstack-service-checks-0/charm/reactive/service_checks.py", line 236, in create_endpoint_checks
endpoints = keystone_client.endpoints.list()
File "/usr/lib/python3/dist-packages/keystoneclient/v3/endpoints.py", line 125, in list
**kwargs)
File "/usr/lib/python3/dist-packages/keystoneclient/base.py", line 75, in func
return f(*args, **new_kwargs)
File "/usr/lib/python3/dist-packages/keystoneclient/base.py", line 397, in list
self.collection_key)
File "/usr/lib/python3/dist-packages/keystoneclient/base.py", line 125, in _list
resp, body = self.client.get(url, **kwargs)
File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 304, in get
return self.request(url, 'GET', **kwargs)
File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 463, in request
resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 189, in request
return self.session.request(url, method, **kwargs)
File "/usr/lib/python3/dist-packages/keystoneclient/session.py", line 343, in request
auth_headers = self.get_auth_headers(auth)
File "/usr/lib/python3/dist-packages/keystoneclient/session.py", line 660, in get_auth_headers
return auth.get_headers(self, **kwargs)
File "/usr/lib/python3/dist-packages/keystoneclient/auth/base.py", line 159, in get_headers
token = self.get_token(session)
File "/usr/lib/python3/dist-packages/keystoneclient/auth/identity/base.py", line 206, in get_token
return self.get_access(session).auth_token
File "/usr/lib/python3/dist-packages/keystoneclient/auth/identity/base.py", line 252, in get_access
self.auth_ref = self.get_auth_ref(session)
File "/usr/lib/python3/dist-packages/keystoneclient/auth/identity/v3/base.py", line 189, in get_auth_ref
authenticated=False, log=False, **rkwargs)
File "/usr/lib/python3/dist-packages/keystoneclient/session.py", line 543, in post
return self.request(url, 'POST', **kwargs)
File "/usr/lib/python3/dist-packages/keystoneclient/session.py", line 443, in request
raise exceptions.from_response(resp, method, url)
keystoneauth1.exceptions.http.InternalServerError: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-58a1d5ba-d15c-41ea-8f94-cb6b01a16356)
2019-02-28 05:35:23 ERROR juju.worker.uniter.operation runhook.go:132 hook "update-status" failed: exit status 1
Full crashdump is attached
I suspect this is something to do with the recent updates to the charm.
It's not clear if keystone is ready or not. In Alex's run, it shows as not in juju status, but all of the units have said they are ready at some point:
http:// paste.ubuntu. com/p/Pz64Yt8rB v/