charm errors on scaleback during leader-settings-change (probably when last unit)

Bug #1912625 reported by Alex Kavanagh
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Keystone Charm
New
Undecided
Unassigned

Bug Description

Model Controller Cloud/Region Version SLA Timestamp Notes
zaza-80eaad86f3aa auto-osci-sv00 serverstack/serverstack 2.8.7 unsupported 20:29:28Z attempt 30 to destroy model failed (will retry): model not empty, found 1 machine, 8 applications (model not empty)

App Version Status Scale Charm Store Rev OS Notes
cinder-volume unknown 0 cinder local 137 ubuntu
glance unknown 0 glance jujucharms 441 ubuntu
keystone 17.0.0 active 1 keystone jujucharms 528 ubuntu
keystone-mysql-router unknown 0 mysql-router jujucharms 33 ubuntu
neutron-api unknown 0 neutron-api jujucharms 464 ubuntu
nova-cloud-controller unknown 0 nova-cloud-controller jujucharms 512 ubuntu
placement unknown 0 placement jujucharms 37 ubuntu
vault unknown 0 vault jujucharms 128 ubuntu

Unit Workload Agent Machine Public address Ports Message
keystone/0* error idle 7 172.17.100.17 5000/tcp hook failed: "leader-settings-changed"

Machine State DNS Inst id Series AZ Message
7 stopped 172.17.100.17 b0fd0d77-9e74-429d-a7b5-3479764806d8 focal nova ACTIVE

Keystone log:
2021-01-20 19:05:45 ERROR juju-log The call within manager.py failed with the error: 'Not Found (HTTP 404)'. The call was: path=['list_services'], args=(), kwargs={}, api_version=None
2021-01-20 19:05:45 WARNING leader-settings-changed Traceback (most recent call last):
2021-01-20 19:05:45 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/leader-settings-changed", line 937, in <module>
2021-01-20 19:05:45 WARNING leader-settings-changed main()
2021-01-20 19:05:45 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/leader-settings-changed", line 930, in main
2021-01-20 19:05:45 WARNING leader-settings-changed hooks.execute(sys.argv)
2021-01-20 19:05:45 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/core/hookenv.py", line 945, in execute
2021-01-20 19:05:45 WARNING leader-settings-changed self._hooks[hook_name]()
2021-01-20 19:05:45 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/contrib/openstack/utils.py", line 1718, in wrapped_f
2021-01-20 19:05:45 WARNING leader-settings-changed return restart_on_change_helper(
2021-01-20 19:05:45 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/core/host.py", line 747, in restart_on_change_helper
2021-01-20 19:05:45 WARNING leader-settings-changed r = lambda_f()
2021-01-20 19:05:45 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/contrib/openstack/utils.py", line 1719, in <lambda>
2021-01-20 19:05:45 WARNING leader-settings-changed (lambda: f(*args, **kwargs)), __restart_map_cache['cache'],
2021-01-20 19:05:45 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/leader-settings-changed", line 590, in leader_settings_changed
2021-01-20 19:05:45 WARNING leader-settings-changed update_all_identity_relation_units()
2021-01-20 19:05:45 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/leader-settings-changed", line 340, in update_all_identity_relation_units
2021-01-20 19:05:45 WARNING leader-settings-changed identity_changed(relation_id=rid, remote_unit=unit)
2021-01-20 19:05:45 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/contrib/openstack/utils.py", line 1718, in wrapped_f
2021-01-20 19:05:45 WARNING leader-settings-changed return restart_on_change_helper(
2021-01-20 19:05:45 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/core/host.py", line 747, in restart_on_change_helper
2021-01-20 19:05:45 WARNING leader-settings-changed r = lambda_f()
2021-01-20 19:05:45 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/contrib/openstack/utils.py", line 1719, in <lambda>
2021-01-20 19:05:45 WARNING leader-settings-changed (lambda: f(*args, **kwargs)), __restart_map_cache['cache'],
2021-01-20 19:05:45 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/leader-settings-changed", line 445, in identity_changed
2021-01-20 19:05:45 WARNING leader-settings-changed add_service_to_keystone(relation_id, remote_unit)
2021-01-20 19:05:45 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1896, in add_service_to_keystone
2021-01-20 19:05:45 WARNING leader-settings-changed _list_services_result = manager.list_services()
2021-01-20 19:05:45 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1212, in __call__
2021-01-20 19:05:45 WARNING leader-settings-changed return _proxy_manager_call(self._path, self.api_version, args, kwargs)
2021-01-20 19:05:45 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/core/decorators.py", line 40, in _retry_on_exception_inner_2
2021-01-20 19:05:45 WARNING leader-settings-changed return f(*args, **kwargs)
2021-01-20 19:05:45 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1256, in _proxy_manager_call
2021-01-20 19:05:45 WARNING leader-settings-changed raise e
2021-01-20 19:05:45 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1250, in _proxy_manager_call
2021-01-20 19:05:45 WARNING leader-settings-changed raise RuntimeError(s)
2021-01-20 19:05:45 WARNING leader-settings-changed RuntimeError: The call within manager.py failed with the error: 'Not Found (HTTP 404)'. The call was: path=['list_services'], args=(), kwargs={}, api_version=None
2021-01-20 19:05:45 WARNING leader-settings-changed Traceback (most recent call last):
2021-01-20 19:05:45 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 716, in <module>
2021-01-20 19:05:45 WARNING leader-settings-changed manager = get_manager(
2021-01-20 19:05:45 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 647, in get_manager
2021-01-20 19:05:45 WARNING leader-settings-changed _keystone_manager['manager'] = get_keystone_manager(
2021-01-20 19:05:45 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 105, in _retry_on_exception_inner_2
2021-01-20 19:05:45 WARNING leader-settings-changed return f(*args, **kwargs)
2021-01-20 19:05:45 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 156, in get_keystone_manager
2021-01-20 19:05:45 WARNING leader-settings-changed for svc in manager.api.services.list():
2021-01-20 19:05:45 WARNING leader-settings-changed File "/usr/lib/python3/dist-packages/keystoneclient/v3/services.py", line 90, in list
2021-01-20 19:05:45 WARNING leader-settings-changed return super(ServiceManager, self).list(
2021-01-20 19:05:45 WARNING leader-settings-changed File "/usr/lib/python3/dist-packages/keystoneclient/base.py", line 86, in func
2021-01-20 19:05:45 WARNING leader-settings-changed return f(*args, **new_kwargs)
2021-01-20 19:05:45 WARNING leader-settings-changed File "/usr/lib/python3/dist-packages/keystoneclient/base.py", line 448, in list
2021-01-20 19:05:45 WARNING leader-settings-changed list_resp = self._list(url_query, self.collection_key)
2021-01-20 19:05:45 WARNING leader-settings-changed File "/usr/lib/python3/dist-packages/keystoneclient/base.py", line 141, in _list
2021-01-20 19:05:45 WARNING leader-settings-changed resp, body = self.client.get(url, **kwargs)
2021-01-20 19:05:45 WARNING leader-settings-changed File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 386, in get
2021-01-20 19:05:45 WARNING leader-settings-changed return self.request(url, 'GET', **kwargs)
2021-01-20 19:05:45 WARNING leader-settings-changed File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 545, in request
2021-01-20 19:05:45 WARNING leader-settings-changed resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2021-01-20 19:05:45 WARNING leader-settings-changed File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 248, in request
2021-01-20 19:05:45 WARNING leader-settings-changed return self.session.request(url, method, **kwargs)
2021-01-20 19:05:45 WARNING leader-settings-changed File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 772, in request
2021-01-20 19:05:45 WARNING leader-settings-changed auth_headers = self.get_auth_headers(auth)
2021-01-20 19:05:45 WARNING leader-settings-changed File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1183, in get_auth_headers
2021-01-20 19:05:45 WARNING leader-settings-changed return auth.get_headers(self, **kwargs)
2021-01-20 19:05:45 WARNING leader-settings-changed File "/usr/lib/python3/dist-packages/keystoneauth1/plugin.py", line 95, in get_headers
2021-01-20 19:05:45 WARNING leader-settings-changed token = self.get_token(session)
2021-01-20 19:05:45 WARNING leader-settings-changed File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token
2021-01-20 19:05:45 WARNING leader-settings-changed return self.get_access(session).auth_token
2021-01-20 19:05:45 WARNING leader-settings-changed File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access
2021-01-20 19:05:45 WARNING leader-settings-changed self.auth_ref = self.get_auth_ref(session)
2021-01-20 19:05:45 WARNING leader-settings-changed File "/usr/lib/python3/dist-packages/keystoneauth1/identity/v3/base.py", line 183, in get_auth_ref
2021-01-20 19:05:45 WARNING leader-settings-changed resp = session.post(token_url, json=body, headers=headers,
2021-01-20 19:05:45 WARNING leader-settings-changed File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1131, in post
2021-01-20 19:05:45 WARNING leader-settings-changed return self.request(url, 'POST', **kwargs)
2021-01-20 19:05:45 WARNING leader-settings-changed File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 968, in request
2021-01-20 19:05:45 WARNING leader-settings-changed raise exceptions.from_response(resp, method, url)
2021-01-20 19:05:45 WARNING leader-settings-changed keystoneauth1.exceptions.http.NotFound: Not Found (HTTP 404)

Tags: scaleback
Revision history for this message
Corey Bryant (corey.bryant) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.