I think this is still an open issue. When upgrading keystone with keystone-ldap from bionic distro to cloud:bionic-rocky, the keystone/leader unit goes into error state not able to reach the proxy manager: unit-keystone-0: 19:38:43 ERROR unit.keystone/0.juju-log The call within manager.py failed with the error: 'An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-c57e4eb3-7d3c-4580-8667-f2324e6d92a9)'. The call was: path=['resolve_domain_id'], args=('default',), kwargs={}, api_version=None unit-keystone-0: 19:38:43 DEBUG unit.keystone/0.config-changed An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-c57e4eb3-7d3c-4580-8667-f2324e6d92a9) unit-keystone-0: 19:38:43 INFO unit.keystone/0.juju-log Retrying '_proxy_manager_call' 5 more times (delay=3) unit-keystone-0: 19:38:47 ERROR unit.keystone/0.juju-log The call within manager.py failed with the error: 'An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-60aa9ecd-56b6-489e-ab3b-4fe0d4c96059)'. The call was: path=['resolve_domain_id'], args=('default',), kwargs={}, api_version=None unit-keystone-0: 19:38:47 DEBUG unit.keystone/0.config-changed An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-60aa9ecd-56b6-489e-ab3b-4fe0d4c96059) unit-keystone-0: 19:38:47 INFO unit.keystone/0.juju-log Retrying '_proxy_manager_call' 4 more times (delay=6) unit-keystone-0: 19:38:53 ERROR unit.keystone/0.juju-log The call within manager.py failed with the error: 'An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-9842ec31-848e-4e9c-a924-81174fed4ab6)'. The call was: path=['resolve_domain_id'], args=('default',), kwargs={}, api_version=None unit-keystone-0: 19:38:53 DEBUG unit.keystone/0.config-changed An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-9842ec31-848e-4e9c-a924-81174fed4ab6) unit-keystone-0: 19:38:54 INFO unit.keystone/0.juju-log Retrying '_proxy_manager_call' 3 more times (delay=9) unit-keystone-0: 19:39:03 ERROR unit.keystone/0.juju-log The call within manager.py failed with the error: 'An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-e3d06cd2-b4c6-48bf-a61e-36a9a5edf806)'. The call was: path=['resolve_domain_id'], args=('default',), kwargs={}, api_version=None unit-keystone-0: 19:39:03 DEBUG unit.keystone/0.config-changed An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-e3d06cd2-b4c6-48bf-a61e-36a9a5edf806) unit-keystone-0: 19:39:03 INFO unit.keystone/0.juju-log Retrying '_proxy_manager_call' 2 more times (delay=12) unit-keystone-0: 19:39:16 ERROR unit.keystone/0.juju-log The call within manager.py failed with the error: 'An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-311c11ac-a7c1-47bb-842b-ef9f8721544e)'. The call was: path=['resolve_domain_id'], args=('default',), kwargs={}, api_version=None unit-keystone-0: 19:39:16 DEBUG unit.keystone/0.config-changed An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-311c11ac-a7c1-47bb-842b-ef9f8721544e) unit-keystone-0: 19:39:16 INFO unit.keystone/0.juju-log Retrying '_proxy_manager_call' 1 more times (delay=15) unit-keystone-0: 19:39:32 ERROR unit.keystone/0.juju-log The call within manager.py failed with the error: 'An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-d0f711ee-62a8-4919-a863-accb017ee0fb)'. The call was: path=['resolve_domain_id'], args=('default',), kwargs={}, api_version=None unit-keystone-0: 19:39:32 DEBUG unit.keystone/0.config-changed An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-d0f711ee-62a8-4919-a863-accb017ee0fb) unit-keystone-0: 19:39:32 DEBUG unit.keystone/0.config-changed OpenStack Release: {} unit-keystone-0: 19:39:32 DEBUG unit.keystone/0.config-changed OpenStack Release: {} unit-keystone-0: 19:39:32 DEBUG unit.keystone/0.config-changed DEPRECATION WARNING: Function maybe_do_policyd_overrides_on_config_changed is being removed : Use maybe_do_poliyd_overrrides instead unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed Traceback (most recent call last): unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 937, in unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed main() unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 930, in main unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed hooks.execute(sys.argv) unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/core/hookenv.py", line 945, in execute unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed self._hooks[hook_name]() unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/contrib/openstack/utils.py", line 1767, in wrapped_f unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed stopstart, restart_functions) unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/core/host.py", line 749, in restart_on_change_helper unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed r = lambda_f() unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/contrib/openstack/utils.py", line 1766, in unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed (lambda: f(*args, **kwargs)), __restart_map_cache['cache'], unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/contrib/hardening/harden.py", line 93, in _harden_inner2 unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed return f(*args, **kwargs) unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 246, in config_changed unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed config_changed_postupgrade() unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/contrib/openstack/utils.py", line 1767, in wrapped_f unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed stopstart, restart_functions) unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/core/host.py", line 749, in restart_on_change_helper unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed r = lambda_f() unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/contrib/openstack/utils.py", line 1766, in unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed (lambda: f(*args, **kwargs)), __restart_map_cache['cache'], unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/contrib/hardening/harden.py", line 93, in _harden_inner2 unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed return f(*args, **kwargs) unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 292, in config_changed_postupgrade unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed ensure_initial_admin(config) unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1579, in ensure_initial_admin unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed return _ensure_initial_admin(config) unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/core/decorators.py", line 40, in _retry_on_exception_inner_2 unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed return f(*args, **kwargs) unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1517, in _ensure_initial_admin unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed default_domain_id = create_or_show_domain(DEFAULT_DOMAIN) unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1108, in create_or_show_domain unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed domain_id = manager.resolve_domain_id(name) unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1212, in __call__ unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed return _proxy_manager_call(self._path, self.api_version, args, kwargs) unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/core/decorators.py", line 40, in _retry_on_exception_inner_2 unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed return f(*args, **kwargs) unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1249, in _proxy_manager_call unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed raise RetryProxyManagerCall() unit-keystone-0: 19:39:32 WARNING unit.keystone/0.config-changed keystone_utils.RetryProxyManagerCall unit-keystone-0: 19:39:32 ERROR juju.worker.uniter.operation hook "config-changed" (via explicit, bespoke hook script) failed: exit status 1 In the keystone.log I see: (keystone.common.wsgi): 2021-03-16 19:41:53,528 ERROR No module named 'ldappool' Traceback (most recent call last): File "/usr/lib/python3/dist-packages/keystone/common/wsgi.py", line 275, in _inner return method(self, request) File "/usr/lib/python3/dist-packages/keystone/middleware/auth.py", line 152, in process_request resp = super(AuthContextMiddleware, self).process_request(request) File "/usr/lib/python3/dist-packages/keystonemiddleware/auth_token/__init__.py", line 405, in process_request allow_expired=allow_expired) File "/usr/lib/python3/dist-packages/keystonemiddleware/auth_token/__init__.py", line 437, in _do_fetch_token data = self.fetch_token(token, **kwargs) File "/usr/lib/python3/dist-packages/keystone/middleware/auth.py", line 47, in fetch_token token_model = self.token_provider_api.validate_token(token) File "/usr/lib/python3/dist-packages/keystone/common/manager.py", line 116, in wrapped __ret_val = __f(*args, **kwargs) File "/usr/lib/python3/dist-packages/keystone/token/provider.py", line 143, in validate_token token = self._validate_token(token_id) File "/usr/lib/python3/dist-packages/dogpile/cache/region.py", line 1220, in decorate should_cache_fn) File "/usr/lib/python3/dist-packages/dogpile/cache/region.py", line 825, in get_or_create async_creator) as value: File "/usr/lib/python3/dist-packages/dogpile/lock.py", line 154, in __enter__ return self._enter() File "/usr/lib/python3/dist-packages/dogpile/lock.py", line 94, in _enter generated = self._enter_create(createdtime) File "/usr/lib/python3/dist-packages/dogpile/lock.py", line 145, in _enter_create created = self.creator() File "/usr/lib/python3/dist-packages/dogpile/cache/region.py", line 792, in gen_value created_value = creator() File "/usr/lib/python3/dist-packages/dogpile/cache/region.py", line 1216, in creator return fn(*arg, **kw) File "/usr/lib/python3/dist-packages/keystone/token/provider.py", line 176, in _validate_token token.mint(token_id, issued_at) File "/usr/lib/python3/dist-packages/keystone/models/token_model.py", line 533, in mint self._validate_token_user() File "/usr/lib/python3/dist-packages/keystone/models/token_model.py", line 465, in _validate_token_user if not self.user_domain.get('enabled'): File "/usr/lib/python3/dist-packages/keystone/models/token_model.py", line 138, in user_domain if self.user: File "/usr/lib/python3/dist-packages/keystone/models/token_model.py", line 132, in user self.__user = PROVIDERS.identity_api.get_user(self.user_id) File "/usr/lib/python3/dist-packages/keystone/common/manager.py", line 116, in wrapped __ret_val = __f(*args, **kwargs) File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 415, in wrapper self.driver, PROVIDERS.resource_api) File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 309, in setup_domain_drivers resource_api) File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 164, in _setup_domain_drivers_from_files -len(DOMAIN_CONF_FTAIL)]) File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 130, in _load_config_from_file domain_config['driver'] = self._load_driver(domain_config) File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 93, in _load_driver domain_config['cfg']) File "/usr/lib/python3/dist-packages/keystone/common/manager.py", line 77, in load_driver invoke_args=args) File "/usr/lib/python3/dist-packages/stevedore/driver.py", line 61, in __init__ warn_on_missing_entrypoint=warn_on_missing_entrypoint File "/usr/lib/python3/dist-packages/stevedore/named.py", line 81, in __init__ verify_requirements) File "/usr/lib/python3/dist-packages/stevedore/extension.py", line 203, in _load_plugins self._on_load_failure_callback(self, ep, err) File "/usr/lib/python3/dist-packages/stevedore/extension.py", line 195, in _load_plugins verify_requirements, File "/usr/lib/python3/dist-packages/stevedore/named.py", line 158, in _load_one_plugin verify_requirements, File "/usr/lib/python3/dist-packages/stevedore/extension.py", line 223, in _load_one_plugin plugin = ep.resolve() File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2330, in resolve module = __import__(self.module_name, fromlist=['__name__'], level=0) File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/__init__.py", line 13, in from keystone.identity.backends.ldap.core import * # noqa File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/core.py", line 26, in from keystone.identity.backends.ldap import common as common_ldap File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 25, in import ldappool ModuleNotFoundError: No module named 'ldappool' I think because keystone-ldap charm does not actually perform the openstack upgrades that the keystone charm also needs a patch to install python3-ldappool on upgrade to Rocky.