charm has a hook error on shared-db-relation-changed when database is series-upgrading

[Chris MacNaughton]

2020-04-15 07:55:27 DEBUG shared-db-relation-changed GatewayTimeout: Gateway Timeout (HTTP 504)
2020-04-15 07:55:27 ERROR juju-log shared-db:10: The call within manager.py failed with the error: 'Gateway Timeout (HTTP 504)'. The call was: path=['resolve_user_id'], args=('nova_placement',), kwargs={'user_domain': 'servi
ce_domain'}, api_version=None
2020-04-15 07:55:27 DEBUG shared-db-relation-changed Traceback (most recent call last):
2020-04-15 07:55:27 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-keystone-1/charm/hooks/shared-db-relation-changed", line 928, in <module>
2020-04-15 07:55:27 DEBUG shared-db-relation-changed main()
2020-04-15 07:55:27 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-keystone-1/charm/hooks/shared-db-relation-changed", line 921, in main
2020-04-15 07:55:27 DEBUG shared-db-relation-changed hooks.execute(sys.argv)
2020-04-15 07:55:27 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-keystone-1/charm/charmhelpers/core/hookenv.py", lin
e 934, in execute
2020-04-15 07:55:27 DEBUG shared-db-relation-changed self._hooks[hook_name]()
2020-04-15 07:55:27 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-keystone-1/charm/charmhelpers/contrib/openstack/utils.py", line 1696, in wrapped_f
2020-04-15 07:55:27 DEBUG shared-db-relation-changed stopstart, restart_functions)
2020-04-15 07:55:27 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-keystone-1/charm/charmhelpers/core/host.py", line 741, in restart_on_change_helper
2020-04-15 07:55:27 DEBUG shared-db-relation-changed r = lambda_f()
2020-04-15 07:55:27 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-keystone-1/charm/charmhelpers/contrib/openstack/utils.py", line 1695, in <lambda>
2020-04-15 07:55:27 DEBUG shared-db-relation-changed (lambda: f(*args, **kwargs)), __restart_map_cache['cache'],
2020-04-15 07:55:27 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-keystone-1/charm/hooks/shared-db-relation-changed", line 416, in db_changed
2020-04-15 07:55:27 DEBUG shared-db-relation-changed leader_init_db_if_ready(use_current_context=True)
2020-04-15 07:55:27 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-keystone-1/charm/charmhelpers/contrib/openstack/utils.py", line 1696, in wrapped_f
2020-04-15 07:55:27 DEBUG shared-db-relation-changed stopstart, restart_functions)
2020-04-15 07:55:27 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-keystone-1/charm/charmhelpers/core/host.py", line 741, in restart_on_change_helper
2020-04-15 07:55:27 DEBUG shared-db-relation-changed r = lambda_f()
2020-04-15 07:55:27 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-keystone-1/charm/charmhelpers/contrib/openstack/utils.py", line 1695, in <lambda>
2020-04-15 07:55:27 DEBUG shared-db-relation-changed (lambda: f(*args, **kwargs)), __restart_map_cache['cache'],
2020-04-15 07:55:27 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-keystone-1/charm/hooks/shared-db-relation-changed", line 386, in leader_init_db_if_ready
2020-04-15 07:55:27 DEBUG shared-db-relation-changed update_all_identity_relation_units(check_db_ready=False)
2020-04-15 07:55:27 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-keystone-1/charm/hooks/shared-db-relation-changed", line 346, in update_all_identity_relation_units
2020-04-15 07:55:27 DEBUG shared-db-relation-changed identity_changed(relation_id=rid, remote_unit=unit)
2020-04-15 07:55:27 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-keystone-1/charm/charmhelpers/contrib/openstack/utils.py", line 1696, in wrapped_f
2020-04-15 07:55:27 DEBUG shared-db-relation-changed stopstart, restart_functions)
2020-04-15 07:55:27 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-keystone-1/charm/charmhelpers/core/host.py", line 741, in restart_on_change_helper
2020-04-15 07:55:27 DEBUG shared-db-relation-changed r = lambda_f()
2020-04-15 07:55:27 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-keystone-1/charm/charmhelpers/contrib/openstack/utils.py", line 1695, in <lambda>
2020-04-15 07:55:27 DEBUG shared-db-relation-changed (lambda: f(*args, **kwargs)), __restart_map_cache['cache'],
2020-04-15 07:55:27 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-keystone-1/charm/hooks/shared-db-relation-changed", line 450, in identity_changed
2020-04-15 07:55:27 DEBUG shared-db-relation-changed add_service_to_keystone(relation_id, remote_unit)
2020-04-15 07:55:27 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-keystone-1/charm/hooks/keystone_utils.py", line 1897, in add_service_to_keystone
2020-04-15 07:55:27 DEBUG shared-db-relation-changed new_roles=roles)
2020-04-15 07:55:27 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-keystone-1/charm/hooks/keystone_utils.py", line 1732, in create_service_credentials
2020-04-15 07:55:27 DEBUG shared-db-relation-changed domain=SERVICE_DOMAIN)
2020-04-15 07:55:27 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-keystone-1/charm/hooks/keystone_utils.py", line 1683, in create_user_credentials
2020-04-15 07:55:27 DEBUG shared-db-relation-changed project_domain=domain)
2020-04-15 07:55:27 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-keystone-1/charm/hooks/keystone_utils.py", line 1401, in grant_role
2020-04-15 07:55:27 DEBUG shared-db-relation-changed user_id = manager.resolve_user_id(user, user_domain=user_domain)
2020-04-15 07:55:27 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-keystone-1/charm/hooks/keystone_utils.py", line 1205, in __call__
2020-04-15 07:55:27 DEBUG shared-db-relation-changed return _proxy_manager_call(self._path, self.api_version, args, kwargs)
2020-04-15 07:55:27 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-keystone-1/charm/charmhelpers/core/decorators.py", line 40, in _retry_on_exception_inner_2
2020-04-15 07:55:27 DEBUG shared-db-relation-changed return f(*args, **kwargs)
2020-04-15 07:55:27 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-keystone-1/charm/hooks/keystone_utils.py", line 1249, in _proxy_manager_call
2020-04-15 07:55:27 DEBUG shared-db-relation-changed raise e
2020-04-15 07:55:27 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-keystone-1/charm/hooks/keystone_utils.py", line 1243, in _proxy_manager_call
2020-04-15 07:55:27 DEBUG shared-db-relation-changed raise RuntimeError(s)
2020-04-15 07:55:27 DEBUG shared-db-relation-changed RuntimeError: The call within manager.py failed with the error: 'Gateway Timeout (HTTP 504)'. The call was: path=['resolve_user_id'], args=('nova_placement',), kwargs={'user_domain': 'service_domain'}, api_version=None

[Chris MacNaughton (chris.macnaughton)]

It looks like this is the result of the VIP going down after pausing the followers, but before the leader sets the cluster upgrading flag

[OpenStack Infra (hudson-openstack)]

Change abandoned by Chris MacNaughton (icey) (<email address hidden>) on branch: master
Review: https://review.opendev.org/720151