Keystone hook error on cluster-departed when sharedb relation doesn't exist

[Billy Olsen]

In recent CI runs(such as that seen in https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_amulet_full/openstack/charm-hacluster/497726/2/453/index.html), the shared-db relation is removed before the keystone units are removed. When the keystone units are removed, the cluster-relation-departed hook fires and attempts to perform operations on the keystone service, which fails - causing the hook to fail and the CI tests to fail..

2017-08-25 09:06:22 DEBUG cluster-relation-departed Traceback (most recent call last):
2017-08-25 09:06:22 DEBUG cluster-relation-departed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/cluster-relation-departed", line 860, in <module>
2017-08-25 09:06:22 DEBUG cluster-relation-departed main()
2017-08-25 09:06:22 DEBUG cluster-relation-departed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/cluster-relation-departed", line 853, in main
2017-08-25 09:06:22 DEBUG cluster-relation-departed hooks.execute(sys.argv)
2017-08-25 09:06:22 DEBUG cluster-relation-departed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/hookenv.py", line 767, in execute
2017-08-25 09:06:22 DEBUG cluster-relation-departed self._hooks[hook_name]()
2017-08-25 09:06:22 DEBUG cluster-relation-departed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1887, in wrapped_f
2017-08-25 09:06:22 DEBUG cluster-relation-departed restart_functions)
2017-08-25 09:06:22 DEBUG cluster-relation-departed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/host.py", line 661, in restart_on_change_helper
2017-08-25 09:06:22 DEBUG cluster-relation-departed r = lambda_f()
2017-08-25 09:06:22 DEBUG cluster-relation-departed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1886, in <lambda>
2017-08-25 09:06:22 DEBUG cluster-relation-departed (lambda: f(*args, **kwargs)), restart_map, stopstart,
2017-08-25 09:06:22 DEBUG cluster-relation-departed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1520, in _inner_update_certs_if_available
2017-08-25 09:06:22 DEBUG cluster-relation-departed return f(*args, **kwargs)
2017-08-25 09:06:22 DEBUG cluster-relation-departed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/cluster-relation-departed", line 614, in cluster_changed
2017-08-25 09:06:22 DEBUG cluster-relation-departed update_all_identity_relation_units_force_sync()
2017-08-25 09:06:22 DEBUG cluster-relation-departed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1675, in inner_synchronize_ca_if_changed2
2017-08-25 09:06:22 DEBUG cluster-relation-departed ret = f(*args, **kwargs)
2017-08-25 09:06:22 DEBUG cluster-relation-departed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/cluster-relation-departed", line 364, in update_all_identity_relation_units_force_sync
2017-08-25 09:06:22 DEBUG cluster-relation-departed update_all_identity_relation_units()
2017-08-25 09:06:22 DEBUG cluster-relation-departed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/cluster-relation-departed", line 347, in update_all_identity_relation_units
2017-08-25 09:06:22 DEBUG cluster-relation-departed ensure_initial_admin(config)
2017-08-25 09:06:22 DEBUG cluster-relation-departed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1096, in ensure_initial_admin
2017-08-25 09:06:22 DEBUG cluster-relation-departed return _ensure_initial_admin(config)
2017-08-25 09:06:22 DEBUG cluster-relation-departed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/decorators.py", line 40, in _retry_on_exception_inner_2
2017-08-25 09:06:22 DEBUG cluster-relation-departed return f(*args, **kwargs)
2017-08-25 09:06:22 DEBUG cluster-relation-departed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1051, in _ensure_initial_admin
2017-08-25 09:06:22 DEBUG cluster-relation-departed create_tenant("admin", DEFAULT_DOMAIN)
2017-08-25 09:06:22 DEBUG cluster-relation-departed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 804, in create_tenant
2017-08-25 09:06:22 DEBUG cluster-relation-departed manager = get_manager()
2017-08-25 09:06:22 DEBUG cluster-relation-departed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 892, in get_manager
2017-08-25 09:06:22 DEBUG cluster-relation-departed api_version)
2017-08-25 09:06:22 DEBUG cluster-relation-departed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/decorators.py", line 40, in _retry_on_exception_inner_2
2017-08-25 09:06:22 DEBUG cluster-relation-departed return f(*args, **kwargs)
2017-08-25 09:06:22 DEBUG cluster-relation-departed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 75, in get_keystone_manager
2017-08-25 09:06:22 DEBUG cluster-relation-departed for svc in manager.api.services.list():
2017-08-25 09:06:22 DEBUG cluster-relation-departed File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/services.py", line 32, in list
2017-08-25 09:06:22 DEBUG cluster-relation-departed return self._list("/OS-KSADM/services", "OS-KSADM:services")
2017-08-25 09:06:22 DEBUG cluster-relation-departed File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 106, in _list
2017-08-25 09:06:22 DEBUG cluster-relation-departed resp, body = self.client.get(url)
2017-08-25 09:06:22 DEBUG cluster-relation-departed File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 590, in get
2017-08-25 09:06:22 DEBUG cluster-relation-departed return self._cs_request(url, 'GET', **kwargs)
2017-08-25 09:06:22 DEBUG cluster-relation-departed File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 582, in _cs_request
2017-08-25 09:06:22 DEBUG cluster-relation-departed return self.request(url, method, **kwargs)
2017-08-25 09:06:22 DEBUG cluster-relation-departed File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 564, in request
2017-08-25 09:06:22 DEBUG cluster-relation-departed resp = super(HTTPClient, self).request(url, method, **kwargs)
2017-08-25 09:06:22 DEBUG cluster-relation-departed File "/usr/lib/python2.7/dist-packages/keystoneclient/baseclient.py", line 21, in request
2017-08-25 09:06:22 DEBUG cluster-relation-departed return self.session.request(url, method, **kwargs)
2017-08-25 09:06:22 DEBUG cluster-relation-departed File "/usr/lib/python2.7/dist-packages/keystoneclient/utils.py", line 318, in inner
2017-08-25 09:06:22 DEBUG cluster-relation-departed return func(*args, **kwargs)
2017-08-25 09:06:22 DEBUG cluster-relation-departed File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 251, in request
2017-08-25 09:06:22 DEBUG cluster-relation-departed raise exceptions.from_response(resp, method, url)
2017-08-25 09:06:22 DEBUG cluster-relation-departed keystoneclient.apiclient.exceptions.InternalServerError: An unexpected error prevented the server from fulfilling your request. (OperationalError) (2003, "Can't connect to MySQL server on '172.17.105.17' (113)") None None (HTTP 500)
2017-08-25 09:06:22 ERROR juju.worker.uniter.operation runhook.go:107 hook "cluster-relation-departed" failed: exit status 1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/498041

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/498041
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=97861e9c4646a30f8d4630a3d4e0b4daaec20d39
Submitter: Jenkins
Branch: master

commit 97861e9c4646a30f8d4630a3d4e0b4daaec20d39
Author: Billy Olsen <email address hidden>
Date: Fri Aug 25 11:10:14 2017 -0700

    Remove cluster-change-departed hook

    The cluster-change-departed hook is tied to the cluster-changed
    hook. In the cluster changed hook, there is logic to ensure the
    initial admin exists which makes calls to the keystone service.
    If the remote database has already been removed (as seen in recent
    CI runs), then this will cause the hook to fail execution.

    This is safe to remove as the primary purpose of the cluster-changed
    hook is to sync the SSL keys and update identity relation units.
    There should be no need to sync the keys when a unit is departing
    the cluster relation. Likewise, the update of the identity relations
    are not needed either as the VIP is used for access to the keystone
    services by remote units and the access credentials won't change.

    Change-Id: Id8fed284557f67f5676189ec8951b778cf506c97
    Closes-Bug: #1713108

[Ryan Beisner (1chb1n)]

After this change, seeing similar fail in a different hook, running the same test as above.

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_amulet_full/openstack/charm-hacluster/497660/1/473/consoleText.test_charm_amulet_full_534.txt

2017-08-26 17:42:32 DEBUG leader-settings-changed Traceback (most recent call last):
2017-08-26 17:42:32 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/leader-settings-changed", line 859, in <module>
2017-08-26 17:42:32 DEBUG leader-settings-changed main()
2017-08-26 17:42:32 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/leader-settings-changed", line 852, in main
2017-08-26 17:42:32 DEBUG leader-settings-changed hooks.execute(sys.argv)
2017-08-26 17:42:32 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/hookenv.py", line 768, in execute
2017-08-26 17:42:32 DEBUG leader-settings-changed self._hooks[hook_name]()
2017-08-26 17:42:32 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/leader-settings-changed", line 640, in leader_settings_changed
2017-08-26 17:42:32 DEBUG leader-settings-changed update_all_identity_relation_units()
2017-08-26 17:42:32 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/leader-settings-changed", line 347, in update_all_identity_relation_units
2017-08-26 17:42:32 DEBUG leader-settings-changed ensure_initial_admin(config)
2017-08-26 17:42:32 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1096, in ensure_initial_admin
2017-08-26 17:42:32 DEBUG leader-settings-changed return _ensure_initial_admin(config)
2017-08-26 17:42:32 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/decorators.py", line 40, in _retry_on_exception_inner_2
2017-08-26 17:42:32 DEBUG leader-settings-changed return f(*args, **kwargs)
2017-08-26 17:42:32 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1051, in _ensure_initial_admin
2017-08-26 17:42:32 DEBUG leader-settings-changed create_tenant("admin", DEFAULT_DOMAIN)
2017-08-26 17:42:32 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 804, in create_tenant
2017-08-26 17:42:32 DEBUG leader-settings-changed manager = get_manager()
2017-08-26 17:42:32 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 892, in get_manager
2017-08-26 17:42:32 DEBUG leader-settings-changed api_version)
2017-08-26 17:42:32 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/decorators.py", line 40, in _retry_on_exception_inner_2
2017-08-26 17:42:32 DEBUG leader-settings-changed return f(*args, **kwargs)
2017-08-26 17:42:32 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 75, in get_keystone_manager
2017-08-26 17:...