There is a race condition where SSL certificates for the local unit have not been generated before apache attempts to start. Apache then fails to start. When shared-db-relation-changed runs on the leader unit it cannot authenticate with itself leading to:
Traceback (most recent call last):
File "hooks/shared-db-relation-changed", line 852, in <module>
main()
File "hooks/shared-db-relation-changed", line 845, in main
hooks.execute(sys.argv)
File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/hookenv.py", line 731, in execute
self._hooks[hook_name]()
File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1931, in wrapped_f
restart_functions)
File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/host.py", line 659, in restart_on_change_helper
r = lambda_f()
File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1930, in <lambda>
(lambda: f(*args, **kwargs)), restart_map, stopstart,
File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1624, in inner_synchronize_ca_if_changed2
return f(*args, **kwargs)
File "hooks/shared-db-relation-changed", line 408, in db_changed
leader_init_db_if_ready(use_current_context=True)
File "hooks/shared-db-relation-changed", line 382, in leader_init_db_if_ready
update_all_identity_relation_units(check_db_ready=False)
File "hooks/shared-db-relation-changed", line 344, in update_all_identity_relation_units
ensure_initial_admin(config)
File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1094, in ensure_initial_admin
return _ensure_initial_admin(config)
File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/decorators.py", line 40, in _retry_on_exception_inner_2
return f(*args, **kwargs)
File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1049, in _ensure_initial_admin
create_tenant("admin", DEFAULT_DOMAIN)
File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 802, in create_tenant
manager = get_manager()
File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 890, in get_manager
api_version)
File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/decorators.py", line 40, in _retry_on_exception_inner_2
return f(*args, **kwargs)
File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 75, in get_keystone_manager
for svc in manager.api.services.list():
File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/services.py", line 32, in list
return self._list("/OS-KSADM/services", "OS-KSADM:services")
File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 124, in _list
resp, body = self.client.get(url, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 173, in get
return self.request(url, 'GET', **kwargs)
File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 331, in request
resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 98, in request
return self.session.request(url, method, **kwargs)
File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
return func(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 405, in request
resp = send(**kwargs)
File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 449, in _send_request
raise exceptions.ConnectionRefused(msg)
keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://localhost:35337/v2.0/OS-KSADM/services
The condition occurs when the juju leader is not the elected ssl-cert-master. This may be where the fix needs to occur.
Ignore the bit about the leader disparity.
Between stable and current master this broke. I am focused on
https:/
And specifically, the gate for send_ssl_
https:/
Removing the gate seems to fix things but I would like to get Ed Hope-Morley's input as this was his code.