Deploying HA configuration at the same time as adding all relations fails

[James Page]

Deploying a clustered keystone service at the same time as adding all relations causes hook execution failures in the identity-changed hook; This is because the cluster-changed hooks don't get fired prior to the identity-changed hooks, so the charm looks for the local endpoint on the wrong port (keystone has not been configured to listen on default port -10):

2014-11-16 19:45:21 INFO juju-log cluster:3: Creating requested role: Member
2014-11-16 19:45:21 INFO cluster-relation-changed Traceback (most recent call last):
2014-11-16 19:45:21 INFO cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-3/charm/hooks/cluster-relation-changed", line 380, in <module>
2014-11-16 19:45:21 INFO cluster-relation-changed main()
2014-11-16 19:45:21 INFO cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-3/charm/hooks/cluster-relation-changed", line 374, in main
2014-11-16 19:45:21 INFO cluster-relation-changed hooks.execute(sys.argv)
2014-11-16 19:45:21 INFO cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-3/charm/hooks/charmhelpers/core/hookenv.py", line 508, in execute
2014-11-16 19:45:21 INFO cluster-relation-changed self._hooks[hook_name]()
2014-11-16 19:45:21 INFO cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-3/charm/hooks/charmhelpers/core/host.py", line 268, in wrapped_f
2014-11-16 19:45:21 INFO cluster-relation-changed f(*args)
2014-11-16 19:45:21 INFO cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-3/charm/hooks/cluster-relation-changed", line 251, in cluster_changed
2014-11-16 19:45:21 INFO cluster-relation-changed remote_unit=unit)
2014-11-16 19:45:21 INFO cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-3/charm/hooks/cluster-relation-changed", line 202, in identity_changed
2014-11-16 19:45:21 INFO cluster-relation-changed add_service_to_keystone(relation_id, remote_unit)
2014-11-16 19:45:21 INFO cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-3/charm/hooks/keystone_utils.py", line 664, in add_service_to_keystone
2014-11-16 19:45:21 INFO cluster-relation-changed create_role(role)
2014-11-16 19:45:21 INFO cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-3/charm/hooks/keystone_utils.py", line 425, in create_role
2014-11-16 19:45:21 INFO cluster-relation-changed roles = [r._info for r in manager.api.roles.list()]
2014-11-16 19:45:21 INFO cluster-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/roles.py", line 47, in list
2014-11-16 19:45:21 INFO cluster-relation-changed return self._list("/OS-KSADM/roles", "roles")
2014-11-16 19:45:21 INFO cluster-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 112, in _list
2014-11-16 19:45:21 INFO cluster-relation-changed resp, body = self.client.get(url, **kwargs)
2014-11-16 19:45:21 INFO cluster-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 611, in get
2014-11-16 19:45:21 INFO cluster-relation-changed return self._cs_request(url, 'GET', **kwargs)
2014-11-16 19:45:21 INFO cluster-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 603, in _cs_request
2014-11-16 19:45:21 INFO cluster-relation-changed return self.request(url, method, **kwargs)
2014-11-16 19:45:21 INFO cluster-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 578, in request
2014-11-16 19:45:21 INFO cluster-relation-changed resp = super(HTTPClient, self).request(url, method, **kwargs)
2014-11-16 19:45:21 INFO cluster-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/baseclient.py", line 21, in request
2014-11-16 19:45:21 INFO cluster-relation-changed return self.session.request(url, method, **kwargs)
2014-11-16 19:45:21 INFO cluster-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/utils.py", line 318, in inner
2014-11-16 19:45:21 INFO cluster-relation-changed return func(*args, **kwargs)
2014-11-16 19:45:21 INFO cluster-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 324, in request
2014-11-16 19:45:21 INFO cluster-relation-changed resp = self._send_request(url, method, redirect, log, **kwargs)
2014-11-16 19:45:21 INFO cluster-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 359, in _send_request
2014-11-16 19:45:21 INFO cluster-relation-changed raise exceptions.ConnectionRefused(msg)
2014-11-16 19:45:21 INFO cluster-relation-changed keystoneclient.openstack.common.apiclient.exceptions.ConnectionRefused: Unable to establish connection to http://localhost:35347/v2.0/OS-KSADM/roles
2014-11-16 19:45:21 ERROR juju.worker.uniter uniter.go:486 hook failed: exit status 1

The best workaround I've found is to deploy a single keystone unit, wait for all relation hook events to complete, and then add another two units to complete clustering configuration.

We should probably make the function that determines the port check actual ports, rather than guess based on relation state - or maybe even query the keystone.conf file.

[Chad Smith (chad.smith)]

yep just hit this 4 times in a row using the autopilot. Closing lp:1413857 as duplicate

[Edward Hope-Morley (hopem)]

I think this will be resolved by the fix for bug 1287294. Please re-check this once 15.01 is released and close if resolved.

[Adam Collard (adam-collard)]

Confirmed that we haven't seen this since 15.01