Comment 0 for bug 1536241

glance (next) identity-service-relation-changed hook fails (object-store not found)

I'm hitting this on 100% of o-c-t lxd/next.yaml test runs. It appears to be a race, because retrying later succeeds.

`juju resolved --retry glance/0` results in all-units ready and OK.

hook failed: "identity-service-relation-changed" for keystone:identity-service
...
2016-01-19 02:45:13 INFO identity-service-relation-changed swiftclient.exceptions.ClientException: Endpoint for object-store not found - have you specified a region?
...
2016-01-19 02:45:13 ERROR juju.worker.uniter.operation runhook.go:107 hook "identity-service-relation-changed" failed: exit status 1

# juju stat
http://pastebin.ubuntu.com/14582852/

# glance unit
2016-01-19 02:45:12 INFO worker.uniter.jujuc server.go:172 running hook tool "juju-log" ["-l" "DEBUG" "Generating template context for identity-service"]
2016-01-19 02:45:12 DEBUG worker.uniter.jujuc server.go:173 hook context id "glance/0-identity-service-relation-changed-5172446555253973670"; dir "/var/lib/juju/agents/unit-glance-0/charm"
2016-01-19 02:45:12 DEBUG juju-log identity-service:24: Generating template context for identity-service
2016-01-19 02:45:13 INFO identity-service-relation-changed Traceback (most recent call last):
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 536, in <module>
2016-01-19 02:45:13 INFO identity-service-relation-changed hooks.execute(sys.argv)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/hookenv.py", line 717, in execute
2016-01-19 02:45:13 INFO identity-service-relation-changed self._hooks[hook_name]()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/host.py", line 438, in wrapped_f
2016-01-19 02:45:13 INFO identity-service-relation-changed f(*args, **kwargs)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 329, in keystone_changed
2016-01-19 02:45:13 INFO identity-service-relation-changed object_store_joined()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/host.py", line 438, in wrapped_f
2016-01-19 02:45:13 INFO identity-service-relation-changed f(*args, **kwargs)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 249, in object_store_joined
2016-01-19 02:45:13 INFO identity-service-relation-changed [image_service_joined(rid) for rid in relation_ids('image-service')]
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 229, in image_service_joined
2016-01-19 02:45:13 INFO identity-service-relation-changed 'swift-temp-url-key': swift_temp_url_key(),
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/glance_utils.py", line 485, in swift_temp_url_key
2016-01-19 02:45:13 INFO identity-service-relation-changed account_stats = swift_connection.head_account()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1430, in head_account
2016-01-19 02:45:13 INFO identity-service-relation-changed return self._retry(None, head_account)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1378, in _retry
2016-01-19 02:45:13 INFO identity-service-relation-changed self.url, self.token = self.get_auth()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1332, in get_auth
2016-01-19 02:45:13 INFO identity-service-relation-changed timeout=self.timeout)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 463, in get_auth
2016-01-19 02:45:13 INFO identity-service-relation-changed auth_version=auth_version)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 402, in get_auth_keystone
2016-01-19 02:45:13 INFO identity-service-relation-changed 'have you specified a region?' % service_type)
2016-01-19 02:45:13 INFO identity-service-relation-changed swiftclient.exceptions.ClientException: Endpoint for object-store not found - have you specified a region?
2016-01-19 02:45:13 INFO juju.worker.uniter.context context.go:579 handling reboot
2016-01-19 02:45:13 ERROR juju.worker.uniter.operation runhook.go:107 hook "identity-service-relation-changed" failed: exit status 1
2016-01-19 02:45:13 DEBUG juju.worker.uniter modes.go:31 [AGENT-STATUS] failed: run relation-changed (24; keystone/0) hook
2016-01-19 02:45:14 INFO juju.worker.uniter modes.go:569 ModeAbide exiting
2016-01-19 02:45:14 INFO juju.worker.uniter modes.go:567 ModeHookError starting
2016-01-19 02:45:14 DEBUG juju.worker.uniter.filter filter.go:601 want resolved event
2016-01-19 02:45:14 DEBUG juju.worker.uniter.filter filter.go:595 want forced upgrade true
2016-01-19 02:45:14 DEBUG juju.worker.uniter.filter filter.go:731 no new charm event
2016-01-19 02:45:14 DEBUG juju.worker.uniter modes.go:31 [AGENT-STATUS] error: hook failed: "identity-service-relation-changed"
2016-01-19 02:45:14 DEBUG juju.worker.leadership tracker.go:154 glance/0 got wait request for glance leadership loss
2016-01-19 02:45:14 DEBUG juju.worker.leadership tracker.go:248 glance/0 still has glance leadership