Juju Charms Collection
glance package

glance (next) identity-service-relation-changed hook fails when object-store not present

Bug #1536241 reported by Ryan Beisner on 2016-01-20
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
​ glance (Juju Charms Collection)
Fix Released
High
James Page
Juju Charms Collection 16.01
Also affects project (?) Also affects distribution/package Nominate for series

Bug Description

glance (next) identity-service-relation-changed hook fails (object-store not found)

I'm hitting this on 100% of o-c-t lxd/next.yaml test runs. It appears to be a race, because retrying later succeeds.

`juju resolved --retry glance/0` results in all-units ready and OK.

hook failed: "identity-service-relation-changed" for keystone:identity-service
...
2016-01-19 02:45:13 INFO identity-service-relation-changed swiftclient.exceptions.ClientException: Endpoint for object-store not found - have you specified a region?
...
2016-01-19 02:45:13 ERROR juju.worker.uniter.operation runhook.go:107 hook "identity-service-relation-changed" failed: exit status 1

# juju stat
http://pastebin.ubuntu.com/14582852/

# glance/0 status history
http://pastebin.ubuntu.com/14583722/

# keystone/0 status history
http://paste.ubuntu.com/14583733/

# glance unit
http://pastebin.ubuntu.com/14583061/
2016-01-19 02:45:12 INFO worker.uniter.jujuc server.go:172 running hook tool "juju-log" ["-l" "DEBUG" "Generating template context for identity-service"]
2016-01-19 02:45:12 DEBUG worker.uniter.jujuc server.go:173 hook context id "glance/0-identity-service-relation-changed-5172446555253973670"; dir "/var/lib/juju/agents/unit-glance-0/charm"
2016-01-19 02:45:12 DEBUG juju-log identity-service:24: Generating template context for identity-service
2016-01-19 02:45:13 INFO identity-service-relation-changed Traceback (most recent call last):
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 536, in <module>
2016-01-19 02:45:13 INFO identity-service-relation-changed hooks.execute(sys.argv)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/hookenv.py", line 717, in execute
2016-01-19 02:45:13 INFO identity-service-relation-changed self._hooks[hook_name]()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/host.py", line 438, in wrapped_f
2016-01-19 02:45:13 INFO identity-service-relation-changed f(*args, **kwargs)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 329, in keystone_changed
2016-01-19 02:45:13 INFO identity-service-relation-changed object_store_joined()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/host.py", line 438, in wrapped_f
2016-01-19 02:45:13 INFO identity-service-relation-changed f(*args, **kwargs)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 249, in object_store_joined
2016-01-19 02:45:13 INFO identity-service-relation-changed [image_service_joined(rid) for rid in relation_ids('image-service')]
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 229, in image_service_joined
2016-01-19 02:45:13 INFO identity-service-relation-changed 'swift-temp-url-key': swift_temp_url_key(),
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/glance_utils.py", line 485, in swift_temp_url_key
2016-01-19 02:45:13 INFO identity-service-relation-changed account_stats = swift_connection.head_account()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1430, in head_account
2016-01-19 02:45:13 INFO identity-service-relation-changed return self._retry(None, head_account)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1378, in _retry
2016-01-19 02:45:13 INFO identity-service-relation-changed self.url, self.token = self.get_auth()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1332, in get_auth
2016-01-19 02:45:13 INFO identity-service-relation-changed timeout=self.timeout)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 463, in get_auth
2016-01-19 02:45:13 INFO identity-service-relation-changed auth_version=auth_version)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 402, in get_auth_keystone
2016-01-19 02:45:13 INFO identity-service-relation-changed 'have you specified a region?' % service_type)
2016-01-19 02:45:13 INFO identity-service-relation-changed swiftclient.exceptions.ClientException: Endpoint for object-store not found - have you specified a region?
2016-01-19 02:45:13 INFO juju.worker.uniter.context context.go:579 handling reboot
2016-01-19 02:45:13 ERROR juju.worker.uniter.operation runhook.go:107 hook "identity-service-relation-changed" failed: exit status 1
2016-01-19 02:45:13 DEBUG juju.worker.uniter modes.go:31 [AGENT-STATUS] failed: run relation-changed (24; keystone/0) hook
2016-01-19 02:45:14 INFO juju.worker.uniter modes.go:569 ModeAbide exiting
2016-01-19 02:45:14 INFO juju.worker.uniter modes.go:567 ModeHookError starting
2016-01-19 02:45:14 DEBUG juju.worker.uniter.filter filter.go:601 want resolved event
2016-01-19 02:45:14 DEBUG juju.worker.uniter.filter filter.go:595 want forced upgrade true
2016-01-19 02:45:14 DEBUG juju.worker.uniter.filter filter.go:731 no new charm event
2016-01-19 02:45:14 DEBUG juju.worker.uniter modes.go:31 [AGENT-STATUS] error: hook failed: "identity-service-relation-changed"
2016-01-19 02:45:14 DEBUG juju.worker.leadership tracker.go:154 glance/0 got wait request for glance leadership loss
2016-01-19 02:45:14 DEBUG juju.worker.leadership tracker.go:248 glance/0 still has glance leadership

See original description

Related branches

lp:~james-page/charms/trusty/glance/lp1536241
Ryan Beisner (community): Needs Fixing on 2016-01-21
OpenStack Charmers: Pending requested 2016-01-21
Diff: 64 lines (+24/-16)
1 file modified
hooks/glance_utils.py (+24/-16)
lp:~1chb1n/charms/trusty/glance/next-lp1536241-addl
OpenStack Charmers: Pending requested 2016-01-21
Diff: 66 lines (+26/-16)
1 file modified
hooks/glance_utils.py (+26/-16)
Ryan Beisner (1chb1n) on 2016-01-20
description: updated
Ryan Beisner (1chb1n) wrote :

I've substituted the stable glance charm in the o-c-t lxd/next.yaml, and it deploys consistently and successfully. Something in glance/next appears to be causing this.

Ryan Beisner (1chb1n) wrote :

It looks like swift_temp_url_key() or image_service_joined() may need some sort of retry resilience.

http://bazaar.launchpad.net/~openstack-charmers/charms/trusty/glance/next/revision/152

Ryan Beisner (1chb1n) on 2016-01-20
description: updated
description: updated
James Page (james-page) on 2016-01-21
Changed in glance (Juju Charms Collection):
status: New → In Progress
importance: Undecided → High
assignee: nobody → James Page (james-page)
milestone: none → 16.01
Ryan Beisner (1chb1n) wrote :

FYI - Exercising the next/lxd.yaml bundle, substituting James' proposed glance/next branch...

Ryan Beisner (1chb1n) wrote :
Download full text (3.6 KiB)

Proposed glance/next errors out on:
AttributeError: 'module' object has no attribute 'exceptions'

...

2016-01-21 18:27:57 INFO identity-service-relation-changed Traceback (most recent call last):
2016-01-21 18:27:57 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 534, in <module>
2016-01-21 18:27:57 INFO identity-service-relation-changed hooks.execute(sys.argv)
2016-01-21 18:27:57 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/hookenv.py", line 717, in execute
2016-01-21 18:27:57 INFO identity-service-relation-changed self._hooks[hook_name]()
2016-01-21 18:27:57 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/host.py", line 438, in wrapped_f
2016-01-21 18:27:57 INFO identity-service-relation-changed f(*args, **kwargs)
2016-01-21 18:27:57 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 327, in keystone_changed
2016-01-21 18:27:57 INFO identity-service-relation-changed object_store_joined()
2016-01-21 18:27:57 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/host.py", line 438, in wrapped_f
2016-01-21 18:27:57 INFO identity-service-relation-changed f(*args, **kwargs)
2016-01-21 18:27:57 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 247, in object_store_joined
2016-01-21 18:27:57 INFO identity-service-relation-changed [image_service_joined(rid) for rid in relation_ids('image-service')]
2016-01-21 18:27:57 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 227, in image_service_joined
2016-01-21 18:27:57 INFO identity-service-relation-changed 'swift-temp-url-key': swift_temp_url_key(),
2016-01-21 18:27:57 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/glance_utils.py", line 489, in swift_temp_url_key
2016-01-21 18:27:57 INFO identity-service-relation-changed exc_type=client.exceptions.ClientException)
2016-01-21 18:27:57 INFO identity-service-relation-changed AttributeError: 'module' object has no attribute 'exceptions'
2016-01-21 18:27:57 ERROR juju.worker.uniter.operation runhook.go:107 hook "identity-service-relation-changed" failed: exit status 1

ubuntu@beisner-bastion:~/bzr/openstack-charm-testing$ bzr info wily/glance
Lightweight checkout (format: 2a)
Location:
  light checkout root: wily/glance
   checkout of branch: bzr+ssh://bazaar.launchpad.net/~james-page/charms/trusty/glance/lp1536241/
    shared repository: bzr+ssh://bazaar.launchpad.net/~james-page/charms/trusty/glance/lp1536241/
             ...

Read more...

Ryan Beisner (1chb1n) wrote :

FYI, see my piggy-back which includes these changes and resolves the import error.

https://code.launchpad.net/~1chb1n/charms/trusty/glance/next-lp1536241-addl/+merge/283540

...
glance/0 active idle 1.25.3 2 9292/tcp 172.18.99.102 Unit is ready
...

James Page (james-page) wrote :

Ryan

Thanks for fixing my bad code - landed into next!

Changed in glance (Juju Charms Collection):
status: In Progress → Fix Committed
tags: added: openstack
James Page (james-page) on 2016-01-28
Changed in glance (Juju Charms Collection):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.
Subscribing...

Other bug subscribers