Juju Charms Collection
glance package

glance (next) identity-service-relation-changed hook fails when object-store not present

Bug #1536241 reported by Ryan Beisner on 2016-01-20

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	glance (Juju Charms Collection)	Fix Released	High	James Page	Juju Charms Collection 16.01

Bug Description

glance (next) identity-service-relation-changed hook fails (object-store not found)

I'm hitting this on 100% of o-c-t lxd/next.yaml test runs. It appears to be a race, because retrying later succeeds.

`juju resolved --retry glance/0` results in all-units ready and OK.

hook failed: "identity-service-relation-changed" for keystone:identity-service
...
2016-01-19 02:45:13 INFO identity-service-relation-changed swiftclient.exceptions.ClientException: Endpoint for object-store not found - have you specified a region?
...
2016-01-19 02:45:13 ERROR juju.worker.uniter.operation runhook.go:107 hook "identity-service-relation-changed" failed: exit status 1

# juju stat
http://pastebin.ubuntu.com/14582852/

# glance/0 status history
http://pastebin.ubuntu.com/14583722/

# keystone/0 status history
http://paste.ubuntu.com/14583733/

# glance unit
http://pastebin.ubuntu.com/14583061/
2016-01-19 02:45:12 INFO worker.uniter.jujuc server.go:172 running hook tool "juju-log" ["-l" "DEBUG" "Generating template context for identity-service"]
2016-01-19 02:45:12 DEBUG worker.uniter.jujuc server.go:173 hook context id "glance/0-identity-service-relation-changed-5172446555253973670"; dir "/var/lib/juju/agents/unit-glance-0/charm"
2016-01-19 02:45:12 DEBUG juju-log identity-service:24: Generating template context for identity-service
2016-01-19 02:45:13 INFO identity-service-relation-changed Traceback (most recent call last):
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 536, in <module>
2016-01-19 02:45:13 INFO identity-service-relation-changed hooks.execute(sys.argv)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/hookenv.py", line 717, in execute
2016-01-19 02:45:13 INFO identity-service-relation-changed self._hooks[hook_name]()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/host.py", line 438, in wrapped_f
2016-01-19 02:45:13 INFO identity-service-relation-changed f(*args, **kwargs)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 329, in keystone_changed
2016-01-19 02:45:13 INFO identity-service-relation-changed object_store_joined()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/host.py", line 438, in wrapped_f
2016-01-19 02:45:13 INFO identity-service-relation-changed f(*args, **kwargs)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 249, in object_store_joined
2016-01-19 02:45:13 INFO identity-service-relation-changed [image_service_joined(rid) for rid in relation_ids('image-service')]
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 229, in image_service_joined
2016-01-19 02:45:13 INFO identity-service-relation-changed 'swift-temp-url-key': swift_temp_url_key(),
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/glance_utils.py", line 485, in swift_temp_url_key
2016-01-19 02:45:13 INFO identity-service-relation-changed account_stats = swift_connection.head_account()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1430, in head_account
2016-01-19 02:45:13 INFO identity-service-relation-changed return self._retry(None, head_account)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1378, in _retry
2016-01-19 02:45:13 INFO identity-service-relation-changed self.url, self.token = self.get_auth()
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1332, in get_auth
2016-01-19 02:45:13 INFO identity-service-relation-changed timeout=self.timeout)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 463, in get_auth
2016-01-19 02:45:13 INFO identity-service-relation-changed auth_version=auth_version)
2016-01-19 02:45:13 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 402, in get_auth_keystone
2016-01-19 02:45:13 INFO identity-service-relation-changed 'have you specified a region?' % service_type)
2016-01-19 02:45:13 INFO identity-service-relation-changed swiftclient.exceptions.ClientException: Endpoint for object-store not found - have you specified a region?
2016-01-19 02:45:13 INFO juju.worker.uniter.context context.go:579 handling reboot
2016-01-19 02:45:13 ERROR juju.worker.uniter.operation runhook.go:107 hook "identity-service-relation-changed" failed: exit status 1
2016-01-19 02:45:13 DEBUG juju.worker.uniter modes.go:31 [AGENT-STATUS] failed: run relation-changed (24; keystone/0) hook
2016-01-19 02:45:14 INFO juju.worker.uniter modes.go:569 ModeAbide exiting
2016-01-19 02:45:14 INFO juju.worker.uniter modes.go:567 ModeHookError starting
2016-01-19 02:45:14 DEBUG juju.worker.uniter.filter filter.go:601 want resolved event
2016-01-19 02:45:14 DEBUG juju.worker.uniter.filter filter.go:595 want forced upgrade true
2016-01-19 02:45:14 DEBUG juju.worker.uniter.filter filter.go:731 no new charm event
2016-01-19 02:45:14 DEBUG juju.worker.uniter modes.go:31 [AGENT-STATUS] error: hook failed: "identity-service-relation-changed"
2016-01-19 02:45:14 DEBUG juju.worker.leadership tracker.go:154 glance/0 got wait request for glance leadership loss
2016-01-19 02:45:14 DEBUG juju.worker.leadership tracker.go:248 glance/0 still has glance leadership

See original description

Tags:

Related branches

lp:~james-page/charms/trusty/glance/lp1536241

Rejected for merging into lp:~openstack-charmers-archive/charms/trusty/glance/next

Ryan Beisner (community): Needs Fixing on 2016-01-21

OpenStack Charmers: Pending requested 2016-01-21

lp:~1chb1n/charms/trusty/glance/next-lp1536241-addl

Merged into lp:~openstack-charmers-archive/charms/trusty/glance/next at revision 164

OpenStack Charmers: Pending requested 2016-01-21

Ryan Beisner (1chb1n) on 2016-01-20

description:

updated

Revision history for this message

Ryan Beisner (1chb1n) wrote on 2016-01-20:

I've substituted the stable glance charm in the o-c-t lxd/next.yaml, and it deploys consistently and successfully. Something in glance/next appears to be causing this.

Revision history for this message

Ryan Beisner (1chb1n) wrote on 2016-01-20:

It looks like swift_temp_url_key() or image_service_joined() may need some sort of retry resilience.

http://bazaar.launchpad.net/~openstack-charmers/charms/trusty/glance/next/revision/152

Ryan Beisner (1chb1n) on 2016-01-20

description:	updated
description:	updated

James Page (james-page) on 2016-01-21

Changed in glance (Juju Charms Collection):
status:	New → In Progress
importance:	Undecided → High
assignee:	nobody → James Page (james-page)
milestone:	none → 16.01

Revision history for this message

Ryan Beisner (1chb1n) wrote on 2016-01-21:

FYI - Exercising the next/lxd.yaml bundle, substituting James' proposed glance/next branch...

Revision history for this message

Ryan Beisner (1chb1n) wrote on 2016-01-21:

Download full text (3.6 KiB)

Proposed glance/next errors out on:
AttributeError: 'module' object has no attribute 'exceptions'

...

2016-01-21 18:27:57 INFO identity-service-relation-changed Traceback (most recent call last):
2016-01-21 18:27:57 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 534, in <module>
2016-01-21 18:27:57 INFO identity-service-relation-changed hooks.execute(sys.argv)
2016-01-21 18:27:57 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/hookenv.py", line 717, in execute
2016-01-21 18:27:57 INFO identity-service-relation-changed self._hooks[hook_name]()
2016-01-21 18:27:57 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/host.py", line 438, in wrapped_f
2016-01-21 18:27:57 INFO identity-service-relation-changed f(*args, **kwargs)
2016-01-21 18:27:57 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 327, in keystone_changed
2016-01-21 18:27:57 INFO identity-service-relation-changed object_store_joined()
2016-01-21 18:27:57 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/host.py", line 438, in wrapped_f
2016-01-21 18:27:57 INFO identity-service-relation-changed f(*args, **kwargs)
2016-01-21 18:27:57 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 247, in object_store_joined
2016-01-21 18:27:57 INFO identity-service-relation-changed [image_service_joined(rid) for rid in relation_ids('image-service')]
2016-01-21 18:27:57 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 227, in image_service_joined
2016-01-21 18:27:57 INFO identity-service-relation-changed 'swift-temp-url-key': swift_temp_url_key(),
2016-01-21 18:27:57 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/glance_utils.py", line 489, in swift_temp_url_key
2016-01-21 18:27:57 INFO identity-service-relation-changed exc_type=client.exceptions.ClientException)
2016-01-21 18:27:57 INFO identity-service-relation-changed AttributeError: 'module' object has no attribute 'exceptions'
2016-01-21 18:27:57 ERROR juju.worker.uniter.operation runhook.go:107 hook "identity-service-relation-changed" failed: exit status 1

Proposed glance/next errors out on:
AttributeError: 'module' object has no attribute 'exceptions'

...

2016-01-21 18:27:57 INFO identity-service-relation-changed Traceback (most recent call last):
2016-01-21 18:27:57 INFO identity-service-relation-changed   File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 534, in <module>
2016-01-21 18:27:57 INFO identity-service-relation-changed     hooks.execute(sys.argv)
2016-01-21 18:27:57 INFO identity-service-relation-changed   File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/hookenv.py", line 717, in execute
2016-01-21 18:27:57 INFO identity-service-relation-changed     self._hooks[hook_name]()
2016-01-21 18:27:57 INFO identity-service-relation-changed   File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/host.py", line 438, in wrapped_f
2016-01-21 18:27:57 INFO identity-service-relation-changed     f(*args, **kwargs)
2016-01-21 18:27:57 INFO identity-service-relation-changed   File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 327, in keystone_changed
2016-01-21 18:27:57 INFO identity-service-relation-changed     object_store_joined()
2016-01-21 18:27:57 INFO identity-service-relation-changed   File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/host.py", line 438, in wrapped_f
2016-01-21 18:27:57 INFO identity-service-relation-changed     f(*args, **kwargs)
2016-01-21 18:27:57 INFO identity-service-relation-changed   File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 247, in object_store_joined
2016-01-21 18:27:57 INFO identity-service-relation-changed     [image_service_joined(rid) for rid in relation_ids('image-service')]
2016-01-21 18:27:57 INFO identity-service-relation-changed   File "/var/lib/juju/agents/unit-glance-0/charm/hooks/identity-service-relation-changed", line 227, in image_service_joined
2016-01-21 18:27:57 INFO identity-service-relation-changed     'swift-temp-url-key': swift_temp_url_key(),
2016-01-21 18:27:57 INFO identity-service-relation-changed   File "/var/lib/juju/agents/unit-glance-0/charm/hooks/glance_utils.py", line 489, in swift_temp_url_key
2016-01-21 18:27:57 INFO identity-service-relation-changed     exc_type=client.exceptions.ClientException)
2016-01-21 18:27:57 INFO identity-service-relation-changed AttributeError: 'module' object has no attribute 'exceptions'
2016-01-21 18:27:57 ERROR juju.worker.uniter.operation runhook.go:107 hook "identity-service-relation-changed" failed: exit status 1

ubuntu@beisner-bastion:~/bzr/openstack-charm-testing$ bzr info wily/glance
Lightweight checkout (format: 2a)                                                                                                                                                                                                                                         
Location:
  light checkout root: wily/glance
   checkout of branch: bzr+ssh://bazaar.launchpad.net/~james-page/charms/trusty/glance/lp1536241/
    shared repository: bzr+ssh://bazaar.launchpad.net/~james-page/charms/trusty/glance/lp1536241/
                                                                                                                                                                                                                                                                          
Related branches:
  parent branch: bzr+ssh://bazaar.launchpad.net/~openstack-charmers/charms/trusty/glance/next/
     stacked on: /+branch-id/905405

ubuntu@beisner-bastion:~/bzr/openstack-charm-testing$ bzr revno wily/glance
163

Revision history for this message

Ryan Beisner (1chb1n) wrote on 2016-01-21:

FYI, see my piggy-back which includes these changes and resolves the import error.

https://code.launchpad.net/~1chb1n/charms/trusty/glance/next-lp1536241-addl/+merge/283540

...
glance/0 active idle 1.25.3 2 9292/tcp 172.18.99.102 Unit is ready
...

Revision history for this message

James Page (james-page) wrote on 2016-01-22:

Ryan

Thanks for fixing my bad code - landed into next!

Changed in glance (Juju Charms Collection):
status:	In Progress → Fix Committed
tags:	added: openstack

James Page (james-page) on 2016-01-28

Changed in glance (Juju Charms Collection):
status:	Fix Committed → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Juju Charms Collectionglance package

glance (next) identity-service-relation-changed hook fails when object-store not present

Bug Description

Related branches

Other bug subscribers

Remote bug watches

Juju Charms Collection
glance package