cs:~openstack-charmers-next/trusty/glance-24
liberty
I have two OSA runs where glance units fell over right away when calling connect_and_post():
unit-glance-1[958]: 2016-01-24 00:15:37 INFO unit.glance/1.juju-log server.go:268 identity-service:35: Connecting swift client...
unit-glance-1[958]: 2016-01-24 00:15:38 INFO unit.glance/1.identity-service-relation-changed logger.go:40 /usr/lib/python2.7/dist-packages/keystoneclient/service_catalog.py:196: UserWarning: Providing attr without filter_value to get_urls() is deprecated as of the 1.7.0 release and may be removed in the 2.0.0 release. Either both should be provided or neither should be provided.
unit-glance-1[958]: 2016-01-24 00:15:38 INFO unit.glance/1.identity-service-relation-changed logger.go:40 'Providing attr without filter_value to get_urls() is '
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 Traceback (most recent call last):
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-glance-1/charm/hooks/identity-service-relation-changed", line 534, in <module>
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 hooks.execute(sys.argv)
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-glance-1/charm/hooks/charmhelpers/core/hookenv.py", line 717, in execute
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 self._hooks[hook_name]()
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-glance-1/charm/hooks/charmhelpers/core/host.py", line 438, in wrapped_f
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 f(*args, **kwargs)
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-glance-1/charm/hooks/identity-service-relation-changed", line 327, in keystone_changed
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 object_store_joined()
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-glance-1/charm/hooks/charmhelpers/core/host.py", line 438, in wrapped_f
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 f(*args, **kwargs)
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-glance-1/charm/hooks/identity-service-relation-changed", line 247, in object_store_joined
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 [image_service_joined(rid) for rid in relation_ids('image-service')]
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-glance-1/charm/hooks/identity-service-relation-changed", line 227, in image_service_joined
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 'swift-temp-url-key': swift_temp_url_key(),
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-glance-1/charm/hooks/glance_utils.py", line 509, in swift_temp_url_key
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 return connect_and_post()
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-glance-1/charm/hooks/charmhelpers/core/decorators.py", line 42, in _retry_on_exception_inner_2
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 return f(*args, **kwargs)
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 File "/var/lib/juju/agents/unit-glance-1/charm/hooks/glance_utils.py", line 499, in connect_and_post
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 account_stats = swift_connection.head_account()
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1430, in head_account
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 return self._retry(None, head_account)
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1390, in _retry
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 service_token=self.service_token, **kwargs)
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 583, in head_account
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 conn.request(method, parsed.path, '', headers)
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 250, in request
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 files=files, **self.requests_args)
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 233, in _request
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 return self.request_session.request(*arg, **kwarg)
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 File "/usr/share/python-wheels/requests-2.2.1-py2.py3-none-any.whl/requests/sessions.py", line 455, in request
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 File "/usr/share/python-wheels/requests-2.2.1-py2.py3-none-any.whl/requests/sessions.py", line 558, in send
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 File "/usr/share/python-wheels/requests-2.2.1-py2.py3-none-any.whl/requests/adapters.py", line 378, in send
unit-glance-1[958]: 2016-01-24 00:16:09 INFO unit.glance/1.identity-service-relation-changed logger.go:40 requests.exceptions.ConnectionError: HTTPConnectionPool(host='10.96.13.16', port=8080): Max retries exceeded with url: /v1/AUTH_4594dd2216584ce29d972177639d5dfb (Caused by <class 'httplib.BadStatusLine'>: '')
unit-glance-1[958]: 2016-01-24 00:16:09 ERROR juju.worker.uniter.operation runhook.go:107 hook "identity-service-relation-changed" failed: exit status 1
connect_and_post() is decorated with @retry_on_exception(), but only for swiftclient.exceptions.ClientException. I'm assuming the ConnectionError above isn't one of those.
We can see the retry working on another unit of this same deployment until it works:
unit-glance-0[952]: 2016-01-24 00:20:25 INFO unit.glance/0.juju-log server.go:268 image-service:82: Connecting swift client...
unit-glance-0[952]: 2016-01-24 00:20:26 INFO unit.glance/0.image-service-relation-joined logger.go:40 /usr/lib/python2.7/dist-packages/keystoneclient/service_catalog.py:196: UserWarning: Providing attr without filter_value to get_urls() is deprecated as of the 1.7.0 release and may be removed in the 2.0.0 release. Either both should be provided or neither should be provided.
unit-glance-0[952]: 2016-01-24 00:20:26 INFO unit.glance/0.image-service-relation-joined logger.go:40 'Providing attr without filter_value to get_urls() is '
unit-glance-0[952]: 2016-01-24 00:20:59 INFO unit.glance/0.juju-log server.go:268 image-service:82: Retrying 'connect_and_post' 15 more times (delay=10)
unit-glance-0[952]: 2016-01-24 00:21:09 INFO unit.glance/0.juju-log server.go:268 image-service:82: Connecting swift client...
unit-glance-0[952]: 2016-01-24 00:21:41 INFO unit.glance/0.juju-log server.go:268 image-service:82: Retrying 'connect_and_post' 14 more times (delay=20)
unit-glance-0[952]: 2016-01-24 00:22:01 INFO unit.glance/0.juju-log server.go:268 image-service:82: Connecting swift client...
unit-glance-0[952]: 2016-01-24 00:22:33 INFO unit.glance/0.juju-log server.go:268 image-service:82: Retrying 'connect_and_post' 13 more times (delay=30)
unit-glance-0[952]: 2016-01-24 00:23:03 INFO unit.glance/0.juju-log server.go:268 image-service:82: Connecting swift client...
Can you please let us know what openstack release you ran on that bug? There are multiple supported Trusty OpenStack version combos. Thank you.