nova gives up on glance connection failures too easily

[Johannes Erdfelt]

nova doesn't retry when connections to glance fail because it is being restarted.

Services can restart at any time and we should be more tolerant of that and retry for a configurable period of time.

2012-02-10 12:25:05,849 ERROR nova.rpc [-] Exception during message handling
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/usr/lib/python2.6/dist-packages/nova/rpc/amqp.py", line 249, in _process_data
(nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc): TRACE: File "/usr/lib/python2.6/dist-packages/nova/exception.py", line 126, in wrapped
(nova.rpc): TRACE: return f(*args, **kw)
(nova.rpc): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/manager.py", line 150, in decorated_function
(nova.rpc): TRACE: function(self, context, instance_uuid, *args, **kwargs)
(nova.rpc): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/manager.py", line 173, in decorated_function
(nova.rpc): TRACE: self.add_instance_fault_from_exc(context, instance_uuid, e)
(nova.rpc): TRACE: File "/usr/lib/python2.6/contextlib.py", line 23, in __exit__
(nova.rpc): TRACE: self.gen.next()
(nova.rpc): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/manager.py", line 168, in decorated_function
(nova.rpc): TRACE: return function(self, context, instance_uuid, *args, **kwargs)
(nova.rpc): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/manager.py", line 1309, in finish_resize
(nova.rpc): TRACE: image_meta = _get_image_meta(context, instance_ref['image_ref'])
(nova.rpc): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/manager.py", line 180, in _get_image_meta
(nova.rpc): TRACE: return image_service.show(context, image_id)
(nova.rpc): TRACE: File "/usr/lib/python2.6/dist-packages/nova/image/glance.py", line 218, in show
(nova.rpc): TRACE: image_meta = self._get_client(context).get_image_meta(image_id)
(nova.rpc): TRACE: File "/usr/lib/python2.6/dist-packages/glance/client.py", line 97, in get_image_meta
(nova.rpc): TRACE: res = self.do_request("HEAD", "/images/%s" % image_id)
(nova.rpc): TRACE: File "/usr/lib/python2.6/dist-packages/glance/common/client.py", line 51, in wrapped
(nova.rpc): TRACE: return func(self, *args, **kwargs)
(nova.rpc): TRACE: File "/usr/lib/python2.6/dist-packages/glance/common/client.py", line 341, in do_request
(nova.rpc): TRACE: headers=headers)
(nova.rpc): TRACE: File "/usr/lib/python2.6/dist-packages/glance/common/client.py", line 68, in wrapped
(nova.rpc): TRACE: return func(self, method, url, body, headers)
(nova.rpc): TRACE: File "/usr/lib/python2.6/dist-packages/glance/common/client.py", line 463, in _do_request
(nova.rpc): TRACE: raise exception.ClientConnectionError(e)
(nova.rpc): TRACE: ClientConnectionError: There was an error connecting to a server
(nova.rpc): TRACE: Details: [Errno 111] ECONNREFUSED
(nova.rpc): TRACE:

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/4013

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/4013
Committed: http://github.com/openstack/nova/commit/9974dcbd304a4e092e952e937f4dc78f9500e304
Submitter: Jenkins
Branch: master

commit 9974dcbd304a4e092e952e937f4dc78f9500e304
Author: Johannes Erdfelt <email address hidden>
Date: Fri Feb 10 17:40:10 2012 +0000

    Extend glance retries to show() as well

    Fixes bug 930245

    nova/image/glance.py would already retry get() requests if there was a
    connection error, but didn't do so for show() requests.

    Change-Id: Ifb1a17da18b7e10ddaaec46e124efcc963681c80