OpenStack Compute (Nova)

nova gives up on glance connection failures too easily

Reported by Johannes Erdfelt on 2012-02-10
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Undecided
Johannes Erdfelt

Bug Description

nova doesn't retry when connections to glance fail because it is being restarted.

Services can restart at any time and we should be more tolerant of that and retry for a configurable period of time.

2012-02-10 12:25:05,849 ERROR nova.rpc [-] Exception during message handling
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/usr/lib/python2.6/dist-packages/nova/rpc/amqp.py", line 249, in _process_data
(nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc): TRACE: File "/usr/lib/python2.6/dist-packages/nova/exception.py", line 126, in wrapped
(nova.rpc): TRACE: return f(*args, **kw)
(nova.rpc): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/manager.py", line 150, in decorated_function
(nova.rpc): TRACE: function(self, context, instance_uuid, *args, **kwargs)
(nova.rpc): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/manager.py", line 173, in decorated_function
(nova.rpc): TRACE: self.add_instance_fault_from_exc(context, instance_uuid, e)
(nova.rpc): TRACE: File "/usr/lib/python2.6/contextlib.py", line 23, in __exit__
(nova.rpc): TRACE: self.gen.next()
(nova.rpc): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/manager.py", line 168, in decorated_function
(nova.rpc): TRACE: return function(self, context, instance_uuid, *args, **kwargs)
(nova.rpc): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/manager.py", line 1309, in finish_resize
(nova.rpc): TRACE: image_meta = _get_image_meta(context, instance_ref['image_ref'])
(nova.rpc): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/manager.py", line 180, in _get_image_meta
(nova.rpc): TRACE: return image_service.show(context, image_id)
(nova.rpc): TRACE: File "/usr/lib/python2.6/dist-packages/nova/image/glance.py", line 218, in show
(nova.rpc): TRACE: image_meta = self._get_client(context).get_image_meta(image_id)
(nova.rpc): TRACE: File "/usr/lib/python2.6/dist-packages/glance/client.py", line 97, in get_image_meta
(nova.rpc): TRACE: res = self.do_request("HEAD", "/images/%s" % image_id)
(nova.rpc): TRACE: File "/usr/lib/python2.6/dist-packages/glance/common/client.py", line 51, in wrapped
(nova.rpc): TRACE: return func(self, *args, **kwargs)
(nova.rpc): TRACE: File "/usr/lib/python2.6/dist-packages/glance/common/client.py", line 341, in do_request
(nova.rpc): TRACE: headers=headers)
(nova.rpc): TRACE: File "/usr/lib/python2.6/dist-packages/glance/common/client.py", line 68, in wrapped
(nova.rpc): TRACE: return func(self, method, url, body, headers)
(nova.rpc): TRACE: File "/usr/lib/python2.6/dist-packages/glance/common/client.py", line 463, in _do_request
(nova.rpc): TRACE: raise exception.ClientConnectionError(e)
(nova.rpc): TRACE: ClientConnectionError: There was an error connecting to a server
(nova.rpc): TRACE: Details: [Errno 111] ECONNREFUSED
(nova.rpc): TRACE:

summary: - nova gives up glance connection failures too easily
+ nova gives up on glance connection failures too easily
Changed in nova:
assignee: nobody → Johannes Erdfelt (johannes.erdfelt)

Fix proposed to branch: master
Review: https://review.openstack.org/4013

Changed in nova:
status: New → In Progress

Reviewed: https://review.openstack.org/4013
Committed: http://github.com/openstack/nova/commit/9974dcbd304a4e092e952e937f4dc78f9500e304
Submitter: Jenkins
Branch: master

commit 9974dcbd304a4e092e952e937f4dc78f9500e304
Author: Johannes Erdfelt <email address hidden>
Date: Fri Feb 10 17:40:10 2012 +0000

    Extend glance retries to show() as well

    Fixes bug 930245

    nova/image/glance.py would already retry get() requests if there was a
    connection error, but didn't do so for show() requests.

    Change-Id: Ifb1a17da18b7e10ddaaec46e124efcc963681c80

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2012-02-29
Changed in nova:
milestone: none → essex-4
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2012-04-05
Changed in nova:
milestone: essex-4 → 2012.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers