OpenStack Compute (Nova)

Token expiration during resize causes vm to go to ERROR state

Reported by Mark Washenberger on 2012-02-29
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
High
Mark Washenberger

Bug Description

Sample traceback

2012-02-23 16:27:51,425 ERROR nova.compute.manager [-] Missing required credential: p
assword. Setting instance vm_state to ERROR
2012-02-23 16:27:51,684 DEBUG nova.rpc.common [-] Sending notification on notifications.error... from (pid=30124) notify /usr/lib/python2.6/dist-packages/nova/rpc/amqp.py:358
2012-02-23 16:27:51,701 ERROR nova.rpc.common [-] Exception during message handling
(nova.rpc.common): TRACE: Traceback (most recent call last):
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/rpc/amqp.py", line 249, in _process_data
(nova.rpc.common): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/exception.py", line 125, in wrapped
(nova.rpc.common): TRACE: return f(*args, **kw)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/manager.py", line 150, in decorated_function
(nova.rpc.common): TRACE: function(self, context, instance_uuid, *args, **kwargs)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/manager.py", line 173, in decorated_function
(nova.rpc.common): TRACE: self.add_instance_fault_from_exc(context, instance_uuid, e)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/contextlib.py", line 23, in __exit__
(nova.rpc.common): TRACE: self.gen.next()
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/manager.py", line 168, in decorated_function
(nova.rpc.common): TRACE: return function(self, context, instance_uuid, *args, **kwargs)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/manager.py", line 1332, in finish_resize
(nova.rpc.common): TRACE: self._set_instance_error_state(context, instance_ref.uuid)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/contextlib.py", line 23, in __exit__
(nova.rpc.common): TRACE: self.gen.next()
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/manager.py", line 1327, in finish_resize
(nova.rpc.common): TRACE: disk_info)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/manager.py", line 1294, in _finish_resize
(nova.rpc.common): TRACE: image_meta = _get_image_meta(context, instance_ref['image_ref'])
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/manager.py", line 180, in _get_image_meta
(nova.rpc.common): TRACE: return image_service.show(context, image_id)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/image/glance.py", line 235, in show
(nova.rpc.common): TRACE: image_id)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/image/glance.py", line 144, in _call_retry
(nova.rpc.common): TRACE: return getattr(client, name)(*args, **kwargs)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/glance/client.py", line 97, in get_image_meta
(nova.rpc.common): TRACE: res = self.do_request("HEAD", "/images/%s" % image_id)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/glance/common/client.py", line 60, in wrapped
(nova.rpc.common): TRACE: self._authenticate(force_reauth=True)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/glance/common/client.py", line 367, in _authenticate
(nova.rpc.common): TRACE: auth_plugin.authenticate()
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/glance/common/auth.py", line 107, in authenticate
(nova.rpc.common): TRACE: check_auth_params()
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/glance/common/auth.py", line 89, in check_auth_params
(nova.rpc.common): TRACE: raise exception.MissingCredentialError(required=required)
(nova.rpc.common): TRACE: MissingCredentialError: Missing required credential: password
(nova.rpc.common): TRACE:

Changed in nova:
assignee: nobody → Mark Washenberger (markwash)

Fix proposed to branch: master
Review: https://review.openstack.org/4698

Changed in nova:
status: New → In Progress
Brian Waldon (bcwaldon) on 2012-02-29
Changed in nova:
importance: Undecided → High

Reviewed: https://review.openstack.org/4698
Committed: http://github.com/openstack/nova/commit/65a7a677c0b1956268939637e384f82ef16cd412
Submitter: Jenkins
Branch: master

commit 65a7a677c0b1956268939637e384f82ef16cd412
Author: Mark Washenberger <email address hidden>
Date: Fri Feb 24 08:52:51 2012 -0500

    Perform image show early in the resize process

    This change addresses bug 943421 by performing glance interactions
    much earlier in the resize process. This prevents token expiration
    from breaking resizes.

    - added a comment to the compute api change to give some context and
      help prevent regression on this bug

    Change-Id: I21619f9f6189d5bdec64de47988985cbc3b8016d

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2012-03-20
Changed in nova:
milestone: none → essex-rc1
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2012-04-05
Changed in nova:
milestone: essex-rc1 → 2012.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers