Cannot wrapped_f while the instance is in this state

Bug #906945 reported by Dan Prince
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Dan Prince

Bug Description

Using nova Essex master...

I'm getting the following error when trying to delete a server in 'BUILD' state:

2011-12-17 02:34:52,012 ERROR nova.api.openstack.v2 [00f8bb84-6a79-49f7-8e4f-0b871eff6b4d 31ebeac7428443ae920ba5a0f593df46 706cdbb728a341d79ce94ad08613a25f] Caught error: Instance 26366adc-7d89-4301-a1c3-530f3911775c in vm_state building. Cannot wrapped_f while the instance is in this state.
(nova.api.openstack.v2): TRACE: Traceback (most recent call last):
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/v2/__init__.py", line 62, in __call__
(nova.api.openstack.v2): TRACE: return req.get_response(self.application)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1053, in get_response
(nova.api.openstack.v2): TRACE: application, catch_exc_info=False)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1022, in call_application
(nova.api.openstack.v2): TRACE: app_iter = application(self.environ, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/keystone/middleware/auth_token.py", line 316, in __call__
(nova.api.openstack.v2): TRACE: return self._forward_request(env, start_response, proxy_headers)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/keystone/middleware/auth_token.py", line 499, in _forward_request
(nova.api.openstack.v2): TRACE: return self.app(env, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
(nova.api.openstack.v2): TRACE: return resp(environ, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
(nova.api.openstack.v2): TRACE: return resp(environ, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
(nova.api.openstack.v2): TRACE: resp = self.call_func(req, *args, **self.kwargs)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 208, in call_func
(nova.api.openstack.v2): TRACE: return self.func(req, *args, **kwargs)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 499, in __call__
(nova.api.openstack.v2): TRACE: response = req.get_response(self.application)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1053, in get_response
(nova.api.openstack.v2): TRACE: application, catch_exc_info=False)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1022, in call_application
(nova.api.openstack.v2): TRACE: app_iter = application(self.environ, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__

(nova.api.openstack.v2): TRACE: return resp(environ, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/pymodules/python2.7/routes/middleware.py", line 131, in __call__
(nova.api.openstack.v2): TRACE: response = self.app(environ, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
(nova.api.openstack.v2): TRACE: return resp(environ, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
(nova.api.openstack.v2): TRACE: resp = self.call_func(req, *args, **self.kwargs)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 208, in call_func
(nova.api.openstack.v2): TRACE: return self.func(req, *args, **kwargs)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 567, in __call__
(nova.api.openstack.v2): TRACE: action_result = self.dispatch(request, action, args)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 599, in dispatch
(nova.api.openstack.v2): TRACE: return controller_method(req=request, **action_args)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 43, in new_f
(nova.api.openstack.v2): TRACE: ret = f(*args, **kwargs)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/nova/scheduler/api.py", line 400, in new_f
(nova.api.openstack.v2): TRACE: return f(*args, **kwargs)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/v2/servers.py", line 701, in delete
(nova.api.openstack.v2): TRACE: self._delete(req.environ['nova.context'], id)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/v2/servers.py", line 472, in _delete
(nova.api.openstack.v2): TRACE: self.compute_api.delete(context, instance)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 75, in inner
(nova.api.openstack.v2): TRACE: method=f.__name__)
(nova.api.openstack.v2): TRACE: InstanceInvalidState: Instance 26366adc-7d89-4301-a1c3-530f3911775c in vm_state building. Cannot wrapped_f while the instance is in this state.

---

Looks like we re-worked the instance state checks in commit 0514bc9d5e92ed0eb6671349e8ff37d7f58aab85.

The nested wrapping seems like its causing the problem here:

    @check_instance_state(vm_state=[vm_states.ACTIVE, vm_states.ERROR,
                                    vm_states.RESCUED, vm_states.STOPPED])
    @scheduler_api.reroute_compute("delete")

I see two problems with the implementation of this. First is that 'wrapped_f' should say the name of the calling function and it doesn't. Second do we really need to spew a full exception trace to the log file when these types of errors occur. Seems like we need to rework our error handling when instance state type exceptions occur.

Dan Prince (dan-prince)
Changed in nova:
importance: Undecided → Medium
Dan Prince (dan-prince)
Changed in nova:
assignee: nobody → Dan Prince (dan-prince)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/2521

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/2521
Committed: http://github.com/openstack/nova/commit/f0b54c7b343a752f3577dafc80e3eb5a9ae0754d
Submitter: Jenkins
Branch: master

commit f0b54c7b343a752f3577dafc80e3eb5a9ae0754d
Author: Dan Prince <email address hidden>
Date: Wed Dec 21 13:42:40 2011 -0500

    Make reroute_compute use functools.wraps. Fixes LP bug #906945.

    Change-Id: I043b6543e79f9cc6c7a32c9952113ee009b31377

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → essex-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: essex-3 → 2012.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.