Update OSAPI and EC2's to properly handle InstanceInvalidState exceptions

Bug #911879 reported by Dan Prince on 2012-01-04
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Brian Lamar

Bug Description

The compute API in nova Essex now throws InstanceInvalidState exceptions when an instance is in the wrong state for the selected action.

Trying to delete a server in build (or resize) will cause HTTP 500 exceptions to occur. This is incorrect API behaviour as the SPEC says that HTTP 409 buildInProgress should be returned.

Additionally, the API log files include a full stack trace when the exceptions occur (these are user generated via the API):

3d527481b 75cf5a01f13c4ef09cf4e1601f0c16c6 7535afc6ccc4449b804f7c06e8ebd98e] Caught error: Instance 3beb9aa8-9081-4398-b3a8-55cd1d77b1fd in vm_state building. Cannot delete while the instance is in this state.
(nova.api.openstack.v2): TRACE: Traceback (most recent call last):
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/v2/__init__.py", line 59, in __call__
(nova.api.openstack.v2): TRACE: return req.get_response(self.application)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1053, in get_response
(nova.api.openstack.v2): TRACE: application, catch_exc_info=False)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1022, in call_application
(nova.api.openstack.v2): TRACE: app_iter = application(self.environ, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/keystone/middleware/auth_token.py", line 330, in __call__
(nova.api.openstack.v2): TRACE: return self._forward_request(env, start_response, proxy_headers)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/keystone/middleware/auth_token.py", line 541, in _forward_request
(nova.api.openstack.v2): TRACE: return self.app(env, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
(nova.api.openstack.v2): TRACE: return resp(environ, start_response)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
(nova.api.openstack.v2): TRACE: resp = self.call_func(req, *args, **self.kwargs)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 208, in call_func
(nova.api.openstack.v2): TRACE: return self.func(req, *args, **kwargs)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 568, in __call__
(nova.api.openstack.v2): TRACE: action_result = self.dispatch(request, action, args)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 600, in dispatch
(nova.api.openstack.v2): TRACE: return controller_method(req=request, **action_args)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 52, in new_f
(nova.api.openstack.v2): TRACE: ret = f(*args, **kwargs)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/nova/scheduler/api.py", line 401, in new_f
(nova.api.openstack.v2): TRACE: return f(*args, **kwargs)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/v2/servers.py", line 575, in delete
(nova.api.openstack.v2): TRACE: self._delete(req.environ['nova.context'], id)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/v2/servers.py", line 437, in _delete
(nova.api.openstack.v2): TRACE: self.compute_api.delete(context, instance)
(nova.api.openstack.v2): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 77, in inner
(nova.api.openstack.v2): TRACE: method=f.__name__)
(nova.api.openstack.v2): TRACE: InstanceInvalidState: Instance 3beb9aa8-9081-4398-b3a8-55cd1d77b1fd in vm_state building. Cannot delete while the instance is in this state.


Given the compute API is throwing a new type of exception it appears we need to review the Openstack and EC2 APIs to make sure these exceptions are handled properly.

Dan Prince (dan-prince) on 2012-01-04
Changed in nova:
importance: Undecided → Medium
Brian Lamar (blamar) on 2012-01-04
Changed in nova:
assignee: nobody → Brian Lamar (blamar)
status: New → In Progress

Reviewed: https://review.openstack.org/2846
Committed: http://github.com/openstack/nova/commit/18f72f29a92733e56d8641e48b2099df1b8b30f4
Submitter: Jenkins
Branch: master

commit 18f72f29a92733e56d8641e48b2099df1b8b30f4
Author: Brian Lamar <email address hidden>
Date: Wed Jan 4 18:54:45 2012 -0500

    Return 409s instead of 500s when deleting certain instances.

    Fixes bug 911879

    Change-Id: Ib2ae875ec2d1eeea21d15a756b96c27047b12bee

Changed in nova:
status: In Progress → Fix Committed

Fix proposed to branch: master
Review: https://review.openstack.org/2992

Changed in nova:
status: Fix Committed → In Progress

Reviewed: https://review.openstack.org/2992
Committed: http://github.com/openstack/nova/commit/475691a4bd5795feb50b5c9ccfe98e6487390e58
Submitter: Jenkins
Branch: master

commit 475691a4bd5795feb50b5c9ccfe98e6487390e58
Author: Chris Behrens <email address hidden>
Date: Wed Jan 11 20:25:23 2012 -0800

    catch InstanceInvalidState in more places

    Further fixes to bug 911879

    500s or 400s are returned in the OS API when actions are denied due
    to being in an invalid state. 409 should be returned, instead. A
    previous review (2846) fixed the delete case and this fixes more.

    When writing tests, I found a number of exceptions that are not raised
    anymore, and they were being incorrectly used in tests still. I fixed
    those up.

    Change-Id: I0d5b1ed52e0cc9766be8e2a7de84c8601f4bdf26

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2012-01-25
Changed in nova:
milestone: none → essex-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2012-04-05
Changed in nova:
milestone: essex-3 → 2012.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers