This is using the latest nova from trunk. In our deployment, we had a hypervisor go down and the tenant issued a hard reboot prior. When attempting a reboot on a guest with the state HARD_REBOOT, nova controller throws this error in it's logs and returns 'ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500)' to the user:
2014-03-06 18:21:00,535 (routes.middleware): DEBUG middleware __call__ Matched POST /tenant1/servers/778032b2-469d-445e-abde-7b9b0b673324/action
2014-03-06 18:21:00,536 (routes.middleware): DEBUG middleware __call__ Route path: '/{project_id}/servers/:(id)/action', defaults: {'action': u'action', 'controller': <nova.api.openstack.wsgi.Resource object at 0x5242c90>}
2014-03-06 18:21:00,536 (routes.middleware): DEBUG middleware __call__ Match dict: {'action': u'action', 'controller': <nova.api.openstack.wsgi.Resource object at 0x5242c90>, 'project_id': u'tenant1', 'id': u'778032b2-469d-445e-abde-7b9b0b673324'}
2014-03-06 18:21:00,537 (nova.api.openstack.wsgi): DEBUG wsgi _process_stack Action: 'action', body: {"reboot": {"type": "SOFT"}}
2014-03-06 18:21:00,538 (nova.api.openstack.wsgi): DEBUG wsgi _process_stack Calling method <bound method Controller._action_reboot of <nova.api.openstack.compute.contrib.keypairs.Controller object at 0x4c35a50>>
2014-03-06 18:21:00,747 (nova.api.openstack): ERROR __init__ _error Caught error: Unexpected task state: expecting [None, 'rebooting'] but the actual state is rebooting_hard
Traceback (most recent call last):
File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/api/openstack/__init__.py", line 125, in __call__
return req.get_response(self.application)
File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/request.py", line 1320, in send
application, catch_exc_info=False)
File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/request.py", line 1284, in call_application
app_iter = application(self.environ, start_response)
File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
return resp(environ, start_response)
File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 598, in __call__
return self.app(env, start_response)
File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
return resp(environ, start_response)
File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
return resp(environ, start_response)
File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/routes/middleware.py", line 131, in __call__
response = self.app(environ, start_response)
File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
return resp(environ, start_response)
File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 130, in __call__
resp = self.call_func(req, *args, **self.kwargs)
File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 195, in call_func
return self.func(req, *args, **kwargs)
File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 925, in __call__
content_type, body, accept)
File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 987, in _process_stack
action_result = self.dispatch(meth, request, action_args)
File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 1074, in dispatch
return method(req=request, **action_args)
File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/api/openstack/compute/servers.py", line 1145, in _action_reboot
self.compute_api.reboot(context, instance, reboot_type)
File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/compute/api.py", line 199, in wrapped
return func(self, context, target, *args, **kwargs)
File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/compute/api.py", line 189, in inner
return function(self, context, instance, *args, **kwargs)
File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/compute/api.py", line 170, in inner
return f(self, context, instance, *args, **kw)
File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/compute/api.py", line 2073, in reboot
instance.save(expected_task_state=[None, task_states.REBOOTING])
File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/objects/base.py", line 151, in wrapper
return fn(self, ctxt, *args, **kwargs)
File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/objects/instance.py", line 472, in save
columns_to_join=_expected_cols(expected_attrs))
File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/db/api.py", line 739, in instance_update_and_get_original
columns_to_join=columns_to_join)
File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 128, in wrapper
return f(*args, **kwargs)
File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 2164, in instance_update_and_get_original
columns_to_join=columns_to_join)
File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 2215, in _instance_update
actual=actual_state, expected=expected)
UnexpectedTaskStateError: Unexpected task state: expecting [None, 'rebooting'] but the actual state is rebooting_hard
2014-03-06 18:21:00,750 (nova.api.openstack): INFO __init__ _error http://nova-controller.isg.apple.com:8774/v2/tenant1/servers/778032b2-469d-445e-abde-7b9b0b673324/action returned with HTTP 500
The actual error message back to the user must be something along the lines of 'Unexpected task state: expecting [None, 'rebooting'] but the actual state is rebooting_hard' with a 4xx HTTP code.
Fix proposed to branch: master
Review: https:/