cells: error if deleting an instance while host is being set

[Andrew Laski]

If a delete request gets past some checks to perform a local delete, but then has a host set on it before the local delete finishes it will fail a db constraint check and spew this to the logs:

2015-05-13 11:14:58.666 ERROR nova.api.openstack [req-cb52db9a-d313-49c4-acce-b627b778ccd1 ListServersNegativeTestJSON-1942377486 ListServersNegativeTestJSON-625281866] Caught error: Object action destroy failed because: host changed
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack Traceback (most recent call last):
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack File "/opt/stack/new/nova/nova/api/openstack/__init__.py", line 125, in __call__
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack return req.get_response(self.application)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1317, in send
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack application, catch_exc_info=False)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1281, in call_application
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack return resp(environ, start_response)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 639, in __call__
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack return self._call_app(env, start_response)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 559, in _call_app
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack return self._app(env, _fake_start_response)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack return resp(environ, start_response)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack return resp(environ, start_response)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 136, in __call__
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack response = self.app(environ, start_response)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack return resp(environ, start_response)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 756, in __call__
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack content_type, body, accept)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 821, in _process_stack
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 911, in dispatch
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack return method(req=request, **action_args)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack File "/opt/stack/new/nova/nova/api/openstack/compute/servers.py", line 833, in delete
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack self._delete(req.environ['nova.context'], req, id)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack File "/opt/stack/new/nova/nova/api/openstack/compute/servers.py", line 668, in _delete
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack self.compute_api.delete(context, instance)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack File "/opt/stack/new/nova/nova/compute/cells_api.py", line 212, in delete
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack self._handle_cell_delete(context, instance, 'delete')
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack File "/opt/stack/new/nova/nova/compute/cells_api.py", line 226, in _handle_cell_delete
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack self._do_delete)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack File "/opt/stack/new/nova/nova/compute/api.py", line 1766, in _local_delete
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack instance.destroy()
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack File "/opt/stack/new/nova/nova/objects/base.py", line 205, in wrapper
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack return fn(self, *args, **kwargs)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack File "/opt/stack/new/nova/nova/objects/instance.py", line 648, in destroy
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack reason='host changed')
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack ObjectActionError: Object action destroy failed because: host changed
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack

This should be fixed.

[Andrew Laski (alaski)]

This is one of the remaining cells job failures.

[Matt Riedemann (mriedem)]

Patch is here: https://review.openstack.org/#/c/182772/

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiT2JqZWN0QWN0aW9uRXJyb3I6IE9iamVjdCBhY3Rpb24gZGVzdHJveSBmYWlsZWQgYmVjYXVzZTogaG9zdCBjaGFuZ2VkXCIgQU5EIHRhZ3M6XCJzY3JlZW4tbi1hcGkudHh0XCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MzE2Mjk4MzI0NTUsIm1vZGUiOiIiLCJhbmFseXplX2ZpZWxkIjoiIn0=

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/182772
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d34e0ce254e25d864fec4b19e618660972da6846
Submitter: Jenkins
Branch: master

commit d34e0ce254e25d864fec4b19e618660972da6846
Author: Andrew Laski <email address hidden>
Date: Wed May 13 13:53:34 2015 -0400

    Retry a cell delete if host constraint fails

    The instance.destroy method has a constraint that fails during a local
    delete if the host is not None, because it should be in that case.
    Cells calls local_delete when there's no cell_name on the instance, but
    when this fails it's likely that we have a host and cell_name now so the
    delete method should be attempted again.

    Closes-Bug: 1454839
    Change-Id: I412fab6f43cf2c64239a0292946c6c3bb9837110