cells: deleting an instance with cell_name set that doesn't exist in the cell fails

Bug #1479941 reported by Andrew Laski
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Low
Andrew Laski

Bug Description

2015-07-28 19:29:33.153 23754 ERROR nova.cells.messaging [req-1ce8e3f7-a14f-4687-891c-a1541dfdce41 3742 391232 - - -] Error processing message locally
2015-07-28 19:29:33.153 23754 ERROR nova.cells.messaging Traceback (most recent call last):
2015-07-28 19:29:33.153 23754 ERROR nova.cells.messaging File "/opt/rackstack/rackstack.301.55/nova/lib/python2.7/site-packages/nova/cells/messaging.py", line 200, in _process_locally
2015-07-28 19:29:33.153 23754 ERROR nova.cells.messaging resp_value = self.msg_runner._process_message_locally(self)
2015-07-28 19:29:33.153 23754 ERROR nova.cells.messaging File "/opt/rackstack/rackstack.301.55/nova/lib/python2.7/site-packages/nova/cells/messaging.py", line 1296, in _process_message_locally
2015-07-28 19:29:33.153 23754 ERROR nova.cells.messaging return fn(message, **message.method_kwargs)
2015-07-28 19:29:33.153 23754 ERROR nova.cells.messaging File "/opt/rackstack/rackstack.301.55/nova/lib/python2.7/site-packages/nova/cells/messaging.py", line 1078, in instance_destroy_at_top
2015-07-28 19:29:33.153 23754 ERROR nova.cells.messaging instance=instance)
2015-07-28 19:29:33.153 23754 ERROR nova.cells.messaging File "/opt/rackstack/rackstack.301.55/nova/lib/python2.7/site-packages/oslo_utils/excutils.py", line 119, in __exit__
2015-07-28 19:29:33.153 23754 ERROR nova.cells.messaging six.reraise(self.type_, self.value, self.tb)
2015-07-28 19:29:33.153 23754 ERROR nova.cells.messaging File "/opt/rackstack/rackstack.301.55/nova/lib/python2.7/site-packages/nova/cells/messaging.py", line 1065, in instance_destroy_at_top
2015-07-28 19:29:33.153 23754 ERROR nova.cells.messaging instance.destroy()
2015-07-28 19:29:33.153 23754 ERROR nova.cells.messaging File "/opt/rackstack/rackstack.301.55/nova/lib/python2.7/site-packages/nova/objects/base.py", line 116, in wrapper
2015-07-28 19:29:33.153 23754 ERROR nova.cells.messaging return fn(self, *args, **kwargs)
2015-07-28 19:29:33.153 23754 ERROR nova.cells.messaging File "/opt/rackstack/rackstack.301.55/nova/lib/python2.7/site-packages/nova/objects/instance.py", line 651, in destroy
2015-07-28 19:29:33.153 23754 ERROR nova.cells.messaging reason='already destroyed')
2015-07-28 19:29:33.153 23754 ERROR nova.cells.messaging ObjectActionError: Object action destroy failed because: already destroyed
2015-07-28 19:29:33.153 23754 ERROR nova.cells.messaging

The exception being raised is http://git.openstack.org/cgit/openstack/nova/tree/nova/objects/instance.py?id=a74f07a92356187d3455e5a9e2418fb1ea697f96#n614

What's happening is that the instance does not exist in the cell so instance.destroy is being called with an instance object instantiated like "objects.Instance(context=ctxt, uuid=instance.uuid)" so instance.id is not set. So we should probably try to look up the instance when this failure occurs in instance_destroy_at_top.

Tags: cells
Andrew Laski (alaski)
Changed in nova:
assignee: nobody → Andrew Laski (alaski)
tags: added: cells
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/207587

Changed in nova:
status: New → In Progress
melanie witt (melwitt)
Changed in nova:
importance: Undecided → Low
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/207587
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7ea699cd854c215f53e243b0de90bc040aa65d57
Submitter: Jenkins
Branch: master

commit 7ea699cd854c215f53e243b0de90bc040aa65d57
Author: Andrew Laski <email address hidden>
Date: Thu Jul 30 15:16:09 2015 -0400

    Cells: Handle instance_destroy_at_top failure

    With cells an instance can occasionally get into a situation where a
    cell_name is set for it in a parent cell but the instance does not exist
    in a child cell. It is okay for an instance to not exist in a child
    cell but normally cell_name isn't set when that happens. If cell_name
    isn't set then a delete will broadcast to all cells, just in case, and
    then handle the delete local to the parent cell. When cell_name is set
    but the instance doesn't exist in the child cell a series of exceptions
    leads to a failure to delete in the parent cell and the instance is
    essentially undeletable.

    The exception that occurs in the parent cell is due to the instance.id
    not being set on the object sent up from the child cell. Since the
    instance doesn't exist in the child cell we can't prevent that behavior,
    at least without sending fake data. Instead this catches the exception
    that's raised in the parent and tries to grab the instance from the
    database there and then tries to delete it.

    Change-Id: I470dc1db78e58ca5deaf4234662c1e58f0007123
    Closes-bug: 1479941

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → liberty-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: liberty-3 → 12.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.