nova cells, force-delete VM throws error even if VM gets deleted

Bug #1381425 reported by Rajesh Tailor
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Rajesh Tailor

Bug Description

In nova cells environment, when try to force-delete an instance which is in 'active' state, the instance gets deleted successfully, but in nova-cells service in compute cell (n-cell-child), it throws the following error:
InvalidRequestError: Object '<Instance at 0x7fc5fcf581d0>' is already attached to session '75' (this is '79')

Reproduction steps:
1) Create instance.
2) Wait until instance becomes 'active'.
3) Try to force-delete the instance.
$ nova force-delete <instance_id>

Found this error in nova-cells service in compute cell (n-cell-child service):

2014-10-15 01:59:36.742 ERROR nova.cells.messaging [req-7c1615ad-491d-4af8-88d7-ff83563ef429 admin admin] Error processing message locally: Object '<Ins
tance at 0x7fc5fcf581d0>' is already attached to session '75' (this is '79')
2014-10-15 01:59:36.742 TRACE nova.cells.messaging Traceback (most recent call last):
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/opt/stack/nova/nova/cells/messaging.py", line 199, in _process_locally
2014-10-15 01:59:36.742 TRACE nova.cells.messaging resp_value = self.msg_runner._process_message_locally(self)
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/opt/stack/nova/nova/cells/messaging.py", line 1293, in _process_message_locally
2014-10-15 01:59:36.742 TRACE nova.cells.messaging return fn(message, **message.method_kwargs)
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/opt/stack/nova/nova/cells/messaging.py", line 698, in run_compute_api_method
2014-10-15 01:59:36.742 TRACE nova.cells.messaging return fn(message.ctxt, *args, **method_info['method_kwargs'])
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/opt/stack/nova/nova/compute/api.py", line 219, in wrapped
2014-10-15 01:59:36.742 TRACE nova.cells.messaging return func(self, context, target, *args, **kwargs)
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/opt/stack/nova/nova/compute/api.py", line 209, in inner
2014-10-15 01:59:36.742 TRACE nova.cells.messaging return function(self, context, instance, *args, **kwargs)
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/opt/stack/nova/nova/compute/api.py", line 190, in inner
2014-10-15 01:59:36.742 TRACE nova.cells.messaging return f(self, context, instance, *args, **kw)
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/opt/stack/nova/nova/compute/api.py", line 1836, in force_delete
2014-10-15 01:59:36.742 TRACE nova.cells.messaging self._delete_instance(context, instance)
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/opt/stack/nova/nova/compute/api.py", line 1790, in _delete_instance2014-10-15 01:59:36.742 TRACE nova.cells.messaging task_state=task_states.DELETING)
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/opt/stack/nova/nova/compute/api.py", line 1622, in _delete
2014-10-15 01:59:36.742 TRACE nova.cells.messaging quotas.rollback()
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/usr/local/lib/python2.7/dist-packages/oslo/utils/excutils.py", line 82, in __exit__
2014-10-15 01:59:36.742 TRACE nova.cells.messaging six.reraise(self.type_, self.value, self.tb)
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/opt/stack/nova/nova/compute/api.py", line 1550, in _delete
2014-10-15 01:59:36.742 TRACE nova.cells.messaging instance.save()
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/opt/stack/nova/nova/db/sqlalchemy/models.py", line 52, in save
2014-10-15 01:59:36.742 TRACE nova.cells.messaging super(NovaBase, self).save(session=session)
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/usr/local/lib/python2.7/dist-packages/oslo/db/sqlalchemy/models.py", line 47, in save
2014-10-15 01:59:36.742 TRACE nova.cells.messaging session.add(self)
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1399, in add
2014-10-15 01:59:36.742 TRACE nova.cells.messaging self._save_or_update_state(state)
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1411, in _save_or_update_state
2014-10-15 01:59:36.742 TRACE nova.cells.messaging self._save_or_update_impl(state)
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1667, in _save_or_update_impl
2014-10-15 01:59:36.742 TRACE nova.cells.messaging self._update_impl(state)
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1661, in _update_impl
2014-10-15 01:59:36.742 TRACE nova.cells.messaging self._attach(state)
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1749, in _attach
2014-10-15 01:59:36.742 TRACE nova.cells.messaging state.session_id, self.hash_key))
2014-10-15 01:59:36.742 TRACE nova.cells.messaging InvalidRequestError: Object '<Instance at 0x7fc5fcf581d0>' is already attached to session '75' (this is '79')

Note: This issue occurs intermitently.

Tags: cells
Changed in nova:
assignee: nobody → Rajesh Tailor (rajesh-tailor)
description: updated
Joe Gordon (jogo)
tags: added: cells
Changed in nova:
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/135202

Changed in nova:
status: Confirmed → In Progress
melanie witt (melwitt)
tags: removed: ntt
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/135202
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=580e8f6a5e8721b0535ea5fb1d414f6e9ede3070
Submitter: Jenkins
Branch: master

commit 580e8f6a5e8721b0535ea5fb1d414f6e9ede3070
Author: Rajesh Tailor <email address hidden>
Date: Fri Oct 17 05:46:41 2014 -0700

    Remove cell api overrides for force-delete

    The force_delete method of nova/compute/cells_api module is actually
    not needed, as this method handle objects and is triggered when
    force-delete api is used for instance deletion in cells environment.

    It actually causes a race condition with instance deletion as it
    triggers 2 instance deletion from a compute cell for single
    force-delete request.

    Modified delete-api implementation so that force-delete request
    triggers single instance deletion call in cells environment.

    Change-Id: I812f27761fc4d4f64563d79f793c484d03aa641f
    Closes-Bug: #1381425

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → liberty-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: liberty-3 → 12.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.