nova cells, force-delete VM throws error even if VM gets deleted

[Rajesh Tailor]

In nova cells environment, when try to force-delete an instance which is in 'active' state, the instance gets deleted successfully, but in nova-cells service in compute cell (n-cell-child), it throws the following error:
InvalidRequestError: Object '<Instance at 0x7fc5fcf581d0>' is already attached to session '75' (this is '79')

Reproduction steps:
1) Create instance.
2) Wait until instance becomes 'active'.
3) Try to force-delete the instance.
$ nova force-delete <instance_id>

Found this error in nova-cells service in compute cell (n-cell-child service):

2014-10-15 01:59:36.742 ERROR nova.cells.messaging [req-7c1615ad-491d-4af8-88d7-ff83563ef429 admin admin] Error processing message locally: Object '<Ins
tance at 0x7fc5fcf581d0>' is already attached to session '75' (this is '79')
2014-10-15 01:59:36.742 TRACE nova.cells.messaging Traceback (most recent call last):
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/opt/stack/nova/nova/cells/messaging.py", line 199, in _process_locally
2014-10-15 01:59:36.742 TRACE nova.cells.messaging resp_value = self.msg_runner._process_message_locally(self)
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/opt/stack/nova/nova/cells/messaging.py", line 1293, in _process_message_locally
2014-10-15 01:59:36.742 TRACE nova.cells.messaging return fn(message, **message.method_kwargs)
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/opt/stack/nova/nova/cells/messaging.py", line 698, in run_compute_api_method
2014-10-15 01:59:36.742 TRACE nova.cells.messaging return fn(message.ctxt, *args, **method_info['method_kwargs'])
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/opt/stack/nova/nova/compute/api.py", line 219, in wrapped
2014-10-15 01:59:36.742 TRACE nova.cells.messaging return func(self, context, target, *args, **kwargs)
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/opt/stack/nova/nova/compute/api.py", line 209, in inner
2014-10-15 01:59:36.742 TRACE nova.cells.messaging return function(self, context, instance, *args, **kwargs)
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/opt/stack/nova/nova/compute/api.py", line 190, in inner
2014-10-15 01:59:36.742 TRACE nova.cells.messaging return f(self, context, instance, *args, **kw)
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/opt/stack/nova/nova/compute/api.py", line 1836, in force_delete
2014-10-15 01:59:36.742 TRACE nova.cells.messaging self._delete_instance(context, instance)
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/opt/stack/nova/nova/compute/api.py", line 1790, in _delete_instance2014-10-15 01:59:36.742 TRACE nova.cells.messaging task_state=task_states.DELETING)
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/opt/stack/nova/nova/compute/api.py", line 1622, in _delete
2014-10-15 01:59:36.742 TRACE nova.cells.messaging quotas.rollback()
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/usr/local/lib/python2.7/dist-packages/oslo/utils/excutils.py", line 82, in __exit__
2014-10-15 01:59:36.742 TRACE nova.cells.messaging six.reraise(self.type_, self.value, self.tb)
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/opt/stack/nova/nova/compute/api.py", line 1550, in _delete
2014-10-15 01:59:36.742 TRACE nova.cells.messaging instance.save()
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/opt/stack/nova/nova/db/sqlalchemy/models.py", line 52, in save
2014-10-15 01:59:36.742 TRACE nova.cells.messaging super(NovaBase, self).save(session=session)
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/usr/local/lib/python2.7/dist-packages/oslo/db/sqlalchemy/models.py", line 47, in save
2014-10-15 01:59:36.742 TRACE nova.cells.messaging session.add(self)
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1399, in add
2014-10-15 01:59:36.742 TRACE nova.cells.messaging self._save_or_update_state(state)
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1411, in _save_or_update_state
2014-10-15 01:59:36.742 TRACE nova.cells.messaging self._save_or_update_impl(state)
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1667, in _save_or_update_impl
2014-10-15 01:59:36.742 TRACE nova.cells.messaging self._update_impl(state)
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1661, in _update_impl
2014-10-15 01:59:36.742 TRACE nova.cells.messaging self._attach(state)
2014-10-15 01:59:36.742 TRACE nova.cells.messaging File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1749, in _attach
2014-10-15 01:59:36.742 TRACE nova.cells.messaging state.session_id, self.hash_key))
2014-10-15 01:59:36.742 TRACE nova.cells.messaging InvalidRequestError: Object '<Instance at 0x7fc5fcf581d0>' is already attached to session '75' (this is '79')

Note: This issue occurs intermitently.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/135202

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/135202
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=580e8f6a5e8721b0535ea5fb1d414f6e9ede3070
Submitter: Jenkins
Branch: master

commit 580e8f6a5e8721b0535ea5fb1d414f6e9ede3070
Author: Rajesh Tailor <email address hidden>
Date: Fri Oct 17 05:46:41 2014 -0700

    Remove cell api overrides for force-delete

    The force_delete method of nova/compute/cells_api module is actually
    not needed, as this method handle objects and is triggered when
    force-delete api is used for instance deletion in cells environment.

    It actually causes a race condition with instance deletion as it
    triggers 2 instance deletion from a compute cell for single
    force-delete request.

    Modified delete-api implementation so that force-delete request
    triggers single instance deletion call in cells environment.

    Change-Id: I812f27761fc4d4f64563d79f793c484d03aa641f
    Closes-Bug: #1381425