nova force-delete can't delete instance in vm_state block_device_mapping

[blkart]

Description
===========
when instance in block_device_mapping task state, can't force-delete it by nova force-delete command.

Steps to reproduce
==================
* Boot an instance, as soon as instance task state changed to block_device_mapping, run nova force-delete INSTANCE_UUID command, get some error:

# nova force-delete 08ed7e71-4470-4746-ae65-d17c659f29c9
/usr/lib/python2.7/site-packages/novaclient/client.py:278: UserWarning: The 'tenant_id' argument is deprecated in Ocata and its use may result in errors in future releases. As 'project_id' is provided, the 'tenant_id' argument will be ignored.
  warnings.warn(msg)
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.InstanceInvalidState'> (HTTP 500) (Request-ID: req-b185ecd7-5f0d-441e-8263-59af8b66aae7)

Expected result
===============
force delete instance sucessfully

Actual result
=============
get some error and force delete instance failed

Environment
===========
# rpm -qa | grep nova
openstack-nova-api-15.0.0-1.el7.noarch
openstack-nova-console-15.0.0-1.el7.noarch
openstack-nova-common-15.0.0-1.el7.noarch
openstack-nova-conductor-15.0.0-1.el7.noarch
openstack-nova-novncproxy-15.0.0-1.el7.noarch
puppet-nova-10.3.0-1.el7.noarch
python-nova-15.0.0-1.el7.noarch
openstack-nova-cert-15.0.0-1.el7.noarch
openstack-nova-placement-api-15.0.0-1.el7.noarch
openstack-nova-compute-15.0.0-1.el7.noarch
openstack-nova-scheduler-15.0.0-1.el7.noarch
python2-novaclient-7.1.0-1.el7.noarch

Logs & Configs
==============
2017-07-18 15:47:22.159 36543 DEBUG nova.api.openstack.wsgi [req-b185ecd7-5f0d-441e-8263-59af8b66aae7 - - - - -] Action: 'action', calling method: <bound method DeferredDeleteController._force_delete of <nova.api.openstack.compute.deferred_delete.DeferredDeleteController object at 0x83f4c10>>, body: {"forceDelete": null} _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:623
2017-07-18 15:47:22.159 36543 DEBUG nova.compute.api [req-b185ecd7-5f0d-441e-8263-59af8b66aae7 - - - - -] [instance: 08ed7e71-4470-4746-ae65-d17c659f29c9] Fetching instance by UUID get /usr/lib/python2.7/site-packages/nova/compute/api.py:2313
2017-07-18 15:47:22.172 36543 DEBUG oslo_db.sqlalchemy.engines [req-b185ecd7-5f0d-441e-8263-59af8b66aae7 - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:261
2017-07-18 15:47:22.212 36543 ERROR nova.api.openstack.extensions [req-b185ecd7-5f0d-441e-8263-59af8b66aae7 - - - - -] Unexpected exception in API method
2017-07-18 15:47:22.212 36543 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2017-07-18 15:47:22.212 36543 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapped
2017-07-18 15:47:22.212 36543 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2017-07-18 15:47:22.212 36543 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/deferred_delete.py", line 64, in _force_delete
2017-07-18 15:47:22.212 36543 ERROR nova.api.openstack.extensions self.compute_api.force_delete(context, instance)
2017-07-18 15:47:22.212 36543 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 170, in inner
2017-07-18 15:47:22.212 36543 ERROR nova.api.openstack.extensions return function(self, context, instance, *args, **kwargs)
2017-07-18 15:47:22.212 36543 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 143, in inner
2017-07-18 15:47:22.212 36543 ERROR nova.api.openstack.extensions method=f.__name__)
2017-07-18 15:47:22.212 36543 ERROR nova.api.openstack.extensions InstanceInvalidState: Instance 08ed7e71-4470-4746-ae65-d17c659f29c9 in task_state block_device_mapping. Cannot force_delete while the instance is in this state.
2017-07-18 15:47:22.212 36543 ERROR nova.api.openstack.extensions
2017-07-18 15:47:22.219 36543 INFO nova.api.openstack.wsgi [req-b185ecd7-5f0d-441e-8263-59af8b66aae7 - - - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.InstanceInvalidState'>
2017-07-18 15:47:22.220 36543 DEBUG nova.api.openstack.wsgi [req-b185ecd7-5f0d-441e-8263-59af8b66aae7 - - - - -] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.InstanceInvalidState'> __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1039
2017-07-18 15:47:22.221 36543 INFO nova.osapi_compute.wsgi.server [req-b185ecd7-5f0d-441e-8263-59af8b66aae7 - - - - -] 192.168.2.145 "POST /v2.1/227c2369e7e9475f93c3ec7f07082937/servers/08ed7e71-4470-4746-ae65-d17c659f29c9/action HTTP/1.1" status: 500 len: 578 time: 0.0697980

[Takashi Natsume (natsume-takashi)]

This bug report has an assignee. So set its status to 'In Progress.'.

[Sylvain Bauza (sylvain-bauza)]

We only assign bugs to "In progress" if there is an uploaded change for reviewing.

Unless there is a Gerrit change, putting it back to Confirmed.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/527951

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/529212

[Raghad Qutteneh (raghadq)]

I have submitted a fix for this defect, how it's has been taken.
I have assigned this defect to me, As there was no activity on it since August.

Please advise

[OpenStack Infra (hudson-openstack)]

Change abandoned by blkart (<email address hidden>) on branch: master
Review: https://review.openstack.org/529212

[Rupesh Chowdhary (rupeshrams)]

Here are the recommendation to validate in the environment,

1. Check the nova services using nova service-list
2. Check the rabbit mq connectivity from the controller nodes
3. Check the quota for particular project which you are trying to launch the instance.
4. Also check the network connectivity between controllers and compute nodes.

[Ghanshyam Mann (ghanshyammann)]

This is duplicate of https://bugs.launchpad.net/nova/+bug/1741000

It is fixed by https://review.openstack.org/#/c/530879/ and fix is backported till stable/pike.

Marking as duplicate of 1741000

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/527951