failed nova deletes have a stale task state 'deleting'

[Robert Collins]

E.g. nova delete an ironic instance and it fails to delete for some reason (e.g. IPMI timeout). This sets the instance state to ERROR, but leaves the task state as deleting - which is a wedge that requires admin intervention to fix.

[Dmitry Tantsur (divius)]

Hi Robert! I know, there was some work on fixing this kind of situation. Is it still reproducable? If so, could you please post logs for it?

[Dmitry Tantsur (divius)]

We cannot solve the issue you reported without more information. Could you please provide the requested information?

[Launchpad Janitor (janitor)]

[Expired for Ironic because there has been no activity for 60 days.]

[Jerry Zhao (zhaoxinyu)]

I met with the same problem.
the steps i took before the problem:
1. register a baremetal node with a wrong mac address, which i found out later.
2. heat stack-create, then stack creation failed because reported event "Message: No valid host was found. , Code: 500"
3. with 'nova list', i saw two nova instances one in error and the other in BUILDING state.
4. i deleted the one with error status but the other one with BUILDING status can't be deleted.

nova reset-state and nova delete didn't work either.

[Jerry Zhao (zhaoxinyu)]

an update happened when i made the prevoius comment:

Oct 22 14:14:09 undercloud nova-compute: 2014-10-22 14:14:09.162 6754 ERROR nova.openstack.common.loopingcall [-] in fixed duration looping call
Oct 22 14:14:09 undercloud nova-compute: 2014-10-22 14:14:09.162 6754 TRACE nova.openstack.common.loopingcall Traceback (most recent call last):
Oct 22 14:14:09 undercloud nova-compute: 2014-10-22 14:14:09.162 6754 TRACE nova.openstack.common.loopingcall File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/loopingcall.py", line 81, in _inner
Oct 22 14:14:09 undercloud nova-compute: 2014-10-22 14:14:09.162 6754 TRACE nova.openstack.common.loopingcall self.f(*self.args, **self.kw)
Oct 22 14:14:09 undercloud nova-compute: 2014-10-22 14:14:09.162 6754 TRACE nova.openstack.common.loopingcall File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/virt/ironic/driver.py", line 350, in _wait_for_active
Oct 22 14:14:09 undercloud nova-compute: 2014-10-22 14:14:09.162 6754 TRACE nova.openstack.common.loopingcall raise exception.InstanceDeployFailure(msg)
Oct 22 14:14:09 undercloud nova-compute: 2014-10-22 14:14:09.162 6754 TRACE nova.openstack.common.loopingcall InstanceDeployFailure: Failed to provision instance 784642e7-0ff3-43bb-9016-9ccb3614cf85: Timeout reached while waiting for callback for node d922a6e3-72ca-418c-8051-fa6e319c43a9
Oct 22 14:14:09 undercloud nova-compute: 2014-10-22 14:14:09.162 6754 TRACE nova.openstack.common.loopingcall
Oct 22 14:14:09 undercloud nova-compute: 2014-10-22 14:14:09.178 6754 ERROR nova.virt.ironic.driver [req-a07c650e-075e-4ca6-aa0e-3b66f128b6f6 None] Error deploying instance 784642e7-0ff3-43bb-9016-9ccb3614cf85 on baremetal node d922a6e3-72ca-418c-8051-fa6e319c43a9.
Oct 22 14:14:25 undercloud nova-compute: 2014-10-22 14:14:25.461 6754 ERROR nova.compute.manager [req-a07c650e-075e-4ca6-aa0e-3b66f128b6f6 None] [instance: 784642e7-0ff3-43bb-9016-9ccb3614cf85] Instance failed to spawn
Oct 22 14:14:25 undercloud nova-compute: 2014-10-22 14:14:25.461 6754 TRACE nova.compute.manager [instance: 784642e7-0ff3-43bb-9016-9ccb3614cf85] Traceback (most recent call last):
Oct 22 14:14:25 undercloud nova-compute: 2014-10-22 14:14:25.461 6754 TRACE nova.compute.manager [instance: 784642e7-0ff3-43bb-9016-9ccb3614cf85] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 2233, in _build_resources
Oct 22 14:14:25 undercloud nova-compute: 2014-10-22 14:14:25.461 6754 TRACE nova.compute.manager [instance: 784642e7-0ff3-43bb-9016-9ccb3614cf85] yield resources
Oct 22 14:14:25 undercloud nova-compute: 2014-10-22 14:14:25.461 6754 TRACE nova.compute.manager [instance: 784642e7-0ff3-43bb-9016-9ccb3614cf85] File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 2103, in _build_and_run_instance
Oct 22 14:14:25 undercloud nova-compute: 2014-10-22 14:14:25.461 6754 TRACE nova.compute.manager [instance: 784642e7-0ff3-43bb-9016-9ccb3614cf85] block_device_info=block_device_info)
Oct 22 14:14:25 undercloud nova-compute: 2014-10-22 14:14:25.461 6754 TRACE nova.compute.manager [instance: 784642e7-0ff3-43bb-901...

[Dmitry Tantsur (divius)]

Jerry, I guess you problem could be solved if https://bugs.launchpad.net/ironic/+bug/1406181 is fixed. This bug is about incorrect behavior during _deletion_, and we can't work on it without more information.