[Nova][Ironic] Instance stuck at deleting state

[Vasyl Saienko]

When trying to delete instance from Nova, sometimes instances stuck at deleting state. It can stuck for different period of time (6hr, day and even more).

I've tried to reset-state and force-delete but it didn't help.

There is logs from:
nova-compute: https://paste.mirantis.net/show/1932/
nova-api: https://paste.mirantis.net/show/1933/

On nova compute at eventlet backdoor pgt() shows a lot of the following calls: https://paste.mirantis.net/show/1935/

[Vasyl Saienko (vsaienko)]

Upstream bug reported https://bugs.launchpad.net/nova/+bug/1552446

[Roman Podoliaka (rpodolyaka)]

It's being fixed in Ironic in upstream: https://review.openstack.org/#/c/287498/

[Vasyl Saienko (vsaienko)]

The fix consists of two parts, there is nova related patch https://review.openstack.org/#/c/287485/

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

version

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Pavlo Shchelokovskyy (pshchelo)]

both patches (in Ironic and Nova) are landed in upstream. Neither is in stable/mitaka. Should we cherry-pick those ourselves or try pushing them in stable and wait for sync?

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 9.0/mitaka
Change author: Vasyl Saienko <email address hidden>
Review: https://review.fuel-infra.org/20573

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 9.0/mitaka
Change author: Vasyl Saienko <email address hidden>
Review: https://review.fuel-infra.org/20574

[Kyrylo Romanenko (kromanenko)]

Does it depend of ironic driver used?

[Pavlo Shchelokovskyy (pshchelo)]

Kyrylo, no, it does not. This is a general Nova-Ironic coordination thing.

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/20573
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 2aed3c4822b5568e6342957ab1dca77bdfc5f000
Author: Vasyl Saienko <email address hidden>
Date: Thu May 12 11:04:23 2016

Allow to update resource per single node.

At the moment during Ironic instance termination resources are updated
for all nodes. As result with high numner of nodes, removing N instances
initiates N x M requests, where M is the total number of nodes.
This commit allows to update resources per specific node only.

Co-Authored-By: Yuriy Zveryanskyy (<email address hidden>)
Closes-Bug: #1552120
Related-Bug: #1552446
Change-Id: Iffad4a6ab1aaa5fea591c19f6d330dc861c5675d
(cherry picked from commit 2005b47c92edb41b08ec401ecdbfe816cab13afb)

[Roman Podoliaka (rpodolyaka)]

^ fix merged

[Roman Podoliaka (rpodolyaka)]

Fix merged to Newton: https://review.openstack.org/#/c/287485/ , thus 10.0 is not affected

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/20574
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: a15427f036f7c742813f8abfbdb5a8b20c468915
Author: Vasyl Saienko <email address hidden>
Date: Thu May 12 11:04:57 2016

Update resources only for specific node during deletion.

At the moment during Nova instance termination resources are
updated for all Ironic nodes. As result with high number of nodes,
removing N instances initiates N x M requests, where M is the total
number of nodes. This commit allows to update resources per specific
node on termination.

Depends-On: Iffad4a6ab1aaa5fea591c19f6d330dc861c5675d

Closes-Bug: #1552120
Closes-Bug: #1552446
Change-Id: Ib54741c8564e18f23c4f50bed190dfc66db7ba76
(cherry picked from commit 16d8819891c4bef58760c62c30aa2b68868087b9)

[Sergii Turivnyi (sturivnyi)]

Waiting for scale lab for verification

[Kyrylo Romanenko (kromanenko)]

As far as i reproduced this bug on small test env previously, not on scale - i`ll try to verify it also on small environment.

[Kyrylo Romanenko (kromanenko)]

Tested on
cat /etc/fuel_build_id:
 427
cat /etc/fuel_build_number:
 427
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6347.noarch
 fuel-misc-9.0.0-1.mos8415.noarch
 fuel-bootstrap-cli-9.0.0-1.mos284.noarch
 fuel-migrate-9.0.0-1.mos8415.noarch
 rubygem-astute-9.0.0-1.mos747.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8718.noarch
 network-checker-9.0.0-1.mos72.x86_64
 fuel-mirror-9.0.0-1.mos137.noarch
 fuel-openstack-metadata-9.0.0-1.mos8718.noarch
 fuel-notify-9.0.0-1.mos8415.noarch
 nailgun-mcagents-9.0.0-1.mos747.noarch
 python-fuelclient-9.0.0-1.mos319.noarch
 fuelmenu-9.0.0-1.mos270.noarch
 fuel-9.0.0-1.mos6347.noarch
 fuel-utils-9.0.0-1.mos8415.noarch
 fuel-setup-9.0.0-1.mos6347.noarch
 fuel-library9.0-9.0.0-1.mos8415.noarch
 shotgun-9.0.0-1.mos90.noarch
 fuel-agent-9.0.0-1.mos284.noarch
 fuel-ui-9.0.0-1.mos2710.noarch
 fuel-ostf-9.0.0-1.mos934.noarch
 python-packetary-9.0.0-1.mos137.noarch
 fuel-nailgun-9.0.0-1.mos8718.noarch