stack_delete fails/times out when soft-delete is enabled in nova

[Sam Whyte]

When reclaim_instance_interval is non-zero in nova configuration, deleting a stack will hang until timeout is exceeded, while waiting for the first VM to delete.

During this hang, if a force-delete is issued against the VM that Heat is "deleting", then Heat will acknowledge the state change and try to delete the next VM. In this way, a stack can be properly deleted if force-deletes are issued in the background.

This was first encountered in openstack-heat-engine-2014.1-1.0.el6.noarch and confirmed to still exist in openstack-heat-engine-2014.1.2-1.0.el6.noarch

[Steve Baker (steve-stevebaker)]

what state does a nova server go to when it is soft-deleted?

[Sam Whyte (swhyte-v)]

This is the output of nova show for a recently soft-deleted VM:

[root@XXX ~(keystone_admin)]# nova show b1667bd3-72ab-4173-80c0-5311d4c12a80
+--------------------------------------+----------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | XXX |
| OS-EXT-SRV-ATTR:hypervisor_hostname | XXX |
| OS-EXT-SRV-ATTR:instance_name | instance-000025ca |
| OS-EXT-STS:power_state | 4 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | soft-delete |
| OS-SRV-USG:launched_at | 2014-10-07T16:30:31.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2014-10-07T16:29:56Z |
| flavor | m3.medium (301) |
| hostId | 20c72d81510bb46cadf9d01c33647a623ece760db6656fa66988a050 |
| id | b1667bd3-72ab-4173-80c0-5311d4c12a80 |
| image | UBNT-1204-LTS (20be54f6-b08f-43e7-b03f-eb7920572b35) |
| key_name | XXX |
| metadata | {} |
| name | sw-0929 |
| net-451 network | 172.24.72.9, 172.24.2.76 |
| os-extended-volumes:volumes_attached | [] |
| security_groups | default |
| status | SOFT_DELETED |
| tenant_id | a9071e35ed3044bfa97e7f37008ae9fc |
| updated | 2014-10-07T16:38:15Z |
| user_id | 5ad302819d0b4060b515f28fd7524199 ...

[Steve Baker (steve-stevebaker)]

http://git.openstack.org/cgit/openstack/heat/tree/heat/engine/resources/nova_utils.py#n274 needs to me modified to check for status SOFT_DELETED

maybe a check for endswith DELETED would suffice

[Sam Whyte (swhyte-v)]

Erring on the side of caution, I'd say an OR is probably a better bet, just in case some crazy person adds a "FAIL_DELETED" status.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/127822

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/127822
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=633a6bb14413b6a8523b6c142df32a7289e05a93
Submitter: Jenkins
Branch: master

commit 633a6bb14413b6a8523b6c142df32a7289e05a93
Author: Rakesh H S <email address hidden>
Date: Sun Oct 12 08:18:01 2014 +0530

    stack delete fails when instance is soft deleted

    In an environment where reclaim_instance_interval is set to
    non-zero in nova, an delete request will only soft delete an
    instance.(status=SOFT_DELETED, so that the instance can be reclaimed
    if required by using 'nova restore <instance_id>')

    Now in the above environment, when an delete stack is requested by
    user, stack will hang until reclaim_instance_inerval or timeout.
    Hence status=SOFT_DELETED for an instance should be handled by heat, which
    this patch proposes.

    Closes-Bug: #1377290

    Change-Id: I86ea788131f113e3deb7aebfccecd7f5e4957ae5

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/243783

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/243783
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=788b2e3a7e68cd1190baebbb99e3149c864ee3b8
Submitter: Jenkins
Branch: stable/juno

commit 788b2e3a7e68cd1190baebbb99e3149c864ee3b8
Author: Rakesh H S <email address hidden>
Date: Sun Oct 12 08:18:01 2014 +0530

    stack delete fails when instance is soft deleted

    In an environment where reclaim_instance_interval is set to
    non-zero in nova, an delete request will only soft delete an
    instance.(status=SOFT_DELETED, so that the instance can be reclaimed
    if required by using 'nova restore <instance_id>')

    Now in the above environment, when an delete stack is requested by
    user, stack will hang until reclaim_instance_inerval or timeout.
    Hence status=SOFT_DELETED for an instance should be handled by heat, which
    this patch proposes.

    Closes-Bug: #1377290

    Change-Id: I86ea788131f113e3deb7aebfccecd7f5e4957ae5
    (cherry picked from commit 633a6bb14413b6a8523b6c142df32a7289e05a93)