Status would never change from DELETE_IN_PROGRESS when instance deletion fails while deleting a stack

[Mitsuru Kanabuchi]

[Issue]

I'm checking the behavior when system failure cases.
Stack status never changed from DELETE_IN_PROGRESS when cinder-api is in down.

[How to reproduce]

1) Create stack from following template, this template is using boot from volume.

$ cat template
{
  "AWSTemplateFormatVersion" : "2010-09-09",
  "Resources": {
    "instance": {
      "Type": "OS::Nova::Server",
      "Properties": {
        "flavor": "m1.tiny",
        "block_device_mapping": [
          {
            "device_name": "vda",
            "volume_id": { "Ref": "volume" }
          }
        ]
      }
    },
    "volume": {
      "Type": "OS::Cinder::Volume",
      "Properties": {
        "size": 1,
        "image": "36e138bb-d9c4-4c3b-a328-a18ab6d8fb10"
      }
    }
  }
}
$ heat stack-create -f template test
+--------------------------------------+------------+--------------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+--------------------+----------------------+
| 895bfcb1-9565-45b3-aec1-1a225a6fd6f2 | test | CREATE_IN_PROGRESS | 2014-06-19T04:02:32Z |
+--------------------------------------+------------+--------------------+----------------------+
$ heat stack-list
+--------------------------------------+------------+-----------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+-----------------+----------------------+
| 895bfcb1-9565-45b3-aec1-1a225a6fd6f2 | test | CREATE_COMPLETE | 2014-06-19T04:02:32Z |
+--------------------------------------+------------+-----------------+----------------------+

2) kill cinder-api processes.

$ ps aux|grep cinder-api|grep -v grep
devstack 517 1.9 0.7 51724 15332 pts/46 S 12:51 0:17 /usr/bin/python /opt/stack/cinder/bin/cinder-api --config-file /etc/cinder/cinder.conf
devstack 523 1.1 2.5 79812 53108 pts/46 S 12:51 0:10 /usr/bin/python /opt/stack/cinder/bin/cinder-api --config-file /etc/cinder/cinder.conf
$ kill -SIGKILL 517 523
$ ps aux|grep cinder-api|grep -v grep
$

3) Delete the stack.

$ heat stack-delete 895bfcb1-9565-45b3-aec1-1a225a6fd6f2
+--------------------------------------+------------+--------------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+--------------------+----------------------+
| 895bfcb1-9565-45b3-aec1-1a225a6fd6f2 | test | DELETE_IN_PROGRESS | 2014-06-19T04:02:32Z |
+--------------------------------------+------------+--------------------+----------------------+

4) Stack status never changed from DELETE_IN_PROGRESS, however nova's instance status already set ERROR.

$ heat stack-delete 895bfcb1-9565-45b3-aec1-1a225a6fd6f2
+--------------------------------------+------------+--------------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+--------------------+----------------------+
| 895bfcb1-9565-45b3-aec1-1a225a6fd6f2 | test | DELETE_IN_PROGRESS | 2014-06-19T04:02:32Z |
+--------------------------------------+------------+--------------------+----------------------+
$ nova list
+--------------------------------------+----------------------------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+----------------------------+--------+------------+-------------+----------+
| 387c4bcd-9d6c-44dd-ae4d-2f5e4a13a3d2 | test-instance-2bcs5qpg7lit | ERROR | - | Running | |
+--------------------------------------+----------------------------+--------+------------+-------------+----------+
$ heat stack-list
+--------------------------------------+------------+--------------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+--------------------+----------------------+
| 895bfcb1-9565-45b3-aec1-1a225a6fd6f2 | test | DELETE_IN_PROGRESS | 2014-06-19T04:02:32Z |
+--------------------------------------+------------+--------------------+----------------------+

[etc]

In addition, we can reproduce same behavior when neutron-server is in down.

In my understanding, heat-engine recognize instance deletion complete by 404.
But in case of instance deletion failed, nova would never reply 404 because instance isn't gone yet.

I think, nova_utils.py should check instance's status.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/104454

[Rakesh H S (rh-s)]

I was able to reproduce the issue, and have proposed the fix for the same.

[Rakesh H S (rh-s)]

https://review.openstack.org/#/c/102637/ has fixed this defect, I am abandoning my patch https://review.openstack.org/#/c/104454/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Rakesh H S (<email address hidden>) on branch: master
Review: https://review.openstack.org/104454
Reason: While rebasing, figured out that https://review.openstack.org/#/c/102637/ has fixed this defect.