Status would never change from DELETE_IN_PROGRESS when instance deletion fails while deleting a stack

Bug #1331916 reported by Mitsuru Kanabuchi
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
Undecided
Rakesh H S

Bug Description

[Issue]

I'm checking the behavior when system failure cases.
Stack status never changed from DELETE_IN_PROGRESS when cinder-api is in down.

[How to reproduce]

1) Create stack from following template, this template is using boot from volume.

$ cat template
{
  "AWSTemplateFormatVersion" : "2010-09-09",
  "Resources": {
    "instance": {
      "Type": "OS::Nova::Server",
      "Properties": {
        "flavor": "m1.tiny",
        "block_device_mapping": [
          {
            "device_name": "vda",
            "volume_id": { "Ref": "volume" }
          }
        ]
      }
    },
    "volume": {
      "Type": "OS::Cinder::Volume",
      "Properties": {
        "size": 1,
        "image": "36e138bb-d9c4-4c3b-a328-a18ab6d8fb10"
      }
    }
  }
}
$ heat stack-create -f template test
+--------------------------------------+------------+--------------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+--------------------+----------------------+
| 895bfcb1-9565-45b3-aec1-1a225a6fd6f2 | test | CREATE_IN_PROGRESS | 2014-06-19T04:02:32Z |
+--------------------------------------+------------+--------------------+----------------------+
$ heat stack-list
+--------------------------------------+------------+-----------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+-----------------+----------------------+
| 895bfcb1-9565-45b3-aec1-1a225a6fd6f2 | test | CREATE_COMPLETE | 2014-06-19T04:02:32Z |
+--------------------------------------+------------+-----------------+----------------------+

2) kill cinder-api processes.

$ ps aux|grep cinder-api|grep -v grep
devstack 517 1.9 0.7 51724 15332 pts/46 S 12:51 0:17 /usr/bin/python /opt/stack/cinder/bin/cinder-api --config-file /etc/cinder/cinder.conf
devstack 523 1.1 2.5 79812 53108 pts/46 S 12:51 0:10 /usr/bin/python /opt/stack/cinder/bin/cinder-api --config-file /etc/cinder/cinder.conf
$ kill -SIGKILL 517 523
$ ps aux|grep cinder-api|grep -v grep
$

3) Delete the stack.

$ heat stack-delete 895bfcb1-9565-45b3-aec1-1a225a6fd6f2
+--------------------------------------+------------+--------------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+--------------------+----------------------+
| 895bfcb1-9565-45b3-aec1-1a225a6fd6f2 | test | DELETE_IN_PROGRESS | 2014-06-19T04:02:32Z |
+--------------------------------------+------------+--------------------+----------------------+

4) Stack status never changed from DELETE_IN_PROGRESS, however nova's instance status already set ERROR.

$ heat stack-delete 895bfcb1-9565-45b3-aec1-1a225a6fd6f2
+--------------------------------------+------------+--------------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+--------------------+----------------------+
| 895bfcb1-9565-45b3-aec1-1a225a6fd6f2 | test | DELETE_IN_PROGRESS | 2014-06-19T04:02:32Z |
+--------------------------------------+------------+--------------------+----------------------+
$ nova list
+--------------------------------------+----------------------------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+----------------------------+--------+------------+-------------+----------+
| 387c4bcd-9d6c-44dd-ae4d-2f5e4a13a3d2 | test-instance-2bcs5qpg7lit | ERROR | - | Running | |
+--------------------------------------+----------------------------+--------+------------+-------------+----------+
$ heat stack-list
+--------------------------------------+------------+--------------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+--------------------+----------------------+
| 895bfcb1-9565-45b3-aec1-1a225a6fd6f2 | test | DELETE_IN_PROGRESS | 2014-06-19T04:02:32Z |
+--------------------------------------+------------+--------------------+----------------------+

[etc]

In addition, we can reproduce same behavior when neutron-server is in down.

In my understanding, heat-engine recognize instance deletion complete by 404.
But in case of instance deletion failed, nova would never reply 404 because instance isn't gone yet.

I think, nova_utils.py should check instance's status.

description: updated
Rakesh H S (rh-s)
Changed in heat:
assignee: nobody → Rakesh H S (rh-s)
Rakesh H S (rh-s)
Changed in heat:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/104454

Revision history for this message
Rakesh H S (rh-s) wrote : Re: Status would never changed from DELETE_IN_PROGRESS when instance deletion failed

I was able to reproduce the issue, and have proposed the fix for the same.

summary: - Status would never changed from DELETE_IN_PROGRESS when instance
- deletion failed
+ Status would never change from DELETE_IN_PROGRESS when instance deletion
+ fails while deleting a stack
Revision history for this message
Rakesh H S (rh-s) wrote :
Changed in heat:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on heat (master)

Change abandoned by Rakesh H S (<email address hidden>) on branch: master
Review: https://review.openstack.org/104454
Reason: While rebasing, figured out that https://review.openstack.org/#/c/102637/ has fixed this defect.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.