| 2014-03-31 05:36:45 |
Ryo Miki |
description |
I created stack with using attached file: before.template.
Then, I tried to update subnet with template that contain invalid property, attached file: after.template.
The update processing was failed.
It's normal behavior but Heat forgot "physical_resource_id" after update failed.
====================
$ heat resource-show test subnet
+------------------------+-----------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+------------------------+-----------------------------------------------------------------------------------------------------------------------------+
| description | |
| links | http://127.0.0.1:8004/v1/ee22f5b151084ebdb2f7f3ce787b9795/stacks/test/bbc3dd09-20bf-45d9-a41a-7556db84c554/resources/subnet |
| | http://127.0.0.1:8004/v1/ee22f5b151084ebdb2f7f3ce787b9795/stacks/test/bbc3dd09-20bf-45d9-a41a-7556db84c554 |
| logical_resource_id | subnet |
| physical_resource_id | |
| required_by | |
| resource_name | subnet |
| resource_status | CREATE_FAILED |
| resource_status_reason | NeutronClientException: Invalid input for allocation_pools. Reason: '192.168.10.300' is not a valid IP address. |
| resource_type | OS::Neutron::Subnet |
| updated_time | 2014-01-20T09:34:29Z |
+------------------------+-----------------------------------------------------------------------------------------------------------------------------+
====================
Apparently, this behavior would occur only case of UpdateReplace.
For this reason, deleting subnet is failed with incorrect API request.
====================
REQ: curl -i http://127.0.0.1:9696//v2.0/subnets/None.json -X DELETE -H "X-Auth-Token: 7c263797d2094de181e8c2f3b4a6b54d" -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-neutronclient"
from (pid=28018) http_log_req /opt/stack/python-neutronclient/neutronclient/common/utils.py:176
2014-01-20 18:34:58.667 DEBUG neutronclient.client [-] RESP:{'date': 'Mon, 20 Jan 2014 09:34:58 GMT', 'status': '404', 'content-length': '52', 'content-type': 'text/plain; charset=UTF-8'} 404 Not Found
The resource could not be found.
from (pid=28018) http_log_resp /opt/stack/python-neutronclient/neutronclient/common/utils.py:182
2014-01-20 18:34:58.668 DEBUG neutronclient.v2_0.client [-] Error message: 404 Not Found
The resource could not be found.
from (pid=28018) _handle_fault_response /opt/stack/python-neutronclient/neutronclient/v2_0/client.py:1126
====================
Heat received 404 and Heat misunderstood what the resource was already deleted.
But, resource was exist at that time.
Eventually, Heat delete network and subnet would deleted together.
However, this behavior isn't correct. |
=============================================================
This is updated bug description that is belong to discussion.
We find there are two ploblems about Heat:
1) when UpdateReplace is failed, cannot delete the Stack if there are
dependency between backup_stack and existing_stack.
2) cannot access backup_stack via API.
This bug report treats 1). New bug report about 2) will be created.
=============================================================
* Problem and Reproduce
When deleting the stack after UPDATE_FAILED, it causes DELETE_FAILED and "Error
deleting backup resources" in the Stack. This is an example.
devstack@heat-test-env:~/work_heat/updatereplace$ cat template.before
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Resources" : {
"Net": {
"Type": "OS::Neutron::Net",
"Properties": {
"name": "network"
}
},
"SubNet": {
"Type": "OS::Neutron::Subnet",
"Properties": {
"network_id": {"Ref": "Net"},
"ip_version": 4,
"cidr": "10.0.1.0/24",
"allocation_pools": [{"start": "10.0.1.20", "end": "10.0.1.150"}]
}
},
"Port": {
"Type": "OS::Neutron::Port",
"Properties": {
"network_id": {"Ref": "Net"},
"fixed_ips": [{"subnet_id": {"Ref": "SubNet"}, "ip_address": "10.0.1.30"}]
}
}
}
}
devstack@heat-test-env:~/work_heat/updatereplace$ heat stack-create -f template.before stack1
+--------------------------------------+------------+--------------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+--------------------+----------------------+
| 8ea7366e-a870-495b-9854-86b007f4d927 | stack1 | CREATE_IN_PROGRESS | 2014-03-28T05:24:15Z |
+--------------------------------------+------------+--------------------+----------------------+
devstack@heat-test-env:~/work_heat/updatereplace$ heat stack-list
+--------------------------------------+------------+-----------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+-----------------+----------------------+
| 8ea7366e-a870-495b-9854-86b007f4d927 | stack1 | CREATE_COMPLETE | 2014-03-28T05:24:15Z |
+--------------------------------------+------------+-----------------+----------------------+
devstack@heat-test-env:~/work_heat/updatereplace$ cat template.after
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Resources" : {
"Net": {
"Type": "OS::Neutron::Net",
"Properties": {
"name": "network"
}
},
"SubNet": {
"Type": "OS::Neutron::Subnet",
"Properties": {
"network_id": {"Ref": "Net"},
"ip_version": 4,
"cidr": "10.0.2.0/24",
"allocation_pools": [{"start": "10.0.2.20", "end": "10.0.2.999"}]
}
},
"Port": {
"Type": "OS::Neutron::Port",
"Properties": {
"network_id": {"Ref": "Net"},
"fixed_ips": [{"subnet_id": {"Ref": "SubNet"}, "ip_address": "10.0.2.30"}]
}
}
}
}
devstack@heat-test-env:~/work_heat/updatereplace$ heat stack-update -f template.after stack1
+--------------------------------------+------------+--------------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+--------------------+----------------------+
| 8ea7366e-a870-495b-9854-86b007f4d927 | stack1 | UPDATE_IN_PROGRESS | 2014-03-28T05:24:15Z |
+--------------------------------------+------------+--------------------+----------------------+
devstack@heat-test-env:~/work_heat/updatereplace$ heat stack-list
+--------------------------------------+------------+---------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+---------------+----------------------+
| 8ea7366e-a870-495b-9854-86b007f4d927 | stack1 | UPDATE_FAILED | 2014-03-28T05:24:15Z |
+--------------------------------------+------------+---------------+----------------------+
devstack@heat-test-env:~/work_heat/updatereplace$ heat stack-delete stack1
+--------------------------------------+------------+--------------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+--------------------+----------------------+
| 8ea7366e-a870-495b-9854-86b007f4d927 | stack1 | DELETE_IN_PROGRESS | 2014-03-28T05:24:15Z |
+--------------------------------------+------------+--------------------+----------------------+
devstack@heat-test-env:~/work_heat/updatereplace$ heat stack-list
+--------------------------------------+------------+---------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+---------------+----------------------+
| 8ea7366e-a870-495b-9854-86b007f4d927 | stack1 | DELETE_FAILED | 2014-03-28T05:24:15Z |
+--------------------------------------+------------+---------------+----------------------+
devstack@heat-test-env:~/work_heat/updatereplace$ heat stack-show stack1
+----------------------+-------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+----------------------+-------------------------------------------------------------------------------------------------------------------------------+
| capabilities | [] |
| creation_time | 2014-03-28T05:24:15Z |
| description | No description |
| disable_rollback | True |
| id | 8ea7366e-a870-495b-9854-86b007f4d927 |
| links | http://192.168.10.99:8004/v1/c1685cb61c4243efa3660550a0b99627/stacks/stack1/8ea7366e-a870-495b-9854-86b007f4d927 |
| notification_topics | [] |
| parameters | { |
| | "AWS::StackId": "arn:openstack:heat::c1685cb61c4243efa3660550a0b99627:stacks/stack1/8ea7366e-a870-495b-9854-86b007f4d927", |
| | "AWS::Region": "ap-southeast-1", |
| | "AWS::StackName": "stack1" |
| | } |
| stack_name | stack1 |
| stack_status | DELETE_FAILED |
| stack_status_reason | Failed to DELETE : Error deleting backup resources: |
| | Resource DELETE failed: NeutronClientException: |
| | 409-{u'NeutronError': {u'message': u'Unable to complete |
| | operation on subnet b06cdbb0-8898-4ebb-999e- |
| | 5e7145322912. One or more ports have an IP allocation f |
| template_description | No description |
| timeout_mins | 60 |
| updated_time | 2014-03-28T05:25:21Z |
+----------------------+-------------------------------------------------------------------------------------------------------------------------------+
devstack@heat-test-env:~/work_heat/updatereplace$ heat resource-list stack1
+---------------+---------------------+-----------------+----------------------+
| resource_name | resource_type | resource_status | updated_time |
+---------------+---------------------+-----------------+----------------------+
| Net | OS::Neutron::Net | CREATE_COMPLETE | 2014-03-28T05:24:15Z |
| Port | OS::Neutron::Port | CREATE_COMPLETE | 2014-03-28T05:24:16Z |
| SubNet | OS::Neutron::Subnet | CREATE_FAILED | 2014-03-28T05:25:24Z |
+---------------+---------------------+-----------------+----------------------+
devstack@heat-test-env:~/work_heat/updatereplace$ heat resource-show stack1 SubNet
+------------------------+-----------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+------------------------+-----------------------------------------------------------------------------------------------------------------------------------+
| description | |
| links | http://192.168.10.99:8004/v1/c1685cb61c4243efa3660550a0b99627/stacks/stack1/8ea7366e-a870-495b-9854-86b007f4d927/resources/SubNet |
| | http://192.168.10.99:8004/v1/c1685cb61c4243efa3660550a0b99627/stacks/stack1/8ea7366e-a870-495b-9854-86b007f4d927 |
| logical_resource_id | SubNet |
| physical_resource_id | |
| required_by | Port |
| resource_name | SubNet |
| resource_status | CREATE_FAILED |
| resource_status_reason | NeutronClientException: Invalid input for allocation_pools. Reason: '10.0.2.999' is not a valid IP address. |
| resource_type | OS::Neutron::Subnet |
| updated_time | 2014-03-28T05:25:24Z |
+------------------------+-----------------------------------------------------------------------------------------------------------------------------------+
* IMO
When deleting stack, Heat attempts to delete backup_stack if it exists.
( please see delete method on parser.py )
There is only SubNet in the backup_stack that has the Port. So the Subnet
cannot be deleted (cannot delete subnets before all ports has been deleted).
mysql> select id, nova_instance, name, status, stack_id from resource;
+--------------------------------------+--------------------------------------+--------+----------+--------------------------------------+
| id | nova_instance | name | status | stack_id |
+--------------------------------------+--------------------------------------+--------+----------+--------------------------------------+
| 21d8983e-4978-435c-9b71-028d405d4b4a | 3a3c6ecf-de94-4a6e-a64c-d6ae65af4fb4 | Net | COMPLETE | 9972abb2-686e-43b2-828a-f933b65824ce |
| 58d92166-e6aa-4d6d-be0d-778e96be8e52 | e67aebe4-7dfb-4a95-ac2f-d0f2cd5cfbe9 | SubNet | FAILED | 1eb8a9ee-772b-4c98-a364-846fb917e054 |
| 8969e3bb-e9be-4cef-8a0d-12f005769a6c | d328113e-5c28-4d38-9f69-19cc5bc40ad9 | Port | COMPLETE | 9972abb2-686e-43b2-828a-f933b65824ce |
| ed16b409-a875-4c1c-b3b5-986fcbe9d88f | NULL | SubNet | FAILED | 9972abb2-686e-43b2-828a-f933b65824ce |
+--------------------------------------+--------------------------------------+--------+----------+--------------------------------------+
4 rows in set (0.00 sec) |
|
