OpenStack Heat

More instances will be created when reupdate the stack twice after stack update failed the first time

Bug #1381834 reported by Shunli Zhou
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
Opinion
Undecided
Unassigned

Bug Description

1, My heat template as follow
heat_template_version: 2013-05-23

resources:
  her_policy_group:
    type: OS::Nova::ServerGroup
    # A Nova ServerGroup
    properties:
      name: nova-server-group
      policies: [affinity]

  her_asg:
    type: OS::Heat::AutoScalingGroup
    properties:
      resource:
        type: OS::Nova::Server
        properties:
            image: "tmpl"
            flavor: test
            scheduler_hints: {group: {get_resource: her_policy_group}}
      min_size: 1
      desired_capacity: 7
      max_size: 11

  scale_up_policy:
    type: OS::Heat::ScalingPolicy
    properties:
      adjustment_type: change_in_capacity
      auto_scaling_group_id: {get_resource: her_asg}
      cooldown: 60
      scaling_adjustment: 1

  scale_down_policy:
    type: OS::Heat::ScalingPolicy
    properties:
      adjustment_type: change_in_capacity
      auto_scaling_group_id: {get_resource: her_asg}
      cooldown: 60
      scaling_adjustment: '-1'
2, Create stack by follow command:
[root@cmo-controller opt]#heat stack-create -f auto.tmpl auto

3,[root@cmo-controller opt]# heat stack-list
+--------------------------------------+------------+-----------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+-----------------+----------------------+
| 57e15a8a-b891-4f23-8819-5d2ca0ff7495 | auto | CREATE_COMPLETE | 2014-10-13T18:15:28Z |

4,[root@cmo-controller opt]# nova list
+--------------------------------------+-----------------------------------------------------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-----------------------------------------------------+--------+------------+-------------+----------+
| 7dbd50d3-b7e9-4c60-95e4-4b6d2d272597 | auto-her_asg-qmy7o4tea5lq-47nhpnd2mfps-2geqw2htib2v | ACTIVE | - | Running | |
| a2dc1e34-b118-4064-bf8a-d8f3fea4f96d | auto-her_asg-qmy7o4tea5lq-5xxuynobudmx-tur2ffwcc427 | ACTIVE | - | Running | |
| fc7b5de6-251b-4e2b-a101-1391803dd534 | auto-her_asg-qmy7o4tea5lq-ay5mz5dtlkya-cemfuoztnowl | ACTIVE | - | Running | |
| 62790180-b271-4365-add3-96ddbdc0f058 | auto-her_asg-qmy7o4tea5lq-g3jvhionlf7v-dusk2xwyjpel | ACTIVE | - | Running | |
| 54c752ec-1bcc-49b6-b667-f0ef058e4632 | auto-her_asg-qmy7o4tea5lq-ief4jql3a3t5-26oztahhkh2y | ACTIVE | - | Running | |
| 2d833de9-327f-4b56-903b-d74d6c02e137 | auto-her_asg-qmy7o4tea5lq-mmuiddwncmoh-d43u4hrdn4ak | ACTIVE | - | Running | |
| f8e1b800-35f0-4530-9b37-1cf1a4a77bfe | auto-her_asg-qmy7o4tea5lq-zwwktumi557g-jq2tlhtqaq4z | ACTIVE | - | Running | |
+--------------------------------------+-----------------------------------------------------+--------+------------+-------------+----------+
+--------------------------------------+------------+-----------------+----------------------+

There are 7 instances.

5, modify the auto.tmpl
heat_template_version: 2013-05-23

resources:
  her_policy_group:
    type: OS::Nova::ServerGroup
    # A Nova ServerGroup
    properties:
      name: nova-server-group
      policies: [affinity]

  her_asg:
    type: OS::Heat::AutoScalingGroup
    properties:
      resource:
        type: OS::Nova::Server
        properties:
            image: "tmpl"
            flavor: test
            scheduler_hints: {group: {get_resource: her_policy_group}}
      min_size: 1
      desired_capacity: 11 =========================> 7->11, which is exceed quota 10
      max_size: 11

  scale_up_policy:
    type: OS::Heat::ScalingPolicy
    properties:
      adjustment_type: change_in_capacity
      auto_scaling_group_id: {get_resource: her_asg}
      cooldown: 60
      scaling_adjustment: 1

  scale_down_policy:
    type: OS::Heat::ScalingPolicy
    properties:
      adjustment_type: change_in_capacity
      auto_scaling_group_id: {get_resource: her_asg}
      cooldown: 60
      scaling_adjustment: '-1'

6.[root@cmo-controller opt]# heat stack-update -f auto.tmpl auto
+--------------------------------------+------------+--------------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+--------------------+----------------------+
| 57e15a8a-b891-4f23-8819-5d2ca0ff7495 | auto | UPDATE_IN_PROGRESS | 2014-10-13T18:15:28Z |
+--------------------------------------+------------+--------------------+----------------------+
 [root@cmo-controller opt]# heat stack-list
+--------------------------------------+------------+---------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+---------------+----------------------+
| 57e15a8a-b891-4f23-8819-5d2ca0ff7495 | auto | UPDATE_FAILED | 2014-10-13T18:15:28Z |
+--------------------------------------+------------+---------------+----------------------+

7,[root@cmo-controller opt]# nova list
+--------------------------------------+-----------------------------------------------------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-----------------------------------------------------+--------+------------+-------------+----------+
| 7dbd50d3-b7e9-4c60-95e4-4b6d2d272597 | auto-her_asg-qmy7o4tea5lq-47nhpnd2mfps-2geqw2htib2v | ACTIVE | - | Running | |
| a2dc1e34-b118-4064-bf8a-d8f3fea4f96d | auto-her_asg-qmy7o4tea5lq-5xxuynobudmx-tur2ffwcc427 | ACTIVE | - | Running | |
| fc7b5de6-251b-4e2b-a101-1391803dd534 | auto-her_asg-qmy7o4tea5lq-ay5mz5dtlkya-cemfuoztnowl | ACTIVE | - | Running | |
| 11c2a2c2-04a0-4642-9047-45e015ef8394 | auto-her_asg-qmy7o4tea5lq-bqnruy5up3vb-gk7pfhdsfycz | ACTIVE | - | Running | |
| f3f65cf4-55ba-427d-8bbd-d099e9a4fa12 | auto-her_asg-qmy7o4tea5lq-budyamquc524-ylteh6x33vst | ACTIVE | - | Running | |
| 62790180-b271-4365-add3-96ddbdc0f058 | auto-her_asg-qmy7o4tea5lq-g3jvhionlf7v-dusk2xwyjpel | ACTIVE | - | Running | |
| 54c752ec-1bcc-49b6-b667-f0ef058e4632 | auto-her_asg-qmy7o4tea5lq-ief4jql3a3t5-26oztahhkh2y | ACTIVE | - | Running | |
| c76420a2-e34d-40e4-a2d5-c76c5ee62dd5 | auto-her_asg-qmy7o4tea5lq-kd2p3pkhyohp-pazeigou4lbf | ACTIVE | - | Running | |
| 2d833de9-327f-4b56-903b-d74d6c02e137 | auto-her_asg-qmy7o4tea5lq-mmuiddwncmoh-d43u4hrdn4ak | ACTIVE | - | Running | |
| f8e1b800-35f0-4530-9b37-1cf1a4a77bfe | auto-her_asg-qmy7o4tea5lq-zwwktumi557g-jq2tlhtqaq4z | ACTIVE | - | Running | |
+--------------------------------------+-----------------------------------------------------+--------+------------+-------------+----------+

There are 10 instances. One instance missing due to exceeding the quota. Stack status in the failed status.

8, 7. [root@cmo-controller opt]# nova delete 62790180-b271-4365-add3-96ddbdc0f058 54c752ec-1bcc-49b6-b667-f0ef058e4632 c76420a2-e34d-40e4-a2d5-c76c5ee62dd5 2d833de9-327f-4b56-903b-d74d6c02e137 f8e1b800-35f0-4530-9b37-1cf1a4a77bfe
Request to delete server 62790180-b271-4365-add3-96ddbdc0f058 has been accepted.
Request to delete server 54c752ec-1bcc-49b6-b667-f0ef058e4632 has been accepted.
Request to delete server c76420a2-e34d-40e4-a2d5-c76c5ee62dd5 has been accepted.
Request to delete server 2d833de9-327f-4b56-903b-d74d6c02e137 has been accepted.
Request to delete server f8e1b800-35f0-4530-9b37-1cf1a4a77bfe has been accepted.

Delete 5 instances from the stack.

[root@cmo-controller opt]# nova list
+--------------------------------------+-----------------------------------------------------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-----------------------------------------------------+--------+------------+-------------+----------+
| 7dbd50d3-b7e9-4c60-95e4-4b6d2d272597 | auto-her_asg-qmy7o4tea5lq-47nhpnd2mfps-2geqw2htib2v | ACTIVE | - | Running | |
| a2dc1e34-b118-4064-bf8a-d8f3fea4f96d | auto-her_asg-qmy7o4tea5lq-5xxuynobudmx-tur2ffwcc427 | ACTIVE | - | Running | |
| fc7b5de6-251b-4e2b-a101-1391803dd534 | auto-her_asg-qmy7o4tea5lq-ay5mz5dtlkya-cemfuoztnowl | ACTIVE | - | Running | |
| 11c2a2c2-04a0-4642-9047-45e015ef8394 | auto-her_asg-qmy7o4tea5lq-bqnruy5up3vb-gk7pfhdsfycz | ACTIVE | - | Running | |
| f3f65cf4-55ba-427d-8bbd-d099e9a4fa12 | auto-her_asg-qmy7o4tea5lq-budyamquc524-ylteh6x33vst | ACTIVE | - | Running | |
+--------------------------------------+-----------------------------------------------------+--------+------------+-------------+----------+

9,Modify the template

[root@cmo-controller opt]# cat auto.tmpl
heat_template_version: 2013-05-23

resources:
  her_policy_group:
    type: OS::Nova::ServerGroup
    # A Nova ServerGroup
    properties:
      name: nova-server-group
      policies: [affinity]

  her_asg:
    type: OS::Heat::AutoScalingGroup
    properties:
      resource:
        type: OS::Nova::Server
        properties:
            image: "tmpl"
            flavor: test
            scheduler_hints: {group: {get_resource: her_policy_group}}
      min_size: 1
      desired_capacity: 8 ============> change from 11->8
      max_size: 11

  scale_up_policy:
    type: OS::Heat::ScalingPolicy
    properties:
      adjustment_type: change_in_capacity
      auto_scaling_group_id: {get_resource: her_asg}
      cooldown: 60
      scaling_adjustment: 1

  scale_down_policy:
    type: OS::Heat::ScalingPolicy
    properties:
      adjustment_type: change_in_capacity
      auto_scaling_group_id: {get_resource: her_asg}
      cooldown: 60
      scaling_adjustment: '-1'

10,[root@cmo-controller opt]# heat stack-update -f auto.tmpl auto
+--------------------------------------+------------+--------------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+--------------------+----------------------+
| 57e15a8a-b891-4f23-8819-5d2ca0ff7495 | auto | UPDATE_IN_PROGRESS | 2014-10-13T18:15:28Z |
+--------------------------------------+------------+--------------------+----------------------+

11.[root@cmo-controller opt]# nova list
+--------------------------------------+-----------------------------------------------------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-----------------------------------------------------+--------+------------+-------------+----------+
| 7dbd50d3-b7e9-4c60-95e4-4b6d2d272597 | auto-her_asg-qmy7o4tea5lq-47nhpnd2mfps-2geqw2htib2v | ACTIVE | - | Running | |
| a2dc1e34-b118-4064-bf8a-d8f3fea4f96d | auto-her_asg-qmy7o4tea5lq-5xxuynobudmx-tur2ffwcc427 | ACTIVE | - | Running | |
| fc7b5de6-251b-4e2b-a101-1391803dd534 | auto-her_asg-qmy7o4tea5lq-ay5mz5dtlkya-cemfuoztnowl | ACTIVE | - | Running | |
| 11c2a2c2-04a0-4642-9047-45e015ef8394 | auto-her_asg-qmy7o4tea5lq-bqnruy5up3vb-gk7pfhdsfycz | ACTIVE | - | Running | |
| f3f65cf4-55ba-427d-8bbd-d099e9a4fa12 | auto-her_asg-qmy7o4tea5lq-budyamquc524-ylteh6x33vst | ACTIVE | - | Running | |
| 6628cf98-9aa4-425c-854a-ca4f9b58f924 | auto-her_asg-r7hamhba2ch3-6oqwxo4pn5kd-3knnpqapowd4 | ACTIVE | - | Running | |
| 0f818d17-b368-46f9-939a-02674618ab71 | auto-her_asg-r7hamhba2ch3-cbfeah5n4ux5-qantck72s2p4 | ACTIVE | - | Running | |
| 354625a9-22b7-4677-aa21-f516bc2ed192 | auto-her_asg-r7hamhba2ch3-lpg3vs72j3of-j5lqz2atbzyl | ACTIVE | - | Running | |
| 33f40511-9825-4fd1-b6fa-9acb4c1a8e14 | auto-her_asg-r7hamhba2ch3-mc2nsxva6kbx-pigchqqzvzso | ACTIVE | - | Running | |
| ed79de42-8576-4d10-b775-e2bb304aaf05 | auto-her_asg-r7hamhba2ch3-yee3eiig7h64-bkha2frftmzf | ACTIVE | - | Running | |
+--------------------------------------+-----------------------------------------------------+--------+------------+-------------+----------+

There are 10 instance created!!!

I expected that the stack update will create 8 instances in the stack, but 10 instances created. Seems stack update cached the last update desired_capacity, not use the latest desired_capacity in the template.

Revision history for this message
Shunli Zhou (shunliz) wrote :

Also notice that when i debug the heat\heat\engine\resources\autoscaling.py::handle_update, update to 8, code even not run to autoscaling.py::handle_update.

Revision history for this message
XiaoLiang Hu (xlhuxa) wrote :

hello, all , any update on this bug?

Revision history for this message
Steve Baker (steve-stevebaker) wrote :

This might be a duplicate of bug #1383709 which is being investigated.

Revision history for this message
Angus Salkeld (asalkeld) wrote :

So the issue here is you are deleting servers outside of heat. This is currently not supported.
See: https://bugs.launchpad.net/heat/+bug/1260167

Angus Salkeld (asalkeld)
Changed in heat:
status: New → Opinion
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.