stack update action doesn't take effect

[XiaoLiang Hu]

Just try "stack-update" case in order to increase/reduce the nova instances on below resource type, It seems heat engine will base on the original instance request number (at first create in template) of the resource to judge whether need adjustment (create/delete nova instance) or ignore the request.

1. OS::Heat::InstanceGroup : The attribute "Size" in resource properties represents the request number of nova instance which belong to this resource when create/update stack
2. OS::Heat::AutoScalingGroup : The attribute "desired_capacity" in resource properties represents the request number of nova instance which belong to this resource when create/update stack
3. OS::Heat::ResourceGroup: The attribute "Count" in resource properties represents the request number of nova instance which belong to this resource when create/update stack
4. AWS::AutoScaling::AutoScalingGroup: The attribute "DesiredCapacity" in resource properties represents the request number of nova instance which belong to this resource when create/update stack

For example test on "OS::Heat::InstanceGroup" :

step 1: create stack with below template

[root@xhu01 os_instance_gp]# more my_instance_group_base.template

heat_template_version: 2013-05-23

resources:
  my_inst_gp:
    type: OS::Heat::InstanceGroup
    properties:
        AvailabilityZones: [nova]
        Size: 1
        LaunchConfigurationName: {Ref: launchconfig_my}

  she_inst_gp:
    type: OS::Heat::InstanceGroup
    properties:
        AvailabilityZones: [nova]
        Size: 1
        LaunchConfigurationName: {Ref: launchconfig_she}

  launchconfig_my:
    type: AWS::AutoScaling::LaunchConfiguration
    properties:
       ImageId: yy
       InstanceType: xhu.test

  launchconfig_she:
    type: AWS::AutoScaling::LaunchConfiguration
    properties:
       ImageId: yy
       InstanceType: xhu.test

[root@xhu01 os_instance_gp]# heat stack-create -f my_instance_group_base.template xx
+--------------------------------------+------------+--------------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+--------------------+----------------------+
| 61688284-c599-4f3c-8427-63fb6c2f91ac | xx | CREATE_IN_PROGRESS | 2014-10-05T16:22:56Z |
+--------------------------------------+------------+--------------------+----------------------+
[root@xhu01 os_instance_gp]#

step 2: check nova instance list, it is ok, each resource "OS::Heat::InstanceGroup" has one nova instance

[root@xhu01 ~]# nova list
+--------------------------------------+-------------------------------------------------------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------------------------------------------------------+--------+------------+-------------+----------+
| 6e614afe-4d71-4f94-a8a5-901a6ca8824c | xx-my_inst_gp-eo6qkxl7pvop-yqdb6nc5pp4a-7v5wo3ca5dal | ACTIVE | - | Running | |
| 6ee6fa06-71f0-492d-b7fa-5620407b9d63 | xx-she_inst_gp-tnpq4adxcq4d-sp42vvhilkj3-inzgpjl6goir | ACTIVE | - | Running | |
+--------------------------------------+-------------------------------------------------------+--------+------------+-------------+----------+
[root@xhu01 ~]#

step 3: update template, change 'Size' from 1 to 3 on the two "OS::Heat::InstanceGroup" resource

step 4: run stack-update command to update stack with the latest template

[root@xhu01 os_instance_gp]# heat stack-update -f my_instance_group_base.template xx
+--------------------------------------+------------+--------------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+--------------------+----------------------+
| 61688284-c599-4f3c-8427-63fb6c2f91ac | xx | UPDATE_IN_PROGRESS | 2014-10-05T16:22:56Z |
+--------------------------------------+------------+--------------------+----------------------+
[root@xhu01 os_instance_gp]#

step 5: check nova list, it is ok, each "OS::Heat::InstanceGroup" resource has 3 nova instances.

[root@xhu01 ~]# nova list
+--------------------------------------+-------------------------------------------------------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------------------------------------------------------+--------+------------+-------------+----------+
| c8e79939-d0d7-4441-9571-15992a323a86 | xx-my_inst_gp-eo6qkxl7pvop-im25urfnirtn-bd6gky4igmzh | ACTIVE | - | Running | |
| 6e614afe-4d71-4f94-a8a5-901a6ca8824c | xx-my_inst_gp-eo6qkxl7pvop-yqdb6nc5pp4a-7v5wo3ca5dal | ACTIVE | - | Running | |
| c8528622-0a05-4127-9b6f-9a9ad9afb7da | xx-my_inst_gp-eo6qkxl7pvop-z6po2se2wchp-bgyarrhrfh5e | ACTIVE | - | Running | |
| 6ee6fa06-71f0-492d-b7fa-5620407b9d63 | xx-she_inst_gp-tnpq4adxcq4d-sp42vvhilkj3-inzgpjl6goir | ACTIVE | - | Running | |
| f3ec7eb0-7f6f-4e02-9b4a-960ba0725516 | xx-she_inst_gp-tnpq4adxcq4d-x6zq5af6amb5-w5upnt6kgjp5 | ACTIVE | - | Running | |
| 9c8d6e91-fc01-4ec3-b625-d81f978ea86f | xx-she_inst_gp-tnpq4adxcq4d-ybfo6vv527ue-wvmdxap6lxha | ACTIVE | - | Running | |
+--------------------------------------+-------------------------------------------------------+--------+------------+-------------+----------+
[root@xhu01 ~]#

step 6: update template, change 'Size' from 3 to original value 1 on the two "OS::Heat::InstanceGroup" resource

step 7: run stack-update command to update stack with the latest template

[root@xhu01 os_instance_gp]# heat stack-update -f my_instance_group_base.template xx
+--------------------------------------+------------+--------------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+--------------------+----------------------+
| 61688284-c599-4f3c-8427-63fb6c2f91ac | xx | UPDATE_IN_PROGRESS | 2014-10-05T16:22:56Z |
+--------------------------------------+------------+--------------------+----------------------+
[root@xhu01 os_instance_gp]#

step 7: check nova list, no change for a long time.

>>>>>> >>> there is something wrong here: I think heat engine should delete 2 (3-1) on each "OS::Heat::InstanceGroup" resource for this case.

step 8: update template, change 'Size' from 1 to 2 on the two "OS::Heat::InstanceGroup" resource

step 9: run stack-update command to update stack with the latest template

[root@xhu01 os_instance_gp]# heat stack-update -f my_instance_group_base.template xx
+--------------------------------------+------------+--------------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+--------------------+----------------------+
| 61688284-c599-4f3c-8427-63fb6c2f91ac | xx | UPDATE_IN_PROGRESS | 2014-10-05T16:22:56Z |
+--------------------------------------+------------+--------------------+----------------------+
[root@xhu01 os_instance_gp]#

step 10: check nova list, it is ok for me, since heat engine deletes 1 (3-2) nova instance on each "OS::Heat::InstanceGroup" resource

[root@xhu01 ~]# nova list
+--------------------------------------+-------------------------------------------------------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------------------------------------------------------+--------+------------+-------------+----------+
| c8e79939-d0d7-4441-9571-15992a323a86 | xx-my_inst_gp-eo6qkxl7pvop-im25urfnirtn-bd6gky4igmzh | ACTIVE | - | Running | |
| c8528622-0a05-4127-9b6f-9a9ad9afb7da | xx-my_inst_gp-eo6qkxl7pvop-z6po2se2wchp-bgyarrhrfh5e | ACTIVE | - | Running | |
| f3ec7eb0-7f6f-4e02-9b4a-960ba0725516 | xx-she_inst_gp-tnpq4adxcq4d-x6zq5af6amb5-w5upnt6kgjp5 | ACTIVE | - | Running | |
| 9c8d6e91-fc01-4ec3-b625-d81f978ea86f | xx-she_inst_gp-tnpq4adxcq4d-ybfo6vv527ue-wvmdxap6lxha | ACTIVE | - | Running | |
+--------------------------------------+-------------------------------------------------------+--------+------------+-------------+----------+
[root@xhu01 ~]#

According to my test, the resource "OS::Heat::AutoScalingGroup", "OS::Heat::ResourceGroup" and "AWS::AutoScaling::AutoScalingGroup" have same issue when run "stack-update" follow above steps 1-10. So I think it should be a bug on heat engine for this case, heat engine should base on below conditions to handle stack update case:

1. a = the current instance number(base on "nova list") on the heat resource, such as "OS::Heat::InstanceGroup"
2. b = the request instance number in template, such as "Size" on "OS::Heat::InstanceGroup"

if b > a, create (b-a) nova instances for this resource
if a > b, delete (a-b) nova instances for this resource
if a ==b , ignore the request on this resource

[XiaoLiang Hu (xlhuxa)]

Hi, all, any comments on this issue?

[Angus Salkeld (asalkeld)]

I'll try reproduce now.

[Angus Salkeld (asalkeld)]

I can confirm the following behaviour:
start with 0 size
update to 2
update to 0 (says success, but still 2 servers)

I'll have a look at the code and see if this is intentional.

[Angus Salkeld (asalkeld)]

So the mechanism of detecting property changes seems broken:

I put some logging in here: https://github.com/openstack/heat/blob/master/heat/engine/resource.py#L648

And I get:
 WARNING heat.engine.resource [-] compare before:{u'Size': 1, u'AvailabilityZones': [u'nova'], u'LaunchConfigurationName': u'one-launchconfig_my-3uwu5xqmnr5n'} != after:{u'Size': 1, u'AvailabilityZones': [u'nova'], u'LaunchConfigurationName': <heat.engine.cfn.functions.ResourceRef {Ref: u'launchconfig_my'} -> u'one-launchconfig_my-3uwu5xqmnr5n'>}
2014-10-09 02:44:27.159 WARNING heat.engine.resource [-] InstanceGroup "my_inst_gp" [d9b17591-9ea6-489d-94a7-a35c95b70f5f] Stack "one" [ddd5bde5-c0f4-4dc7-b3d0-ec98de14d37f] not _needs_update

This is when I change Size from 1 to 0 or 0 to 1
For some wacky reason when I go from 1 to 2 it works. Could this be some interpretation of the integer 0 as bool or a default?

[Angus Salkeld (asalkeld)]

I did the following:
create stack (Size=0)
=================
  heat.engine.resource [-] _stored_properties_data {u'AvailabilityZones': [u'nova'], u'LaunchConfigurationName': u'one-launchconfig_my-c2duwc6qjnbw', u'Size': 0}

update stack (Size=1)
=================
heat.engine.reheat.engine.resource [-] ** frozen_definition {'properties': {u'Size': 0, u'AvailabilityZones': [u'nova'], u'LaunchConfigurationName': u'one-launchconfig_my-c2duwc6qjnbw'}}

...

2014-10-09 07:46:52.625 WARNING heat.engine.resources.autoscaling [-] ***handle_update prop_diff {'Size': 1}
2014-10-09 07:46:52.633 WARNING heat.engine.resources.autoscaling [-] current 0 requested 1
2014-10-09 07:46:52.633 WARNING heat.engine.resources.autoscaling [-] resize 1

2014-10-09 07:47:18.552 WARNING heat.engine.resource [-] _stored_properties_data {u'AvailabilityZones': [u'nova'], u'LaunchConfigurationName': u'one-launchconfig_my-c2duwc6qjnbw', u'Size': 1}

update stack (Size=2)
=================

2014-10-09 07:50:09.603 WARNING heat.engine.resource [-] ** frozen_definition {'properties': {u'Size': 0, u'AvailabilityZones': [u'nova'], u'LaunchConfigurationName': u'one-launchconfig_my-c2duwc6qjnbw'}}

!! Note above: "Size: 0", wtf it should be "1" - where did that come from?

2014-10-09 07:50:12.633 WARNING heat.engine.resource [-] ** update 0 2
2014-10-09 07:50:12.633 INFO heat.engine.resource [-] updating InstanceGroup "my_inst_gp" [bd47b301-3111-47c1-af6b-1067ae109ee7] Stack "one" [a725c21a-4f32-4b41-89e0-31234701cde1]
2014-10-09 07:50:12.652 WARNING heat.engine.resources.autoscaling [-] ***handle_update prop_diff {'Size': 2}
2014-10-09 07:50:12.668 WARNING heat.engine.resources.autoscaling [-] current 1 requested 2
# autoscaling counts it's own instances so it figures out there is actually 1 ^
2014-10-09 07:50:12.669 WARNING heat.engine.resources.autoscaling [-] resize 2

2014-10-09 07:50:44.876 WARNING heat.engine.resource [-] _stored_properties_data {u'AvailabilityZones': [u'nova'], u'LaunchConfigurationName': u'one-launchconfig_my-c2duwc6qjnbw', u'Size': 2}

With all the stacks lying around (existing, current, new, backup) we could loading the properties from the wrong stack (one that is too old).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/127393

[Angus Salkeld (asalkeld)]

Fixing this bug causes: https://bugs.launchpad.net/heat/+bug/1379619

I think we should only submit this to juno-rc3 if 1379619 can also be fixed - waiting for shardy to give that a go.

[Thierry Carrez (ttx)]

OK, let's punt it back to rc-potential then

[Zane Bitter (zaneb)]

IMHO badness(every stack update is wrong) >> badness(stack updates of exactly one resource type are wrong)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/127393
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=d32370233eaf2a5c32888f269bd1dc5e0e787467
Submitter: Jenkins
Branch: master

commit d32370233eaf2a5c32888f269bd1dc5e0e787467
Author: Angus Salkeld <email address hidden>
Date: Fri Oct 10 12:00:37 2014 +1000

    Make sure that the properties are stored on updates

    Previously properties_data was only stored on creation, now
    this is getting passed to the update mechanism too.
    Later we can look at reworking this into a single mechanism.

    Closes-bug: #1377681
    Change-Id: If3d476f34b9e61a3c99f63ba33734a875353c8fc

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: proposed/juno
Review: https://review.openstack.org/127707

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/127707
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=5aa66555209eb5a59abdb189b0a5d1224e73b566
Submitter: Jenkins
Branch: proposed/juno

commit 5aa66555209eb5a59abdb189b0a5d1224e73b566
Author: Angus Salkeld <email address hidden>
Date: Fri Oct 10 12:00:37 2014 +1000

    Make sure that the properties are stored on updates

    Previously properties_data was only stored on creation, now
    this is getting passed to the update mechanism too.
    Later we can look at reworking this into a single mechanism.

    Closes-bug: #1377681
    Change-Id: If3d476f34b9e61a3c99f63ba33734a875353c8fc
    (cherry picked from commit d32370233eaf2a5c32888f269bd1dc5e0e787467)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/128908

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/128908
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=91c0e2329982df57e9520c23cda44930fa1b2cf4
Submitter: Jenkins
Branch: master

commit c06772571a81383aa37336a78d72d455067b75e8
Author: Zane Bitter <email address hidden>
Date: Fri Oct 10 15:43:16 2014 -0400

    Don't update a LoadBalancer under autoscaling control

    Unfortunately, Autoscaling currently uses the update() method of a
    LoadBalancer resource to do updates, with the result that the current
    member list gets persisted (good) and used to compare in the event of
    future *stack* updates (bad).

    With this patch, we assume that LoadBalancers under the control of
    Autoscaling will never have a members list property supplied in the
    template. We then ignore any updates to Autoscaling LoadBalancers that
    don't actually modify the template.

    The test changes revert the changes made in order to be able to merge
    d32370233eaf2a5c32888f269bd1dc5e0e787467, before which LoadBalancers were
    behaving correctly.

    Change-Id: I9c02ab3d3dfbee0a8a90dd0ba345a5acdaf8a610
    Closes-Bug: #1379619
    (cherry picked from commit 7506d2c22fcd3d6e825ee660a4b20c5947cc7e9a)

commit 5aa66555209eb5a59abdb189b0a5d1224e73b566
Author: Angus Salkeld <email address hidden>
Date: Fri Oct 10 12:00:37 2014 +1000

    Make sure that the properties are stored on updates

    Previously properties_data was only stored on creation, now
    this is getting passed to the update mechanism too.
    Later we can look at reworking this into a single mechanism.

    Closes-bug: #1377681
    Change-Id: If3d476f34b9e61a3c99f63ba33734a875353c8fc
    (cherry picked from commit d32370233eaf2a5c32888f269bd1dc5e0e787467)

commit 6e1ad898d887514267e3a429c291b2a067ea7e03
Author: huangtianhua <email address hidden>
Date: Wed Sep 24 15:19:24 2014 +0800

    Do not attempt a stack update when it is deleting

    If a stack is in any delete state(inprogress, failed),
    don't attempt to update it.

    Closes-bug: #1379113
    Change-Id: I1de99702a385ac8ddeffc568270d2d3b51674323

commit 1e9b2cdd9004c5e1fa3bb1a27830d75ebaebe20c
Author: Brant Knudson <email address hidden>
Date: Wed Oct 8 20:15:54 2014 -0500

    Update references to auth_token middleware

    There's references to the auth_token middleware in keystoncelient.
    The auth_token middleware has been moved to keystonemiddleware and
    the version in keystoneclient shouldn't be used anymore.

    If these references aren't updated, then when options are changed in
    keystonemiddleware.auth_token the heat-api will fail to start because
    there's duplicate options in keystoneclient.middleware.auth_token.

    Change-Id: I04573aa5ff967afe3e00329f797fcc71b779e7b3
    Closes-Bug: #1379082

commit 170069d540342ced4dc3fdb3df6619395dd56c6f
Author: Angus Salkeld <email address hidden>
Date: Thu Oct 9 15:35:34 2014 +1000

    Add missing extra "greenthread" arg to remove_event()

    See: http://eventlet.net/doc/modules/greenthread.html
    the callback gets passed a "gt" arg.

    Change-Id: I9bd44857662e45a1da2d4287017e...