stack is always in UPDATE_IN_PROGRESS

Bug #1478480 reported by huangtianhua
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
High
huangtianhua
Kilo
Fix Released
Undecided
Unassigned

Bug Description

The steps:
1. create a stack with aws autoscaling group:
********************
heat_template_version: 2015-04-30
parameters:
  subnet:
    type: string
resources:
  my_aws_asg:
    type: AWS::AutoScaling::AutoScalingGroup
    properties:
      AvailabilityZones: ['nova']
      LaunchConfigurationName: {get_resource: lc}
      MaxSize: 4
      MinSize: 1
      VPCZoneIdentifier:
      - get_param: subnet
  lc:
    type: AWS::AutoScaling::LaunchConfiguration
    properties:
      ImageId: cirros-0.3.0-x86_64-disk
      InstanceType: 1
***********************
2. the stack create complete
3. update the stack, pass an invalid 'subnet'
4. the stack update failed, the stack is in UPDATE_FAILED
5. update with valid subnet again, then the stack is always in UPDATE_IN_PROGRESS, due _check_status_complete() in loop

I found that, when we update the stack, first we will remove backup resources
****************************
      cleanup_prev = scheduler.DependencyTaskGroup(
            self.previous_stack.dependencies,
            self._remove_backup_resource,
            reverse=True)
************************
  def delete_nested(self):
        '''
        Delete the nested stack.
        '''
        stack_identity = identifier.HeatIdentifier(
            self.context.tenant_id,
            self.physical_resource_name(),
            self.resource_id)

        try:
            self.rpc_client().delete_stack(self.context, stack_identity)
 *************************
here we set the stack_name as self.physical_resource_name(), such as {'stack_name': u'FF*-my_aws_asg-id222tz7mole', 'stack_id': '9e6cee9e-a501-43f1-b098-ae7d0269f7e4', 'tenant': u'30be1fc9a03c4e94ab05c403a8a377f2', 'path': ''},
and it's not match the name of nested_stack, so we can delete the stack always

Changed in heat:
assignee: nobody → huangtianhua (huangtianhua)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/206002

Changed in heat:
status: New → In Progress
Changed in heat:
importance: Undecided → Medium
importance: Medium → High
tags: added: kilo-backport-potential
Revision history for this message
huangtianhua (huangtianhua) wrote :

@Steve Baker:
Thanks, in fact, there is another problem, if we update with the invalid subnet again(instead of the above step5), even if after this change( correct the stack identity), the stack will be UPDATE_FAILED, but the instance has been deleted, I think user will confuse this result. So how to solve this problem, why we delete the backup resource in second update not in first update? why we delete the backup resource before the new one be created successful?

Changed in heat:
milestone: none → liberty-2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/206002
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=ab234eb65871be3607a5702d6a01a6be5f6b59d5
Submitter: Jenkins
Branch: master

commit ab234eb65871be3607a5702d6a01a6be5f6b59d5
Author: huangtianhua <email address hidden>
Date: Mon Jul 27 18:26:04 2015 +0800

    Correct stack identity when delete nested

    1. Update an aws asg with invalid 'VPCZoneIdentifier',
    the stack update failed.
    2. Update again, the stack is always in 'UPDATE_IN_PROGRESS'.

    We remove the backup resources first when updating,
    but the backup of stack_resource can't be deleted,
    due we set an incorrect name of stack_identity before
    rpc calling 'delete_stack', due the name is not match the
    nested_stack's name.

    Change-Id: I89966d0124607c233ac9a82f1ce14f4f2deb3f9b
    Closes-Bug: #1478480

Changed in heat:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/206910

Thierry Carrez (ttx)
Changed in heat:
status: Fix Committed → Fix Released
Revision history for this message
huangtianhua (huangtianhua) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (stable/kilo)

Reviewed: https://review.openstack.org/206910
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=4a8282c42cc0306d9e2c155c700a5adb1bb59ae7
Submitter: Jenkins
Branch: stable/kilo

commit 4a8282c42cc0306d9e2c155c700a5adb1bb59ae7
Author: huangtianhua <email address hidden>
Date: Mon Jul 27 18:26:04 2015 +0800

    Correct stack identity when delete nested

    1. Update an aws asg with invalid 'VPCZoneIdentifier',
    the stack update failed.
    2. Update again, the stack is always in 'UPDATE_IN_PROGRESS'.

    We remove the backup resources first when updating,
    but the backup of stack_resource can't be deleted,
    due we set an incorrect name of stack_identity before
    rpc calling 'delete_stack', due the name is not match the
    nested_stack's name.

    Change-Id: I89966d0124607c233ac9a82f1ce14f4f2deb3f9b
    Closes-Bug: #1478480
    (cherry picked from commit ab234eb65871be3607a5702d6a01a6be5f6b59d5)

tags: added: in-stable-kilo
tags: removed: kilo-backport-potential
Thierry Carrez (ttx)
Changed in heat:
milestone: liberty-2 → 5.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.