Can't rollback update with nested stack in progress

[Zane Bitter]

When an in-progress stack update fails (and rollback is enabled) or is cancelled, we attempt to roll back the stack to it's previous state. For nested stacks we do this by issuing a new update call with the previous template and parameters, without attempting to cancel a current update that may be in progress. As a result, if the nested stack is in progress at the time we are attempting to roll back the parent, the attempt to update the nested stack will fail and therefore the rollback will fail.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/175606

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/175606
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=ad4c00e672fdd2c1c8461d7e4813dff4b093ae0f
Submitter: Jenkins
Branch: master

commit ad4c00e672fdd2c1c8461d7e4813dff4b093ae0f
Author: Zane Bitter <email address hidden>
Date: Mon Apr 20 19:12:07 2015 -0400

    Cancel nested stack updates when appropriate

    If a stack update is cancelled while the update of a nested stack is being
    processed, the nested stack update is no longer stopped since the
    implementation changed in Kilo to use the RPC API to control nested stacks.

    To alleviate this, use the RPC API to cancel the (nested) stack update when
    the resource update is cancelled. This results in a rollback of the nested
    stack, so if a subsequent attempt to roll back the ensted stack by doing a
    stack update fails due to the rollback already being in progress, just
    ignore the error and wait for the existing operation to complete.

    Change-Id: I3eacf216194517c995732a3f4de876551b70fe91
    Closes-Bug: #1446252

[Zane Bitter (zaneb)]

I'm no longer sure this was the right fix. It kicks in not only when the stack is being rolled back, but on any sibling resource failure that causes the update to be cancelled (even if rollback is disabled). See bug 1475057 for details.

[Angus Salkeld (asalkeld)]

This is now needed purely as a code dependency for bug 1475057 to get merged.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/225527

[OpenStack Infra (hudson-openstack)]

Change abandoned by Angus Salkeld (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/225527

[Zane Bitter (zaneb)]

Note: this fix didn't work. See bug 1591337.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/332967

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/332967
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=97a3670462d791587764b05c2e50a2bbd45521b0
Submitter: Jenkins
Branch: master

commit 97a3670462d791587764b05c2e50a2bbd45521b0
Author: Zane Bitter <email address hidden>
Date: Fri Jul 22 11:14:54 2016 -0400

    Use handle_update_cancel() to cancel nested stack updates

    The code designed to deal with the case where an in-progress StackResource
    update is cancelled by stopping the update of the nested stack has never
    worked because it was erroneously trying to catch StopIteration exceptions
    instead of the actual exception raised when a generator is closed,
    GeneratorExit.

    This change moves the handling code to the new handle_update_cancel()
    mechanism instead. Using this mechanism also means that StackResources will
    be cancelled immediately, rather than waiting for up to error_wait_time
    before cancelling the nested stack update. The previous patch adds a grace
    period for IN_PROGRESS resources within the nested stack to complete, so we
    want to notify it straight away to begin that period (and to not start any
    new work).

    Since the grace period on the child stack means that the child is likely to
    still be IN_PROGRESS at the time we want to roll back, handle rollback by
    sending an update-cancel with rollback message to the child. Only if the
    child is no longer IN_PROGRESS will we start an update with the previous
    template. If it is still IN_PROGRESS then just wait for it to reach
    ROLLBACK_COMPLETE.

    Change-Id: Ide210f695446fe8de2057b3b74c276165a6f9f3f
    Closes-Bug: #1591337
    Closes-Bug: #1446252

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat 7.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/490810

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/490897

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/490898

[OpenStack Infra (hudson-openstack)]

Change abandoned by Rico Lin (<email address hidden>) on branch: master
Review: https://review.openstack.org/490810
Reason: After discussion in previous PTG, we already provide other way to do this

[OpenStack Infra (hudson-openstack)]

Change abandoned by Rico Lin (<email address hidden>) on branch: master
Review: https://review.openstack.org/490898
Reason: in prefer of https://review.openstack.org/#/q/status:open+project:openstack/heat+branch:master+topic:story/1669608

[OpenStack Infra (hudson-openstack)]

Change abandoned by Rico Lin (<email address hidden>) on branch: master
Review: https://review.openstack.org/490897
Reason: in prefer of https://review.openstack.org/#/q/status:open+project:openstack/heat+branch:master+topic:story/1669608