Child stack updates are not rolled back immediately in convergence

Bug #1623201 reported by Zane Bitter
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
High
Zane Bitter

Bug Description

Thanks to various bug fixes, the stack-cancel-update command now works under convergence. However, it does not recursively cancel nested stacks. The nested stacks will eventually be updated as part of the rollback of the parent stack, but in the meantime they will continue updating resources that they'll eventually just have to roll back, and we won't be able to start said rollback at least until the current update has completed or failed. It would be more optimal if we could recursively stop the traversals on nested stacks as soon as we receive the stack-cancel-update command, so that we wouldn't do any unnecessary/unwanted work prior to the rollback starting.

We now do basically that in the legacy path: where most resources have a grace period of 4 minutes to complete or fail before we cancel them (note: in convergence the grace period effectively extends until the stack timeout), resources that provide a handle_<action>_cancel() method are cancelled immediately, causing that method to be called. (Although in practice nested stacks are currently the only resources to make use of the mechanism, it is generic enough that other resource types that can be safely cancelled can take advantage.)

An option I discussed with Anant in http://eavesdrop.openstack.org/irclogs/%23heat/%23heat.2016-08-29.log.html#t2016-08-29T19:53:48 would be to piggy-back on the same mechanism for convergence. We could send a message to all resources in a stack to hard-cancel (i.e. kill the thread) *only* if the resource provides a handle_<action>_cancel() method. That way the initial soft-cancel (i.e. stopping the traversal) would propagate immediately down through all the nested stacks.

Zane Bitter (zaneb)
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/370123

Changed in heat:
status: New → In Progress
Changed in heat:
assignee: Anant Patil (ananta) → Zane Bitter (zaneb)
Zane Bitter (zaneb)
tags: added: newton-rc-potential
Changed in heat:
milestone: next → newton-rc2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/370123
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=bc2e136fe34e25185f344ebce4fb171243ae7dc9
Submitter: Jenkins
Branch: master

commit bc2e136fe34e25185f344ebce4fb171243ae7dc9
Author: Anant Patil <email address hidden>
Date: Wed Sep 14 18:59:36 2016 +0530

    Cancel traversal of nested stack

    The stack cancel update would halt the parent stack from propagating but
    the nested stacks kept on going till they either failed or completed.
    This is not desired, the cancel update should stop all the nested stacks
    from moving further, albeit, it shouldn't abruptly stop the currently
    running workers.

    Change-Id: I3e1c58bbe4f92e2d2bfea539f3d0e861a3a7cef1
    Co-Authored-By: Zane Bitter <email address hidden>
    Closes-Bug: #1623201

Changed in heat:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/373269

Zane Bitter (zaneb)
Changed in heat:
importance: Medium → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (stable/newton)

Reviewed: https://review.openstack.org/373269
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=25fbedc726aebcb8179faa5c4a3d85de2a9067c2
Submitter: Jenkins
Branch: stable/newton

commit 25fbedc726aebcb8179faa5c4a3d85de2a9067c2
Author: Anant Patil <email address hidden>
Date: Wed Sep 14 18:59:36 2016 +0530

    Cancel traversal of nested stack

    The stack cancel update would halt the parent stack from propagating but
    the nested stacks kept on going till they either failed or completed.
    This is not desired, the cancel update should stop all the nested stacks
    from moving further, albeit, it shouldn't abruptly stop the currently
    running workers.

    Change-Id: I3e1c58bbe4f92e2d2bfea539f3d0e861a3a7cef1
    Co-Authored-By: Zane Bitter <email address hidden>
    Closes-Bug: #1623201
    (cherry picked from commit bc2e136fe34e25185f344ebce4fb171243ae7dc9)

tags: added: in-stable-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/heat 7.0.0.0rc2

This issue was fixed in the openstack/heat 7.0.0.0rc2 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/heat 7.0.0

This issue was fixed in the openstack/heat 7.0.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/heat 8.0.0.0b1

This issue was fixed in the openstack/heat 8.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.