Environments can be left in an inconsistent state if update does not finish normally

Bug #1544348 reported by Zane Bitter
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
High
Zane Bitter

Bug Description

The fix for bug 1477812 fixes the case where a stack update fails part way through, leaving some old and some new resources behind, by merging the old and new environments and storing them with the stack, in order to try to keep the state consistent. (It's not entirely successful at that - see bug 1543685 - but pretty close).

However, the environment is updated only *after* the update has finished. This means that if the update process is not run to completion, we're back where we started. An example would be if heat-engine were restarted during the update.

One easy-ish improvement would be to put the code inside a 'finally' block, so that if things at least exit in an orderly fashion (by raising an exit exception in each thread), the code would still run.

That would still leave a gap in the solution in cases where the exit is not orderly (e.g. kill -9). To completely resolve this, we likely need to write the merged environment before the update starts, or at least at the same time as the first resource is updated.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/278806

Changed in heat:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/278831

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/278840

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/278806
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=2c24badd55cdf90ba79bedc6ea18516ba0b5e6ca
Submitter: Jenkins
Branch: master

commit 2c24badd55cdf90ba79bedc6ea18516ba0b5e6ca
Author: Zane Bitter <email address hidden>
Date: Wed Feb 10 19:14:16 2016 -0500

    Always do cleanup after a failed stack update

    Move the cleanup code (that sets the status to FAILED and merges the new
    environment into the old one) so that it runs on any exception, including
    exit exceptions (due to e.g. the thread being cancelled or the engine
    restarted). In the case of the environment this isn't as good as merging
    them before the update starts (since it's possible for Heat to die without
    time to run exception handlers), but at least it leaves things consistent
    in the orderly case.

    Change-Id: Ia4d2deca393ac9e250932c5cdb3691b6ee2b3ef4
    Partial-bug: #1544348
    Related-bug: #1492433

Zane Bitter (zaneb)
Changed in heat:
assignee: Zane Bitter (zaneb) → nobody
status: In Progress → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (stable/liberty)

Reviewed: https://review.openstack.org/278831
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=f3eab3ff449768f5f683cf10fce70cdec4a6c967
Submitter: Jenkins
Branch: stable/liberty

commit f3eab3ff449768f5f683cf10fce70cdec4a6c967
Author: Zane Bitter <email address hidden>
Date: Wed Feb 10 19:14:16 2016 -0500

    Always do cleanup after a failed stack update

    Move the cleanup code (that sets the status to FAILED and merges the new
    environment into the old one) so that it runs on any exception, including
    exit exceptions (due to e.g. the thread being cancelled or the engine
    restarted). In the case of the environment this isn't as good as merging
    them before the update starts (since it's possible for Heat to die without
    time to run exception handlers), but at least it leaves things consistent
    in the orderly case.

    Change-Id: Ia4d2deca393ac9e250932c5cdb3691b6ee2b3ef4
    Partial-bug: #1544348
    Related-bug: #1492433
    (cherry picked from commit 2c24badd55cdf90ba79bedc6ea18516ba0b5e6ca)

tags: added: in-stable-liberty
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on heat (stable/kilo)

Change abandoned by Dave Walker (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/278840
Reason:
stable/kilo closed for 2015.1.4

This release is now pending its final release and no freeze exception has
been seen for this changeset. Therefore, I am now abandoning this change.

If this is not correct, please urgently raise a thread on openstack-dev.

More details at: https://wiki.openstack.org/wiki/StableBranch

Zane Bitter (zaneb)
Changed in heat:
status: Triaged → Fix Released
assignee: nobody → Zane Bitter (zaneb)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.