Convergence: resource is not completed when worker dies

[Anant Patil]

When a worker provisioning a resource dies and another worker takes it up, it should run the check_{action}_complete for the in progress resource. Currently, just update is called on the exiting in progress resource and it will be no-op as the needs update will return as it will find no changes in resource definitions. This needs to be handled by again doing check_action_complete.

The change will be in addition to https://review.openstack.org/#/c/224402/.

[Anant Patil (ananta)]

If a user issues an update on the stack which has either failed due to a failed worker, or is stuck waiting for resource to complete (from a failed worker). then heat should mark those resources as FAILED and continue to provision it. Fix is to mark the IN_PROGRESS resources from non-responding workers as FAILED, and start check_resource.

Since concurrent updates are allowed in heat with convergence, when a worker fails, user can issue an update on in progress stack with same template and this fix should complete the stack.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/264115

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/264115
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=b84417b6cea85229f78a8b371ee6eb7e00e3411c
Submitter: Jenkins
Branch: master

commit b84417b6cea85229f78a8b371ee6eb7e00e3411c
Author: Anant Patil <email address hidden>
Date: Wed Jan 6 14:24:01 2016 +0530

    Convergence: Pick resource from dead engine worker

    When a engine worker crashes or is restarted, the resources being
    provisioned in it remain in IN_PROGRESS state. Next stack update should
    pick these resources and work on them. The implementation is to set the
    status of resource as FAILED and re-trigger check_resource.

    Change-Id: Ib7fd73eadd0127f8fae47881b59388b31131daf4
    Closes-Bug: #1501161

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/heat 6.0.0.0b2 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/heat 6.0.0.0b2 development milestone.