Convergence stack stuck in update/rollback in progress for concurrent update

[Rakesh H S]

Consider the following scenario:

template1:
rsrc_A

template2:
rsrc_B

NOTE that when updating with template2, rsrc_A is DELETE_IN_PROGRESS and taking time to delete.

In the mean time another update is triggered:
template3:
rsrc_A

Now since rsrc_A is still in DB, while calculating graph we need not store the resource in DB again.
However there is possibility that after calculating the graph template2 successfully deletes rsrc_A (which clears the rsrc from DB)

Now template3 will be stuck in progress because of:
https://github.com/openstack/heat/blob/master/heat/engine/worker.py#L289

[Anant Patil (ananta)]

While searching for best candidate from DB to represent a resource from current template, we should ignore the ones marked for deletion and add a new resource in DB. The creation of this new resource should depend on deletion of old resource which was marked for deletion.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/278959

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/279520

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/278959
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=9cf021ac7eeb2fe2a1dfd230f4eaa1ed3030cc77
Submitter: Jenkins
Branch: master

commit 9cf021ac7eeb2fe2a1dfd230f4eaa1ed3030cc77
Author: Rakesh H S <email address hidden>
Date: Thu Feb 11 15:22:09 2016 +0530

    Convergence: DB api to get all active resources of stack

    This patch reverts change I6a212da19a774239f014163774e75fe11dfe272c
    and adds new DB api resource_get_all_active_by_stack to be used by
    convergence.

    The new DB api will be used while generating graph for convergence
    stack and will fetch all resources of stack from DB excluding
    the DELETE COMPLETE resources, if any.

    Change-Id: I303ef2c9b5b6a0a49253425c00565c8981cc6825
    Partial-Bug: #1528560

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/279520
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=906a0ed6fd94dcb25bdc345ef09b5e5a9d1e1816
Submitter: Jenkins
Branch: master

commit 906a0ed6fd94dcb25bdc345ef09b5e5a9d1e1816
Author: Rakesh H S <email address hidden>
Date: Fri Feb 12 15:41:29 2016 +0530

    Convergence: Fix concurrent update resource delete

    In convergence, wherein concurrent updates are possible, if a resource
    is deleted (by previous traversal) after dependency graph is created
    for new traversal, the resource remains in graph but wouldn't be
    available in DB for processing.
    It is prerequisite to have resources in DB before any action can be
    taken on them.

    Hence during convergence resource delete action, the resource entry
    from DB is not deleted i.e soft deleted, so that the latest/new update
    can find the entry.
    All of these soft deleted resources will be deleted when the stack has
    completed its operation.

    Closes-Bug: #1528560
    Change-Id: I0b36ce098022560d7fe01623ce7b66d1d5b38d55

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/heat 7.0.0.0b2 development milestone.