Invalid Templete Reference when delete failed stack

[Kairat Kushaev]

During testing of Murano we faced with the following issue:
Heat stack is handing on in UPDATE_IN_PROGRESS state or DELETE_IN_PROGRESS case when we are trying to create and update the stack in series.
The case is the following:
 1. Create the stack with murano_first template (using neutron router in devstack).
 2. Update the stack with murano_second template using m1.express flavor( flavor RAM value should be enough to raise No valid host found in Nova)
   The stack is failing because of no valid host found error.
 3. Update the failed stack with murano_third (using neutron public net as floating network).
  The stack is failing again because of no valid host found error.
 4. Delete the stack.
Expected result: the stack has been deleted successfully.
Actual result: the stack is hanging on in DELETE_IN_PROGRESS
This is specialty of Murano that it sends update requests in a bunch and doesn't check that the previous update was successful.
But we definitely need to treat properly such kind of cases.
When I looked into the heat log I found that because of some reason heat generated Invalid Template Reference error.

[Kairat Kushaev (kkushaev)]

After deep analysis it turned out that Heat does not create backup resources properly in the case below.
That happens because of following:
0. Server (..ks41) in the second template has a reference to Neutron Port.
1. When Heat is trying to update the server first time everything is working correctly.
1. When Heat is trying to update the failed server (...ks41) second time it is trying to re-create it.
2. During re-creation Heat is backing up failed server independently of stack-update result and saves that resource to DB. (but the Port was not saved because there were no updates for it)
3. When Heat is trying to delete stack we have got an error because the server has a reference to the port but the port does not exist in the template.
So I suppose we should prohibit backing up of failed resource in the case above.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/175868

[Kairat Kushaev (kkushaev)]

Folks, I also prepared the diagram that explains the root cause and proposed solution.
Hope, it will be useful for guys who will face with same issues.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/175868
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=f2edd0d68ea84f8ca212df0ec1d6d92463a030aa
Submitter: Jenkins
Branch: master

commit f2edd0d68ea84f8ca212df0ec1d6d92463a030aa
Author: kairat_kushaev <email address hidden>
Date: Tue May 5 19:22:32 2015 +0300

    Save updated-in-place resources to backup stack

    The patch changes the approach of resource backup during stack
    update. It stores the definition of resource that needs to be
    updated in-place to backup stack(if it was not created before).

    Without this functionality, InvalidTemplateReference will be thrown
    every time when update-replaced resource has a reference to updated
    in-place resource and stack update was failed. In this case, backup
    stack has a resource that refers to not-presented another resource
    and backup stack deletion generates an exception.

    Change-Id: I436c44a579bb4df3031d1a17b9ca5b62da37afeb
    Closes-bug: #1446575

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/182599

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/182599
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=da3777342a344a2dafb9244b7afca8722a398caa
Submitter: Jenkins
Branch: stable/kilo

commit da3777342a344a2dafb9244b7afca8722a398caa
Author: kairat_kushaev <email address hidden>
Date: Tue May 5 19:22:32 2015 +0300

    Save updated-in-place resources to backup stack

    The patch changes the approach of resource backup during stack
    update. It stores the definition of resource that needs to be
    updated in-place to backup stack(if it was not created before).

    Without this functionality, InvalidTemplateReference will be thrown
    every time when update-replaced resource has a reference to updated
    in-place resource and stack update was failed. In this case, backup
    stack has a resource that refers to not-presented another resource
    and backup stack deletion generates an exception.

    Change-Id: I436c44a579bb4df3031d1a17b9ca5b62da37afeb
    Closes-bug: #1446575
    (cherry-picked from commit f2edd0d68ea84f8ca212df0ec1d6d92463a030aa)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/230130

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/230130
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=2a6e29113d9f68afa3e1e8c42af842a1495879cc
Submitter: Jenkins
Branch: stable/juno

commit 2a6e29113d9f68afa3e1e8c42af842a1495879cc
Author: kairat_kushaev <email address hidden>
Date: Tue May 5 19:22:32 2015 +0300

    Save updated-in-place resources to backup stack

    The patch changes the approach of resource backup during stack
    update. It stores the definition of resource that needs to be
    updated in-place to backup stack(if it was not created before).

    Without this functionality, InvalidTemplateReference will be thrown
    every time when update-replaced resource has a reference to updated
    in-place resource and stack update was failed. In this case, backup
    stack has a resource that refers to not-presented another resource
    and backup stack deletion generates an exception.

    Change-Id: I436c44a579bb4df3031d1a17b9ca5b62da37afeb
    Closes-bug: #1446575
    (cherry picked from commits da3777342a344a2dafb9244b7afca8722a398caa
                            and 3786262aed62d81ad35de3d4f703c38223907ffd)