Heat is hanging on in DELETE_IN_PROGRESS state

[Kairat Kushaev]

During testing of Murano we faced with the following issue:
Heat stack is handing on in UPDATE_IN_PROGRESS state or DELETE_IN_PROGRESS case when we are trying to create and update the stack in series.
The case is the following:
 1. Create the stack with murano_first template (using neutron router in devstack).
 2. Update the stack with murano_second template using m1.express flavor( flavor RAM value should be enough to raise No valid host found in Nova)
   The stack is failing because of no valid host found error.
 3. Update the failed stack with murano_third (using neutron public net as floating network).
  The stack is failing again because of no valid host found error.
 4. Delete the stack.
Expected result: the stack has been deleted successfully.
Actual result: the stack is hanging on in DELETE_IN_PROGRESS
This is specialty of Murano that it sends update requests in a bunch and doesn't check that the previous update was successful.
But we definitely need to treat properly such kind of cases.
When I looked into the heat log I found that because of some reason heat generated Invalid Template Reference error.

[Kairat Kushaev (kkushaev)]

Hi guys,
I detected the place where the issue appears. Still have some problems with solution because the involved part is the trickiest part of heat but I expect the final issue to be provided today evening. Hope, i can ask some of core reviewers to review it here(https://review.openstack.org/#/c/175868/).
After that the plan to fix the issue is the following:
1) Fix the issue in heat master.
2) Backport it manually inside MOS
3) Try to backport it to Juno.
Please note that we also proposed a workaround for Murano how to avoid an issue:
they should not try to update stack if it has failed.

[Kairat Kushaev (kkushaev)]

Hello,
I proposed a fix here:
https://review.openstack.org/#/c/175868/.
I will respond as soon as I get any updates.

[Kairat Kushaev (kkushaev)]

Hi guys,
The current status is the following:
I tested the patch with Juno and Kilo releases.
Also i tested the issue for MOS 6.1 and it seems that the patch solves the problem.
I am waiting for review approval and going to backport it asap.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: master
Change author: Kairat Kushaev <email address hidden>
Review: https://review.fuel-infra.org/6465

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Kairat Kushaev <email address hidden> on branch: master
Review: https://review.fuel-infra.org/6465
Reason: Wrong branch.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-6.1/2014.2
Change author: Kairat Kushaev <email address hidden>
Review: https://review.fuel-infra.org/6475

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/6475
Submitter: mos-infra-ci <>
Branch: openstack-ci/fuel-6.1/2014.2

Commit: 9bc1a97647231bcda0310971e2b5012a8b39ef18
Author: Kairat Kushaev <email address hidden>
Date: Fri May 8 07:15:30 2015

Save updated-in-place resources to backup stack

The patch changes the approach of resource backup during stack
update. It stores the definition of resource that needs to be
updated in-place to backup stack(if it was not created before).

Without this functionality, InvalidTemplateReference will be thrown
every time when update-replaced resource has a reference to updated
in-place resource and stack update was failed. In this case, backup
stack has a resource that refers to not-presented another resource
and backup stack deletion generates an exception.

The similar patch proposed to upstream in this review:
https://review.openstack.org/#/c/175868/

Closes-bug: #1446678
Change-Id: I80bb6b192f50fbb7e7b41bbed416f8031b7a0662

[Kairat Kushaev (kkushaev)]

Previously bug was closed accidentally.

[Yaroslav Lobankov (ylobankov)]

Verified on

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  openstack_version: "2014.2.2-6.1"
  api: "1.0"
  build_number: "432"
  build_id: "2015-05-18_03-43-53"
  nailgun_sha: "076566b5df37f681c3fd5b139c966d680d81e0a5"
  python-fuelclient_sha: "38765563e1a7f14f45201fd47cf507393ff5d673"
  astute_sha: "cb655a9a9ad26848bcd9d9ace91857b6f4a0ec15"
  fuel-library_sha: "1621cb350af744f497c35f2b3bb889c2041465d8"
  fuel-ostf_sha: "9ce1800749081780b8b2a4a7eab6586583ffaf33"
  fuelmain_sha: "0e970647a83d9a7d336c4cc253606d4dd0d59a60"

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-6.0-updates/2014.2
Change author: Kairat Kushaev <email address hidden>
Review: https://review.fuel-infra.org/7230

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/7230
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-6.0-updates/2014.2

Commit: 1d9325bea6be2495411ddc7dfe29fc2def8e0971
Author: Kairat Kushaev <email address hidden>
Date: Mon Jun 1 15:14:24 2015

Save updated-in-place resources to backup stack

The patch changes the approach of resource backup during stack
update. It stores the definition of resource that needs to be
updated in-place to backup stack(if it was not created before).

Without this functionality, InvalidTemplateReference will be thrown
every time when update-replaced resource has a reference to updated
in-place resource and stack update was failed. In this case, backup
stack has a resource that refers to not-presented another resource
and backup stack deletion generates an exception.

The similar patch proposed to upstream in this review:
https://review.openstack.org/#/c/175868/

Closes-bug: #1446678
Change-Id: I80bb6b192f50fbb7e7b41bbed416f8031b7a0662
(cherry picked from commit 9bc1a97647231bcda0310971e2b5012a8b39ef18)

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Kairat Kushaev <email address hidden>
Review: https://review.fuel-infra.org/8131

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Sergey Kraynev <email address hidden> on branch: openstack-ci/fuel-7.0/2015.1.0
Review: https://review.fuel-infra.org/8131
Reason: This fix was merged as a part of sync-merge https://review.fuel-infra.org/#/c/8483/

[Roman Rufanov (rrufanov)]

Customer found on 6.1 - added 6.1 series. Urgently need fix for 6.1 and MU.

[Vitaly Sedelnik (vsedelnik)]

This bug was already Fix Released for 6.1 set in default series for MOS project, the fix was merged by Kairat at 5/8 and landed to 6.1 release ISO. I fixed wrong nomination to 6.1 series.

Roman - please check the status and milestone set in the default series before targetting the bug to another series.

Regarding this particular issue - please provide more information regarding customer issue. If the bug is actually not fixed or fixed partially we have to reopen the bug, complete the fix, provide the fix in updates and fix it 7.0 and 8.0. If this is another issue please file another bug. Thanks!

[Denis Klepikov (dklepikov)]

Please pay attention on bug https://bugs.launchpad.net/heat/+bug/1384750
It is related to this problem too