Exception during execution of .destroy on <io.murano.Environment/0.0.0 > Unexpected stack state UPDATE_COMPLETE

[Anastasia Kuznetsova]

During execution of tearDown method in functional tests on the murano-ci, one of resources wasn't deleted:
2015-12-15 16:05:40.730 21677 WARNING murano.dsl.executor [-] Muted exception during execution of .destroy on <io.murano.Environment/0.0.0 eddb91a079f945d993d1b55d9dc79440 (139797918624464)>: [exceptions.EnvironmentError]: Unexpected stack state UPDATE_COMPLETE: Stack UPDATE completed successfully
2015-12-15 16:05:40.730 21677 ERROR murano.dsl.executor Traceback (most recent call last):
2015-12-15 16:05:40.730 21677 ERROR murano.dsl.executor File "/opt/stack/murano/murano/dsl/executor.py", line 211, in cleanup
2015-12-15 16:05:40.730 21677 ERROR murano.dsl.executor method.invoke(self, obj, (), {}, None)
2015-12-15 16:05:40.730 21677 ERROR murano.dsl.executor File "/opt/stack/murano/murano/dsl/murano_method.py", line 115, in invoke
2015-12-15 16:05:40.730 21677 ERROR murano.dsl.executor context, args, kwargs, skip_stub)
2015-12-15 16:05:40.730 21677 ERROR murano.dsl.executor File "/opt/stack/murano/murano/dsl/executor.py", line 79, in invoke_method
2015-12-15 16:05:40.730 21677 ERROR murano.dsl.executor yaql_engine, method_context, this.real_this)(*args, **kwargs)
2015-12-15 16:05:40.730 21677 ERROR murano.dsl.executor File "/usr/local/lib/python2.7/dist-packages/yaql/language/specs.py", line 60, in func
2015-12-15 16:05:40.730 21677 ERROR murano.dsl.executor return self.get_delegate(receiver, engine, context, args, kwargs)()
2015-12-15 16:05:40.730 21677 ERROR murano.dsl.executor File "/usr/local/lib/python2.7/dist-packages/yaql/language/specs.py", line 341, in func
2015-12-15 16:05:40.730 21677 ERROR murano.dsl.executor six.iteritems(keyword_args)))
2015-12-15 16:05:40.730 21677 ERROR murano.dsl.executor File "/opt/stack/murano/murano/dsl/yaql_integration.py", line 198, in payload
2015-12-15 16:05:40.730 21677 ERROR murano.dsl.executor executor, __sender, args, kwargs, __context, True)
2015-12-15 16:05:40.730 21677 ERROR murano.dsl.executor File "/opt/stack/murano/murano/dsl/murano_method.py", line 115, in invoke
2015-12-15 16:05:40.730 21677 ERROR murano.dsl.executor context, args, kwargs, skip_stub)
2015-12-15 16:05:40.730 21677 ERROR murano.dsl.executor File "/opt/stack/murano/murano/dsl/executor.py", line 112, in invoke_method
2015-12-15 16:05:40.730 21677 ERROR murano.dsl.executor result = call()
2015-12-15 16:05:40.730 21677 ERROR murano.dsl.executor File "/opt/stack/murano/murano/dsl/executor.py", line 107, in call
2015-12-15 16:05:40.730 21677 ERROR murano.dsl.executor else method.body.execute(context))
2015-12-15 16:05:40.730 21677 ERROR murano.dsl.executor File "/opt/stack/murano/murano/dsl/macros.py", line 55, in execute
2015-12-15 16:05:40.730 21677 ERROR murano.dsl.executor super(MethodBlock, self).execute(new_context)
2015-12-15 16:05:40.730 21677 ERROR murano.dsl.executor File "/opt/stack/murano/murano/dsl/macros.py", line 37, in execute
2015-12-15 16:05:40.730 21677 ERROR murano.dsl.executor expr.execute(context)
2015-12-15 16:05:40.730 21677 ERROR murano.dsl.executor File "/opt/stack/murano/murano/dsl/expressions.py", line 76, in execute
2015-12-15 16:05:40.730 21677 ERROR murano.dsl.executor e, context)
2015-12-15 16:05:40.730 21677 ERROR murano.dsl.executor MuranoPlException: [exceptions.EnvironmentError]: Unexpected stack state UPDATE_COMPLETE: Stack UPDATE completed successfully
2015-12-15 16:05:40.730 21677 ERROR murano.dsl.executor

Full murano log can be found here: https://murano-ci.mirantis.com/logs/65/257365/2/check/gate-murano-ubuntu/c145abe/artifacts/devstack/murano-engine.log

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/260278

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/260278
Committed: https://git.openstack.org/cgit/openstack/murano/commit/?id=cc3739d021c5393a12f41e95655c982c4500efe6
Submitter: Jenkins
Branch: master

commit cc3739d021c5393a12f41e95655c982c4500efe6
Author: Stan Lagun <email address hidden>
Date: Tue Dec 22 03:15:19 2015 +0300

    Wait for state progress of Heat stack upon delete

    When Murano deletes Heat stack it waits for stack to
    disappear handling intermediate DELETE_IN_PROGRESS
    status. However if Heat doesn't immediately starts stack
    deletion the first status observed might be UPDATE_COMPLETE
    or CREATE_COMPLETE. This unexpected stack will cause exception
    to be thrown and internal variable holding stack content no to
    be reset. As a result subsequent stack.push() calls can recreate the
    stack.

    The solution is to wait for stack progress before waiting for
    terminal state. This ability was added earlier and used for stack updates
    but not for delete(). However delete() brings another corner case that
    need to be handled - attempt to delete stack that is not exist. In this
    case there is not going to be any status progress so Murano must not
    wait for it (because otherwise it will wait forever). This case needs special
    handling.

    Change-Id: I6f4782879b85ac5691d3c3a7203e8fa60fdbbeaf
    Closes-Bug: #1526876

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/267304

[Nikolay Starodubtsev (starodubcevna)]

need to backport this in kilo, as far as this bug possibly affects gate jobs

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/267304
Committed: https://git.openstack.org/cgit/openstack/murano/commit/?id=27e120a7a82b3a617df59f7e25c253117fb57cbc
Submitter: Jenkins
Branch: stable/kilo

commit 27e120a7a82b3a617df59f7e25c253117fb57cbc
Author: Stan Lagun <email address hidden>
Date: Tue Dec 22 03:15:19 2015 +0300

    Wait for state progress of Heat stack upon delete

    When Murano deletes Heat stack it waits for stack to
    disappear handling intermediate DELETE_IN_PROGRESS
    status. However if Heat doesn't immediately starts stack
    deletion the first status observed might be UPDATE_COMPLETE
    or CREATE_COMPLETE. This unexpected stack will cause exception
    to be thrown and internal variable holding stack content no to
    be reset. As a result subsequent stack.push() calls can recreate the
    stack.

    The solution is to wait for stack progress before waiting for
    terminal state. This ability was added earlier and used for stack updates
    but not for delete(). However delete() brings another corner case that
    need to be handled - attempt to delete stack that is not exist. In this
    case there is not going to be any status progress so Murano must not
    wait for it (because otherwise it will wait forever). This case needs special
    handling.

    Change-Id: I6f4782879b85ac5691d3c3a7203e8fa60fdbbeaf
    Closes-Bug: #1526876

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/267988

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/267988
Committed: https://git.openstack.org/cgit/openstack/murano/commit/?id=0e56e3293382841ee2cd776e953b1de001ef13f1
Submitter: Jenkins
Branch: stable/liberty

commit 0e56e3293382841ee2cd776e953b1de001ef13f1
Author: Stan Lagun <email address hidden>
Date: Tue Dec 22 03:15:19 2015 +0300

    Wait for state progress of Heat stack upon delete

    When Murano deletes Heat stack it waits for stack to
    disappear handling intermediate DELETE_IN_PROGRESS
    status. However if Heat doesn't immediately starts stack
    deletion the first status observed might be UPDATE_COMPLETE
    or CREATE_COMPLETE. This unexpected stack will cause exception
    to be thrown and internal variable holding stack content no to
    be reset. As a result subsequent stack.push() calls can recreate the
    stack.

    The solution is to wait for stack progress before waiting for
    terminal state. This ability was added earlier and used for stack updates
    but not for delete(). However delete() brings another corner case that
    need to be handled - attempt to delete stack that is not exist. In this
    case there is not going to be any status progress so Murano must not
    wait for it (because otherwise it will wait forever). This case needs special
    handling.

    Change-Id: I6f4782879b85ac5691d3c3a7203e8fa60fdbbeaf
    Closes-Bug: #1526876
    (cherry picked from commit cc3739d021c5393a12f41e95655c982c4500efe6)

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/murano 2015.1.1 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/murano 1.0.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/murano 2015.1.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/murano 1.0.3 release.