Any component in murano environment hangs in state 'Creating VM for ...'

[Victor Ryzhenkin]

Using master branch for all components (affects liberty-rc1 too)

Steps:
1. Deploy Murano with devstack
2. Add any app to environment
3. Deploy environment.

Expected: Application successfully deployed

Actual:
Murano environment hangs in 'DEPLOYING' state.
Component status hangs with state 'Creating VM for Apache Server'

Instance is ready and active (metadata injected successfully, keypair on it, murano-agent armed, config of murano-agent is correct)
Heat stack state is 'Update Complete'

In murano logs we can see endless requests to heat, like this:

2015-09-27 00:11:38.826 19802 DEBUG heatclient.common.http [-]
HTTP/1.1 302 Found
content-length: 208
connection: keep-alive
location: http://192.168.111.10:8004/v1/d42dbd0145344e3fa4132d52e14fb2f9/stacks/murano-lkiutif1ozv3a2/bbff7349-3a4d-4f3e-a0cb-e10fe76b29e2
date: Sun, 27 Sep 2015 00:11:38 GMT
content-type: text/plain; charset=UTF-8
x-openstack-request-id: req-531aa109-6036-41d3-a601-c21ad304e349

302 Found

The resource was found at http://192.168.111.10:8004/v1/d42dbd0145344e3fa4132d52e14fb2f9/stacks/murano-lkiutif1ozv3a2/bbff7349-3a4d-4f3e-a0cb-e10fe76b29e2; you should be redirected automatically.
 log_http_response /usr/local/lib/python2.7/dist-packages/heatclient/common/http.py:142
2015-09-27 00:11:38.826 19802 DEBUG heatclient.common.http [-] curl -g -i -X GET -H 'User-Agent: python-heatclient' -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'X-Auth-Token: {SHA1}7953d
b33f752d12f9df93cbd75e647839b8fa14a' http://192.168.111.10:8004/v1/d42dbd0145344e3fa4132d52e14fb2f9/stacks/murano-lkiutif1ozv3a2/bbff7349-3a4d-4f3e-a0cb-e10fe76b29e2 log_curl_request /usr/local/lib/python2.7/dis
t-packages/heatclient/common/http.py:129

Finally, after an ~hour we got a traceback which connected with auth:

2015-09-27 00:11:50.284 19802 DEBUG keystoneclient.session [-] Request returned failure status: 403 request /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:400
2015-09-27 00:11:50.377 19802 ERROR murano.common.engine [-]
  keystoneclient.exceptions.AuthorizationFailure: Authorization failed: User is not a trustee. (Disable debug mode to suppress these details.) (HTTP 403) (Request-ID: req-73cdef62-3412-4cd4-8866-98363379472b)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/228255

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/228287

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/228255
Committed: https://git.openstack.org/cgit/openstack/murano/commit/?id=8b7e07d8f01a83bc37b01b44888cdca113313286
Submitter: Jenkins
Branch: master

commit 8b7e07d8f01a83bc37b01b44888cdca113313286
Author: Stan Lagun <email address hidden>
Date: Sun Sep 27 18:22:40 2015 +0300

    Murano waited indefinitely for UPDATE_COMPLETE stack status

    On recent Heat version last modification timestamp changes on
    update start rather than finish. As a result when stack exits from
    IN_PROGRESS state timestamp doesn't change any longer.
    But Murano also waited for timestamp to change. As a result
    Murano waited indefinitely and deployment hung.

    With this change IN_PROGRESS status will no longer set
    last modification timestamps. So now we will be comparing
    timestamps between 2 terminal statuses rather than between
    last UPDATE_IN_PROGRESS and UPDATE_COMPLETE that
    now have the same timestamp.

    Change-Id: Idd933d453ef7715e409439a53a3baf9eaa202370
    Closes-Bug: #1500200

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/228287
Committed: https://git.openstack.org/cgit/openstack/murano/commit/?id=e130ed3c42ef6e837f5f42c7182f0806d4eba104
Submitter: Jenkins
Branch: stable/liberty

commit e130ed3c42ef6e837f5f42c7182f0806d4eba104
Author: Stan Lagun <email address hidden>
Date: Sun Sep 27 18:22:40 2015 +0300

    Murano waited indefinitely for UPDATE_COMPLETE stack status

    On recent Heat version last modification timestamp changes on
    update start rather than finish. As a result when stack exits from
    IN_PROGRESS state timestamp doesn't change any longer.
    But Murano also waited for timestamp to change. As a result
    Murano waited indefinitely and deployment hung.

    With this change IN_PROGRESS status will no longer set
    last modification timestamps. So now we will be comparing
    timestamps between 2 terminal statuses rather than between
    last UPDATE_IN_PROGRESS and UPDATE_COMPLETE that
    now have the same timestamp.

    Change-Id: Idd933d453ef7715e409439a53a3baf9eaa202370
    Closes-Bug: #1500200
    (cherry picked from commit 8b7e07d8f01a83bc37b01b44888cdca113313286)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/267303

[Nikolay Starodubtsev (starodubcevna)]

need to backport this in kilo, as far as this bug affects gate jobs

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/267303
Committed: https://git.openstack.org/cgit/openstack/murano/commit/?id=4a3fc9b07775313e9802c2e11e4f388d75731950
Submitter: Jenkins
Branch: stable/kilo

commit 4a3fc9b07775313e9802c2e11e4f388d75731950
Author: Stan Lagun <email address hidden>
Date: Sun Sep 27 18:22:40 2015 +0300

    Murano waited indefinitely for UPDATE_COMPLETE stack status

    On recent Heat version last modification timestamp changes on
    update start rather than finish. As a result when stack exits from
    IN_PROGRESS state timestamp doesn't change any longer.
    But Murano also waited for timestamp to change. As a result
    Murano waited indefinitely and deployment hung.

    With this change IN_PROGRESS status will no longer set
    last modification timestamps. So now we will be comparing
    timestamps between 2 terminal statuses rather than between
    last UPDATE_IN_PROGRESS and UPDATE_COMPLETE that
    now have the same timestamp.

    Change-Id: Idd933d453ef7715e409439a53a3baf9eaa202370
    Closes-Bug: #1500200

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/murano 2015.1.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/murano 2015.1.1 release.