stack creation is stuck in IN_PROGRESS state

[Removed by request]

Due to misconfiguration with a tenant's roles Heat engine received 401 from Keystone as soon as the command was issues. However the stack status remains perpetually in IN_PROGRESS state:

2014-12-23 11:21:58.878 23656 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.35.160.29
2014-12-23 11:21:58.945 23656 DEBUG urllib3.connectionpool [-] "POST /v3/auth/tokens HTTP/1.1" 401 114_make_request /usr/lib/python2.7/site-packages/urllib3/connectionpool.py:357
2014-12-23 11:21:58.945 23656 DEBUG keystoneclient.session [-] Request returned failure status: 401 request /usr/lib/python2.7/site-packages/keystoneclient/session.py:345
2014-12-23 11:21:58.946 23656 DEBUG keystoneclient.v3.client [-] Authorization failed. get_raw_token_from_identity_service /usr/lib/python2.7/site-packages/keystoneclient/v3/client.py:267
2014-12-23 11:21:58.959 23656 DEBUG heat.engine.stack_lock [-] Engine 9b39ddc4-e203-4567-b3a7-c85669d7adfa released lock on stack 6bcda96b-1c41-440a-bfb3-37f59331019b release /usr/lib/python2.7/site-packages/heat/engine/stack_lock.py:122

Tested on 0.2.12

[Steve Baker (steve-stevebaker)]

can you delete the stack?

[Removed by request (removed3381998)]

Yes. Also both 'list' and 'show' did not give any indication that something is wrong.

[Removed by request (removed3381998)]

Some more information from Keystone's side here: https://ask.openstack.org/en/question/57044/auth-token-not-in-the-request-header-when-running-heat-stack-create/

[Launchpad Janitor (janitor)]

[Expired for heat because there has been no activity for 60 days.]

[rajiv (rajiv-kumar)]

I installed heat using packstack and after installation, i am not able to create stack. It stucks in CREATE_IN_PROGRESS state. I checked heat-engine log, i got the same, as in this bug. I checked keystone log too. I looked at the request processed at the keystone, here is what i got

WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from 10.0.8.18

i checked role assigned to admin(i am creating stack from admin account)

+----------------------------------+------------------+----------------------------------+----------------------------------+
| id | name | user_id | tenant_id |
+----------------------------------+------------------+----------------------------------+----------------------------------+
| 9fe2ff9ee4384b1894a90878d3e92bab | _member_ | 69ce4cc2167e435ca845f9fd5cd222be | 21d2a58fc01045858947986fad8f9ed1 |
| 10b25e2ce56c4efda109dc3086b0cda9 | admin | 69ce4cc2167e435ca845f9fd5cd222be | 21d2a58fc01045858947986fad8f9ed1 |
| 0ea68e0c13c64d07818421aaf80eaf7b | heat_stack_owner | 69ce4cc2167e435ca845f9fd5cd222be | 21d2a58fc01045858947986fad8f9ed1 |
+----------------------------------+------------------+----------------------------------+----------------------------------+

I listed roles too. There is one more role "heat_stack_user" and this is not assigned to admin. is it the reason of the problem?

I think, it is not bug, it is just misconfiguration of role. If anyone has solved this problem please let me know.

[Steve Baker (steve-stevebaker)]

On 28/03/15 02:36, rajiv wrote:
>
> I listed roles too. There is one more role "heat_stack_user" and this
> is not assigned to admin. is it the reason of the problem?
>
>
No, never do this. The heat install guide[1] says this:

   NOTE: The Orchestration service automatically assigns
the|heat_stack_user|role to users that it creates during stack
deployment. By default, this role restrictsAPI
<http://docs.openstack.org/juno/install-guide/install/apt/content/heat-install-controller-node.html#>operations.
To avoid conflicts, do not add this role to users with
the|heat_stack_owner|role.

[1]
http://docs.openstack.org/juno/install-guide/install/apt/content/heat-install-controller-node.html

[rajiv (rajiv-kumar)]

I want to work on this bug, can you (@steve) please direct me in right direction. How to debug or found root cause of this problem.

[Alex Tesch (gatesch)]

Same problem here,

It seems to be some kind of a problem with the trust and keystone v3,

soon after issuing the "heat stack-create" command, keystone.log reports

"POST /v3/OS-TRUST/trusts HTTP/1.1" 201 851 0.121289
2015-04-13 16:42:20.156 14860 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication.

The heat-engine.log shows the same message highlighted by Rajiv above.

Adding to that, the log pointed that all the resources are "validating", however it never starts "creating" the resources in the stack. Below is an abstract.

2015-04-13 16:42:19.465 15050 INFO heat.engine.resource [req-24ddd1a8-da86-4d83-
921b-6371fe3ed77f None] Validating RouterInterface "heat_router_int0"
2015-04-13 16:42:19.466 15050 INFO heat.engine.resource [req-24ddd1a8-da86-4d83-
921b-6371fe3ed77f None] Validating RouterGateway "heat_router_01_gw"
2015-04-13 16:42:19.511 15050 DEBUG keystoneclient.auth.identity.v3 [req-24ddd1a
8-da86-4d83-921b-6371fe3ed77f ] Making authentication request to http://192.168.
1.66:35357/v3/auth/tokens get_auth_ref /usr/lib/python2.7/site-packages/keystone
client/auth/identity/v3.py:117
2015-04-13 16:42:19.513 15050 INFO urllib3.connectionpool [req-24ddd1a8-da86-4d8
3-921b-6371fe3ed77f ] Starting new HTTP connection (1): 192.168.1.66
2015-04-13 16:42:19.710 15050 DEBUG urllib3.connectionpool [req-24ddd1a8-da86-4d
83-921b-6371fe3ed77f ] "POST /v3/auth/tokens HTTP/1.1" 201 9611 _make_request /u
sr/lib/python2.7/site-packages/urllib3/connectionpool.py:357
2015-04-13 16:42:19.713 15050 DEBUG keystoneclient.session [req-24ddd1a8-da86-4d
83-921b-6371fe3ed77f ] REQ: curl -i -X POST http://192.168.1.66:35357/v3/OS-TRUST/trusts -H "User-Agent: python-keystoneclient" -H "Content-Type: application/json" -H "X-Auth-Token: TOKEN_REDACTED" -d '{"trust": {"impersonation": true, "project_id": "05d9f647b8044b21814b18801a55fe84", "trustor_user_id": "67237a17be794ad29fc28e1ce3b0dca9", "roles": [{"name": "heat_stack_owner"}], "trustee_user_id": "46887b9a3fe74864a11e3f9a5827cf21"}}' _http_log_request /usr/lib/python2.7/site-packages/keystoneclient/session.py:155

This was obtained on RHOS 6 after a successful packstack installation.

Alex

[sara (ichi-sara)]

Hey, I have the same error and didn't find any clue to debug this. Did you solve the problem?

[John Haller (john-haller)]

I suspect that the original bug report expected that the state of the stack should be FAILED when misconfigured, as opposed to being continuously in IN_PROGRESS. The stack create should fail with an authorization failure when keystone authorization fails.

For completeness, and people reading the bug history expecting information about the misconfiguration and how to fix, This has fixed the problem for ichi-sara using a packstack install of RDO, which did not fully set up Keystone v3 for Heat:
$ heat-keystone-setup-domain \
--stack-user-domain-name heat_user_domain \
--stack-domain-admin heat_domain_admin \
--stack-domain-admin-password heat_domain_password
(obviously, pick a different password than the above)

Add/update the following to heat.conf with the following in [DEFAULT]

stack_user_domain_id=UUID of heat_user_domain
stack_domain_admin=heat_domain_admin
stack_domain_admin_password=heat_domain_password

It does not appear that all of the above is in the Install guide, in particular, the stack_user_domain_id, stack_domain_admin, and stack_domain_admin_password configurations are not mentioned, which is probably worth a bug report on the install guide.

Additionally, a bug on the Puppet scripts used by packstack to properly configure Heat for Keystone V3 would also be warranted.