heat stack creation failed with keystone v3

[Narinder Gupta]

After doing the deployment with latest heat charm running a simple test to create the resource under heat template fails on authentication failure.

cat autoscaling_policy.yaml.template
heat_template_version: 2017-02-24

resources:
  test_group:
    type: OS::Heat::AutoScalingGroup
    properties:
      desired_capacity: 0
      max_size: 0
      min_size: 0
      resource:
        type: OS::Heat::RandomString
  test_policy:
    type: OS::Heat::ScalingPolicy
    properties:
      adjustment_type: change_in_capacity
      auto_scaling_group_id: { get_resource: test_group }
      scaling_adjustment: 1

ubuntu@etsi-ubuntu-jh:~$

heat stack-create --template-file autoscaling_policy.yaml.template heat_st
WARNING (shell) "heat stack-create" is deprecated, please use "openstack stack create" instead
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
+--------------------------------------+------------+--------------------+----------------------+--------------+
| id | stack_name | stack_status | creation_time | updated_time |
+--------------------------------------+------------+--------------------+----------------------+--------------+
| c0a3cb1a-176b-47a6-bef3-b3d146990b38 | heat_st | CREATE_IN_PROGRESS | 2017-09-06T18:35:58Z | None |
+--------------------------------------+------------+--------------------+----------------------+--------------+
ubuntu@etsi-ubuntu-jh:~$

 heat stack-show heat_st
WARNING (shell) "heat stack-show" is deprecated, please use "openstack stack show" instead
+-----------------------+------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+-----------------------+------------------------------------------------------------------------------------------------------------------------+
| capabilities | [] |
| creation_time | 2017-09-06T18:35:58Z |
| deletion_time | None |
| description | No description |
| disable_rollback | True |
| id | c0a3cb1a-176b-47a6-bef3-b3d146990b38 |
| links | http://192.0.5.194:8004/v1/546499c82a8249fdafb1982979a0baa1/stacks/heat_st/c0a3cb1a-176b-47a6-bef3-b3d146990b38 (self) |
| notification_topics | [] |
| outputs | [] |
| parameters | { |
| | "OS::project_id": "546499c82a8249fdafb1982979a0baa1", |
| | "OS::stack_id": "c0a3cb1a-176b-47a6-bef3-b3d146990b38", |
| | "OS::stack_name": "heat_st" |
| | } |
| parent | None |
| stack_name | heat_st |
| stack_owner | None |
| stack_status | CREATE_FAILED |
| stack_status_reason | Resource CREATE failed: AuthorizationFailure: |
| | resources.test_policy: Authorization failed. |
| stack_user_project_id | None |
| tags | null |
| template_description | No description |
| timeout_mins | None |
| updated_time | None |
+-----------------------+------------------------------------------------------------------------------------------------------------------------+

[Narinder Gupta (narindergupta)]

heat log sho the below error.
2017-09-06 10:56:49.663 27620 WARNING oslo_config.cfg [-] Option "rabbit_password" from group "oslo_messaging_rabbit" is deprecated for removal. Its value may be silently ignored in the future.
2017-09-06 17:01:16.177 27616 WARNING oslo_config.cfg [req-d4e1387c-9a1a-4957-9e19-cc66fe5fa44c - admin - - -] Option "auth_plugin" from group "trustee" is deprecated. Use option "auth_type" from group "trustee".
2017-09-06 17:01:16.925 27616 ERROR heat.engine.clients.keystoneclient [req-d4e1387c-9a1a-4957-9e19-cc66fe5fa44c - admin - - -] Domain admin client authentication failed
2017-09-06 17:01:17.673 27617 ERROR heat.engine.clients.keystoneclient [req-d4e1387c-9a1a-4957-9e19-cc66fe5fa44c - admin - - -] Domain admin client authentication failed
2017-09-06 17:01:19.341 27619 ERROR heat.engine.clients.keystoneclient [req-d4e1387c-9a1a-4957-9e19-cc66fe5fa44c - admin - - -] Domain admin client authentication failed
2017-09-06 17:03:21.484 27620 WARNING oslo_config.cfg [req-729f984d-ac3b-4cba-b679-22fa46711ece - admin - - -] Option "auth_plugin" from group "trustee" is deprecated. Use option "auth_type" from group "trustee".
2017-09-06 17:03:22.165 27620 ERROR heat.engine.clients.keystoneclient [req-729f984d-ac3b-4cba-b679-22fa46711ece - admin - - -] Domain admin client authentication failed
2017-09-06 17:03:22.892 27618 ERROR heat.engine.clients.keystoneclient [req-729f984d-ac3b-4cba-b679-22fa46711ece - admin - - -] Domain admin client authentication failed
2017-09-06 17:03:24.497 27618 ERROR heat.engine.clients.keystoneclient [req-729f984d-ac3b-4cba-b679-22fa46711ece - admin - - -] Domain admin client authentication failed
2017-09-06 18:32:27.573 27617 WARNING oslo_config.cfg [req-3b132976-9f5a-4e68-8719-9d1597be163e - admin - - -] Option "auth_plugin" from group "trustee" is deprecated. Use option "auth_type" from group "trustee".
2017-09-06 18:32:28.194 27617 ERROR heat.engine.clients.keystoneclient [req-3b132976-9f5a-4e68-8719-9d1597be163e - admin - - -] Domain admin client authentication failed
2017-09-06 18:32:28.948 27620 ERROR heat.engine.clients.keystoneclient [req-3b132976-9f5a-4e68-8719-9d1597be163e - admin - - -] Domain admin client authentication failed
2017-09-06 18:32:30.553 27620 ERROR heat.engine.clients.keystoneclient [req-3b132976-9f5a-4e68-8719-9d1597be163e - admin - - -] Domain admin client authentication failed
2017-09-06 18:35:58.046 27616 ERROR heat.engine.clients.keystoneclient [req-2ccc34cd-821a-47cd-be03-4350c1f880f6 - admin - - -] Domain admin client authentication failed
2017-09-06 18:35:58.759 27617 ERROR heat.engine.clients.keystoneclient [req-2ccc34cd-821a-47cd-be03-4350c1f880f6 - admin - - -] Domain admin client authentication failed
2017-09-06 18:36:00.306 27617 ERROR heat.engine.clients.keystoneclient [req-2ccc34cd-821a-47cd-be03-4350c1f880f6 - admin - - -] Domain admin client authentication failed

complete logs can be found at

sudo pastebinit < /var/log/heat/heat-engine.log
http://paste.ubuntu.com/25479762/

[Narinder Gupta (narindergupta)]

cat joid_config/admin-openrc
export OS_AUTH_URL=http://192.0.5.229:5000/v3
export OS_USER_DOMAIN_NAME=admin_domain
export OS_PROJECT_DOMAIN_NAME=admin_domain
export OS_USERNAME=admin
export OS_TENANT_NAME=admin
export OS_PROJECT_NAME=admin
export OS_PASSWORD=openstack
export OS_IDENTITY_API_VERSION=3
export OS_REGION_NAME=RegionOne

[Jason Hobbs (jason-hobbs)]

I tested this on our lab and was able to reproduce the issue, just as narinder reported above.

[Nobuto Murata (nobuto)]

Out of curiosity, did you execute the heat action below before trying to use Heat?

$ juju run-action heat/0 domain-setup

[Narinder Gupta (narindergupta)]

yes thats part of post deployment step we execute and I can confirm that was executed as we have CI in OPNFV community and we are able to reproduce there also.

[huangsm (huangsm)]

I test this on my lab, heat_template_version: 2016-04-08, not 2017-02-24.
But the issue don't be reproduced.
can you show your heat.conf

[Narinder Gupta (narindergupta)]

here is my heat.conf

http://paste.ubuntu.com/25488358/
 and openrc file

cat joid_config/admin-openrc
export OS_AUTH_URL=http://192.0.5.229:5000/v3
export OS_USER_DOMAIN_NAME=admin_domain
export OS_PROJECT_DOMAIN_NAME=admin_domain
export OS_USERNAME=admin
export OS_TENANT_NAME=admin
export OS_PROJECT_NAME=admin
export OS_PASSWORD=openstack
export OS_IDENTITY_API_VERSION=3
export OS_REGION_NAME=RegionOne

[Narinder Gupta (narindergupta)]

This is for OPNFV E release which is based on Ocata and current charms integrated in this is 17.08 bundle file is as follows.
http://paste.ubuntu.com/25532647/

Where charm has been downloaded locally first before deployment. Charm download script is available at:

https://gerrit.opnfv.org/gerrit/gitweb?p=joid.git;a=blob;f=ci/nosdn/fetch-charms.sh

[James Page (james-page)]

The role assignment for the heat_domain_admin was incorrectly scoped for v3 deployments; raising a fix now.

[James Page (james-page)]

https://review.openstack.org/504219

[Narinder Gupta (narindergupta)]

I can confirm the patch posted by James fixes the problem here is the result after the fixes.

ubuntu@etsi-ubuntu-jh:~$ heat stack-show heat_st
WARNING (shell) "heat stack-show" is deprecated, please use "openstack stack show" instead
+-----------------------+------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+-----------------------+------------------------------------------------------------------------------------------------------------------------+
| capabilities | [] |
| creation_time | 2017-09-14T21:27:09Z |
| deletion_time | None |
| description | No description |
| disable_rollback | True |
| id | 81694895-0fd0-49ec-a468-0e2f150ccc21 |
| links | http://192.0.5.159:8004/v1/2f236fb1e55c4db9841f64b7b386426c/stacks/heat_st/81694895-0fd0-49ec-a468-0e2f150ccc21 (self) |
| notification_topics | [] |
| outputs | [] |
| parameters | { |
| | "OS::project_id": "2f236fb1e55c4db9841f64b7b386426c", |
| | "OS::stack_id": "81694895-0fd0-49ec-a468-0e2f150ccc21", |
| | "OS::stack_name": "heat_st" |
| | } |
| parent | None |
| stack_name | heat_st |
| stack_owner | None |
| stack_status |...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/504219
Committed: https://git.openstack.org/cgit/openstack/charm-heat/commit/?id=501e3415b346322769947ddfb5665e56490b3e3f
Submitter: Jenkins
Branch: master

commit 501e3415b346322769947ddfb5665e56490b3e3f
Author: James Page <email address hidden>
Date: Thu Sep 14 14:20:49 2017 -0600

    Correctly scope role assignment for heat_domain_admin

    The admin role assignment for the heat_domain_admin user
    needs to be scoped to the heat domain; update domain-setup
    action to scope the role assigment correctly.

    This change also stops ignoring the return code of the
    role assignment; re-assigning a role already granted is
    idemponent in openstackclient.

    Change-Id: Ia5ba409e2f566614e6db9bbf9540c563af8e4a82
    Closes-Bug: 1715465

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/17.08
Review: https://review.openstack.org/505614

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/505614
Committed: https://git.openstack.org/cgit/openstack/charm-heat/commit/?id=f30723b839fb91f0f50873b16cc724c02845e324
Submitter: Jenkins
Branch: stable/17.08

commit f30723b839fb91f0f50873b16cc724c02845e324
Author: James Page <email address hidden>
Date: Thu Sep 14 14:20:49 2017 -0600

    Correctly scope role assignment for heat_domain_admin

    The admin role assignment for the heat_domain_admin user
    needs to be scoped to the heat domain; update domain-setup
    action to scope the role assigment correctly.

    This change also stops ignoring the return code of the
    role assignment; re-assigning a role already granted is
    idemponent in openstackclient.

    Change-Id: Ia5ba409e2f566614e6db9bbf9540c563af8e4a82
    Closes-Bug: 1715465
    (cherry picked from commit 501e3415b346322769947ddfb5665e56490b3e3f)