Heat Stack: Authorization failed

[Andrew Kalach]

Description
===========
During 9.2 Certification Testing Sutie run, the Heat tasks failed with error:

HTTPInternalServerError: ERROR: Authorization failed.

Steps to reproduce
==================
Install Rally on masternode and run specified rally tasks (see attachment):

1. cd /opt/stack
2. source .venv/bin/activate
3. rally task start <task YAML file>

Expected result
===============
Task run without errors

Actual result
=============
The following error logs:

2016-11-30 08:38:06.475 11985 ERROR rally.task.runner [-] ERROR: Authorization failed.
2016-11-30 08:38:06.475 11985 ERROR rally.task.runner Traceback (most recent call last):
2016-11-30 08:38:06.475 11985 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/runner.py", line 73, in _run_scenario_once
2016-11-30 08:38:06.475 11985 ERROR rally.task.runner getattr(scenario_inst, method_name)(**scenario_kwargs)
2016-11-30 08:38:06.475 11985 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/plugins/openstack/scenarios/heat/stacks.py", line 87, in run
2016-11-30 08:38:06.475 11985 ERROR rally.task.runner files, environment)
2016-11-30 08:38:06.475 11985 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/task/atomic.py", line 84, in func_atomic_actions
2016-11-30 08:38:06.475 11985 ERROR rally.task.runner f = func(self, *args, **kwargs)
2016-11-30 08:38:06.475 11985 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/rally/plugins/openstack/scenarios/heat/utils.py", line 142, in _create_stack
2016-11-30 08:38:06.475 11985 ERROR rally.task.runner stack_id = self.clients("heat").stacks.create(**kw)["stack"]["id"]
2016-11-30 08:38:06.475 11985 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/heatclient/v1/stacks.py", line 172, in create
2016-11-30 08:38:06.475 11985 ERROR rally.task.runner data=kwargs, headers=headers)
2016-11-30 08:38:06.475 11985 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 193, in post
2016-11-30 08:38:06.475 11985 ERROR rally.task.runner return self.request(url, 'POST', **kwargs)
2016-11-30 08:38:06.475 11985 ERROR rally.task.runner File "/opt/stack/.venv/lib/python2.7/site-packages/heatclient/common/http.py", line 318, in request
2016-11-30 08:38:06.475 11985 ERROR rally.task.runner raise exc.from_response(resp)
2016-11-30 08:38:06.475 11985 ERROR rally.task.runner HTTPInternalServerError: ERROR: Authorization failed.
2016-11-30 08:38:06.475 11985 ERROR rally.task.runner
2016-11-30 08:38:06.479 11985 INFO rally.task.runner [-] Task b13018f6-8624-4691-bc26-59a965c6c04f | ITER: 1 END: Error HTTPInternalServerError: ERROR: Authorization failed.

Environment
===========
Scale 200-node lab (ENV-10):
  3 controllers
  200 compute nodes
  20 ceph nodes
Image build:
  Build-9.0.0-495_snap_557
Diagnostic snapshot:
  http://mos-scale-share.mirantis.com/fuel-dsc-161208.tar.gz

[Vitaly Sedelnik (vsedelnik)]

Andrew, please provide diagnostic snapshot.

[Andrew Kalach (akndex)]

Diagnostic snapshot added

[Sergey Kraynev (skraynev)]

Right now, we have lab with 9.0. So we need to update our lab to 9.2 to reproduce mentioned issue.
So current status is WIP upgrading lab to 9.2

[Peter Razumovsky (prazumovsky)]

Next results were found:

Heat uses trusts for auth. /etc/heat/heat.conf has configured section trustee, which has trustee/project_domain_id = Default. This is incorrect, because default keystone domain has id "default" and name "Default". So, authentication cannot find domain with id "Default" and fails.

project_domain_id specified in puppet-heat init.pp manifest [1].
In Nov 30, when this error has been raised, fuel-library [2] references to 8.2.0 puppet-heat, which has next values of domain variables: [3]. This is incorrect due to issue description above.

Since Oct 13 puppet-heat has fix for wrong values [4]. But in fuel-library this fix merged Dec 1 [5]. So, Nov 30 puppet-heat still configured wrong values for domain variables, but now this issue has been fixed.

[1] https://github.com/openstack/puppet-heat/blob/stable/mitaka/manifests/init.pp#L282-L285
[2] https://github.com/openstack/fuel-library/blob/1959ba7d674fada96a0a6bf30c79850ff77b5fb2/deployment/puppet/openstack_tasks/Puppetfile#L45-L47
[3] https://github.com/fuel-infra/puppet-heat/blob/8.2.0/manifests/init.pp#L277-L280
[4] https://review.openstack.org/#/c/385179/
[5] https://review.openstack.org/#/c/402699/

[Vitaly Sedelnik (vsedelnik)]

Setting to Fix Committed per above comment. Link to the fix - https://review.openstack.org/#/c/402699/