Can't deploy overcloud of Mitaka on CentOS

[Andrey Pavlov]

CentOS 7.2
Undercloud deployed normally by tripleo instructions.

keystone -
Version : 9.2.1
Release : 0.20161007011449.012bc3d.el7.centos

heat-api -
Version : 6.0.1
Release : 0.20160829124409.ed46562.el7.centos

These numbers tell us that undercloud installed from Mitaka repository

But overcloud deploy fails with error -
[stack@myhost ~]$ openstack overcloud deploy --templates --neutron-tunnel-types vxlan --neutron-network-type vxlan --ntp-server pool.ntp.org --control-scale 1 --compute-scale 1 --block-storage-scale 3 --control-flavor control --compute-flavor compute --block-storage-flavor block-storage -e overcloud/scaleio-env.yaml
Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates
ERROR: Authorization failed.

keystone logs contains error: domain Default can not be found

workaround - change domain from 'Default' to 'default' in heat-api.conf and then overcloud deploy can be started...

[Ben Nemec (bnemec)]

This is also causing stable/mitaka failures in CI.

[Ben Nemec (bnemec)]

Note that the domains in heat-api.conf seem to be set to Default on master as well, but for some reason it's working there. :-/

This is most likely https://github.com/openstack/keystone/commit/eccd428870f06507234b682ad2d645f7904fb181

[Ben Nemec (bnemec)]

Confirmed locally that reverting the above patch fixes this. I'm not sure why it's only affecting Mitaka, so going to add keystone to the bug and see if they have any thoughts.

[Ben Nemec (bnemec)]

In case it helps, my keystone database looks like this:

MariaDB [keystone]> select * from project
    -> ;
+----------------------------------+--------------------------+-------+-----------------------------------+---------+--------------------------+-----------+-----------+
| id | name | extra | description | enabled | domain_id | parent_id | is_domain |
+----------------------------------+--------------------------+-------+-----------------------------------+---------+--------------------------+-----------+-----------+
| 3bf3b0bfcc334f4e96edb7ff25dbd0b5 | heat_stack | {} | | 1 | <<keystone.domain.root>> | NULL | 1 |
| 85520d55b2c34ba8b925728eedfab5e9 | admin | {} | admin tenant | 1 | default | default | 0 |
| <<keystone.domain.root>> | <<keystone.domain.root>> | {} | | 0 | <<keystone.domain.root>> | NULL | 1 |
| c33d2ac55f5941b78bda5835dfbdd3c8 | service | {} | Tenant for the openstack services | 1 | default | default | 0 |
| default | Default | {} | The default domain | 1 | <<keystone.domain.root>> | NULL | 1 |
+----------------------------------+--------------------------+-------+-----------------------------------+---------+--------------------------+-----------+-----------+

Note that we're using the default domain from the puppet modules, so this isn't specific to tripleo. I see the id is "default" and the name is "Default", and puppet configures heat to use "Default". This works fine on Newton and Master, but is breaking on Mitaka.

[Ben Nemec (bnemec)]

Sorry, that paste looks terrible. Try this one: http://paste.openstack.org/show/585382/

[Ben Nemec (bnemec)]

https://review.openstack.org/#/c/385114/ confirms that reverting the keystone patch fixes our mitaka deploys. Just need to figure out what the correct fix is.

[Ben Nemec (bnemec)]

Ah, I see the problem. In Mitaka, we're setting project_domain_id to Default. In master, we set project_domain_name to Default (instead of _id), which is correct. Looking into the right way to fix that.

[Ben Nemec (bnemec)]

Fix proposed in puppet-heat: https://review.openstack.org/385179

[Steve Martinelli (stevemar)]

Thanks for the quick analysis here Ben. Looking at newton and future releases, if you are using the "keystone-manage bootstrap" option to setup keystone, then the domain ID won't be "default" it'll be some UUID. Your best bet going forward is to use the domain name only, it'll always be "Default" (capital D).

Marking this as Invalid for keystone.

[Andrey Pavlov (apavlov-e)]

@Ben, please help me to understand when it will be available in packages/production?

[Andrey Pavlov (apavlov-e)]

verified. now it works.