Contrail Ocata :: R4.0.1.0 build 34 :: keystone v3 not supported.

[Ritam Gangopadhyay]

Keystone v3 provisioning does not bring up all necessary services and conf/policy files properly. Here are the errors and discrepancies I see.

You can bring up any ocata setup with below config in cluster json to reproduce the issue.

"parameters": {
    "provision": {
        "openstack": {
            "keystone": {
                "version": "v3"

************************************
************************************

Authentication Failure

************************************
************************************

root@nodei19:~# grep -rn "Authorization failed" /var/lib/docker/volumes/kolla_logs/_data/keystone/keystone.log
460:2017-09-06 15:37:49.509 24 WARNING keystone.common.wsgi [req-4d2afbb4-2bce-4d95-a394-bd94ded2dfbc - - - - -] Authorization failed. The request you have made requires authentication. from 192.168.100.15
472:2017-09-06 15:37:58.752 25 WARNING keystone.common.wsgi [req-5176c15a-543d-4beb-a624-cbf55fda0de2 - - - - -] Authorization failed. The request you have made requires authentication. from 192.168.100.15
495:2017-09-06 15:38:08.276 24 WARNING keystone.common.wsgi [req-550a52e0-130f-41f4-8f95-95b50a5f4704 - - - - -] Authorization failed. The request you have made requires authentication. from 192.168.100.15
10896:2017-09-06 18:35:07.188 23 WARNING keystone.common.wsgi [req-b560472e-87a9-4655-b059-916ab7454307 - - - - -] Authorization failed. The request you have made requires authentication. from 192.168.100.11
10948:2017-09-06 18:37:10.149 25 WARNING keystone.common.wsgi [req-3c030d00-fa11-420e-88cb-713b398a5627 - - - - -] Authorization failed. The request you have made requires authentication. from 192.168.100.11
11287:2017-09-06 18:40:32.038 18 WARNING keystone.common.wsgi [req-623e3f7b-e187-4be2-8a3e-e29349ec90ac df318ee8467e497a970ac8989eacf378 - - default -] Authorization failed. The request you have made requires authentication. from 192.168.100.15
12353:2017-09-06 18:47:31.701 27 WARNING keystone.common.wsgi [req-38bfae01-a6fc-4552-bf62-b5ffbdd218f5 - - - - -] Authorization failed. The request you have made requires authentication. from 192.168.100.15
13034:2017-09-06 18:58:10.537 22 WARNING keystone.common.wsgi [req-da681b9b-0510-4fdc-b41b-82c5ffeea906 - - - - -] Authorization failed. The request you have made requires authentication. from 192.168.100.15
13332:2017-09-06 19:00:27.640 20 WARNING keystone.common.wsgi [req-ae56b0fb-7a85-4ae8-aa84-b837e437b7f6 df318ee8467e497a970ac8989eacf378 - - default -] Authorization failed. The request you have made requires authentication. from 192.168.100.15
15209:2017-09-06 19:32:16.574 20 WARNING keystone.common.wsgi [req-474260eb-d77c-4496-9b1b-ea4d9d1d766c df318ee8467e497a970ac8989eacf378 - - default -] Authorization failed. The request you have made requires authentication. from 192.168.100.15
16624:2017-09-06 19:52:40.214 22 WARNING keystone.common.wsgi [req-cbf21453-6b9b-4977-87f1-99dfcfbabda0 df318ee8467e497a970ac8989eacf378 - - default -] Authorization failed. The request you have made requires authentication. from 192.168.100.15
19364:2017-09-06 20:24:10.624 22 WARNING keystone.common.wsgi [req-6e580f13-0c44-498c-8fa6-ffe4d0b5e676 - - - - -] Authorization failed. The request you have made requires authentication. from 192.168.100.15
20417:2017-09-06 20:38:54.021 22 WARNING keystone.common.wsgi [req-fbb4883b-4644-46c2-b0e9-0fd09429ed66 df318ee8467e497a970ac8989eacf378 - - default -] Authorization failed. The request you have made requires authentication. from 192.168.100.15
21248:2017-09-06 20:50:50.379 23 WARNING keystone.common.wsgi [req-f51d288b-2257-4525-b290-ebcd5de33a77 - - - - -] Authorization failed. The request you have made requires authentication. from 192.168.100.11
root@nodei19:~#

***************************************************
***************************************************

Missing Configuration and provisioning issues

***************************************************
***************************************************

1. Horizon doesn't come up with the option of domain and keystone v3 domain context enabled.

2. No v3 policy file or cloud_admin in keystone policy file /etc/keystone/policy.json

(keystone)[root@nodei19 /]# grep -rn cloud /etc/keystone/policy.json
(keystone)[root@nodei19 /]#

3. config.identityManager.apiVersion = ['v2.0']; in webui /etc/contrail/config.global.js not configured to v3

root@nodec28(controller):/# grep -rn "identityManager.apiVersion =" /etc/contrail/config.global.js
156:config.identityManager.apiVersion = ['v2.0'];
root@nodec28(controller):/#

4. /etc/openstack-dashboard/local_settings does not have multi domain and v3 parameters set

(horizon)[root@nodei19 /]# grep -rn OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT /etc/openstack-dashboard/local_settings
72:#OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT = False
(horizon)[root@nodei19 /]# grep -rn OPENSTACK_KEYSTONE_URL /etc/openstack-dashboard/local_settings
170:OPENSTACK_KEYSTONE_URL = "http://192.168.100.20:5000"
(horizon)[root@nodei19 /]#

5. None of the services are configured to authenticate with keystone in v3

(nova-api)[nova@nodei19 /]$ cat /etc/nova/nova.conf | grep keystone -A13
[keystone_authtoken]
auth_uri = http://192.168.100.20:5000
auth_url = http://192.168.100.20:35357
auth_type = password
project_domain_id = default
user_domain_id = default
project_name = service
username = nova
password = contrail123
memcache_security_strategy = ENCRYPT
memcache_secret_key = contrail123
memcached_servers = 192.168.100.15:11211

(nova-api)[nova@nodei19 /]$

6. keystone endpoints are configured in v3

root@nodei19:~# openstack endpoint list | grep keystone
| 46de2f2005cb4b319f57d51b487fd97d | RegionOne | keystone | identity | True | admin | http://192.168.100.20:35357 |
| 794616380a674acf94aea2facb0853d4 | RegionOne | keystone | identity | True | internal | http://192.168.100.20:5000 |
| 7a9415c0fa4d413ba32bb8ccdf1c6865 | RegionOne | keystone | identity | True | public | http://10.204.217.184:5000 |
root@nodei19:~#

root@nodei19:~# openstack domain create --description "New Domain" NewDomain
2017-09-06 20:22:23.584 23 INFO keystone.common.wsgi [req-044c65c2-55be-4f8f-b4c5-27172882cc06 - - - - -] GET http://192.168.100.20:35357/v3/
2017-09-06 20:22:23.592 27 INFO keystone.common.wsgi [req-d44c6ce6-0ba4-4f6e-92dd-84e037ac5768 - - - - -] POST http://192.168.100.20:35357/v3/auth/tokens
2017-09-06 20:22:23.719 24 INFO keystone.common.wsgi [req-eae6823f-8abf-4572-b56e-a0cb7e49d7ed - - - - -] POST http://192.168.100.20:35357/v3/auth/tokens
2017-09-06 20:22:23.838 22 INFO keystone.common.wsgi [req-82c78b6a-3ebe-4d55-a424-47811c43bc2b df318ee8467e497a970ac8989eacf378 - - default -] GET http://192.168.100.20:5000/
2017-09-06 20:22:23.878 21 INFO keystone.common.wsgi [req-0e0752f7-275e-47a4-acb3-e18b69c78590 df318ee8467e497a970ac8989eacf378 7bb15144662e4958be579e6b539af017 - default default] POST http://192.168.100.20:5000/v3/domains
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | New Domain |
| enabled | True |
| id | 87ffa7a8de60437e83adb0058a8831ef |
| name | NewDomain |
+-------------+----------------------------------+
root@nodei19:~#

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35416
Submitter: Ramprakash R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35417
Submitter: Ramprakash R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35420
Submitter: Ramprakash R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35421
Submitter: Ramprakash R (<email address hidden>)

[Ramprakash R (ramprakash)]

@Ritam: With the above commit, please note the following (number corresponds to the item number in the bug description above):

1. Horizon now comes up with option for Domain when version is set to v3 (see attachment)?field.comment=@Ritam: With the above commit, please note the following (number corresponds to the item number in the bug description above):

1. Authorization errors are not seen now. Horizon now comes up with option for Domain when version is set to v3 (see attachment)

2. For keystone policy.json, the way Kolla handles this is the user crafts his own policy.json based on https://github.com/openstack/keystone/blob/master/etc/policy.v3cloudsample.json and then if this is placed in /etc/kolla/configs/keystone/policy.json of the SM machine (where sensible is going to be run) this will be copied into the container during provision. So if you need a policy with cloud_admin, you can craft one yourself and place it in the directory mentioned above and then issue provision. If there is a change, you can change it and then issue retrovision for it to take effect.

3. The provisioning code seems to set the version information correctly, but I suspect the internal Ansible is not handling this to set in the config.global.js file OR it may not be required.

*** Please raise a separate bug and track this with the config/controller team. ***

4. openstack-dashboard/localsettings will now have MULTIDOMAIN configs enabled when v3 is enabled.

5. nova.conf seems to be coming directly form upstream kolla nova templates. Please check if this is causing operational issues and then we can debug/fix if required.

6. keystone endpoints are now v3 if version is set to v3:
root@server8:~# openstack endpoint list | grep keystone
| 1179f3aa560c400b979c9e90297f8c79 | RegionOne | keystone | identity | True | internal | http://192.168.10.100:5000/v3 |
| 24e8fd5155ed4360a1b46c03eed36b82 | RegionOne | keystone | identity | True | public | http://192.168.1.100:5000/v3 |
| 8a8a5e7f59fe4a4085cb73804bc20563 | RegionOne | keystone | identity | True | admin | http://192.168.10.100:35357/v3 |

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35416
Submitter: Ramprakash R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35417
Submitter: Ramprakash R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/35417
Committed: http://github.com/Juniper/contrail-server-manager/commit/3bd688b7ab393472eb90bd8da0a259eea4c3e0f2
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 3bd688b7ab393472eb90bd8da0a259eea4c3e0f2
Author: Ramprakash Ram Mohan <email address hidden>
Date: Fri Sep 8 16:51:49 2017 -0700

Derive value for enable_keystone_v3 based on keystone version

Change-Id: Ia61af991ca18e364903925d698e6c93498a8ac0c
Partial-bug: #1715427

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/35420
Committed: http://github.com/Juniper/contrail-ansible/commit/a193b65f8c8d75ceb3b430235ccc0c419127813f
Submitter: Zuul (<email address hidden>)
Branch: master

commit a193b65f8c8d75ceb3b430235ccc0c419127813f
Author: Ramprakash Ram Mohan <email address hidden>
Date: Fri Sep 8 16:54:25 2017 -0700

Add support for keystone v3 in ocata

1. group_vars/all.yml - Add default values for enable_keystone_v3,
keystone_admin_project_name and keystone_admin_project_domain_name
2. horizon/templates/local_settings.j2 - If enable_keystone_v3 is true, then
enable the OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT and
OPENSTACK_KEYSTONE_DEFAULT_DOMAIN variables
3. keystone/tasks/config.yml - Upstream bug - typo in the notify tag
4. keystone/tasks/register.yml - Assign admin user to default domain for v3
5. keystone/template/keystone.conf.j2 - Fix for Bug #1710739 - setting admin
project details in keystone.conf
6. neutron_opencontrail.conf.j2 - No need to use keystone_admin_user variable,
use the existing openstack_auth dictionary instead
7. globals.yml.original - update sample kolla_globals with the newly introduced
variables
8. roles/openstack/compute/tasks/configure.yml - transport_url was not being
populated with all the rabbitmq hosts in the case of oenstack ha

Closes-bug: #1710739
Closes-bug: #1715427

Change-Id: I13f1b8a5f615496f3fbfb30ce611a2e32a7965d5

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/35421
Committed: http://github.com/Juniper/contrail-ansible/commit/beb5da163a76f5b9d6ac6769af90f98ddbaf9005
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit beb5da163a76f5b9d6ac6769af90f98ddbaf9005
Author: Ramprakash Ram Mohan <email address hidden>
Date: Fri Sep 8 16:54:25 2017 -0700

Add support for keystone v3 in ocata

1. group_vars/all.yml - Add default values for enable_keystone_v3,
keystone_admin_project_name and keystone_admin_project_domain_name
2. horizon/templates/local_settings.j2 - If enable_keystone_v3 is true, then
enable the OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT and
OPENSTACK_KEYSTONE_DEFAULT_DOMAIN variables
3. keystone/tasks/config.yml - Upstream bug - typo in the notify tag
4. keystone/tasks/register.yml - Assign admin user to default domain for v3
5. keystone/template/keystone.conf.j2 - Fix for Bug #1710739 - setting admin
project details in keystone.conf
6. neutron_opencontrail.conf.j2 - No need to use keystone_admin_user variable,
use the existing openstack_auth dictionary instead
7. globals.yml.original - update sample kolla_globals with the newly introduced
variables
8. roles/openstack/compute/tasks/configure.yml - transport_url was not being
populated with all the rabbitmq hosts in the case of oenstack ha

Closes-bug: #1710739
Closes-bug: #1715427

Change-Id: I13f1b8a5f615496f3fbfb30ce611a2e32a7965d5

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/35416
Committed: http://github.com/Juniper/contrail-server-manager/commit/54fbde6d9dd13bb19ba37dcfe1079b45fd263202
Submitter: Zuul (<email address hidden>)
Branch: master

commit 54fbde6d9dd13bb19ba37dcfe1079b45fd263202
Author: Ramprakash Ram Mohan <email address hidden>
Date: Fri Sep 8 16:51:49 2017 -0700

Derive value for enable_keystone_v3 based on keystone version

Change-Id: Ia61af991ca18e364903925d698e6c93498a8ac0c
Partial-bug: #1715427