Deploy will fail if keystone.conf has '[oslo_policy]/enforce_scope=true'

[Niklas Hagman]

In current Kolla master (train) keystone permission system has not been adapted to the new scope thinking.

$ cat /etc/kolla/config/keystone/keystone.conf
[oslo_policy]
enforce_scope = True

$ kolla-ansible -i multinode deploy
...
TASK [service-ks-register : keystone | Creating services] ************************************************************************************
...
failed: [control1.example.com -> control1.example.com] (item={u'service_type': u'identity', u'name': u'keystone'}) => {"action": "os_keystone_service", "ansible_loop_var": "item", "attempts": 5, "changed": false, "item": {"description": "Openstack Identity Service", "endpoints": [{"interface": "admin", "url": "http://vip.example.com:35357"}, {"interface": "internal", "url": "http://vip.example.com:5000"}, {"interface": "public", "url": "https://openstack.example.com:5000"}], "name": "keystone", "type": "identity"}, "msg": "Failed to list services: Client Error for url: http://vip.example.com:35357/v3/services, You are not authorized to perform the requested action: identity:list_services."}

== https://docs.openstack.org/releasenotes/keystone/en_GB/train.html ==
This release leverages oslo.policy’s policy-in-code feature to modify the default check strings and scope types for nearly all of keystone’s API policies. These changes make the policies more precise than they were before, using the reader, member, and admin roles where previously only the admin role and a catch-all rule was available. The changes also take advantage of system, domain, and project scope, allowing you to create role assignments for your users that are appropriate to the actions they need to perform. Eventually this will allow you to set [oslo_policy]/enforce_scope=true in your keystone configuration, which simplifies access control management by ensuring that oslo.policy checks both the role and the scope on API requests.

[bug 1806762] [bug 1630434] The entire policy.v3cloudsample.json file has been removed. If you were using this policy file to supply overrides in your deployment, you should consider using the defaults in code and setting keystone.conf [oslo_policy] enforce_scope=True. The new policy defaults are more flexible, they’re tested extensively, and they solve all the problems the policy.v3cloudsample.json file was trying to solve.

[Niklas Hagman (blinkiz)]

Some more information if the URL changes in the future.

== https://docs.openstack.org/releasenotes/keystone/en_GB/train.html ==
[bug 1806762] The user policies in policy.v3cloudsample.json policy file have been removed in favor of better defaults in code. These policies weren’t tested exhaustively and were misleading to users and operators.

[bug 1806762] The grant policies in policy.v3cloudsample.json policy file have been removed in favor of better defaults in code. These policies weren’t tested exhaustively and were misleading to users and operators.

[bug 1804462] The group policies in policy.v3cloudsample.json policy file have been removed in favor of better defaults in code. These policies weren’t tested exhaustively and were misleading to users and operators.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/692179

[Radosław Piliszek (yoctozepto)]

The proposed changes tests it in CI.

[Radosław Piliszek (yoctozepto)]

Reproduced.

[Radosław Piliszek (yoctozepto)]

So trying to get auth not scoped to a project but domain instead, I get this:
failed: [primary] (item={u'service_type': u'identity', u'name': u'keystone'}) => {
    "action": "os_keystone_service",
    "attempts": 5,
    "changed": false,
    "invocation": {
        "module_args": {
            "api_version": "auto",
            "module_args": {
                "auth": {
                    "auth_url": "http://192.0.2.10:35357",
                    "domain_name": "default",
                    "password": "9PJVm6kJI1k00JgNzhXpRAosMAXBkIqSSmDYDwR3",
                    "user_domain_name": "default",
                    "username": "admin"
                },
                "cacert": "",
                "description": "Openstack Identity Service",
                "interface": "admin",
                "name": "keystone",
                "region_name": "RegionOne",
                "service_type": "identity"
            },
            "module_extra_vars": null,
            "module_name": "os_keystone_service",
            "timeout": 180,
            "user": null
        }
    },
    "item": {
        "description": "Openstack Identity Service",
        "endpoints": [
            {
                "interface": "admin",
                "url": "http://192.0.2.10:35357"
            },
            {
                "interface": "internal",
                "url": "http://192.0.2.10:5000"
            },
            {
                "interface": "public",
                "url": "http://192.0.2.10:5000"
            }
        ],
        "name": "keystone",
        "type": "identity"
    },
    "module_stderr": "Traceback (most recent call last):\n File \"/tmp/ansible-tmp-1572531912.06-54869509402289/AnsiballZ_os_keystone_service.py\", line 114, in <module>\n _ansiballz_main()\n File \"/tmp/ansible-tmp-1572531912.06-54869509402289/AnsiballZ_os_keystone_service.py\", line 106, in _ansiballz_main\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n File \"/tmp/ansible-tmp-1572531912.06-54869509402289/AnsiballZ_os_keystone_service.py\", line 49, in invoke_module\n imp.load_module('__main__', mod, module, MOD_DESC)\n File \"/tmp/ansible_os_keystone_service_payload_RpqMjI/__main__.py\", line 194, in <module>\n File \"/tmp/ansible_os_keystone_service_payload_RpqMjI/__main__.py\", line 153, in main\n File \"/opt/ansible/lib/python2.7/site-packages/openstack/cloud/_identity.py\", line 510, in search_services\n services = self.list_services()\n File \"/opt/ansible/lib/python2.7/site-packages/openstack/cloud/_identity.py\", line 485, in list_services\n if self._is_client_version('identity', 2):\n File \"/opt/ansible/lib/python2.7/site-packages/openstack/cloud/openstackcloud.py\", line 459, in _is_client_version\n client = getattr(self, client_name)\n File \"/opt/ansible/lib/python2.7/site-packages/openstack/cloud/_identity.py\", line 32, in _identity_client\n 'identity', min_version=2, max_version='3.latest')\n File \"/opt/ansible/lib/python2.7/site-packages/openstack/cloud/openstackcloud.py\", line 422, in _get_versioned_client\n endpoint_override=self.config.get_endpoint(service_type))\n File \"/o...

[Radosław Piliszek (yoctozepto)]

Hey keystoners, if you could take a look at the situation we are in. I have no idea how to bend openstacksdk (as used by Ansible) to let me create service endpoints. It either ends up having Client Error with admin-project-scoped auth or 401 with supposedly defalt-domain-scoped auth.

[Radosław Piliszek (yoctozepto)]

service logs:
2019-10-31 14:25:13.159673 2019-10-31 14:25:13.158 22 DEBUG keystone.server.flask.request_processing.req_logging [req-610327c9-f08d-46b8-b9b7-6f6d18120f41 - - - - -] REQUEST_METHOD: `GET` log_request_info /var/lib/kolla/venv/lib/python2.7/site-packages/keystone/server/flask/request_processing/req_logging.py:27\x1b[00m
2019-10-31 14:25:13.160259 2019-10-31 14:25:13.159 22 DEBUG keystone.server.flask.request_processing.req_logging [req-610327c9-f08d-46b8-b9b7-6f6d18120f41 - - - - -] SCRIPT_NAME: `` log_request_info /var/lib/kolla/venv/lib/python2.7/site-packages/keystone/server/flask/request_processing/req_logging.py:28\x1b[00m
2019-10-31 14:25:13.160798 2019-10-31 14:25:13.160 22 DEBUG keystone.server.flask.request_processing.req_logging [req-610327c9-f08d-46b8-b9b7-6f6d18120f41 - - - - -] PATH_INFO: `/` log_request_info /var/lib/kolla/venv/lib/python2.7/site-packages/keystone/server/flask/request_processing/req_logging.py:29\x1b[00m
2019-10-31 14:25:13.168445 2019-10-31 14:25:13.167 22 DEBUG keystone.server.flask.request_processing.req_logging [req-2eed28d1-00fd-4878-8acc-9d5eee838a93 - - - - -] REQUEST_METHOD: `POST` log_request_info /var/lib/kolla/venv/lib/python2.7/site-packages/keystone/server/flask/request_processing/req_logging.py:27\x1b[00m
2019-10-31 14:25:13.169235 2019-10-31 14:25:13.168 22 DEBUG keystone.server.flask.request_processing.req_logging [req-2eed28d1-00fd-4878-8acc-9d5eee838a93 - - - - -] SCRIPT_NAME: `` log_request_info /var/lib/kolla/venv/lib/python2.7/site-packages/keystone/server/flask/request_processing/req_logging.py:28\x1b[00m
2019-10-31 14:25:13.170971 2019-10-31 14:25:13.169 22 DEBUG keystone.server.flask.request_processing.req_logging [req-2eed28d1-00fd-4878-8acc-9d5eee838a93 - - - - -] PATH_INFO: `/v3/auth/tokens` log_request_info /var/lib/kolla/venv/lib/python2.7/site-packages/keystone/server/flask/request_processing/req_logging.py:29\x1b[00m
2019-10-31 14:25:13.495747 2019-10-31 14:25:13.494 22 DEBUG keystone.auth.core [req-2eed28d1-00fd-4878-8acc-9d5eee838a93 - - - - -] MFA Rules not processed for user `e292abd123704368832b9383f9c8b31d`. Rule list: `[]` (Enabled: `True`). check_auth_methods_against_rules /var/lib/kolla/venv/lib/python2.7/site-packages/keystone/auth/core.py:446\x1b[00m
2019-10-31 14:25:13.498527 2019-10-31 14:25:13.498 22 DEBUG keystone.common.fernet_utils [req-2eed28d1-00fd-4878-8acc-9d5eee838a93 - - - - -] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? load_keys /var/lib/kolla/venv/lib/python2.7/site-packages/keystone/common/fernet_utils.py:289\x1b[00m
2019-10-31 14:25:13.501483 2019-10-31 14:25:13.500 22 DEBUG keystone.models.token_model [req-2eed28d1-00fd-4878-8acc-9d5eee838a93 - - - - -] User e292abd123704368832b9383f9c8b31d has no access to domain default _validate_domain_scope /var/lib/kolla/venv/lib/python2.7/site-packages/keystone/models/token_model.py:511\x1b[00m
2019-10-31 14:25:13.506320 2019-10-31 14:25:13.505 22 WARNING keystone.server.flask.application [req-2eed28d1-00fd-4878-8acc-9d5eee838a93 - - - - -] Authorization failed. The requ...

[Colleen Murphy (krinkle)]

> It either ends up having Client Error with admin-project-scoped auth or 401 with supposedly defalt-domain-scoped auth.

The scope_type for identity:create_endpoint is "system": https://docs.openstack.org/keystone/latest/configuration/policy.html

So neither a project- nor domain-scoped token will be usable with this policy if enforce_scope is set to true. You either need a system-scoped token or you need to set enforce_scope=false (which is the default).

[Radosław Piliszek (yoctozepto)]

Thanks, Colleen. Indeed I felt like I was missing something obvious there. Will look in that direction then. Though I am not sure whether I saw system-scope toggle there...

[Niklas Hagman (blinkiz)]

Settings:
[oslo_policy]
enforce_scope = True

GET /v3/services
identity:list_services

"openstack_auth": {
  "auth_url": "http://192.0.2.10:35357",
  "username": "admin",
  "password": "9PJVm6kJI1k00JgNzhXpRAosMAXBkIqSSmDYDwR3"
}

Error:
keystoneauth1.exceptions.http.BadRequest: Expecting to find domain in user. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-ea92e74e-2cf1-4524-9c78-a02f24a180a1)

Goal:
Getting auth into scope 'system'. I can not figure out how to do this.

-------------------------

Traceback (most recent call last):
  File "/tmp/ansible-tmp-1573484557.4130447-82566091548974/AnsiballZ_os_keystone_service.py", line 114, in <module>
    _ansiballz_main()
  File "/tmp/ansible-tmp-1573484557.4130447-82566091548974/AnsiballZ_os_keystone_service.py", line 106, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/tmp/ansible-tmp-1573484557.4130447-82566091548974/AnsiballZ_os_keystone_service.py", line 49, in invoke_module
    imp.load_module('__main__', mod, module, MOD_DESC)
  File "/opt/ansible/lib/python3.6/imp.py", line 235, in load_module
    return load_source(name, filename, file)
  File "/opt/ansible/lib/python3.6/imp.py", line 170, in load_source
    module = _exec(spec, sys.modules[name])
  File "<frozen importlib._bootstrap>", line 618, in _exec
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmp/ansible_os_keystone_service_payload_4370z2m9/__main__.py", line 194, in <module>
  File "/tmp/ansible_os_keystone_service_payload_4370z2m9/__main__.py", line 153, in main
  File "/opt/ansible/lib/python3.6/site-packages/openstack/cloud/_identity.py", line 510, in search_services
    services = self.list_services()
  File "/opt/ansible/lib/python3.6/site-packages/openstack/cloud/_identity.py", line 485, in list_services
    if self._is_client_version('identity', 2):
  File "/opt/ansible/lib/python3.6/site-packages/openstack/cloud/openstackcloud.py", line 459, in _is_client_version
    client = getattr(self, client_name)
  File "/opt/ansible/lib/python3.6/site-packages/openstack/cloud/_identity.py", line 32, in _identity_client
    'identity', min_version=2, max_version='3.latest')
  File "/opt/ansible/lib/python3.6/site-packages/openstack/cloud/openstackcloud.py", line 422, in _get_versioned_client
    endpoint_override=self.config.get_endpoint(service_type))
  File "/opt/ansible/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 345, in get_api_major_version
    return self.session.get_api_major_version(auth or self.auth, **kwargs)
  File "/opt/ansible/lib/python3.6/site-packages/keystoneauth1/session.py", line 1233, in get_api_major_version
    return auth.get_api_major_version(self, **kwargs)
  File "/opt/ansible/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 500, in get_api_major_version
    data = get_endpoint_data(discover_versions=discover_versions)
  File "/opt/ansible/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 2...

[Radosław Piliszek (yoctozepto)]

Yeah, had the same issue in that review I linked to above.

We had a chat with openstacksdkers and it is probably not doable from ansible side atm.

If you could raise the issue with ansible module, it would help resolving this (actually, that's the only path I can think of).

[Niklas Hagman (blinkiz)]

A bug ticket has been open against openstacksdk. https://storyboard.openstack.org/#!/story/2006839

openstacksdk: https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/cloud/openstackcloud.py
keystoneauth1: https://opendev.org/openstack/keystoneauth/src/branch/master/keystoneauth1/identity/v3/base.py
python project openstacksdk is using python project keystoneauth1 for authentication.

Openstacksdk is using keystoneauth1. keystoneauth1 can handle the argument 'system_scope' with value 'all'. Problem is that this is lost somehow along the path to keystoneauth1 from kolla-ansible and it's 'openstack_auth' setting. Maybe kolla-ansible is calling the wrong function/class in openstacksdk. Maybe openstacksdk is calling wrong function/class in keystoneauth1.

[Niklas Hagman (blinkiz)]

Hi. I do think I have solved this problem. Can you please create a new patch set with openstack_auth as below? Tell Zuul to run it.

Currently I do think it is required to write "Default" and not "default".
It is important to not specify domain_id, domain_name, project_id, project_name, trust_id or unscoped.

In ansible/group_vars/all.yml edit:
openstack_auth:
  auth_url: "{{ keystone_admin_url }}"
  username: "{{ keystone_admin_user }}"
  password: "{{ keystone_admin_password }}"
  user_domain_name: "Default"
  system_scope: "all"

[Radosław Piliszek (yoctozepto)]

Done. I had "system-scope". I will rotfl if it passes because it was one character off. ;D

[Radosław Piliszek (yoctozepto)]

Passes. Rotfl. ;D

Still, we have various dependencies on this variable so it needs a little bit of more work but at least we have a fix for the very issue. Thanks for investigating. I was literally one character from the fix.

[Niklas Hagman (blinkiz)]

Next fix is to give nova-cell system scope permission. openstackcli seems to put into scope system if no domain, projects (and so on) is not specified.

share/kolla-ansible/ansible/roles/nova-cell/tasks/discover_computes.yml.
Remove:
--os-project-domain-name
--os-tenant-name
Change:
--os-user-domain-name {{ openstack_auth.user_domain_name }}

Later in the deploy play I have problem with missing config file "/var/lib/kolla/config_files/config.json", TASK [neutron : Running Neutron bootstrap container] found in file share/kolla-ansible/ansible/roles/nova/tasks/bootstrap_service.yml. Currently I do not know if it is related to the change above. Still troubleshooting.

[Niklas Hagman (blinkiz)]

I have pushed successful build into https://review.opendev.org/692179
Please review this patch.

[Mark Goddard (mgoddard)]

Given the current size of the patch, we'll drop this from 9.0.0.

[Florian Faltermeier (florianfa)]

Hello,

this affect kolla-ansible/wallaby, too.

[Helder Marques (hlourencomarques)]

Hi,
I'm having the same issue deploying the Xena release.
I'm using kolla-ansible 13.0.1.
Thank you!