Domain-specific domain ID resolution breaks with system-scoped tokens

[Lance Bragstad]

System-scope was introduced in Queens [0] but recently we discovered a weird case where system users aren't able to do things they should be able to with system-scoped tokens when domain-specific drivers are enabled.

For example, they are unable to list groups or users because the API logic for GET /v3/groups and GET /v3/users tries to resolve a domain ID from the request [1]. If domain-specific drivers are enabled and there isn't a domain ID associated to the request (either with a domain-scoped token or a project-scoped token) the API returns a 401, which makes no sense from the context of a system user [2].

You can recreate this locally by enabling domain-specific drivers in keystone.conf [3] and running the test_groups or test_users v3 protection tests using:

  $ tox -e py37 -- keystone.tests.unit.protection.v3.test_groups

Observed failures: https://pasted.tech/pastes/b45c6b015b97c865018c4b3290f60e0456fe304a.raw

This isn't blocking the gate because domain-specific drivers are off by default and the logic short-circuits [4].

[0] http://specs.openstack.org/openstack/keystone-specs/specs/keystone/queens/system-scope.html
[1] https://opendev.org/openstack/keystone/src/branch/master/keystone/api/groups.py#L84
[2] https://opendev.org/openstack/keystone/src/branch/master/keystone/server/flask/common.py#L917-L943
[3] https://pasted.tech/pastes/e8ffce7a3377b960dd88de8c88e2ccfd173ec726.raw
[4] https://opendev.org/openstack/keystone/src/branch/master/keystone/server/flask/common.py#L924-L926

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/681833

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/681841

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/681849

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/681907

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/681833
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=8f43b9cab00c86a455b2a9700b434e98b2e9c2d8
Submitter: Zuul
Branch: master

commit 8f43b9cab00c86a455b2a9700b434e98b2e9c2d8
Author: Lance Bragstad <email address hidden>
Date: Thu Sep 12 16:46:26 2019 +0000

    Make system tokens work with domain-specific drivers

    When calling certain group or user APIs, keystone logic would attempt
    to figure out the domain to scope responses to. This was specific to
    enabling domain-specific driver support, where each domain is backed
    by a different identity store. This functionality is turned off by
    default. Since system-scoped tokens are not associated to a domain
    (unlike project-scoped tokens or domain-scoped tokens), the logic to
    determine a domain from a system-scoped token was breaking and
    returning an erroneous HTTP 401 Unauthorized when system users
    attempted to list users or groups.

    This commit adds support for domain detection with system-scoped
    tokens.

    Change-Id: I8f0f7a623a1741f461493d872849fae7ef3e8077
    Closes-Bug: 1843609

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystone 16.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/681841
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=e9612a672b88867e8df82d443a5ca08b1343a411
Submitter: Zuul
Branch: stable/stein

commit e9612a672b88867e8df82d443a5ca08b1343a411
Author: Lance Bragstad <email address hidden>
Date: Thu Sep 12 16:46:26 2019 +0000

    Make system tokens work with domain-specific drivers

    When calling certain group or user APIs, keystone logic would attempt
    to figure out the domain to scope responses to. This was specific to
    enabling domain-specific driver support, where each domain is backed
    by a different identity store. This functionality is turned off by
    default. Since system-scoped tokens are not associated to a domain
    (unlike project-scoped tokens or domain-scoped tokens), the logic to
    determine a domain from a system-scoped token was breaking and
    returning an erroneous HTTP 401 Unauthorized when system users
    attempted to list users or groups.

    This commit adds support for domain detection with system-scoped
    tokens.

    Change-Id: I8f0f7a623a1741f461493d872849fae7ef3e8077
    Closes-Bug: 1843609
    (cherry picked from commit 8f43b9cab00c86a455b2a9700b434e98b2e9c2d8)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/681849
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=417d2c0e6e6bef39f447681325ae5b0ba46b2e2c
Submitter: Zuul
Branch: stable/rocky

commit 417d2c0e6e6bef39f447681325ae5b0ba46b2e2c
Author: Lance Bragstad <email address hidden>
Date: Thu Sep 12 16:46:26 2019 +0000

    Make system tokens work with domain-specific drivers

    When calling certain group or user APIs, keystone logic would attempt
    to figure out the domain to scope responses to. This was specific to
    enabling domain-specific driver support, where each domain is backed
    by a different identity store. This functionality is turned off by
    default. Since system-scoped tokens are not associated to a domain
    (unlike project-scoped tokens or domain-scoped tokens), the logic to
    determine a domain from a system-scoped token was breaking and
    returning an erroneous HTTP 401 Unauthorized when system users
    attempted to list users or groups.

    This commit adds support for domain detection with system-scoped
    tokens.

    Conflicts:
          keystone/server/flask/common.py

    This backport has conflicts with keystone/server/flask/common.py due to
    a massive refactor to get keystone off python-paste that started in
    Rocky and spilled over into Stein. The change is functionally equivalent
    to the patch merged to Train but done in keystone/common/controller.py
    instead of keystone/server/flask/common.py. There was also some changes
    to the test since it didn't have a utility method to obtain a
    system-scoped token.

    Change-Id: I8f0f7a623a1741f461493d872849fae7ef3e8077
    Closes-Bug: 1843609
    (cherry picked from commit 8f43b9cab00c86a455b2a9700b434e98b2e9c2d8)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/681907
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=65cb669e78521ad9012cba16f3071d741b8672c5
Submitter: Zuul
Branch: stable/queens

commit 65cb669e78521ad9012cba16f3071d741b8672c5
Author: Lance Bragstad <email address hidden>
Date: Thu Sep 12 16:46:26 2019 +0000

    Make system tokens work with domain-specific drivers

    When calling certain group or user APIs, keystone logic would attempt
    to figure out the domain to scope responses to. This was specific to
    enabling domain-specific driver support, where each domain is backed
    by a different identity store. This functionality is turned off by
    default. Since system-scoped tokens are not associated to a domain
    (unlike project-scoped tokens or domain-scoped tokens), the logic to
    determine a domain from a system-scoped token was breaking and
    returning an erroneous HTTP 401 Unauthorized when system users
    attempted to list users or groups.

    This commit adds support for domain detection with system-scoped
    tokens.

    Conflicts:
          keystone/server/flask/common.py

    This backport has conflicts with keystone/server/flask/common.py due to
    the ``token_ref`` variable being renamed to ``token``. This conflict is
    resolved by continuing to use the old name, but the change is
    functionally equivalent to what was proposed to all other branches.

    This backport modifies the unit test to use the pre-flask-compatible
    self.admin_request method instead of flask's test_client() context
    manager.

    Change-Id: I8f0f7a623a1741f461493d872849fae7ef3e8077
    Closes-Bug: 1843609
    (cherry picked from commit 8f43b9cab00c86a455b2a9700b434e98b2e9c2d8)
    (cherry picked from commit 417d2c0e6e6bef39f447681325ae5b0ba46b2e2c)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystone 13.0.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystone 14.2.0 release.