[SRU] Horizon Identity Domain Panel is broken in Caracal+

[Pavlo Shchelokovskyy]

[Impact]

Since Caracal, when using domain-scoped token, keystone only returns the domain the token is scoped to when listing domains.

Since Horizon does some behind-the-scenes swap of token scope when doing some requests to Keystone, this breaks the Identity->Domains panel for admins.

The fix forces the domain_list call to always use the original auth scope, w/o a swap to the domain-scoped token.

[Test Case]

This issue can be easily reproduced by following these steps:

1, Set up an openstack Caracal env with horizon
2, Log in to horizon dashboard as an admin user
3, Navigate to the 'Identity->Domains' panel

You will notice that only the admin user is listed - https://imgur.com/a/e6dbh2v
other domains can be listed after applying the fix - https://imgur.com/a/vLd6Mcr

[Where problems could occur]

We are changing the domain_list call to always use the original auth scope, so any regression issues will be limited to 'Identity->Domains' panel only.

[Other info]

This issue started with Caracal release, and this was fixed upstream by:

$ git tag --contains 964623e16baaf8d2902e6000b2cec62bea14d15d
25.2.0
25.3.0
25.4.0
$ git branch -r --contains 23d0b9525f7c11288d503123e29db0bd66f9ca88
  origin/stable/2024.2
$ git tag --contains 23d0b9525f7c11288d503123e29db0bd66f9ca88
<empty>
$ git tag --contains b06ce1c2a1baa6bd53e70f407cd2194aadcf169e
24.0.1

For UA, the fix is already in Questing(ubuntu 25.10, 4:25.3.0-0ubuntu1), Plucky(ubuntu 25.04, 4:25.3.0-0ubuntu1), backporting is still required for: oracular(ubuntu 24.10, 4:25.1.0-0ubuntu1.1) and noble(ubuntu 24.04, 4:24.0.0-0ubuntu1.3)

For UCA, the fix is already in flamingo(2025.2, 4:25.3.0-0ubuntu1~cloud0), eproxy(2025.1, 4:25.3.0-0ubuntu1~cloud0), backporting is still required for: dalmatian(2024.2, 4:25.1.0-0ubuntu1.1~cloud0) and caracal(2024.1 4:24.0.0-0ubuntu1.3~cloud0), but no debdiff needed due to inclusion in UA.

== ORIGINAL DESCRIPTION ==

Starting with Caracal release, Identity Domains Panel is broken, as it only ever lists that domain that the user belongs to.

Devstack/Master, logged as admin (devstack-admin creds in /etc/openstack/clouds.yaml).

With default Horizon settings, I only ever see Default domain, even if I manually create some more. And I do not have an option to create domains from UI as well. This is because AFAIU the ability to create domains is tied to OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT (False by default), which is waaay legacy IMO. This option is quite overloaded in Horizon code, but that's a different question.

When I enable the OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT in my local_settings.py, I can create domains from UI, but I still can not see any other domain other than the domain of the user.

I tracked it to this piece of code that replaces the scope to the domain one for admins
https://opendev.org/openstack/horizon/src/branch/stable/2024.1/openstack_dashboard/api/keystone.py#L153-L163 ,
plus a recent change in Keystone https://review.opendev.org/c/openstack/keystone/+/900028 that started forcing domain tokens to only be able to list their own domains.

[Charles Smith (drabbit679)]

Did you figure out a workaround for this issue other than just falling back to the CLI for cross-domain user administrative activities? It's looking like that keystone change you posted got cherry backed into bobcat and antelope, so now all current supported releases are bugged :( . Debating playing around with reverting the keystone diff but I have no idea what else in the system might get hosed by that.

[Charles Smith (drabbit679)]

Can confirm, reversing the changes to domains.py fixes the gui. No clue what other chaos might come as a result of doing so, but guess I'm rolling with this hack for now, lol

[Markus Hentsch (mhen)]

I am not familiar with the implementation and goal of Horizon regarding this matter (need to look into that) but I'll gladly share some context around the Keystone changes referenced above.

The change causing this behavior was introduced by https://bugs.launchpad.net/keystone/+bug/2041611 to prevent the exposure of other tenants'/customers' domains for policy models which allow specific non-admin roles the listing of domains in Keystone.

Ultimately, this was preliminary work for the domain manager persona as later introduced by https://bugs.launchpad.net/keystone/+bug/2045974 which extends the "manager" role in Keystone with functionalities centered around identity self-service for customers using the new non-admin "manager" role.
It allows editing user/project/group/role relationships within a single domain without the need of an admin. For that it was important that all endpoints in Keystone properly scope on domain level as well for tenant separation, including the domains endpoint itself.

Again, I'm unfamiliar with the related Horizon implementation and can only guess based on what was described above but I think Horizon might be using a token with the wrong scope or role when making the Keystone request to retrieve the domain list. Based on the current defaults, a token with admin role or system-level reader should do.

Reverting the change in domains.py should only affect domain manager users currently as they will have domain names and IDs of foreign domains exposed to them.

[Gilles Mocellin (gilles-mocellin)]

Hi !

Perhaps the system scope should replace domain scope in Horizon, to allow domains administration.

But I tried, activating system scope in Horizon like showed in this bug :
https://bugs.launchpad.net/horizon/+bug/1971592

But no, I'm always seeing my own domain only...

For now, to allow our internal operators (authenticated with AD, in its own domain) to manage external user accounts in an internal SQL domain, only patching keystone domains.py works.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/horizon/+/920395
Committed: https://opendev.org/openstack/horizon/commit/964623e16baaf8d2902e6000b2cec62bea14d15d
Submitter: "Zuul (22348)"
Branch: master

commit 964623e16baaf8d2902e6000b2cec62bea14d15d
Author: Pavlo Shchelokovskyy <email address hidden>
Date: Fri May 24 13:57:17 2024 +0000

    Force scope when listing domains

    since Caracal, when using domain-scoped token, keystone only returns
    the domain the token is scoped to when listing domains [0].

    Since Horizon does some behind-the-scenes swap of token scope when
    doing some requests to Keystone, this breaks the Identity->Domains panel
    for admins.

    This patch forces the domain_list call to always use the original
    auth scope, w/o a swap to the domain-scoped token.

    [0] https://review.opendev.org/c/openstack/keystone/+/900028

    Closes-Bug: #2067075
    Change-Id: I4ff5f2de01c0bb13cfbb5136f40afe8187135686

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2024.2
Review: https://review.opendev.org/c/openstack/horizon/+/940460

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/horizon/+/940461

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/horizon/+/940460
Committed: https://opendev.org/openstack/horizon/commit/23d0b9525f7c11288d503123e29db0bd66f9ca88
Submitter: "Zuul (22348)"
Branch: stable/2024.2

commit 23d0b9525f7c11288d503123e29db0bd66f9ca88
Author: Pavlo Shchelokovskyy <email address hidden>
Date: Fri May 24 13:57:17 2024 +0000

    Force scope when listing domains

    since Caracal, when using domain-scoped token, keystone only returns
    the domain the token is scoped to when listing domains [0].

    Since Horizon does some behind-the-scenes swap of token scope when
    doing some requests to Keystone, this breaks the Identity->Domains panel
    for admins.

    This patch forces the domain_list call to always use the original
    auth scope, w/o a swap to the domain-scoped token.

    [0] https://review.opendev.org/c/openstack/keystone/+/900028

    Closes-Bug: #2067075
    Change-Id: I4ff5f2de01c0bb13cfbb5136f40afe8187135686
    (cherry picked from commit 964623e16baaf8d2902e6000b2cec62bea14d15d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/horizon/+/940461
Committed: https://opendev.org/openstack/horizon/commit/b06ce1c2a1baa6bd53e70f407cd2194aadcf169e
Submitter: "Zuul (22348)"
Branch: stable/2024.1

commit b06ce1c2a1baa6bd53e70f407cd2194aadcf169e
Author: Pavlo Shchelokovskyy <email address hidden>
Date: Fri May 24 13:57:17 2024 +0000

    Force scope when listing domains

    since Caracal, when using domain-scoped token, keystone only returns
    the domain the token is scoped to when listing domains [0].

    Since Horizon does some behind-the-scenes swap of token scope when
    doing some requests to Keystone, this breaks the Identity->Domains panel
    for admins.

    This patch forces the domain_list call to always use the original
    auth scope, w/o a swap to the domain-scoped token.

    [0] https://review.opendev.org/c/openstack/keystone/+/900028

    Closes-Bug: #2067075
    Change-Id: I4ff5f2de01c0bb13cfbb5136f40afe8187135686
    (cherry picked from commit 964623e16baaf8d2902e6000b2cec62bea14d15d)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/horizon 25.2.0 Epoxy release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/horizon 24.0.1 Caracal release.

[Guillaume Boutry (gboutry)]

https://bugs.launchpad.net/cloud-archive/+bug/2110279 will contain the noble patch, as it is a point release update to horizon. Going from 24.0.0 to 24.0.1.

Content of that release: https://review.opendev.org/c/openstack/releases/+/949317

$ git log --oneline --no-merges 24.0.0..24.0.1
8687f2053 Fix typo in metadef for Cinder Volume Type
b06ce1c2a Force scope when listing domains
013de42a6 Move deprecated since/reason to deprecated_rule object
9a3effbb9 Fix generation of inventory capacities
680e0729e Imported Translations from Zanata
a761ec8d3 Delete container show duplicate toast notifications
04a5b33af Fix Placement statistics display
88c087e10 Sort image source choices by name for volume
b5b592ead Fix floating IP associated to unbound port
5a7a49923 Fix allowed address pair row unique ID
d4ec2786c Show Created At column for backups table
72ea53cdd Show availability zone for volume backups
5e6b36f73 Imported Translations from Zanata
dcaf0cc51 Respect SSL settings in placement API
4936fec3a Fix Users/Groups tab list when a domain context is set
597b37c62 Imported Translations from Zanata
e415e8dad Update TOX_CONSTRAINTS_FILE for stable/2024.1
5968e7d8e Update .gitreview for stable/2024.1