On further debugging, it appears that there is an issue in horizon (caused due to a change in keystone) with the use of a scoped tokens for the admin user when multi-domain is enabled. The scenario is as follows: 1. Multi domain is enabled. 2. The admin user is logged in with credentials using an admin domain. 3. Domain context is set to a domain in which the admin user is not a member. 4. The admin user attempts to list the projects or users. 5. A domain scoped token is used by horizon to list the projects, due to the code in [1] 6. No users are returned from keystone because, due to change [2] the users are filtered as the token contains the admin domain, not the target domain or users to list. It's quite involved! I'm not sure if the issue is: 1. Keystone shouldn't be filtering this list. 2. Horizon shouldn't be using a domain scoped token for the admin user (e.g. the openstack CLI doesn't use a domain scoped token to list the users in the domain, or an admin user). 3. Something else. Horizon appears to only start using the domain scoped token after the domain context is set. Also, it only appears (in my testing) to use it for the user list and (maybe) project list -- I focussed on the user list. It looks like a new token is requested to perform the user list and that this one is domain scoped. I can do further testing as necessary. References: [1] Horizon, openstack_dashboard/api/keystone.py (def keystoneclient:) https://github.com/openstack/horizon/blob/stable/stein/openstack_dashboard/api/keystone.py#L167 [2] Keystone, change Id: I60b2e2b8af172c369eab0eb2c29f056f5c98ad16, https://review.opendev.org/#/c/647587/ (for user list) Debugging info: I added some debug LOG lines to the various bits of horizon and keystone to try to work out what was going on. The following is a comparison between Horizon and the OpenStack CLI in listing users for a domain "test-domain": Preamble: The test set up: The test is listing users for the "test-domain" on the OpenStack CLI and using the Horizon dashboard. Domain list: +----------------------------------+----------------+---------+-----------------+ | ID | Name | Enabled | Description | +----------------------------------+----------------+---------+-----------------+ | 4c97d83fd8f34507aa5849710218272e | default | True | Created by Juju | | 917f251e6fc24c389f1e3f3624d701d1 | admin_domain | True | Created by Juju | | be5450b76a2348c48df0d0571295de40 | test-domain2 | True | | | c9ca71bd88894017a6b6448dfcffeb68 | test-domain | True | | | ecb1e99a62534253a5b515dcfc218733 | service_domain | True | Created by Juju | +----------------------------------+----------------+---------+-----------------+ The "admin" user is in the admin_domain. Project List: +----------------------------------+---------------+ | ID | Name | +----------------------------------+---------------+ | 1014c1815147453b8bd77de578467a80 | demo | | 49ae284fd4aa42208573d9c399a95eee | services | | 7581c43d252848dface4c75e2b921224 | test-project | | 75c183f2aece43e2860be59926e244fb | admin | | 9bc98ed16a7547e0b11d002172ab1d6e | test-project2 | | 9c619796ef91470bba2d30427bd7adc6 | admin | | a7c8c2f4d11844619fb22753ab4d7a80 | services | | b8eb986468684e7ab4c7eb92542d3e58 | alt_demo | +----------------------------------+---------------+ The "admin" users is in the "admin" project. openstack user list +----------------------------------+----------+ | ID | Name | +----------------------------------+----------+ | 8973385dd5ca467fb4be7a3eca7a603f | admin | | 8aeaead88fdc49c6a44a3983d3ff8c63 | demo | | b7beaf7d43b144d5b71acb33f0abb87d | alt_demo | +----------------------------------+----------+ +----------------------------------+------------------+ | ID | Name | +----------------------------------+------------------+ | 9c1fa58637a64cd387922a4b2b8ce522 | test-domain-user | +----------------------------------+------------------+ --- OpenStack CLI debug for "openstack user list --domain=test-domain" OS_VARS: OS_AUTH_URL=http://10.5.0.56:5000/v3 OS_DOMAIN_NAME=admin_domain OS_REGION_NAME=RegionOne OS_PROJECT_NAME=admin OS_PROJECT_DOMAIN_NAME=admin_domain OS_USER_DOMAIN_NAME=admin_domain OS_AUTH_VERSION=3 OS_IDENTITY_API_VERSION=3 OS_PASSWORD=openstack OS_USERNAME=admin Token: (keystone.token.provider): 2019-07-17 18:03:04,001 DEBUG ZZZZ - the token: gAAAAABdL2LX_HT3mi4RO0KcwuqYaJ-NoY-gDMQtcKm-QDDJ0o-SsiH1BOaI5LlhbPLVyiKw7amvGcuwwuM9LLCCBb0VGkyIs2cmkTlHAC rOyXvtHAdIcRTwzOVmdQ3wsswwB02jnRL2c49w4a9dfii1eMUhxwtCs-ZDkxE8k52Yf9lkXDnDyzQ contains: (keystone.token.provider): 2019-07-17 18:03:04,001 DEBUG domain: None, domain_scoped: False, user: {'email': 'juju@localhost', 'id': '8973385dd5ca467fb4be7a3eca7a603f', 'name': 'adm in', 'domain_id': '917f251e6fc24c389f1e3f3624d701d1', 'enabled': True, 'password_expires_at': None, 'options': {}} On Keystone using log_foreisic: +20164:5d2f62d7:17|GET /v3/domains/test-domain HTTP/1.1|Host:10.5.0.56%3a35357|User-Agent:python-keystoneclient|Accept-Encoding:gzip, deflate|Accept:application/json|Connection:keep-alive|X-Auth-Token:gAAAAABdL2LX_HT3mi4RO0KcwuqYaJ-NoY-gDMQtcKm-QDDJ0o-SsiH1BOaI5LlhbPLVyiKw7amvGcuwwuM9LLCCBb0VGkyIs2cmkTlHACrOyXvtHAdIcRTwzOVmdQ3wsswwB02jnRL2c49w4a9dfii1eMUhxwtCs-ZDkxE8k52Yf9lkXDnDyzQ -20164:5d2f62d7:17 +20164:5d2f62d7:18|GET /v3/domains?name=test-domain HTTP/1.1|Host:10.5.0.56%3a35357|User-Agent:python-keystoneclient|Accept-Encoding:gzip, deflate|Accept:application/json|Connection:keep-alive|X-Auth-Token:gAAAAABdL2LX_HT3mi4RO0KcwuqYaJ-NoY-gDMQtcKm-QDDJ0o-SsiH1BOaI5LlhbPLVyiKw7amvGcuwwuM9LLCCBb0VGkyIs2cmkTlHACrOyXvtHAdIcRTwzOVmdQ3wsswwB02jnRL2c49w4a9dfii1eMUhxwtCs-ZDkxE8k52Yf9lkXDnDyzQ -20164:5d2f62d7:18 +20164:5d2f62d7:19|GET /v3/users?domain_id=c9ca71bd88894017a6b6448dfcffeb68 HTTP/1.1|Host:10.5.0.56%3a35357|User-Agent:python-keystoneclient|Accept-Encoding:gzip, deflate|Accept:application/json|Connection:keep-alive|X-Auth-Token:gAAAAABdL2LX_HT3mi4RO0KcwuqYaJ-NoY-gDMQtcKm-QDDJ0o-SsiH1BOaI5LlhbPLVyiKw7amvGcuwwuM9LLCCBb0VGkyIs2cmkTlHACrOyXvtHAdIcRTwzOVmdQ3wsswwB02jnRL2c49w4a9dfii1eMUhxwtCs-ZDkxE8k52Yf9lkXDnDyzQ -20164:5d2f62d7:19 I.e. the OpenStack CLI used a non-domain scoped token to access the /v3/users?domain_id=... endpoint --- In Horizon/dashboard: (listing users for test-domain domain, with user "admin" logged in with "admin_domain" and a password): Token: (keystone.token.provider): 2019-07-17 17:52:15,538 DEBUG ZZZZ - the token: gAAAAABdL2BMm4RFbxzvoHI897QYkhLQLvfdAMKN25pThs5XCMLrpQRm9VmjXYvh52MAiK_6XUKRyBDBC9OIlJPbEPGU981kRIdwp5T8e1bby4UJeerBKoEB1BQ69c8T23HsoGsN0Kwr4iWST7ECXKfep6k7-C3TsZU74xmsflTlkA1DyDCPGZSN1g_JiPyfVyPTqa96AzMN contains: (keystone.token.provider): 2019-07-17 17:52:15,539 DEBUG domain: {'id': '917f251e6fc24c389f1e3f3624d701d1', 'name': 'admin_domain', 'description': 'Created by Juju', 'enabled': True, 'tags': []}, domain_scoped: True, user: {'email': 'juju@localhost', 'id': '8973385dd5ca467fb4be7a3eca7a603f', 'name': 'admin', 'domain_id': '917f251e6fc24c389f1e3f3624d701d1', 'enabled': True, 'password_expires_at': None, 'options': {}} (note that it is domain_scoped) Notice that the first two queries use the domain_scoped token: +20165:5d2f6184:17|GET /v3/users?domain_id=c9ca71bd88894017a6b6448dfcffeb68 HTTP/1.1|Host:10.5.0.56%3a35357|User-Agent:python-keystoneclient|Accept-Encoding:gzip, deflate|Accept:application/json|Connection:keep-alive|X-Auth-Token:gAAAAABdL2BMm4RFbxzvoHI897QYkhLQLvfdAMKN25pThs5XCMLrpQRm9VmjXYvh52MAiK_6XUKRyBDBC9OIlJPbEPGU981kRIdwp5T8e1bby4UJeerBKoEB1BQ69c8T23HsoGsN0Kwr4iWST7ECXKfep6k7-C3TsZU74xmsflTlkA1DyDCPGZSN1g_JiPyfVyPTqa96AzMN|Forwarded:for=10.5.0.105;by=python-keystoneclient -20165:5d2f6184:17 +20164:5d2f6184:d|GET /v3/domains HTTP/1.1|Host:10.5.0.56%3a35357|User-Agent:python-keystoneclient|Accept-Encoding:gzip, deflate|Accept:application/json|Connection:keep-alive|X-Auth-Token:gAAAAABdL2BMm4RFbxzvoHI897QYkhLQLvfdAMKN25pThs5XCMLrpQRm9VmjXYvh52MAiK_6XUKRyBDBC9OIlJPbEPGU981kRIdwp5T8e1bby4UJeerBKoEB1BQ69c8T23HsoGsN0Kwr4iWST7ECXKfep6k7-C3TsZU74xmsflTlkA1DyDCPGZSN1g_JiPyfVyPTqa96AzMN|Forwarded:for=10.5.0.105;by=python-keystoneclient -20164:5d2f6184:d (but then next is using a different token) +20164:5d2f6184:e|GET /v3/users/8973385dd5ca467fb4be7a3eca7a603f/projects HTTP/1.1|Host:10.5.0.56%3a5000|User-Agent:python-keystoneclient|Accept-Encoding:gzip, deflate|Accept:application/json|Connection:keep-alive|X-Auth-Token:gAAAAABdL2BKK3ew3ilF-JOVVZbnVDLRxLmS3VlJTmJY8ENYFrv4lqdcJf2LALE1BvcG5D5ay_1Ce1NJVdr4CTr7haHa0oP8_bcJ7qWMxnYlMEPL-Niw-8ddzlNHdcFtR45dOJgk0snGr1biueO_kEG9O04U_fXQhQ -20164:5d2f6184:e +20164:5d2f6184:f|GET /v3 HTTP/1.1|Host:10.5.0.56%3a5000|User-Agent:python-novaclient keystoneauth1/3.13.1 python-requests/2.18.4 CPython/3.6.8|Accept-Encoding:gzip, deflate|Accept:application/json|Connection:keep-alive -20164:5d2f6184:f Then a token request is made: +20164:5d2f6184:10|POST /v3/auth/tokens HTTP/1.1|Host:10.5.0.56%3a5000|User-Agent:python-novaclient keystoneauth1/3.13.1 python-requests/2.18.4 CPython/3.6.8|Accept-Encoding:gzip, deflate|Accept:application/json|Connection:keep-alive|X-Auth-Token:gAAAAABdL2BPSlVP8HSWBoxcd0u9j2Z1vX5wz2dYvvpFjyq2ixvsVYpC0xpet2vMjE9l6FNrCv4yiYCmww6d4QUSLrna7bSNGc4fkEAIWa-RdJ9fVkC40P3j1litDkCx0aQ2WhlOYUFTa1-Sh7vr0QckzHeuwd3LviSRLKhCi55NccgCW55RCDchW43l-RCLNsJKBmc5W00C|Content-Type:application/json|Content-Length:337 -20164:5d2f6184:10 +20164:5d2f6184:11|GET /v3 HTTP/1.1|Host:10.5.0.56%3a5000|User-Agent:openstack_dashboard keystoneauth1/3.13.1 python-requests/2.18.4 CPython/3.6.8|Accept-Encoding:gzip, deflate|Accept:application/json|Connection:keep-alive|Forwarded:for=10.5.0.105;by=openstack_dashboard keystoneauth1/3.13.1 python-requests/2.18.4 CPython/3.6.8 -20164:5d2f6184:11 And the projects for that "admin" user are listed with that token. +20165:5d2f6185:18|GET /v3/users/8973385dd5ca467fb4be7a3eca7a603f/projects HTTP/1.1|Host:10.5.0.56%3a5000|User-Agent:python-keystoneclient|Accept-Encoding:gzip, deflate|Accept:application/json|Connection:keep-alive|X-Auth-Token:gAAAAABdL2BKK3ew3ilF-JOVVZbnVDLRxLmS3VlJTmJY8ENYFrv4lqdcJf2LALE1BvcG5D5ay_1Ce1NJVdr4CTr7haHa0oP8_bcJ7qWMxnYlMEPL-Niw-8ddzlNHdcFtR45dOJgk0snGr1biueO_kEG9O04U_fXQhQ -20165:5d2f6185:18