stein: openstack-dashboard gui not showing project/users for a selected domain (via "set domain context) for the admin user.

[rafi]

OpenStack-dashboard gui not showing newly created project/users under a new domain, but everything shows fine in CLI. The project and users were created on the horizon gui a notification in green shows successfully but nothing shows in the gui. The OpenStack Stein version (openstack telemetry #53 ) was installed with juju charm bundle running 4 nodes any idea where to look? I'm wondering how the Horizon gui get updated (who's responsible to update the horizon gui) I would appreciate your help Thanks.

[Alex Kavanagh (ajkavanagh)]

If you refresh the page, do the new users/projects appear? i.e. is it an async update issue or a permissions issue in viewing things in the dashboard.

[rafi (rafise)]

Hi Alex,

Yes, the pages were refreshed also the service was restarted. This issue was reproduced by another user at https://ask.openstack.org/en/question/121350/openstack-horizon-gui-not-showing-newly-created-projectusers-under-newly-domain/ another info I'm using the latest stain charms.

[Alex Kavanagh (ajkavanagh)]

Hi Rafael; sorry, I wasn't clear. Did the projects appear AFTER the page was refreshed, or are they still missing? I need to understand whether it's a horizon issue/bug or a config issue around permissions.

[rafi (rafise)]

Sorry for the confusion, the issue still persists I still not able to fix it.

[Alex Kavanagh (ajkavanagh)]

I'm sorry it's still not clear to me: Please could you answer the following questions with yes/no answers:

1. When I add the newly created project/users under a new domain they show up in horizon: (Yes/No).
2. If I refresh the page (Ctrl-R in Chrome) immediately afterwards, they show in horizon: (Yes/No).
3. If I log out and log back in to horizon, the newly created projects/users show in horizon: (Yes/No).
3. If I use the CLI I can see the projects/users: (Yes/No).

If it is a horizon bug, then I anticipate (No, Yes, Yes, Yes)
If it is a permissions/schema bug then I anticipate (No, No, No, Yes)

Please also, could you paste the config for the dashboard charm. Thanks.
Please could you indicate the user/domain that you use to log into dashboard and the CLI. Thanks.

[rafi (rafise)]

1. NO
2. NO
3. NO
3. (CLI) YES
If it is a horizon bug, then I anticipate (No, Yes, Yes, Yes)
NO NO NO
--------------------------
Hi,

 This is a permission issue (Keystone) here's how I know. Reload the cloud again specifying keystone charm 296 from rocky version and it works fine. I'm able to create new domain and project/user and they all show in horizon, although all charms are running stain versions except keystone.

[rafi (rafise)]

Here's the screenshot of the services. (Everything works fine)

App Version Status Scale Charm Store Rev OS Notes
aodh 8.0.0 active 1 aodh jujucharms 27 ubuntu stain
ceilometer 12.0.0 active 1 ceilometer jujucharms 262 ubuntu stain
ceilometer-agent 12.0.0 active 3 ceilometer-agent jujucharms 252 ubuntu stain
ceph-mon 13.2.4+dfsg1 active 3 ceph-mon jujucharms 38 ubuntu stain
ceph-osd 13.2.4+dfsg1 active 3 ceph-osd jujucharms 284 ubuntu stain
ceph-radosgw 13.2.4+dfsg1 active 1 ceph-radosgw jujucharms 270 ubuntu stain
cinder 14.0.0 active 1 cinder jujucharms 284 ubuntu stain
cinder-ceph 14.0.0 active 1 cinder-ceph jujucharms 242 ubuntu stain
glance 18.0.0 active 1 glance jujucharms 281 ubuntu stain
gnocchi 4.3.2 active 1 gnocchi jujucharms 23 ubuntu stain
keystone 14.0.1 active 1 keystone jujucharms 296 ubuntu rocky
memcached active 1 memcached jujucharms 23 ubuntu stain
mysql 5.7.20-29.24 active 1 percona-cluster jujucharms 276 ubuntu stain
neutron-api 14.0.0 active 1 neutron-api jujucharms 273 ubuntu stain
neutron-gateway 14.0.0 active 1 neutron-gateway jujucharms 262 ubuntu stain
neutron-openvswitch 14.0.0 active 3 neutron-openvswitch jujucharms 259 ubuntu stain
nova-cloud-controller 19.0.0 active 1 nova-cloud-controller jujucharms 329 ubuntu stain
nova-compute 19.0.0 active 3 nova-compute jujucharms 302 ubuntu stain
ntp 3.2 active 4 ntp jujucharms 32 ubuntu stain
openstack-dashboard 15.0.0 active 1 openstack-dashboard jujucharms 288 ubuntu stain
rabbitmq-server 3.6.10 active 1 rabbitmq-server jujucharms 89 ubuntu stain

[Alex Kavanagh (ajkavanagh)]

Hi Rafael, I agree, it's almost certainly a permissions/policy issue. However, the charms are also in an odd situation where you have a rocky keystone and stein services; is there a particular reason for that? My concern is that the policy files in the rocky packages don't match the ones in stein.

Also does it really say "stain" on the juju panel in the Notes field; because the current release is "stein".

Anyway, if you can, can you upgrade keystone to stein (assuming that is what the other services are), and see if the problem persists?

[rafi (rafise)]

 Well, I manually forced to install the rocky 296 version because is the only version that works. I did upgrade to the stain version (300) and the project and users disappear again.

[Alex Kavanagh (ajkavanagh)]

Hi Rafael, ah, I think there is a bit of confusion here, (or I misunderstanding something here).

The charm version (296, 300) refers to when they were released:

keystone rev 296 - UploadTime: 2019-04-17T22:22:31.574Z
Keystone rev 300 - UploadTime: 2019-04-17T22:22:31.574Z

The actual keystone charm (and all the other OpenStack charms) are 'multi-series' which means they support Ubuntu Xenial (16.04) through to Ubuntu Disco (19.04) AND multiple versions of OpenStack: in fact the keystone charm is explicitly verified and supported on:
- trusty-mitaka
- xenial-mitaka
- xenial-ocata
- xenial-pike
- xenial-queens
- bionic-queens
- bionic-rocky
- bionic-stein

So, to be clear what you are running, please could you paste the output of "juju status keystone". It will look something like:

Model Controller Cloud/Region Version SLA Timestamp
neutron-gateway serverstack serverstack/serverstack 2.6.2 unsupported 19:45:25Z

App Version Status Scale Charm Store Rev OS Notes
keystone 14.0.1 active 1 keystone jujucharms 445 ubuntu

Unit Workload Agent Machine Public address Ports Message
keystone/0* active idle 1 10.5.0.11 5000/tcp Unit is ready

Machine State DNS Inst id Series AZ Message
1 started 10.5.0.11 7f39a66a-8c7e-4a63-a5fd-4a6f5a27eb27 bionic nova ACTIVE

I'm assuming you are changing the "openstack-origin" config value as well as issuing a 'juju upgrade-charm' command?

From the original juju status you did, I see that a few of the main apps are:

glance 18.0.0 active 1 glance jujucharms 281 ubuntu stain
keystone 14.0.1 active 1 keystone jujucharms 296 ubuntu rocky
openstack-dashboard 15.0.0

In this case, keystone 14.0.1 is a rocky release; however, the 300 rev keystone charm also support rocky, so there's no need to down-grade the charm, eve
openstack-dashboard 15.0.0 is Stein.

So, you're saying that keystone (rocky) works, but keystone (stein) doesn't? It would be useful if you could upgrade the keystone charm to the latest revision (300) BUT leave the keystone release as rocky and see if the error re-occurs or keeps working. This way we can tell if it is a charm issue (in general) or a packaging/charm issue to do with stein explicitly. Thanks.

[rafi (rafise)]

Thanks for your responds Alex.

 If I upgrade just the keystone in rocky release with just with: (juju upgrade-charm keystone) the issue come back I lose my domain and user in the dashboard.

____________________________________________________________________________________________________

Model Controller Cloud/Region Version SLA Timestamp
default default-controller default 2.6.2 unsupported 16:13:21-04:00

App Version Status Scale Charm Store Rev OS Notes
keystone 14.0.1 active 1 keystone jujucharms 296 ubuntu

Unit Workload Agent Machine Public address Ports Message
keystone/0* active idle 3/lxd/1 10.10.10.17 5000/tcp Unit is ready

Machine State DNS Inst id Series AZ Message
3 started 10.10.10.15 os-compute04 bionic default Deployed
3/lxd/1 started 10.10.10.17 juju-388b7d-3-lxd-1 bionic default Container started

[rafi (rafise)]

I notice from 296 and up regardless of what version rocky or stain the issue comes back

[Alex Kavanagh (ajkavanagh)]

Ok, thanks Rafael. I think it is a charm bug (I've added the keystone charm). You "should" be able to upgrade keystone to Stein, but leave the keystone charm at 296.

Oddly, there's not a lot of difference between keystone-296 and keystone-300 that would affect the relationship with horizon (the openstack-dashboard), so I'm at a bit of a loss.

It would be good to collect some log files from the openstack-dashboard and keystone and add them here for analysis. i.e. any errors or warnings in either in either.

I could also be changes in the policy files and configs. If you could also copy the /etc/keystone/keystone.conf, policy.json and keystone.policy.yaml files from keystone at keystone-296 and -300 and see what the differences are.

[rafi (rafise)]

 What's the command to update the keystone to stain but not changing the charm from 296? also is there a website to paste all the logs and files and just give you the links?

[Alex Kavanagh (ajkavanagh)]

Rafael, I think there is still a little confusion about what's going on. I hope I can clear it up a little:

1. There is no "stain"; it's Stein. (note 'e' not 'a').
2. There are two pieces of software in play here:
   a) OpenStack "keystone" which is the OpenStack control-plane software that provides identity services in Openstack.
   b) The keystone "charm"; deployment and lifecycle management software that Juju uses to install, configure and manage the OpenStack keystone control-plane software -- this is the payload of the charm.
3. The keystone charm has revisions (rev 1..300 at the time of writing), and stable releases every three months. Crucially, the current revision (or version) of the keystone charm supports MULTIPLE versions of the Ubuntu (Trusty .. Disco) AND OpenStack (Ocata .. Stein). The exact matrix currently supported is:
- trusty-mitaka
- xenial-mitaka
- xenial-ocata
- xenial-pike
- xenial-queens
- bionic-queens
- bionic-rocky
- bionic-stein
(but note, that cosmic-rocky, cosmic-stein, and disco-stein are continuously tested and has do work ... most of the time).
4. The OpenStack keystone software (the control plane) comes in versions (e.g. mitaka, ocata .. stein), with the current release being "stein".
5. Canonical supports a particular version of OpenStack per Ubuntu LTS. So for Trusty (14.04) it is Mitaka, Xenial (16.04) it is Ocata, and Bionic (18.04) it is Queens. So the OpenStack packages supplied with Bionic (18.04) has Queens packages.

So, the keystone charm revision 300 can support Ubuntu 14.04 to Ubuntu 19.04 AND OpenStack Mitaka to Stein. However, the above combinations are the ones which are supported (at present).

6. The general advice is "always run the latest stable charm that supports the Ubuntu version and OpenStack version". The stable charm is always on the charm store. The in-development/unstable charm is in the "~openstack-charmers" namespace and can be accessed by cs:~openstack-charmers/keystone -- However, we do not recommend using it is unstable and may break.

7. To upgrade the keystone CHARM, the command is "juju upgrade-charm keystone". This ONLY upgrades the charm software and NOT the underlying payload (in this case the OpenStack keystone control plane software).

8. Upgrading the keystone PAYLOAD is controlled through the config option "openstack-origin". Essentially, this is either "distro" (i.e. match the version shipped with the version of Ubuntu), or a string that represents a repository set. e.g. "cloud:bionic-stein" will select the stein version of keystone and upgrade the payload to that.

More details can be found here: https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-upgrade-openstack.html

So the to upgrade to stein, do:

juju config keystone openstack-origin=cloud:bionic-stein

Hope that helps.

[rafi (rafise)]

 Hello Alex, sorry about the typo and thank you for the explanation actually I did some more testing I've loaded the rocky 300 and it's running fine, now when I load the stein 296 or any stein versions the issue comes back.

 Do you know if canonical Ubuntu does have a website that you paste the logs files and copy the links?

[Chris MacNaughton (chris.macnaughton)]

Rafael, the easiest way to attach logs is to attach them directly to this bug using the "Add attachment of patch" link below the comment box.

[rafi (rafise)]

Hi, already upgrade the keystone from rocky to stein and my created domain and user disappear from horizon.

App Version Status Scale Charm Store Rev OS Notes
aodh 8.0.0 active 1 aodh jujucharms 27 ubuntu
ceilometer 12.0.0 active 1 ceilometer jujucharms 262 ubuntu
ceilometer-agent 12.0.0 active 3 ceilometer-agent jujucharms 252 ubuntu
ceph-mon 13.2.4+dfsg1 active 3 ceph-mon jujucharms 38 ubuntu
ceph-osd 13.2.4+dfsg1 active 3 ceph-osd jujucharms 284 ubuntu
ceph-radosgw 13.2.4+dfsg1 active 1 ceph-radosgw jujucharms 270 ubuntu
cinder 14.0.0 active 1 cinder jujucharms 284 ubuntu
cinder-ceph 14.0.0 active 1 cinder-ceph jujucharms 242 ubuntu
glance 18.0.0 active 1 glance jujucharms 281 ubuntu
gnocchi 4.3.2 active 1 gnocchi jujucharms 23 ubuntu
keystone 15.0.0 active 1 keystone jujucharms 300 ubuntu
memcached active 1 memcached jujucharms 23 ubuntu
mysql 5.7.20-29.24 active 1 percona-cluster jujucharms 276 ubuntu
neutron-api 14.0.0 active 1 neutron-api jujucharms 273 ubuntu
neutron-gateway 14.0.0 active 1 neutron-gateway jujucharms 262 ubuntu
neutron-openvswitch 14.0.0 active 3 neutron-openvswitch jujucharms 259 ubuntu
nova-cloud-controller 19.0.0 active 1 nova-cloud-controller jujucharms 329 ubuntu
nova-compute 19.0.0 active 3 nova-compute jujucharms 302 ubuntu
ntp 3.2 active 4 ntp jujucharms 32 ubuntu
openstack-dashboard 15.0.0 active 1 openstack-dashboard jujucharms 288 ubuntu
rabbitmq-server 3.6.10 active 1 rabbitmq-server jujucharms 89 ubuntu

[Alex Kavanagh (ajkavanagh)]

Rafael, if you could just attach the config files requested, that would be great. Thanks.

[rafi (rafise)]

keystone rocky 296

[rafi (rafise)]

keystone stein 300

[Alex Kavanagh (ajkavanagh)]

Rafael, I'm now on leave for a week, but will be back on 10th June. I've noticed that the 296 and 300 policy.json files refer to different projects (if you diff them you'll see:)

diff 296/policy.json-296.txt 300/policy.json-300\ -\ Copy.txt
4c4
< "cloud_admin": "rule:admin_required and (is_admin_project:True or domain_id:ef2674ec5a394a8080fe5306529817d1 or project_id:c59f2093e088412086fd9dbe11f3f5f3)",
---
> "cloud_admin": "rule:admin_required and (is_admin_project:True or domain_id:cce855539fa54878a995cc449ad4f628 or project_id:7c1957f6d99447fc820c60d383d9d248)",

I don't know why they are different (ye), but it should be interesting if you could have a look on your system and see which friendly names have these projects & domain names (I'm guessing "admin" and "admin_domain", but I'm not sure).

[rafi (rafise)]

Heres some images and some out of the openstack list

maas01:~$ openstack domain list
+----------------------------------+----------------+---------+-----------------+
| ID | Name | Enabled | Description |
+----------------------------------+----------------+---------+-----------------+
| 132df88ca3a64041b4c0b671754b1793 | service_domain | True | Created by Juju |
| 2c4ca7320fd24409b73cbe7bfcf98698 | admin_domain | True | Created by Juju |
| d6dc68a01d204377a1786a5968559de8 | default | True | Created by Juju |
| f176142d72314e8c91aff1cbaf9c9675 | test | True | |
+----------------------------------+----------------+---------+-----------------+

maas01:~$ openstack project list
+----------------------------------+--------------+
| ID | Name |
+----------------------------------+--------------+
| 8077fbd62fbf4908a9be7ec6881708fe | admin |
| 82bfd647337244b0a7beca109ec93f03 | test-project |
| a983357060914e6a8876e3d5bd4b1b28 | services |
| f4c29dc067784dac8a253c78d399064c | services |
| fd7a127740404a319d1079d856814231 | admin |
+----------------------------------+--------------+

maas01:~$ openstack user list --domain test
+----------------------------------+-------+
| ID | Name |
+----------------------------------+-------+
| 241fc066c744482ba40e556c99ddf493 | user1 |
+----------------------------------+-------+

[rafi (rafise)]

Actually, the images and output are from full stein installation including Keystone .

keystone 15.0.0 active 1 keystone jujucharms 301 ubuntu

[Matthias (matthiashuether)]

We have the same problem with a new deployment with stein and the newest charm versions.

[Matthias (matthiashuether)]

see here, maybe the same problem: https://bugs.launchpad.net/horizon/+bug/1826114

[Przemyslaw Hausman (phausman)]

I also experience this issue on a new Stein deployment.

[Przemyslaw Hausman (phausman)]

I'm subscribing field-high as we're experiencing this issue in a field deployment.

[rafi (rafise)]

I just notice the stein version has been removed from juju charm store.

On Thu, Jul 4, 2019, 11:10 AM Przemyslaw Hausman <email address hidden>
wrote:

> I'm subscribing field-high as we're experiencing this issue in a field
> deployment.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1830782
>
> Title:
> openstack-dashboard gui not showing newly created project/users under
> newly domain
>
> Status in OpenStack keystone charm:
> Incomplete
> Status in OpenStack Bundles:
> Incomplete
>
> Bug description:
> OpenStack-dashboard gui not showing newly created project/users under
> a new domain, but everything shows fine in CLI. The project and users
> were created on the horizon gui a notification in green shows
> successfully but nothing shows in the gui. The OpenStack Stein version
> (openstack telemetry #53 ) was installed with juju charm bundle
> running 4 nodes any idea where to look? I'm wondering how the Horizon
> gui get updated (who's responsible to update the horizon gui) I would
> appreciate your help Thanks.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/charm-keystone/+bug/1830782/+subscriptions
>

[Alex Kavanagh (ajkavanagh)]

Rafael, what do you mean? The latest stable version is #288. The unstable/development version is now at #425: https://jaas.ai/u/openstack-charmers-next/openstack-dashboard/425

[Alex Kavanagh (ajkavanagh)]

Note that when a fix is found, we should update this question: https://ask.openstack.org/en/question/121350/openstack-horizon-gui-not-showing-newly-created-projectusers-under-newly-domain/

[rafi (rafise)]

Alex,

I Notice the stein version has been removed from juju charm store rocky version is back, that explains everything.

[Ryan Beisner (1chb1n)]

FYI Marked the bundle bug task as invalid as this is a charm or charm payload issue. Thank you for your work on this.

[Alex Kavanagh (ajkavanagh)]

After a lot of checking, the issue is also apparent with the rocky dashboard on a stein system. This could be changes in keystone between rocky and stein, rather than the openstack dashboard.

[rafi (rafise)]

Hi Alex,

I rebuild with rocky and everything is working fine.

On Tue, Jul 9, 2019, 10:16 AM Alex Kavanagh <email address hidden>
wrote:

> After a lot of checking, the issue is also apparent with the rocky
> dashboard on a stein system. This could be changes in keystone between
> rocky and stein, rather than the openstack dashboard.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1830782
>
> Title:
> openstack-dashboard gui not showing newly created project/users under
> newly domain
>
> Status in OpenStack keystone charm:
> Incomplete
> Status in OpenStack openstack-dashboard charm:
> Confirmed
> Status in OpenStack Bundles:
> Invalid
>
> Bug description:
> OpenStack-dashboard gui not showing newly created project/users under
> a new domain, but everything shows fine in CLI. The project and users
> were created on the horizon gui a notification in green shows
> successfully but nothing shows in the gui. The OpenStack Stein version
> (openstack telemetry #53 ) was installed with juju charm bundle
> running 4 nodes any idea where to look? I'm wondering how the Horizon
> gui get updated (who's responsible to update the horizon gui) I would
> appreciate your help Thanks.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/charm-keystone/+bug/1830782/+subscriptions
>

[David Ames (thedac)]

Alex, is LP Bug#1837104 a duplicate of this bug?

https://bugs.launchpad.net/charm-keystone-ldap/+bug/1837104

[Alex Kavanagh (ajkavanagh)]

David, yes https://bugs.launchpad.net/charm-keystone-ldap/+bug/1837104 is a duplicate of this one; they have slightly different titles, so I'll edit this one, and then mark the other as a duplicate.

[Alex Kavanagh (ajkavanagh)]

On further debugging, it appears that there is an issue in horizon (caused due to a change in keystone) with the use of a scoped tokens for the admin user when multi-domain is enabled.

The scenario is as follows:

1. Multi domain is enabled.
2. The admin user is logged in with credentials using an admin domain.
3. Domain context is set to a domain in which the admin user is not a member.
4. The admin user attempts to list the projects or users.
5. A domain scoped token is used by horizon to list the projects, due to the code in [1]
6. No users are returned from keystone because, due to change [2] the users are filtered as the token contains the admin domain, not the target domain or users to list.

It's quite involved!

I'm not sure if the issue is:

1. Keystone shouldn't be filtering this list.
2. Horizon shouldn't be using a domain scoped token for the admin user (e.g. the openstack CLI doesn't use a domain scoped token to list the users in the domain, or an admin user).
3. Something else.

Horizon appears to only start using the domain scoped token after the domain context is set. Also, it only appears (in my testing) to use it for the user list and (maybe) project list -- I focussed on the user list. It looks like a new token is requested to perform the user list and that this one is domain scoped.

I can do further testing as necessary.

References:
[1] Horizon, openstack_dashboard/api/keystone.py (def keystoneclient:) https://github.com/openstack/horizon/blob/stable/stein/openstack_dashboard/api/keystone.py#L167
[2] Keystone, change Id: I60b2e2b8af172c369eab0eb2c29f056f5c98ad16, https://review.opendev.org/#/c/647587/ (for user list)

Debugging info:

I added some debug LOG lines to the various bits of horizon and keystone to try to work out what was going on. The following is a comparison between Horizon and the OpenStack CLI in listing users for a domain "test-domain":

Preamble: The test set up:

The test is listing users for the "test-domain" on the OpenStack CLI and using the Horizon dashboard.

Domain list:

+----------------------------------+----------------+---------+-----------------+
| ID | Name | Enabled | Description |
+----------------------------------+----------------+---------+-----------------+
| 4c97d83fd8f34507aa5849710218272e | default | True | Created by Juju |
| 917f251e6fc24c389f1e3f3624d701d1 | admin_domain | True | Created by Juju |
| be5450b76a2348c48df0d0571295de40 | test-domain2 | True | |
| c9ca71bd88894017a6b6448dfcffeb68 | test-domain | True | |
| ecb1e99a62534253a5b515dcfc218733 | service_domain | True | Created by Juju |
+----------------------------------+----------------+---------+-----------------+

The "admin" user is in the admin_domain.

Project List:

+----------------------------------+---------------+
| ID | Name |
+----------------------------------+---------------+
| 1014c1815147453b8bd77de578467a80 | demo |
| 49ae284fd4aa42208573d9c399a95eee | services |
| 7581c43d252848dface4c75e2b921224 | test-project |
| 75c183f2aece43e2860be5...

[Alex Kavanagh (ajkavanagh)]

I've removed the openstack dashboard and keystone charms (deployment tool) as they are not the source of the error. I've added in the Horizon project as I think it's a mismatch in behaviour between horizon and keystone, and that https://review.opendev.org/#/c/647587/ has caused the issue to become apparent. The relevant code in keystone is:

    def _list_users(self):
        """List users.

        GET/HEAD /v3/users
        """
        filters = ('domain_id', 'enabled', 'idp_id', 'name', 'protocol_id',
                   'unique_id', 'password_expires_at')
        target = None
        if self.oslo_context.domain_id:
            target = {'domain_id': self.oslo_context.domain_id}
        hints = self.build_driver_hints(filters)
        ENFORCER.enforce_call(
            action='identity:list_users', filters=filters, target_attr=target
        )
        domain = self._get_domain_id_for_list_request()
        if domain is None and self.oslo_context.domain_id:
            domain = self.oslo_context.domain_id
        refs = PROVIDERS.identity_api.list_users(
            domain_scope=domain, hints=hints)

        # If the user making the request used a domain-scoped token, let's make
        # sure we filter out users that are not in that domain. Otherwise, we'd
        # be exposing users in other domains. This if statement is needed in
        # case _get_domain_id_for_list_request() short-circuits due to
        # configuration and protects against information from other domains
        # leaking to people who shouldn't see it.
        if self.oslo_context.domain_id:
            domain_id = self.oslo_context.domain_id
            users = [user for user in refs if user['domain_id'] == domain_id]
        else:
            users = refs

        return self.wrap_collection(users, hints=hints)

(from keystone/api/users.py - stable/stein -- lines 162 - 32).

This code filters out users who are not in the domain self.oslo_context.domain_id, which comes from a domain scoped token; in this case, Horizon is using the domain_scoped token with "admin_domain" as the domain_id (obviously, the ID rather than the text).

[Alex Kavanagh (ajkavanagh)]

See also, that a question has been asked on ask.openstack.org: https://ask.openstack.org/en/question/121350/openstack-horizon-gui-not-showing-newly-created-projectusers-under-newly-domain/

[Drew Freiberger (afreiberger)]

more details from affected user:

topics:
1. The browser being used - see if you get the same issue with another browser
2. any adblockers, anti-malware, popupblockers or scriptblockers being used as plugins on the browser
3. if possible inform us of the skin being used within horizon
4. The OS being used
5. Whether you are able to list all the projects on the CLI

answers:
1) Desktop: Safari - Firefox - Chrome; Mobile Android: Chrome <--all browsers present the same cosmetic issue
2) no plugins
3) themes: Ubuntu & default
4) desktop: Mac OS (Mojave); mobile: android pie
5) yes, CLI works smoothly

[David Coronel (davecore)]

Hi Alex, I'm seeing a different symptom in a Stein deployment with RBAC policies for networks that I believe might be caused by the same issue.

The scenario is:

1. Multi domain is enabled.

2. As admin, create a network (not shared):
$ openstack network create --external --provider-physical-network physnet2 --provider-network-type flat prod-external

3. Share the network with RBAC to a project from another domain (creating the rbac before or after the subnet doesn't seem to matter):
$ openstack network rbac create --target-project <project id> --action access_as_external --type network <network id>
$ openstack subnet create --network <network id> --allocation-pool start=<start ip>,end=<end ip> --dns-nameserver <dns server> --gateway <gateway> --subnet-range <subnet range> prod-external

4. Logged in as a user in the other domain, the prod-external network is visible in the Networks section in Horizon, but the subnet is not

5. Still logged in as that user, unable to launch an instance with that network (the network doesn't show up in the instance creation screen)

6. However, the same user on the CLI is able to launch an instance as well as see the subnet

[Alex Kavanagh (ajkavanagh)]

I've posted upstream about this issue here [1], but not had a response yet. This is a (potentially) tricky bug to fix as it's not really clear what the intention of the authors of horizon have regarding admin and domain-scoped tokens.

[1] http://lists.openstack.org/pipermail/openstack-discuss/2019-August/008170.html

[Vishal Manchanda (vishalmanchanda)]

Hi @rafael, I am not able to reproduce this bug on the master branch. Please find below the steps I have executed to reproduce it:

1. KEYSTONE_MULTIDOMAIN_SUPPORT is enabled.
2. Login with domain = 'default' Username = 'admin' password = 'xxxxxx'
3. Go to Identity -> Project ->> Create Project ->>> created sample Project -->>>sample Project created successfully
   and 'sample' project is visible to the Projects table as well.
4. Go to Identity -> Users ->> Create User ->>> created test User -->>>test user created successfully
   and 'test' user is visible to the Users table as well.

Let me know if I have missed some something

[Alex Kavanagh (ajkavanagh)]

Hi @Vishal,

I think you maybe missed a step (or I've misunderstood what you did).

1. Admin user is in its own domain (in my case admin_domain).
2. multidomain is enabled
3. log in as admin in admin_domain
4. Create a NEW domain (say 'test')
5. Set the domain context to that domain (i.e. click on "Set domain context").
6. Create a user in the test domain.
7. See if you can see the user.
8. Try with a project, etc.

[Alex Kavanagh (ajkavanagh)]

Reading back on my comment, it's not clear that these are the things I did; there are also the logs above which show (sort of; it's not very clear) that a domain scoped token is used to ask for the list of users when the domain context is set.

[Eduardo Pérez (eperezf)]

Hi! It seems to be the same that I reported here a couple of months ago: https://bugs.launchpad.net/horizon/+bug/1826114

As a workaround, my solution was to deploy the rocky version of the OpenStack bundle in the meantime. But can I deploy the Stein release on all the other charms and deploy Rocky in the Horizon charm? Or there are conflicts between versions?

[BN (zatoichy)]

Is there any update on this? What can I do to solve this issue?

Thank you

[Alex Kavanagh (ajkavanagh)]

> Is there any update on this? What can I do to solve this issue?

It's tricky; it's accepted as a bug in the horizon code, which means there needs to be a fix there. I fairly certain there is no work around, except to not use the stein version of horizon.

[rafi (rafise)]

Hi Alex,

Using an older version of the horizon mix with stein does not solve the issue that was one of the earliest tests I did.

[Alex Kavanagh (ajkavanagh)]

Rafael, yes, you're right; you'd also need to use an older version of keystone as its keystone's change that precipitated the issue in horizon. Not that I'd recommend that.

[rafi (rafise)]

Alex- Is there a way to use juju to create domain/project and users? If there is can you share the commands thanks

[Tim Penhey (thumper)]

No way for juju to do this at this stage.

[Tim Penhey (thumper)]

Well... juju core at least. It is possible that there may be some charm action. I only considered this after I sent the last one.

[Alex Kavanagh (ajkavanagh)]

Rafael, you can use the OpenStack CLI client to do all the actions. It is not affected by this issue in horizon. i.e. you can use "openstack user create --domain xxx --project-domain xxx etc" for the user, "openstack domain create ..." and "openstack project create ..." for the creation, and then the "openstack <thing> list" to show the user/domain/project list, and then "openstack <thing> show" to get more detail. Everything works as normal.

[Eduardo Pérez (eperezf)]

Any news on this issue? It's still present in the train release.

[Przemyslaw Hausman (phausman)]

This looks like a duplicate of bug #1826114 (Errors creating users and projects).

[Ziyu Bai (baiziyu-inspur)]

Yes. This commit https://review.opendev.org/#/c/697962/ works for me. And I think this bug could be closed too.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/697962
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=9aca7a94e28589484e312770cc1ee2b8243211bb
Submitter: Zuul
Branch: master

commit 9aca7a94e28589484e312770cc1ee2b8243211bb
Author: Hemanth Nakkina <email address hidden>
Date: Mon Dec 9 14:59:59 2019 +0530

    Fix users/projects list when domain context is changed

    In case of Keystone Multidomain setup, the project and users list
    is empty when the domain context is changed. Horizon uses domain
    scoped token for keystone api calls to get list of projects and users.
    And domain scoped token cannot get information about projects and users
    in other domains, the list is empty.

    This patch modifies the use of domain scoped token only when domain
    context is not modified.

    The bug have 3 parts
    1. Users are not listed on GUI in different domain
    2. Projects are not listed on GUI in different domain
    3. Gui hangs during creation of user/project using + option
    This patch handles case 1 and 2.

    Change-Id: Ibafe3e2eb3ee1ee5c9eb5d2a276a0edfa3e7c607
    Partial-Bug: #1826114
    Closes-Bug: #1830782

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/698866

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/698867

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/horizon 17.1.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/698866
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=02f4d8ab762cd8587873e832bdcb8e4c1ec0656c
Submitter: Zuul
Branch: stable/train

commit 02f4d8ab762cd8587873e832bdcb8e4c1ec0656c
Author: Hemanth Nakkina <email address hidden>
Date: Mon Dec 9 14:59:59 2019 +0530

    Fix users/projects list when domain context is changed

    In case of Keystone Multidomain setup, the project and users list
    is empty when the domain context is changed. Horizon uses domain
    scoped token for keystone api calls to get list of projects and users.
    And domain scoped token cannot get information about projects and users
    in other domains, the list is empty.

    This patch modifies the use of domain scoped token only when domain
    context is not modified.

    The bug have 3 parts
    1. Users are not listed on GUI in different domain
    2. Projects are not listed on GUI in different domain
    3. Gui hangs during creation of user/project using + option
    This patch handles case 1 and 2.

    Change-Id: Ibafe3e2eb3ee1ee5c9eb5d2a276a0edfa3e7c607
    Partial-Bug: #1826114
    Closes-Bug: #1830782
    (cherry picked from commit 9aca7a94e28589484e312770cc1ee2b8243211bb)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/698867
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=f6490abc288f667e9af299264e5eebf7eb931355
Submitter: Zuul
Branch: stable/stein

commit f6490abc288f667e9af299264e5eebf7eb931355
Author: Hemanth Nakkina <email address hidden>
Date: Mon Dec 9 14:59:59 2019 +0530

    Fix users/projects list when domain context is changed

    In case of Keystone Multidomain setup, the project and users list
    is empty when the domain context is changed. Horizon uses domain
    scoped token for keystone api calls to get list of projects and users.
    And domain scoped token cannot get information about projects and users
    in other domains, the list is empty.

    This patch modifies the use of domain scoped token only when domain
    context is not modified.

    The bug have 3 parts
    1. Users are not listed on GUI in different domain
    2. Projects are not listed on GUI in different domain
    3. Gui hangs during creation of user/project using + option
    This patch handles case 1 and 2.

    Change-Id: Ibafe3e2eb3ee1ee5c9eb5d2a276a0edfa3e7c607
    Partial-Bug: #1826114
    Closes-Bug: #1830782
    (cherry picked from commit 9aca7a94e28589484e312770cc1ee2b8243211bb)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/horizon 15.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/horizon 16.1.0 release.

[Hemanth Nakkina (hemanth-n)]

FYI, Fix is now released in Ubuntu Focal, Eoan and UCA Train, UCA Stein
Please check https://bugs.launchpad.net/ubuntu/+source/horizon/+bug/1826114 for more information