No longer able to manage project members in Horizon

[Dan Ackerson]

After upgrading to the 20.05 charm version, clicking on "Manage members" silently fails while trying to visit any other tab (e.g. "Users") displays "Error: Could not find default role "member" in Keystone".

We believe it to be related to the following change: https://github.com/openstack/charm-openstack-dashboard/commit/3f1f985be853b7be65ad70f96890899e688b6a4e

Perhaps just a missing db migration?

[Dan Ackerson (dan.ackerson)]

Workaround: Running `openstack config openstack-dashboard default-role=Member` resolves the issue immediately.

[Alex Kavanagh (ajkavanagh)]

Hi Dan, please could you add the version of Ubuntu and OpenStack that you are using as well; there were some changes in the defaults of the case ("title-case" vs "lower-case") in the roles during recent versions and it might be something that isn't being caught during upgrades. Thanks.

[Dan Ackerson (dan.ackerson)]

Hi Alex,

sure. We had this issue on 2 different clouds. One is running Ubuntu 16.04 + Queens. The other is Ubuntu 18.04 and Stein.

Hope that helps,

Dan

[Liam Young (gnuoy)]

As @dan.ackerson says this down to a mismatch between the member role created by the keystone charm and the one expected by the dashboard charm. As of 20.05 the member role is called Member rather than member.

To reproduce:
Deploy bionic stein with cs:~openstack-charmers-next/keystone-480
Login to horizon
 Domain: admin_domain
 Username: admin
 Password: $(juju run --unit keystone/0 "leader-get admin_passwd")

Identity -> Projects -> Manage Members

Sign Out

juju upgrade-charm --switch cs:~openstack-charmers-next/keystone keystone

Identity -> Projects -> Manage Members # Silently fails

[Liam Young (gnuoy)]

Actually you need to use an earlier version of the dashbaord too to see it working pre-20.05. I used cs:~openstack-charmers-next/openstack-dashboard-470

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/747213

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/747213
Committed: https://git.openstack.org/cgit/openstack/charm-openstack-dashboard/commit/?id=47c1097ea421efaee1d9d04e3549a73b53792ce2
Submitter: Zuul
Branch: master

commit 47c1097ea421efaee1d9d04e3549a73b53792ce2
Author: Liam Young <email address hidden>
Date: Thu Aug 20 15:31:44 2020 +0000

    Check the created_roles reply for keystone

    Check the created_roles reply for keystone to see if the name of
    the role that was created in keystone matches what was requested.
    It may differ in terms of case.

    Change-Id: I5b5216909268ba3bb1b7ad13767647fa1af77cc5
    Closes-Bug: #1890437

[Felipe Reyes (freyes)]

Hello,

I'm re-opening this bug since when deploying with cs:~openstack-charmers-next/openstack-dashboard-486 a bionic-queens environment[0] is still presenting this problem.

$ juju export-bundle | pastebinit -f yaml # cs:~openstack-charmers-next/openstack-dashboard-486
https://paste.ubuntu.com/p/Q2DxNz3DcY/
$ juju ssh openstack-dashboard/0 sudo grep ROLE /etc/openstack-dashboard/local_settings.py
OPENSTACK_KEYSTONE_DEFAULT_ROLE = "member"
Connection to 192.168.11.185 closed.
$ juju run --unit openstack-dashboard/0 'relation-get -r $(relation-ids identity-service) - keystone/0'
admin_domain_id: b2f887e4dacb4f0cabf5a307d301abd8
api_version: "3"
auth_host: 192.168.11.182
auth_port: "35357"
auth_protocol: http
egress-subnets: 192.168.11.182/32
ingress-address: 192.168.11.182
private-address: 192.168.11.182
region: RegionOne
service_host: 192.168.11.182
service_port: "5000"
service_protocol: http

Relevant chunk of the log:

[Sat Sep 12 21:47:49.987162 2020] [wsgi:error] [pid 17286:tid 140054022964992] [remote 192.168.11.185:36648] Recoverable error: Could not find default role "member" in Keystone
[Sat Sep 12 21:47:49.987501 2020] [wsgi:error] [pid 17286:tid 140054022964992] [remote 192.168.11.185:36648] Problem instantiating action class.
[Sat Sep 12 21:47:49.987508 2020] [wsgi:error] [pid 17286:tid 140054022964992] [remote 192.168.11.185:36648] Traceback (most recent call last):
[Sat Sep 12 21:47:49.987510 2020] [wsgi:error] [pid 17286:tid 140054022964992] [remote 192.168.11.185:36648] File "/usr/share/openstack-dashboard/horizon/workflows/base.py", line 396, in action
[Sat Sep 12 21:47:49.987512 2020] [wsgi:error] [pid 17286:tid 140054022964992] [remote 192.168.11.185:36648] context)
[Sat Sep 12 21:47:49.987514 2020] [wsgi:error] [pid 17286:tid 140054022964992] [remote 192.168.11.185:36648] File "/usr/share/openstack-dashboard/openstack_dashboard/dashboards/identity/domains/workflows.py", line 80, in __init__
[Sat Sep 12 21:47:49.987515 2020] [wsgi:error] [pid 17286:tid 140054022964992] [remote 192.168.11.185:36648] redirect=reverse(constants.DOMAINS_INDEX_URL))
[Sat Sep 12 21:47:49.987517 2020] [wsgi:error] [pid 17286:tid 140054022964992] [remote 192.168.11.185:36648] File "/usr/share/openstack-dashboard/horizon/exceptions.py", line 347, in handle
[Sat Sep 12 21:47:49.987518 2020] [wsgi:error] [pid 17286:tid 140054022964992] [remote 192.168.11.185:36648] log_method, log_entry, log_level)
[Sat Sep 12 21:47:49.987527 2020] [wsgi:error] [pid 17286:tid 140054022964992] [remote 192.168.11.185:36648] File "/usr/share/openstack-dashboard/horizon/exceptions.py", line 256, in handle_recoverable
[Sat Sep 12 21:47:49.987529 2020] [wsgi:error] [pid 17286:tid 140054022964992] [remote 192.168.11.185:36648] raise Http302(redirect)
[Sat Sep 12 21:47:49.987530 2020] [wsgi:error] [pid 17286:tid 140054022964992] [remote 192.168.11.185:36648] Http302

[0] https://gist.github.com/freyes/39c7c59183140cb8a3ed341b6ac1cea1

[Liam Young (gnuoy)]

Hi Felipe,
    The bundle you mention contains cs:keystone-317 which does not contain the keystone side of the fix (https://review.opendev.org/#/c/747212/). Please retry with a version of the charm which contains that fix. I am going to mark this back to Fix Committed obviously feel free to reopen if you are still seeing issues.
Thanks
Liam

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/20.08
Review: https://review.opendev.org/756393

[OpenStack Infra (hudson-openstack)]

Change abandoned by Felipe Reyes (<email address hidden>) on branch: stable/20.08
Review: https://review.opendev.org/756393
Reason: 20.10 is around the corner, there is no urgency in having them in stable, we can wait.

[Drew Freiberger (afreiberger)]

Queens is still experiencing this after a charm upgrade of a cloud to:
    charm: cs:keystone-319
    charm: cs:openstack-dashboard-309

I think maybe it's fixed for new deployments but doesn't get fixed in upgrade-charm. It appears that the check for context in openstack-dashboard only checks both entries lower-cased, but this doesn't allow for the relation data to have come in upper-cased.

I feel charm-openstack-dashboard needs to support the relation data providing upper case Member and not accepting the charm default of lower-case member as matching the upper-case Member role set in the relation.

$ juju-run openstack-dashboard/10 -r 133 --remote-unit keystone/2 relation-get |grep created_roles
created_roles: Member

I had to juju config openstack-dashboard default-role=Member to resolve.

[Liam Young (gnuoy)]

(comment below replicated here: https://paste.ubuntu.com/p/B4Qjn2YJ9V/)

Hi Drew,
    I don't seem to be able to reproduce the issue. The IdentityServiceContext
in openstack dashboard does allow for the relation data to have come in
upper-case (or mixed case). The idea is that openstack-dashboard charm tells
keystone the role it wants to use via the identity service relation.

The keystone charm then checks to see if it has a role that matches (case
insensative). If it finds a match it tells the dashboard charm the
case sensative name of the role that matches.

The IdentityServiceContext looks at the names of the roles provided
by keystone using the created_roles key. It does a case insensative
search comparing each role against the charm config option 'default-role'.
If it finds a match it uses the role name provided by keystone preserving
the case as specified by the keystone charm.

In the example of a deployment where the keystone setup was done by pre-20.05
charms, the member role is called Member and this does not change when the
charms are upgraded. However, the dashbaord switches from expecting the
role to be called Member to expecting to be called member. The charms cope
with this. As decribed above after the upgrade the dashboard charm now
expects the role to be called 'member':

$ juju config openstack-dashboard default-role
member

And it tells keystone this is the name of the role it wants:

$ juju run --application openstack-dashboard "relation-get -r identity-service:5 requested_roles openstack-dashboard/0"
member

The keystone charm spots there is already a role with
that name but with different case and tells the dashbaord:

$ juju run --application openstack-dashboard "relation-get -r identity-service:5 created_roles keystone/0"
Member

The dashboard charm IdentityServiceContext resolves this and correctly
identifies it should be using the capatalised member:

$ juju run --application openstack-dashboard "cd hooks; export PYTHONPATH=../; python3 -c 'import horizon_contexts; print(horizon_contexts.IdentityServiceContext()())'"
{'service_port': '5000', 'service_host': '172.20.0.9', 'service_protocol': 'http', 'api_version': '3', 'default_role': 'Member', 'admin_domain_id': '31d65a195f7e4fb2b826720dda0275b7'}

Below are the steps I went through to try and replicate the error, would
you mind taking a look and letting me know if any of it deviates from
your setup?

Thanks
Liam

Deploy pre-20.05 charms:

series: bionic
relations:
  - ["keystone:shared-db", "mysql:shared-db"]
  - ["openstack-dashboard:shared-db", "mysql:shared-db"]
  - ["openstack-dashboard:identity-service", "keystone:identity-service"]
applications:
  mysql:
    charm: cs:~openstack-charmers/percona-cluster
    num_units: 1
  keystone:
    charm: cs:~openstack-charmers/keystone-312
    num_units: 1
  openstack-dashboard:
    charm: cs:~openstack-charmers/openstack-dashboard-302
    num_units: 1

$ charm show cs:~openstack-charmers/openstack-dashboard-302 archive-upload-time
archive-upload-time:
  UploadTime: "2020-04-21T19:53:31.028Z"

$ charm show cs:~openstack-charmers/keystone-312 archive-upload-time
archive-upload-time:
  UploadTime: "2020-04-21T20:02:32.1...

[Bayani Carbone (bcarbone)]

I'm facing this issue in an Ussuri deployment that has been upgraded from Queens (going through each intermediate openstack version in between).

I'm running charms:
 - cs:keystone-323
 - cs:openstack-dashboard-313

I'm wondering if in my case it could be linked to the relation data as this is what I see:

$ juju run --application openstack-dashboard "relation-get -r identity-service:116 - keystone/0"
- Stdout: |
    admin_domain_id: c7f2b8aaedbb4fb0a128ebb4ee03f030
    api_version: "3"
    auth_host: keystone-int.acme.com
    auth_port: "35357"
    auth_protocol: https
    egress-subnets: 10.123.45.6/32
    ingress-address: 10.123.45.6
    private-address: 10.123.45.6
    region: RegionOne
    service_host: keystone.acme.com
    service_port: "5000"
    service_protocol: https
  UnitId: openstack-dashboard/0
- Stdout: |
    admin_domain_id: c7f2b8aaedbb4fb0a128ebb4ee03f030
    api_version: "3"
    auth_host: keystone-int.acme.com
    auth_port: "35357"
    auth_protocol: https
    egress-subnets: 10.123.45.6/32
    ingress-address: 10.123.45.6
    private-address: 10.123.45.6
    region: RegionOne
    service_host: keystone.acme.com
    service_port: "5000"
    service_protocol: https
  UnitId: openstack-dashboard/2
- Stdout: |
    admin_domain_id: c7f2b8aaedbb4fb0a128ebb4ee03f030
    api_version: "3"
    auth_host: keystone-int.acme.com
    auth_port: "35357"
    auth_protocol: https
    egress-subnets: 10.123.45.6/32
    ingress-address: 10.123.45.6
    private-address: 10.123.45.6
    region: RegionOne
    service_host: keystone.acme.com
    service_port: "5000"
    service_protocol: https
  UnitId: openstack-dashboard/3

$ juju run --application openstack-dashboard "relation-get -r identity-service:116 - keystone/1"
- Stdout: |
    admin_domain_id: c7f2b8aaedbb4fb0a128ebb4ee03f030
    api_version: "3"
    auth_host: keystone-int.acme.com
    auth_port: "35357"
    auth_protocol: https
    created_roles: Member
    egress-subnets: 10.123.45.7/32
    ingress-address: 10.123.45.7
    internal_host: keystone-int.acme.com
    internal_port: "5000"
    internal_protocol: https
    private-address: 10.123.45.7
    region: RegionOne
    service_host: keystone.acme.com
    service_port: "5000"
    service_protocol: https
  UnitId: openstack-dashboard/0
- Stdout: |
    admin_domain_id: c7f2b8aaedbb4fb0a128ebb4ee03f030
    api_version: "3"
    auth_host: keystone-int.acme.com
    auth_port: "35357"
    auth_protocol: https
    created_roles: Member
    egress-subnets: 10.123.45.7/32
    ingress-address: 10.123.45.7
    internal_host: keystone-int.acme.com
    internal_port: "5000"
    internal_protocol: https
    private-address: 10.123.45.7
    region: RegionOne
    service_host: keystone.acme.com
    service_port: "5000"
    service_protocol: https
  UnitId: openstack-dashboard/2
- Stdout: |
    admin_domain_id: c7f2b8aaedbb4fb0a128ebb4ee03f030
    api_version: "3"
    auth_host: keystone-int.acme.com
    auth_port: "35357"
    auth_protocol: https
    created_roles: Member
    egress-subnets: 10.123.45.7/32
    ingress-address: 10.123.45.7
    internal_host: keystone-int.acme.com
    internal_port: "5000"
    internal_protocol: https
    privat...