User assigned admin role gets 403 when querying various object types.

[Andy Speagle]

Our users, having been assigned admin role on domain and projects in that domain we're unable to query certain things via the openstack CLI. Ex:

$ openstack user list
You are not authorized to perform the requested action: identity:list_users. (HTTP 403) (Request-ID: req-0c479c91-636d-4b74-b4d1-d18bd1ca4761)
$ openstack group list
You are not authorized to perform the requested action: identity:list_groups. (HTTP 403) (Request-ID: req-c10c217b-a730-4b8c-90f2-daad2d9dc4cb)
$ openstack domain list
You are not authorized to perform the requested action: identity:list_domains. (HTTP 403) (Request-ID: req-5b1d5007-f9dd-4149-bc1c-182f7a0c88b2)
$ openstack role assignment list
You are not authorized to perform the requested action: identity:list_role_assignments. (HTTP 403) (Request-ID: req-a10ff2cb-cb24-4447-b962-6e8b6bd8afd9)

I can view projects however... which is interesting. Our users are granted admin on the domain and projects via group membership.

We're running keystone 17.0.1 in Ussuri.

[Alex Kavanagh (ajkavanagh)]

Re-assigning to the keystone identity package.

[Alex Kavanagh (ajkavanagh)]

Could you verify whether you are running keystone via (Juju) Charmed OpenStack, the packages as part of another deployment system (Ansible, Triple-O, etc.) or the bare packages. Just want to work out where to put this bug/request. Thanks.

[Andy Speagle (aspeagle)]

Aye... this is all part of our charmed openstack deployment via juju.

[Felipe Reyes (freyes)]

Hi Andy, can you share with us the commands used to create the user and assign it the role?, is this part of a domain by any chance?.

[Andy Speagle (aspeagle)]

Yeah, these users are members of an AD group that has been assigned the "admin" role. I'm not sure I have the commands that were used to do the assigning of this role as that has likely been lost to the ether, having been done by another colleague a few years ago.

[Alex Kavanagh (ajkavanagh)]

Okay, so now I know that it's a charmed openstack keystone, I'm fairly certain it should work according to the default policy. From the default policy file (just the relevant bits):

    "admin_required": "role:{{ admin_role }}",
    "cloud_admin": "rule:admin_required and (is_admin_project:True or domain_id:{{ admin_domain_id }} or project_id:{{ service_tenant_id }})",

    "admin_and_matching_user_domain_id": "rule:admin_required and domain_id:%(user.domain_id)s",
    "identity:list_users": "rule:cloud_admin or rule:admin_and_matching_domain_id",

    "identity:list_groups": "rule:cloud_admin or rule:admin_and_matching_domain_id",

    "identity:list_domains": "rule:cloud_admin",

    "admin_on_domain_filter": "rule:admin_required and domain_id:%(scope.domain.id)s",
    "admin_on_project_filter": "rule:admin_required and project_id:%(scope.project.id)s",
    "admin_on_domain_of_project_filter": "rule:admin_required and domain_id:%(target.project.domain_id)s",
    "identity:list_role_assignments": "rule:cloud_admin or rule:admin_on_domain_filter or rule:admin_on_project_filter",

So in theory, the list_users target (openstack user list) the user needs to have the admin_role (which would be 'admin') and the domain being listed should match the users' domain. Listing groups should work the same way.

Listing domains, is only available to the cloud admin; i.e. just having the admin role wouldn't grant that ability; the user would also need to be in the admin domain.

And listing the role assignments, would be having the admin role, and either being in the domain or project.

So, off the top of my head, does "openstack user list --domain <my-domain>" work for the users in question?

Could you also check the roles for the users in keystone. You mention AD (Active Directory) as well?

[Andy Speagle (aspeagle)]

openstack user list --domain <domain> also throws a 403.

I confirmed that my policy.json on keystone/leader looks like this above mostly... this one thing is different... it seems to be hard-coded.

"admin_required": "role:Admin",

When I go through the UI ... I see that our user group has the Admin role selected. This is an Active Directory based group.

Some other inconsistencies... such as ... 'openstack project list' returns a list of projects for my user... but an 'openstack project list --domain <my-domain>' throws a 403.

[Andy Speagle (aspeagle)]

Just to be certain... and remove the group membership from consideration, I added my own user, pulled from AD, as an admin at both the domain and project level... but, sadly, no change in behavior.

[Alex Kavanagh (ajkavanagh)]

Some other things worth checking:

* the juju config for admin-user and admin-role
* Verify the list of roles includes the admin-role from above. i.e. there may be two roles "Admin" and "admin", which might be causing confusion.
* Can your create a user not related to AD (i.e. just inside OpenStack) and see if it is similarly affected?

I'd like to rule in/out if it's AD that's causing the problem (keystone-ldap?) and also make sure that there aren't extra roles or groups with capitalisation that might be causing the problem.

Thanks.

[Andy Speagle (aspeagle)]

I only see one role... "Admin" ... I tried creating a new user in my domain via the horizon UI... but, it won't let me... even as the default admin user. I wonder if that's because users/groups for the domain are locked to AD.

"Error: Unable to create user."

My juju keystone config looks like this:

  admin-role:
    default: Admin
    description: Admin role to be associated with admin and service users.
    source: default
    type: string
    value: Admin
  admin-user:
    default: admin
    description: Default admin user to create and manage.
    source: default
    type: string
    value: admin

[Andy Speagle (aspeagle)]

We're seeing similar behavior as users in this bug... we can do things via Horizon... but not the CLI.

https://bugs.launchpad.net/charms/+source/keystone/+bug/1649106

[Alex Kavanagh (ajkavanagh)]

@aspeagle Are you able to explore the database as per: https://bugs.launchpad.net/charms/+source/keystone/+bug/1649106/comments/8

It would be good to know if it's a misconfiguration (and something the charm has done) or if it's inherently an issue within Keystone (ussuri).

[Andy Speagle (aspeagle)]

Yeah, I can look at the DB... this is what I see:

select a.type, c.name, d.name
    -> from assignment a, local_user b, role c, project d
    -> where a.actor_id = b.user_id
    -> and b.name = 'admin'
    -> and c.id = a.role_id
    -> and d.id = a.target_id;
+-------------+--------+--------------+
| type | name | name |
+-------------+--------+--------------+
| UserProject | Admin | admin |
| UserProject | member | admin |
| UserDomain | Admin | <our domain> |
| UserDomain | Admin | admin_domain |
+-------------+--------+--------------+
4 rows in set (0.00 sec)

[Alex Kavanagh (ajkavanagh)]

@aspeagle; just circling back to this. I've noticed that the local_user.name is 'member' rather that 'Member' (cf 'Admin' in the 'b' column). I'm pretty sure the keystone charm sets up 'Member' rather than 'member', so this *may* be a pointer to the issue?

[Launchpad Janitor (janitor)]

[Expired for OpenStack Keystone Charm because there has been no activity for 60 days.]

[Li Li (li555li)]

I am running into the same issue in a newly deployed openstack (Bobcat) cluster using juju. @aspeagle or @ajkavanagh: can you kindly tell me how to connect to the keystone mysql database and run query to verify? In my deployment, even run "openstack user list" with the built-in "admin" user, it only returns the admin user itself. Thanks for your help!

[Alex Kavanagh (ajkavanagh)]

@li555li

You can try something similar to:

$ juju run -u <mysql-unit>/<num> -- leader-get | grep "mysql.passwd"
$ juju ssh <mysql-unit>/<num>

then on the unit.

$ mysql -u root -p
<paste the password from the first command>

That should get you into the mysql database.