Keystone is unable to remove role-assignment for deleted LDAP users

[Eigil Obrestad]

We are experiencing issues when trying to remove role-assignments for users that were in our LDAP catalog, but now is deleted. We use LDAP as a read-only source of usernames/passwords, and the rest is stored in keystones mysql database. We are currently running keystone version 15.0.0-0ubuntu1.2~cloud0.

When listing role-assignments we get something like this:

$ openstack role assignment list --project project --names
+------------------+-----------------------+-------+---------------------+--------+--------+-----------+
| Role | User | Group | Project | Domain | System | Inherited |
+------------------+-----------------------+-------+---------------------+--------+--------+-----------+
| _member_ | @ | | project@LDAP-DOMAIN | | | False |
| _member_ | @ | | project@LDAP-DOMAIN | | | True |
| heat_stack_owner | @ | | project@LDAP-DOMAIN | | | False |
| heat_stack_owner | @ | | project@LDAP-DOMAIN | | | True |
| _member_ | @ | | project@LDAP-DOMAIN | | | False |
| _member_ | @ | | project@LDAP-DOMAIN | | | True |
| heat_stack_owner | @ | | project@LDAP-DOMAIN | | | False |
| heat_stack_owner | @ | | project@LDAP-DOMAIN | | | True |
| _member_ | defaultuser@Default | | project@LDAP-DOMAIN | | | False |
| _member_ | defaultuser@Default | | project@LDAP-DOMAIN | | | True |
| heat_stack_owner | defaultuser@Default | | project@LDAP-DOMAIN | | | False |
| heat_stack_owner | defaultuser@Default | | project@LDAP-DOMAIN | | | True |
| _member_ | ldapuser@LDAP-DOMAIN | | project@LDAP-DOMAIN | | | False |
| _member_ | ldapuser@LDAP-DOMAIN | | project@LDAP-DOMAIN | | | True |
| heat_stack_owner | ldapuser@LDAP-DOMAIN | | project@LDAP-DOMAIN | | | False |
| heat_stack_owner | ldapuser@LDAP-DOMAIN | | project@LDAP-DOMAIN | | | True |
+------------------+-----------------------+-------+---------------------+--------+--------+-----------+

Here the first 8 lines represents roles to two different users deleted from LDAP. Listing the assignments without --names will give me the users ID's:

$ openstack role assignment list --project project --names
+----------------------------------+------------------------------------------------------------------+-------+----------------------------------+--------+--------+-----------+
| Role | User | Group | Project | Domain | System | Inherited |
+----------------------------------+------------------------------------------------------------------+-------+----------------------------------+--------+--------+-----------+
| 9fe2ff9ee4384b1894a90878d3e92bab | 0f9389d48ed88c24656981beb9605c56346bdbf3a90420a9628db62c1e6241e5 | | 03b552df38d249f1b88a6cda1c008bcd | | | False |
| 9fe2ff9ee4384b1894a90878d3e92bab | 0f9389d48ed88c24656981beb9605c56346bdbf3a90420a9628db62c1e6241e5 | | 03b552df38d249f1b88a6cda1c008bcd | | | True |
| c5b14c3cf7014b4faa1aed52b36291f1 | 0f9389d48ed88c24656981beb9605c56346bdbf3a90420a9628db62c1e6241e5 | | 03b552df38d249f1b88a6cda1c008bcd | | | False |
| c5b14c3cf7014b4faa1aed52b36291f1 | 0f9389d48ed88c24656981beb9605c56346bdbf3a90420a9628db62c1e6241e5 | | 03b552df38d249f1b88a6cda1c008bcd | | | True |
| 9fe2ff9ee4384b1894a90878d3e92bab | 10e5b026e981c99e50dcb9c73d5860ed75a2e292acdf4cd9f3e06283820a84b0 | | 03b552df38d249f1b88a6cda1c008bcd | | | False |
| 9fe2ff9ee4384b1894a90878d3e92bab | 10e5b026e981c99e50dcb9c73d5860ed75a2e292acdf4cd9f3e06283820a84b0 | | 03b552df38d249f1b88a6cda1c008bcd | | | True |
| c5b14c3cf7014b4faa1aed52b36291f1 | 10e5b026e981c99e50dcb9c73d5860ed75a2e292acdf4cd9f3e06283820a84b0 | | 03b552df38d249f1b88a6cda1c008bcd | | | False |
| c5b14c3cf7014b4faa1aed52b36291f1 | 10e5b026e981c99e50dcb9c73d5860ed75a2e292acdf4cd9f3e06283820a84b0 | | 03b552df38d249f1b88a6cda1c008bcd | | | True |
| 9fe2ff9ee4384b1894a90878d3e92bab | abaa86ee4b7d48b2aafb1d31125108c6 | | 03b552df38d249f1b88a6cda1c008bcd | | | False |
| 9fe2ff9ee4384b1894a90878d3e92bab | abaa86ee4b7d48b2aafb1d31125108c6 | | 03b552df38d249f1b88a6cda1c008bcd | | | True |
| c5b14c3cf7014b4faa1aed52b36291f1 | abaa86ee4b7d48b2aafb1d31125108c6 | | 03b552df38d249f1b88a6cda1c008bcd | | | False |
| c5b14c3cf7014b4faa1aed52b36291f1 | abaa86ee4b7d48b2aafb1d31125108c6 | | 03b552df38d249f1b88a6cda1c008bcd | | | True |
| 9fe2ff9ee4384b1894a90878d3e92bab | bcd2364027255505c8c471139e92f75e78446641bbeb6ee18a279d730947be46 | | 03b552df38d249f1b88a6cda1c008bcd | | | False |
| 9fe2ff9ee4384b1894a90878d3e92bab | bcd2364027255505c8c471139e92f75e78446641bbeb6ee18a279d730947be46 | | 03b552df38d249f1b88a6cda1c008bcd | | | True |
| c5b14c3cf7014b4faa1aed52b36291f1 | bcd2364027255505c8c471139e92f75e78446641bbeb6ee18a279d730947be46 | | 03b552df38d249f1b88a6cda1c008bcd | | | False |
| c5b14c3cf7014b4faa1aed52b36291f1 | bcd2364027255505c8c471139e92f75e78446641bbeb6ee18a279d730947be46 | | 03b552df38d249f1b88a6cda1c008bcd | | | True |
+----------------------------------+------------------------------------------------------------------+-------+----------------------------------+--------+--------+-----------+

Trying to delete the roles for one of the deleted users simply gives me an error-message stating that the user dont exist:

$ openstack role remove --user 0f9389d48ed88c24656981beb9605c56346bdbf3a90420a9628db62c1e6241e5 --project 03b552df38d249f1b88a6cda1c008bcd _member_
No user with a name or ID of '0f9389d48ed88c24656981beb9605c56346bdbf3a90420a9628db62c1e6241e5' exists.

I have an understanding of the reasons why role-assignments and id-mappings are not cleaned automaticly to allow the user get its old roles back if it reappear; but as an adminstrator I should be able to remove a role-assignments to prevent a re-appearing user from getting access to a certain project.

[Kristi Nikolla (knikolla)]

Can you please run the `openstack role remove` command with the `--debug` flag?

I believe the above error is coming from the way that the openstack command line client checks to see if the argument provided is a name or an id, and fails because it cannot figure that out.

Can you please try to remove the role assignment directly using the REST API? See here https://docs.openstack.org/api-ref/identity/v3/#unassign-role-from-user-on-project

You can get a token using the command `openstack token issue -f value -c id` and then use that by filling in the token and ids in the below command.

`curl -i -X DELETE -H "X-Auth-Token: <token>" https://<keystone-hostname>/v3/projects/<project_id>/users/<user_id>/roles/<role_id>`

[Vishakha Agarwal (vishakha.agarwal)]

According to the description in the command openstack role remove the --user '0f9389d48ed88c24656981beb9605c56346bdbf3a90420a9628db62c1e6241e5' is deleted?

And could you please check if you passed the correct user_id since it seems invalid id?

[Eigil Obrestad (obrestad)]

Kristi: Running the openstack role command results in the following output: http://paste.openstack.org/show/788865/

(I have search/replaced a little; removing api-urls and similar, and adding bogus-values instead. I have also search/replaced usernames).

Manually deleting the user using curl seems to work fine (http://paste.openstack.org/show/788868/), so it might be a keystoneclient issue more than a keystone-server issue. The role-assignment gets removed with the curl-command.

Vishakha: As the bug initially states, the user is deleted from LDAP, while keystone still keeps an association between the now deleted user and an existing project. The bug is that this association cannot be deleted. The ID 0f9389d48ed88c24656981beb9605c56346bdbf3a90420a9628db62c1e6241e5 is correct, and that can be seen in line 62 of the paste as the openstack-command was able to find the deleted user's username (I have replaced it with REDACTED-USER-NAME).

[Colleen Murphy (krinkle)]

I was able to reproduce this behavior, which surprised me a bit because even if the user is removed from the LDAP backend there is still a shadow record of it in the keystone database. Unfortunately, if you query keystone for the user it calls to the LDAP backend to verify the user exists, so it appears to the user (and to openstackclient) that the user doesn't exist. There's no way that openstackclient could handle this better because it always must query keystone's /v3/users API to check the ID of the user.

Brainstorming ideas, options I see are 1) expose the shadow user table in the API somehow so that a query for a no-longer-existing user returns some kind of result - maybe showing the user as disabled or deleted, or 2) create a new keystone-manage command or enhance the keystone-manage mapping_purge command to purge all deleted users from the shadow table (which should cascade to delete their role assignments as well).