Moving/disabling LDAP users break Keystone queries depending on role ID

Bug #1658641 reported by Tim S on 2017-01-23
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Medium
Kristi Nikolla

Bug Description

- ubuntu 16.04
- keystone 2:9.2.0-ubuntu1 (mitaka)
- python-openstackclient 2.3.0-2
- swift 2.7.0-0ubuntu2 (mitaka)

Hi, I got a Keystone installation with a domain using the LDAP driver to connect to AD (read-only). It's working great, and even though I don't administrate the AD the seperation hasn't been a problem until now. Primary usage is to authenticate users with Swift.

Projects and project members are more or less mapped 1:1 to specific AD groups, generated during setup. An ongoing process has been to keep this up to date with new/old employees/groups. The issue arise with the current company policy, where the user accounts of old employees is not disabled, but moved to a seperate OU. For instance:

| CN=Doe, John, OU=Users, DC=DOMAIN, DC=COM
| CN=Doe, John, OU=Former employees, DC=DOMAIN, DC=COM

Whenever this happens it seems to break the role assignment for the user. Commands such as listing users in the user's project, or looking up the user's details yields the error "Could not find resource <id>".

Does moving users in AD break the identity mapping, and thus their ID with relations stored in Keystone? Is there any possible configuration that can be done to avoid this?

--- keystone.DOMAIN.conf ----

[ldap]
url =
user =
password =
suffix =
query_scope = sub
page_size = 500
user_tree_dn =
user_objectclass = person
user_id_attribute = sAMAccountName
user_name_attribute = sAMAccountName
user_mail_attribute = mail
user_pass_attribute =
user_enabled_attribute = userAccountControl
user_enabled_mask = 2
user_enabled_default = 512
user_attribute_ignore =
user_allow_create = false
user_allow_update = false
user_allow_delete = false

[identity]
driver = ldap

--- openstack_user_list.txt ----

# user's id is listed in response to listing users that belong in a project,
# and while keystone is able to find the correct username based on id, it can't find the user itself
$ openstack user list --debug --project PROJECT --long
...

REQ: curl -g -i -X GET https://domain.com:35357/v3/role_assignments?scope.project.id=<id> -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}<token>"
"GET /v3/role_assignments?scope.project.id=<id> HTTP/1.1" 200 4401
RESP: [200] Content-Length: 4401 Vary: X-Auth-Token Keep-Alive: timeout=5, max=98 Server: Apache/2.4.18 (Ubuntu) Connection: Keep-Alive Date: Fri, 20 Jan 2017 13:54:19 GMT x-openstack-request-id req-<id> Content-Type: application/json X-Distribution: Ubuntu
RESP BODY: {"role_assignments": [{"scope": {"project": {"id": "<id>"}}, "role": {"id": "<id>"}, "user": {"id": "<id>"}, "links": {"assignment": "https://domain.com:35357/v3/projects/<id>/users/<id>/roles/<id>"}}, ...
...

REQ: curl -g -i -X GET https://domain.com:35357/v3/users/<id> -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}<token>"
"GET /v3/users/<id> HTTP/1.1" 404 89
RESP: [404] Content-Length: 89 Vary: X-Auth-Token Keep-Alive: timeout=5, max=89 Server: Apache/2.4.18 (Ubuntu) Connection: Keep-Alive Date: Fri, 20 Jan 2017 13:54:19 GMT x-openstack-request-id: req-<id> Content-Type: application/json X-Distribution: Ubuntu
RESP BODY: {"error": {"message": "Could not find user: <username>", "code": 404, "title": "Not Found"}}
...

Could not find resource <id>

Lance Bragstad (lbragstad) wrote :

I did some checking with other keystone developers that are a bit more familiar with keystone+LDAP integration [0]. It sounds like the short answer is that keystone doesn't support inspecting group assignment changes on the LDAP side.

We could clarify that in our LDAP documentation though.

[0] http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-keystone.2017-01-23.log.html#t2017-01-23T20:20:56

tags: added: documentation
Tim S (timss) wrote :

Thank you Lance, that's very helpful and pretty much what I expected.

Looking at the existing documentation of deleting users[1], it would seem that purging specific Local IDs (of users) using keystone-manage is one way to go. Alternatively, purge the entire domain and repopulate the domain right afterwards.

| While keystone will dynamically maintain the identity mapping, including
| removing entries when entities are deleted via the keystone, for those
| entities in backends that are managed outside of keystone (e.g. a read-only
| LDAP), keystone will not know if entities have been deleted and hence will
| continue to carry stale identity mappings in its table. While benign,
| keystone provides an ability for operators to purge the mapping table of such
| stale entries using the keystone-manage command, for example:
|
| $ keystone-manage mapping_purge --domain-name DOMAINA --local-id <email address hidden>

Looking at the populate function however, it does not seem to support populating only specific entries [2]. Would it be ok to run a full repopulate even after only purging specific entries? Or would they be regenerated automatically?

I think that only minor clarifications are needed in the documentation as it is already quite good. It could possibly underline that any changes to DNs for already mapped entities would break the mapping, not just deleting entities. Depending on how you suggest to repopulate purged specific entries, maybe also some words about that if you feel it's necessary.

Suppose this is a bug for python-keystoneclient, but it would also be nice if it was able to still list the users in a project even if one or more users have invalid mappings. Currently it breaks ungracefully, even if it's able to request information about most users.

[1]: http://docs.openstack.org/developer/keystone/configuration.html#domain-specific-drivers
[2]: https://github.com/openstack/keystone/blob/f2d0f8c9ab38172a6e37b02339eac59da911435c/keystone/cmd/cli.py#L1239

Lance Bragstad (lbragstad) wrote :

I did some digging and this is what I found. The `keystone-manage mapping_populate` command is simply a wrapper that makes a call to list all users [0], since the responsibility to map local and public ID falls within the various layers of the identity API. `keystone-manage` only passes the domain ID of the domain to populate, which in turn is passed to the identity APIs list users method [1]. This is all pretty straight forward, and before we exit that method, we have a list of references [2] from LDAP (in the multi-domain case). This is passed to _set_domain_id_and_mapping() [3], which handles parsing each entity in the list. At the same time, it looks up all current id_mappings (one should exist for every user) and compares them to the list of existing entities. If a reference, recently pulled from LDAP, matches a mapping from the id_mapping table, we pull the public id from the id_mapping and update the reference [4]. At this point, any entity that has already been mapped has its corresponding public ID. Then we make another pass through the list to populate any public ID for entities that haven't been mapped (i.e. these would be the ones that you just explicitly removed using the `keystone-manage mapping_purge`` command) [5][6]. If there isn't a public ID in the entity, we will go ahead and create one from attributes [7].

So, to answer your question. You should be able to remove explicit mappings with the ``keystone-manage mapping_purge`` command, and when you go to repopulate them, existing reference should maintain their original public IDs.

As far as the python-keystoneclient bug, do you have a trace, or are you referencing the one from the description (which would be a server issue I believe)?

Hopefully this helps.

[0] https://github.com/openstack/keystone/blob/f2d0f8c9ab38172a6e37b02339eac59da911435c/keystone/cmd/cli.py#L1258
[1] https://github.com/openstack/keystone/blob/f2d0f8c9ab38172a6e37b02339eac59da911435c/keystone/identity/core.py#L963
[2] https://github.com/openstack/keystone/blob/f2d0f8c9ab38172a6e37b02339eac59da911435c/keystone/identity/core.py#L975
[3] https://github.com/openstack/keystone/blob/f2d0f8c9ab38172a6e37b02339eac59da911435c/keystone/identity/core.py#L522
[4] https://github.com/openstack/keystone/blob/f2d0f8c9ab38172a6e37b02339eac59da911435c/keystone/identity/core.py#L648
[5] https://github.com/openstack/keystone/blob/f2d0f8c9ab38172a6e37b02339eac59da911435c/keystone/identity/core.py#L654-L659
[6] https://github.com/openstack/keystone/blob/f2d0f8c9ab38172a6e37b02339eac59da911435c/keystone/identity/core.py#L580-L588
[7] https://github.com/openstack/keystone/blob/f2d0f8c9ab38172a6e37b02339eac59da911435c/keystone/identity/mapping_backends/sql.py#L74-L76

Tim S (timss) wrote :
Download full text (5.6 KiB)

Great explaination, and thanks for looking into this.

Unfortunately I've not been able to verify if a purge was successful yet due to residing role assignment issues. Using LDAP, should the local ID be "<email address hidden>" (typically capital letters) or just "USERNAME"?

Listing users still yield the error described below. Trying to remove the role assignment using the user's ID (`openstack role remove`) fails for the same reason. Could it be that the mapping is purged successfully, but the role assignment remains in a sort stale state? In that case, how should I proceed?

---

Regarding the possible python-keystoneclient bug I am refering to the one included in part in the bug description. I'm using python-openstackclient, but if I'm not mistaken these operations are a part of python-keystoneclient[1]? Chose to initially not include too much of the trace due to its verbosity and because it might be a seperate bug.

In short, when users are listed with `openstack user list --project PROJECT` or similar, the command seems to issue these API requests but fails at step 4 since it can't find an user with the ID referenced in the role assignments.

1. Authenticate/issue token (if applicable)
2. Get details about the project via `/v3/projects?name=PROJECT` (or by ID)
3. Using now acquired project ID, it asks for the role assignments in the scope of the project via `/v3/role_assignments?scope.project.id=ID`
4. Iterate through and get info about each user found in role assignments via `/v3/users/ID`
5. Print (list) users in the project

Step 4 and out for the user not found (`openstack user list --debug --project PROJECT --long`):

---

REQ: curl -g -i -X GET https://domain.com:35357/v3/users/<ID> -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}<token>"
"GET /v3/users/<ID> HTTP/1.1" 404 89
RESP: [404] Content-Length: 89 Vary: X-Auth-Token Keep-Alive: timeout=5, max=89 Server: Apache/2.4.18 (Ubuntu) Connection: Keep-Alive Date: Fri, 20 Jan 2017 13:54:19 GMT x-openstack-request-id: req-<ID> Content-Type: application/json X-Distribution: Ubuntu
RESP BODY: {"error": {"message": "Could not find user: <username>", "code": 404, "title": "Not Found"}}

Request returned failure status: 404
REQ: curl -g -i -X GET https://domain.com:35357/v3/users?name=<ID> -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}<token>"
"GET /v3/users?name=<ID> HTTP/1.1" 401 114
RESP: [401] Content-Length: 114 Vary: X-Auth-Token Keep-Alive: timeout=5, max=88 X-Distribution: Ubuntu Connection: Keep-Alive Date: Fri, 20 Jan 2017 13:54:19 GMT WWW-Authenticate: Keystone uri="https://domain.com:35357" Server: Apache/2.4.18 (Ubuntu) Content-Type: application/json x-openstack-request-id: req-<ID>
RESP BODY: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}

Request returned failure status 401
REQ: curl -g -i -X GET https://domain.com:35357/v3/users -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}<token>"
"GET /v3/users HTTP/1.1" 401 114
RESP: [401] Content-Length: 114 Vary: X-Auth-Token Keep-Al...

Read more...

Tim S (timss) wrote :

Didn't realize how unreadable launchpad made the trace, so here's a paste[1] of the `--debug` output instead. Again, everything is normal up to this point (see steps), so I chose to leave that out for now.

[1]: http://paste.openstack.org/show/596876/

Matthew Edmonds (edmondsw) wrote :

based on the config you have in the description, I believe a user's local_id would be that user's sAMAccountName.

Andreas Karis (akaris) wrote :
Download full text (7.6 KiB)

Hi,

The same happens if users are deleted or their group membership is changed.

Users that are auto removed from LDAP break roles and cant easily be removed.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Followed: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/9/html/integrate_with_identity_service/sec-active-directory

User test was added to Active Directory:
~~~
[stack@undercloud-6 ~]$ openstack user list --domain redhat
+------------------------------------------------------------------+----------+
| ID | Name |
+------------------------------------------------------------------+----------+
| 853a331554ea0fb6e938f39256beb9f8096625c29f34bc8d88990b4198205f90 | svc-ldap |
| 82ec6ba7034541d55349c62705f750634a1d0d680386444dbe0f7ffd9f15b032 | akaris |
| 39e5b866156f05d6b3f95409a663a44718bec62eeabc9ec6f08ff78ef5fd457d | nalmond |
| f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 | test |
+------------------------------------------------------------------+----------+
[stack@undercloud-6 ~]$ openstack project create demo
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | |
| domain_id | default |
| enabled | True |
| id | 1c3e304811d8457a871a6c67f6f63a75 |
| is_domain | False |
| name | demo |
| parent_id | default |
+-------------+----------------------------------+
[stack@undercloud-6 ~]$ openstack role add --project demo --user f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 _member_
[stack@undercloud-6 ~]$ openstack role assignment list | grep 1c3e304811d8457a871a6c67f6f63a75
| 9fe2ff9ee4384b1894a90878d3e92bab | f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 | | 1c3e304811d8457a871a6c67f6f63a75 | | False |
[stack@undercloud-6 ~]$ openstack role assignment list --names
+---------------+------------------------------------+-------+-----------------+------------+-----------+
| Role | User | Group | Project | Domain | Inherited |
+---------------+------------------------------------+-------+-----------------+------------+-----------+
| admin | cinderv2@Default | | service@Default | | False |
| _member_ | cinderv2@Default | | service@Default | | False |
| admin | ceilometer@Default | | service@Default | | False |
| _member_ | ceilometer@Default | | service@Default | | False |
| ResellerAdmin | ceilometer@Default | | service@Default | | False |
| admin | admin@Default | | admin@Default | | False |
| admin |...

Read more...

Andreas Karis (akaris) wrote :
Download full text (6.9 KiB)

Looking at the database, the id_mapping table and the assignment table seem to be the only ones holding references to this user (given the test which I ran, meaning that I assigned a role to the user)
~~~
[root@overcloud-controller-0 domains]# mysql keystone -e 'show tables;' | awk '{print $1}' | while read t;do echo "XXXXX $t XXXXX"; mysql keystone -e "select * from $t \G" | grep f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 -C5 ; done
XXXXX Tables_in_keystone XXXXX
ERROR 1146 (42S02) at line 1: Table 'keystone.Tables_in_keystone' doesn't exist
XXXXX access_token XXXXX
XXXXX assignment XXXXX
target_id: dfc01178c51b4688be78188b5e8c9581
  role_id: 9fe2ff9ee4384b1894a90878d3e92bab
inherited: 0
*************************** 22. row ***************************
     type: UserProject
 actor_id: f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2
target_id: 1c3e304811d8457a871a6c67f6f63a75
  role_id: 9fe2ff9ee4384b1894a90878d3e92bab
inherited: 0
*************************** 23. row ***************************
     type: UserProject
XXXXX config_register XXXXX
XXXXX consumer XXXXX
XXXXX credential XXXXX
XXXXX domain XXXXX
XXXXX endpoint XXXXX
XXXXX endpoint_group XXXXX
XXXXX federated_user XXXXX
XXXXX federation_protocol XXXXX
XXXXX group XXXXX
ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'group' at line 1
XXXXX id_mapping XXXXX
  public_id: 853a331554ea0fb6e938f39256beb9f8096625c29f34bc8d88990b4198205f90
  domain_id: 210bdf7974a14693843ec7f9b1956105
   local_id: svc-ldap
entity_type: user
*************************** 4. row ***************************
  public_id: f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2
  domain_id: 210bdf7974a14693843ec7f9b1956105
   local_id: test
entity_type: user
XXXXX identity_provider XXXXX
XXXXX idp_remote_ids XXXXX
XXXXX implied_role XXXXX
XXXXX local_user XXXXX
XXXXX mapping XXXXX
XXXXX migrate_version XXXXX
XXXXX password XXXXX
XXXXX policy XXXXX
XXXXX policy_association XXXXX
XXXXX project XXXXX
XXXXX project_endpoint XXXXX
XXXXX project_endpoint_group XXXXX
XXXXX region XXXXX
XXXXX request_token XXXXX
XXXXX revocation_event XXXXX
XXXXX role XXXXX
XXXXX sensitive_config XXXXX
XXXXX service XXXXX
XXXXX service_provider XXXXX
XXXXX token XXXXX
XXXXX trust XXXXX
XXXXX trust_role XXXXX
XXXXX user XXXXX
XXXXX user_group_membership XXXXX
XXXXX whitelisted_config XXXXX
~~~

Before cleanup:
~~~
[stack@undercloud-6 ~]$ openstack role assignment list | grep f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2
| 9fe2ff9ee4384b1894a90878d3e92bab | f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 | | 1c3e304811d8457a871a6c67f6f63a75 | | False |
~~~

In order to clean up the mapping:
~~~
[root@overcloud-controller-0 domains]# mysql keystone -e 'delete from assignment where actor_id="f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2"'
[root@overcloud-controller-0 domains]#
~~~

After this:
~~~
[stack@undercloud-6 ~]$ openstack role assignment list | grep f3f3e1b1c01c79299154f85f...

Read more...

Andreas Karis (akaris) wrote :

Note that I recreated user test in AD:

Output from today:
~~~
[stack@undercloud-6 ~]$ openstack user list --domain redhat
+------------------------------------------------------------------+----------+
| ID | Name |
+------------------------------------------------------------------+----------+
| 853a331554ea0fb6e938f39256beb9f8096625c29f34bc8d88990b4198205f90 | svc-ldap |
| 82ec6ba7034541d55349c62705f750634a1d0d680386444dbe0f7ffd9f15b032 | akaris |
| 39e5b866156f05d6b3f95409a663a44718bec62eeabc9ec6f08ff78ef5fd457d | nalmond |
| f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 | test |
+------------------------------------------------------------------+----------+
~~~

Output from yesterday:
~~~
[stack@undercloud-6 ~]$ openstack user list --domain redhat
+------------------------------------------------------------------+----------+
| ID | Name |
+------------------------------------------------------------------+----------+
| 853a331554ea0fb6e938f39256beb9f8096625c29f34bc8d88990b4198205f90 | svc-ldap |
| 82ec6ba7034541d55349c62705f750634a1d0d680386444dbe0f7ffd9f15b032 | akaris |
| 39e5b866156f05d6b3f95409a663a44718bec62eeabc9ec6f08ff78ef5fd457d | nalmond |
| f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 | test |
+------------------------------------------------------------------+----------+
~~~

"The public ID is automatically generated by keystone when it first encounters the entity. If the local ID of the entity is from a backend that does not guarantee to generate UUIDs, a hash algorithm will generate a public ID for that entity, which is what will be exposed by keystone."

The interesting observation here is that UUIDs are the same, so by recreating a deleted user with the same user name (sAMAccountname ???) one could "fix" keystone. Just a quick observation.

Andreas Karis (akaris) wrote :
Download full text (4.7 KiB)

The same issue by the way happens when a user's membership is changed and thus removes him from the domain:

[stack@undercloud-6 ~]$ ^C
[stack@undercloud-6 ~]$ openstack role add --project demo --user f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 _member_
[stack@undercloud-6 ~]$ openstack role assignment list --names
+---------------+------------------------------------+-------+-----------------+------------+-----------+
| Role | User | Group | Project | Domain | Inherited |
+---------------+------------------------------------+-------+-----------------+------------+-----------+
| admin | cinderv2@Default | | service@Default | | False |
| _member_ | cinderv2@Default | | service@Default | | False |
| admin | ceilometer@Default | | service@Default | | False |
| _member_ | ceilometer@Default | | service@Default | | False |
| ResellerAdmin | ceilometer@Default | | service@Default | | False |
| admin | admin@Default | | admin@Default | | False |
| admin | nova@Default | | service@Default | | False |
| _member_ | nova@Default | | service@Default | | False |
| admin | glance@Default | | service@Default | | False |
| _member_ | glance@Default | | service@Default | | False |
| admin | neutron@Default | | service@Default | | False |
| _member_ | neutron@Default | | service@Default | | False |
| admin | sahara@Default | | service@Default | | False |
| _member_ | sahara@Default | | service@Default | | False |
| admin | gnocchi@Default | | service@Default | | False |
| _member_ | gnocchi@Default | | service@Default | | False |
| ResellerAdmin | gnocchi@Default | | service@Default | | False |
| admin | swift@Default | | service@Default | | False |
| _member_ | swift@Default | | service@Default | | False |
| admin | aodh@Default | | service@Default | | False |
| _member_ | aodh@Default | | service@Default | | False |
| _member_ | test@redhat | | demo@Default | | False |
| admin | cinder@Default | | service@Default | | False |
| _member_ | cinder@Default | | service@Default | | False |
...

Read more...

Andreas Karis (akaris) wrote :
Download full text (5.1 KiB)

I deleted the user to test the mapping_purge ...

~~~
[stack@undercloud-6 ~]$ openstack user list --domain redhat
+------------------------------------------------------------------+----------+
| ID | Name |
+------------------------------------------------------------------+----------+
| 853a331554ea0fb6e938f39256beb9f8096625c29f34bc8d88990b4198205f90 | svc-ldap |
| 82ec6ba7034541d55349c62705f750634a1d0d680386444dbe0f7ffd9f15b032 | akaris |
| 39e5b866156f05d6b3f95409a663a44718bec62eeabc9ec6f08ff78ef5fd457d | nalmond |
+------------------------------------------------------------------+----------+
[stack@undercloud-6 ~]$ openstack role assignment list | grep 1c3e304811d8457a871a6c67f6f63a75
| 9fe2ff9ee4384b1894a90878d3e92bab | 82ec6ba7034541d55349c62705f750634a1d0d680386444dbe0f7ffd9f15b032 | | 1c3e304811d8457a871a6c67f6f63a75 | | False |
| 9fe2ff9ee4384b1894a90878d3e92bab | f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 | | 1c3e304811d8457a871a6c67f6f63a75 | | False |
[stack@undercloud-6 ~]$ openstack role assignment list --names
Could not find user: f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 (HTTP 404) (Request-ID: req-dacdaa34-d07c-40f7-ac7c-2864ca6d6608)
~~~

Note that this does not work:

~~~
[root@overcloud-controller-0 ~]# keystone-manage mapping_purge --domain-name redhat
[root@overcloud-controller-0 ~]#
~~~

~~~
[stack@undercloud-6 ~]$ openstack role assignment list | grep 1c3e304811d8457a871a6c67f6f63a75
| 9fe2ff9ee4384b1894a90878d3e92bab | 82ec6ba7034541d55349c62705f750634a1d0d680386444dbe0f7ffd9f15b032 | | 1c3e304811d8457a871a6c67f6f63a75 | | False |
| 9fe2ff9ee4384b1894a90878d3e92bab | f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 | | 1c3e304811d8457a871a6c67f6f63a75 | | False |
[stack@undercloud-6 ~]$ openstack role assignment list --names
Could not find user: f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 (HTTP 404) (Request-ID: req-e53543a3-2164-4fa1-a86b-55a38d199d57)
~~~

I also tried `keystone-manage mapping_purge --domain-name redhat --local-id f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2`

Yes,. it purges the mapping table, but it does not touch the assignment table:
~~~
[root@overcloud-controller-0 ~]# mysql keystone -e 'show tables;' | awk '{print $1}' | while read t;do echo "XXXXX $t XXXXX"; mysql keystone -e "select * from $t \G" | grep f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2 -C5 ; done
XXXXX Tables_in_keystone XXXXX
ERROR 1146 (42S02) at line 1: Table 'keystone.Tables_in_keystone' doesn't exist
XXXXX access_token XXXXX
XXXXX assignment XXXXX
target_id: dfc01178c51b4688be78188b5e8c9581
  role_id: 9fe2ff9ee4384b1894a90878d3e92bab
inherited: 0
*************************** 23. row ***************************
     type: UserProject
 actor_id: f3f3e1b1c01c79299154f85f0821ceb0f7c149de8d9836f86eceaaa38e9f27c2
target_id: 1c3e304811d8457a871a6c67f6f63a75
  role_id: 9fe...

Read more...

Andreas Karis (akaris) wrote :
Download full text (5.0 KiB)

I tested this again against OSP 11 (Ocata):
~~~
[stack@undercloud-8 ~]$ openstack user list --domain redhat
+------------------------------------------------------------------+----------+
| ID | Name |
+------------------------------------------------------------------+----------+
| 80fc5502450b25ec0691dccc27307efc65ebb5c7f54db131e120f4a8bb6cecf0 | svc-ldap |
| 3edee38198325e08fd8795862575196802ac7572bc1b00d0f0dade56d674a7e3 | akaris |
| f612704506d90104aebdc0989f6120a8db166ad0e80b9990262ea32487209d31 | nalmond |
| 6992dec6afa26e9faac16e0f56748a6b492ada373c21eaad919af87cc89a831e | test |
+------------------------------------------------------------------+----------+
[stack@undercloud-8 ~]$ openstack role add --project demo --user 6992dec6afa26e9faac16e0f56748a6b492ada373c21eaad919af87cc89a831e _member_
[stack@undercloud-8 ~]$ openstack role assignment list --names
+---------------+------------------------------------+-------+-----------------+------------+-----------+
| Role | User | Group | Project | Domain | Inherited |
+---------------+------------------------------------+-------+-----------------+------------+-----------+
| admin | nova@Default | | service@Default | | False |
| admin | aodh@Default | | service@Default | | False |
| admin | glance@Default | | service@Default | | False |
| admin | heat-cfn@Default | | service@Default | | False |
| admin | swift@Default | | service@Default | | False |
| _member_ | test@redhat | | demo@Default | | False |
| ResellerAdmin | ceilometer@Default | | service@Default | | False |
| admin | ceilometer@Default | | service@Default | | False |
| admin | panko@Default | | service@Default | | False |
| admin | gnocchi@Default | | service@Default | | False |
| admin | heat@Default | | service@Default | | False |
| admin | neutron@Default | | service@Default | | False |
| admin | admin@Default | | admin@Default | | False |
| admin | placement@Default | | service@Default | | False |
| admin | cinder@Default | | service@Default | | False |
| admin | heat_stack_domain_admin@heat_stack | | | heat_stack | False |
| admin | admin@Default | | | redhat | False |
+---------------+------------------------------------+-------+-----------------+------------+-----------+
[stack@undercloud-8 ~]$ ...

Read more...

Tim S (timss) wrote :

Thanks for sharing your experiences and tests, @akaris. This is very useful!
Especially the pointers on how to delete the entries manually from the databse and its effect.

Unfortunately I haven't been able to look further into this issue in a while, although it's still a problem in the Keystone installation I'm running. I'd like to put some more time into resolving it, but not sure when as it's been too easy to put into the backlog for now.

Anyway, thanks for the input and possibly confirming the bug.

Lance Bragstad (lbragstad) wrote :

It sounds like a possible solution would be to enhance the ``keystone-manage`` tool. What we could do is add a path to it that accepts a user_id, check to make sure it doesn't exist in the identity backend, and then forcibly removes any role assignments that user has.

Changed in keystone:
status: New → Triaged
importance: Undecided → Medium

Fix proposed to branch: master
Review: https://review.openstack.org/487579

Changed in keystone:
assignee: nobody → Kristi Nikolla (knikolla)
status: Triaged → In Progress
Colleen Murphy (krinkle) on 2018-01-24
Changed in keystone:
milestone: none → queens-rc1
Lance Bragstad (lbragstad) wrote :

We're probably going to have to cut queens-rc1 without this fix. If we get a patch landed we can backport and cut an rc2.

Changed in keystone:
milestone: queens-rc1 → queens-rc2
Lance Bragstad (lbragstad) wrote :

Thinking about this a bit more and it seems we have two different options when using the mapping purge.

1.) We can have keystone-manage mapping_purge accept an argument for a specific user ID. If that user id isn't in a specific domain backend, then it should be removed from the id_mapping table, similiar to what happens today. In addition to the path, keystone could remove the role assignment for that ID if they exist in the assignment tables.

2.) We can add another argument to the mapping_purge command that forces the removal of all assignments associated to purged identities (essentially option #1 but batch processing).

My concern for both approach is that id_mappings can be regenerated because keystone pulls the identity from the domain backend (LDAP/AD/etc...), which it treats as the source of truth. Since it's possible for users to have assignments, when we remove them, there is no recovery option if the users is enabled or recreated in the domain backend. In a way, this makes mapping_purge less idempotent because;

- user is create in ldap
- user is mapped in keystone and given an id based on a hash
- user is granted assignments to various projects
- user is disabled or moves groups in ldap
- the users id and mapping are purged in keystone, along with all of the user's assignments
- the user is enabled or moved back to the original group, the mapping backend maps the user to their previous ID
- user doesn't have any of the assignment they had prior to the purge operation

Is this going to be an issue?

Kristi Nikolla (knikolla) wrote :

Played around for a bit.

1. Created a user in ldap
2. Listed users in the domain Users, new user correctly showed
3. Assigned Member role to that user on project demo
4. Deleted the user directly in ldap
5. Cleaned mappings and restarted keystone to clear cache

http://paste.openstack.org/show/670540/

In the above paste, only the last command unexpectedly fails. That is because the `openstack user list --project <project>` command does the following:

1. It queries the API /role_assignment?scope.project.id=<project> # Works
2. From the list returned above it does /users?name=<user_id> # Works
3. For a user which doesn't exist, the above command will return an empty list # Works
4. If the above call returned an empty list, the openstackclient will throw an exception.

Adam Young (ayoung) wrote :

Sounds like it is a valid return code and message and an error on the part of the client code, then. Should we refile the bug against OSC?

Lance Bragstad (lbragstad) wrote :

Removing this as a queens rc target based on comment number #18. Kristi, thanks for recreating this.

Changed in keystone:
milestone: queens-rc2 → none
tags: added: office-hours
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers