keystone user-role-* operations fails when user no longer exists in backend

[Mike Dorman]

When using an external user catalog (in our case, AD), if the user is removed on the backend catalog, the user-role-* keystone CLI commands no longer work, because keystone cannot look up the user.

The specific situation is a user had been granted roles on some projects, but then that user left the company and was removed from the backend directory. When going back to remove the roles assigned to that user, the keystone commands fail.

It may still be possible to do these operations directly through the API, I didn't check that. But ultimately was able to work around it by directly removing the entries in the keystone user_project_metadata table.

[Dolph Mathews (dolph)]

You're specifically looking for user-role-delete support, correct? (not -add?)

[Mike Dorman (mdorman-m)]

That's right. For our particular use case, user-role-delete is what's needed.

[Dolph Mathews (dolph)]

I've narrowed the scope of this bug to match then -- I think that's completely legitimate. We have a history of getting bug reports from confused users when we allow "invalid" data to enter the database though (bad user_id references, etc) -- you should definitely be able to clean them out though!

[Ajaya Agrawal (ajayaa)]

If they are deleting the user directly and not through the call "openstack user delete" this problem will arise. The deletion of user should also call assignment_api.delete_user().
Alternative:
We could add a periodic job which would check the table assignment and user in keystone schema and remove the assignments which have no corresponding entry in user table.

[Mike Dorman (mdorman-m)]

For our use case, we cannot enforce an "openstack user delete" and/or delete_user() call. We use AD for backend authentication, and only have read-only access to it. User creation and deletion is handled at the corporate level and from the OpenStack perspective, we have no control over that.

[Dolph Mathews (dolph)]

Keystone should attempt the delete the assignment, and only 404 if no records were affected. The input doesn't need to be discretely validated.

[Lance Bragstad (lbragstad)]

Running some tests against this for V3 and I don't think this is an issue with with V3.

http://paste.openstack.org/show/108332/

Also working on a functional test for V3

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/119843

[Lance Bragstad (lbragstad)]

This fix is going to be dependent on Brant's work here:

https://review.openstack.org/#/c/119629/6

which is gating.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/119843
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=ea31841ad4d719840c1afaa3ddf7a4fe7bd39a6c
Submitter: Jenkins
Branch: master

commit ea31841ad4d719840c1afaa3ddf7a4fe7bd39a6c
Author: Lance Bragstad <email address hidden>
Date: Fri Sep 5 20:33:11 2014 +0000

    Allow users to clean up role assignments

    This change allows users the ability to remove role assignments in the
    event the user has been removed from the backend.

    Closes-Bug: #1321378
    Change-Id: I895e6e9718bc48344a53f192d5d56e6990da888e

[Adam Young (ayoung)]

So...this is a continuing Saga. The fix that went in for Keystone only allows the V3 AP call to continue. However, there is currently no way to call that API except for CURL.

Something like:

 curl -X DELETE -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" $OS_AUTH_URL/projects/e9d504e8524e4c8d9876d179420dab89/users/tuser/roles/95a2366f8b514d43a5584342aefe448e

Will work, but there is no way to invoke that from python-keystoneclient or python-openwstackclient as both will attempt to list the users and do a lookup.

We probably need a --userid option that indicates that the passed in value is a userid, and do not attempt to look it up.

[Adam Young (ayoung)]

Reopening the issue against the Keystone server. The fix was not sufficient, as it was just a workaround, and one that we can't apply via the CLI.

The real fix requires avoiding the exception from the identity backend when performing any assignment-backend calls.

[Adam Young (ayoung)]

Note that, while supporting Delete allowed the customer to work around the issue, a change made to LDAP (AD) external to Keystone has the effect of breaking Keystone, and that is the real problem here. It has been seen multiple times now in live deployments.

[Adam Young (ayoung)]

Closing the Keystone server component again, as I just confirmed the user-list error does not happen in this code base, and thus it is a new bug and a regression. Will open a separate ticket for that.

[QiangTang (qtang)]

Does this issue still openstackclient or keystoneclient impact?

[Lance Bragstad (lbragstad)]

This is going to require changes to python-openstackclient

https://storyboard.openstack.org/#!/story/2006635