Unable to delete domains when users was managed by LDAP back-end

Bug #1801873 reported by Izak A Eygelaar
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Fix Released
Medium
Colleen Murphy
Pike
Fix Committed
Undecided
Colleen Murphy
Queens
Fix Committed
Undecided
Colleen Murphy
Rocky
Fix Committed
Undecided
Colleen Murphy
Stein
Fix Released
Medium
Colleen Murphy

Bug Description

Problem description:
We can create domains and assign users from LDAP back-end to the domains and create projects inside of the domains and also assign users out of LDAP access to the project. When we try to delete the domains that made use of the LDAP domain configuration to manage users, the domain deletion fails with a 500 server error.

Reproducing error:
Flow for creating domains:
1.) Create new domain
2.) Add domain configuration for LDAP config to domain
3.) Assign user out of LDAP access to the domain
4.) Create project inside of Domain and assign user access to the relevant project(s)

Flow for deleting domains:
1.) Revoke user access to the projects
2.) Remove all projects assigned in the Domain
3.) Revoke user access on domain level
4.) Remove the LDAP domain configuration for the domain
5.) Set domain status to disabled
6.) Remove domain (fails with server 500 error)

LDAP domain configuration added making use of the API call:
{
    "config": {
        "identity": {
            "driver": "ldap"
        },
        "ldap": {
            "user_mail_attribute": "mail",
            "url": "ldaps://ldap.x.x.x",
            "user_id_attribute": "mail",
            "query_scope": "sub",
            "user_description_attribute": "description",
            "user_enabled_mask": "2",
            "user_enabled_default": "512",
            "user_enabled_invert": "true",
            "user_enabled_attribute": "enabled",
            "user_name_attribute": "mail",
            "user_pass_attribute": "userPassword",
            "user_objectclass": "inetOrgPerson",
            "user_tree_dn": "ou=Openstack,dc=x,dc=x,dc=x"
        }
    }
}

Keystone config:
[identity]
domain_configurations_from_database = True
domain_specific_drivers_enabled = True
driver = sql

[ldap]
group_allow_create = False
group_allow_delete = False
group_allow_update = False
query_scope = sub
user_allow_create = False
user_allow_delete = False
user_allow_update = False

After some further investigation, it looks like remote LDAP users are then also created "locally" on keystone database in the user tables, which maps the users to the domains, and as a result when we remove the LDAP domain configuration it keeps the "locally mapped" users in the Keystone DB, which looks like Openstack performs a sanity check and checks if users are still assigned to domains, before deleting domains.

Keystone DB user table:
+------------------------------------------------------------------+-------------------------------------------------------+---------+----------------------------------+---------------------+----------------+----------------------------------+

| id | extra | enabled | default_project_id | created_at | last_active_at | domain_id |

+------------------------------------------------------------------+-------------------------------------------------------+---------+----------------------------------+---------------------+----------------+----------------------------------+

| 0ed65766eef533144508267d781de2ce70c62241a85ced00b9e07d0fc1a83483 | {"email": "<email address hidden>"} | 1 | NULL | 2018-10-29 16:57:06 | NULL | 28226b751b704ed6b0040016ec14d301 |

| 10adec00e356f1cd5d0470ac8dcc9148867c3913142d6d317621998aaaefb06a | {"email": "<email address hidden>", "description": " "} | 1 | NULL | 2018-10-03 20:13:17 | NULL | 29e6ee57de1f4468821607f33aa69d9a |

| 1b92cfe2d0add50a8b1b04ba532fdb1a467adebcaa6891addcc2fa5263e565a6 | {"email": "<email address hidden>"} | 1 | NULL | 2018-10-29 16:55:10 | NULL | 1dca35888d344a3e8c5b69604ba92806 |

| 1d12c86bafc10dd8aa6da726ec85ac7a2351957da4cda3cc7ad19863ed9bd94d | {"email": "<email address hidden>"} | 1 | NULL | 2018-10-31 06:48:34 | NULL | 2dd53ca43c3144ca86e0b4425d61668e |

Stack trace:
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi [req-ddd4c5b0-54ef-46e1-a267-d7f0c3f75106 8ae9dd0a0bbbe702ff0e0baa0dbeeb287ac6b70af521e4d99ed8bd5ba089c48f - d143c253f5ea437881a4263d3544a046 d143c253f5ea437881a4263d3544a046 -] An unexpected error prevented the server from fulfilling your request.: UnexpectedError: An unexpected error prevented the server from fulfilling your request.
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi Traceback (most recent call last):
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi File "/openstack/venvs/keystone-17.1.2/lib/python2.7/site-packages/keystone/common/wsgi.py", line 226, in __call__
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi result = method(req, **params)
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi File "/openstack/venvs/keystone-17.1.2/lib/python2.7/site-packages/keystone/common/controller.py", line 82, in inner
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi return f(self, request, *args, **kwargs)
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi File "/openstack/venvs/keystone-17.1.2/lib/python2.7/site-packages/keystone/resource/controllers.py", line 75, in delete_domain
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi domain_id, initiator=request.audit_initiator
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi File "/openstack/venvs/keystone-17.1.2/lib/python2.7/site-packages/keystone/common/manager.py", line 116, in wrapped
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi __ret_val = __f(*args, **kwargs)
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi File "/openstack/venvs/keystone-17.1.2/lib/python2.7/site-packages/keystone/resource/core.py", line 779, in delete_domain
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi self._delete_project(domain_id, initiator)
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi File "/openstack/venvs/keystone-17.1.2/lib/python2.7/site-packages/keystone/resource/core.py", line 511, in _delete_project
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi ret = self.driver.delete_project(project_id)
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi File "/openstack/venvs/keystone-17.1.2/lib/python2.7/site-packages/keystone/common/sql/core.py", line 561, in wrapper
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi 'store %s') % conflict_type)
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi UnexpectedError: An unexpected error prevented the server from fulfilling your request.
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi

Similar bugs have been reported, but they do not make use of external authentication for users, like LDAP. For example: https://bugs.launchpad.net/keystone/+bug/1718747

Some more env variables:
Openstack versions affected: Pike and Queens

Keystone version on Pike:
{
    "version": {
        "status": "stable",
        "updated": "2017-02-22T00:00:00Z",
        "media-types": [
            {
                "base": "application/json",
                "type": "application/vnd.openstack.identity-v3+json"
            }
        ],
        "id": "v3.8",
        "links": [
            {
                "href": "https://192.168.101.8:5000/v3/",
                "rel": "self"
            }
        ]
    }
}

Changed in keystone:
status: New → Triaged
importance: Undecided → Medium
status: Triaged → New
Revision history for this message
Morgan Fainberg (mdrnstm) wrote :

This looks like an issue, I need to stand up a local keystone to confirm. Marking it as medium, Once I duplicate this, I'll be able to fix/address the issue. Where this is being raised out/up it is not straightforward. It is probably related to forignkey and lack of cascade deletes.

I expect this is a small error / simple fix but the traceback is fairly opaque and will need more digging.

Revision history for this message
Colleen Murphy (krinkle) wrote :

Morgan, were you able to confirm this?

Revision history for this message
Colleen Murphy (krinkle) wrote :

I was able to reproduce this on Stein.

Changed in keystone:
status: New → Triaged
milestone: none → stein-rc2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/647498

Changed in keystone:
assignee: nobody → Colleen Murphy (krinkle)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (master)

Reviewed: https://review.openstack.org/647498
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=1b16725d066f71a833ca74656c0c3dfe5f815e98
Submitter: Zuul
Branch: master

commit 1b16725d066f71a833ca74656c0c3dfe5f815e98
Author: Colleen Murphy <email address hidden>
Date: Mon Mar 25 15:50:26 2019 +0100

    Delete shadow users when domain is deleted

    Without this change, when an admin tries to delete an LDAP-backed
    domain, it fails due to the foreign key relationship in the users table.
    Previously, we were assuming that LDAP users existed solely in the LDAP
    directory, but this is not true with shadow users. This patch fixes the
    logic to delete the shadow users upon domain deletion.

    Change-Id: I12a08001e3aa08e4db9438cae425ad1a0a8070f7
    Closes-bug: #1801873

Changed in keystone:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.openstack.org/648825

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (stable/stein)

Reviewed: https://review.openstack.org/648825
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=b86df692d0fccc5ce09de7f7840ea420e6d3d664
Submitter: Zuul
Branch: stable/stein

commit b86df692d0fccc5ce09de7f7840ea420e6d3d664
Author: Colleen Murphy <email address hidden>
Date: Mon Mar 25 15:50:26 2019 +0100

    Delete shadow users when domain is deleted

    Without this change, when an admin tries to delete an LDAP-backed
    domain, it fails due to the foreign key relationship in the users table.
    Previously, we were assuming that LDAP users existed solely in the LDAP
    directory, but this is not true with shadow users. This patch fixes the
    logic to delete the shadow users upon domain deletion.

    Change-Id: I12a08001e3aa08e4db9438cae425ad1a0a8070f7
    Closes-bug: #1801873
    (cherry picked from commit 1b16725d066f71a833ca74656c0c3dfe5f815e98)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/649376

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/649377

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/649378

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/keystone 15.0.0.0rc2

This issue was fixed in the openstack/keystone 15.0.0.0rc2 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (stable/pike)

Reviewed: https://review.openstack.org/649378
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=d76a068d1348da467cd1b5db518c7739ce9f7488
Submitter: Zuul
Branch: stable/pike

commit d76a068d1348da467cd1b5db518c7739ce9f7488
Author: Colleen Murphy <email address hidden>
Date: Mon Mar 25 15:50:26 2019 +0100

    Delete shadow users when domain is deleted

    Without this change, when an admin tries to delete an LDAP-backed
    domain, it fails due to the foreign key relationship in the users table.
    Previously, we were assuming that LDAP users existed solely in the LDAP
    directory, but this is not true with shadow users. This patch fixes the
    logic to delete the shadow users upon domain deletion.

    Altered cherry-pick: use injected API provider since provider_api
    mechanism doesn't exist in pike.

    Change-Id: I12a08001e3aa08e4db9438cae425ad1a0a8070f7
    Closes-bug: #1801873
    (cherry picked from commit 1b16725d066f71a833ca74656c0c3dfe5f815e98)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (stable/rocky)

Reviewed: https://review.openstack.org/649376
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=b5af5c9009e19b40fce8496f777c665f8fa3c644
Submitter: Zuul
Branch: stable/rocky

commit b5af5c9009e19b40fce8496f777c665f8fa3c644
Author: Colleen Murphy <email address hidden>
Date: Mon Mar 25 15:50:26 2019 +0100

    Delete shadow users when domain is deleted

    Without this change, when an admin tries to delete an LDAP-backed
    domain, it fails due to the foreign key relationship in the users table.
    Previously, we were assuming that LDAP users existed solely in the LDAP
    directory, but this is not true with shadow users. This patch fixes the
    logic to delete the shadow users upon domain deletion.

    Change-Id: I12a08001e3aa08e4db9438cae425ad1a0a8070f7
    Closes-bug: #1801873
    (cherry picked from commit 1b16725d066f71a833ca74656c0c3dfe5f815e98)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (stable/queens)

Reviewed: https://review.openstack.org/649377
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=86428a6c6688f1756a3c52567061edf34bb89315
Submitter: Zuul
Branch: stable/queens

commit 86428a6c6688f1756a3c52567061edf34bb89315
Author: Colleen Murphy <email address hidden>
Date: Mon Mar 25 15:50:26 2019 +0100

    Delete shadow users when domain is deleted

    Without this change, when an admin tries to delete an LDAP-backed
    domain, it fails due to the foreign key relationship in the users table.
    Previously, we were assuming that LDAP users existed solely in the LDAP
    directory, but this is not true with shadow users. This patch fixes the
    logic to delete the shadow users upon domain deletion.

    Change-Id: I12a08001e3aa08e4db9438cae425ad1a0a8070f7
    Closes-bug: #1801873
    (cherry picked from commit 1b16725d066f71a833ca74656c0c3dfe5f815e98)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/keystone 12.0.3

This issue was fixed in the openstack/keystone 12.0.3 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/keystone 16.0.0.0rc1

This issue was fixed in the openstack/keystone 16.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/keystone 13.0.3

This issue was fixed in the openstack/keystone 13.0.3 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/keystone 14.2.0

This issue was fixed in the openstack/keystone 14.2.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.