[SRU] Can't delete users

Bug #959294 reported by Soren Hansen
38
This bug affects 6 people
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Fix Released
High
Unassigned
keystone (Ubuntu)
Fix Released
Undecided
Unassigned
Precise
Fix Released
Undecided
Chuck Short

Bug Description

I created a user using:

keystone tenant-create --name=services
keystone user-create --name=glance --tenant_id=<the id from the tenant>

Then I decided to delete the user again:
keystone user-delete <id of the user>

I got a massive traceback that ended in

IntegrityError: (IntegrityError) (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`keystone`.`user_tenant_membership`, CONSTRAINT `user_tenant_membership_ibfk_1` FOREIGN KEY (`user_id`) REFERENCES `user` (`id`))') 'DELETE FROM user WHERE user.id = %s' ('443e71fb4f4344ee941f58795e259ac6',)

Joseph Heck (heckj)
Changed in keystone:
status: New → Confirmed
importance: Undecided → High
Joseph Heck (heckj)
tags: added: essex-rc-potential
Dolph Mathews (dolph)
Changed in keystone:
assignee: nobody → Dolph Mathews (dolph)
Joseph Heck (heckj)
Changed in keystone:
milestone: none → essex-rc1
assignee: Dolph Mathews (dolph) → termie (termie)
assignee: termie (termie) → Dolph Mathews (dolph)
Revision history for this message
Dolph Mathews (dolph) wrote :

Unable to reproduce at this point. Running master against a fresh SQL backend, I tried to A) create a user with a default tenant and delete the user, and B) create a user with an explicit role grant on a tenant and delete the user, and was able to delete the user (and then tenant, as well) in both cases.

http://paste.openstack.org/raw/11493/

Revision history for this message
Joseph Heck (heckj) wrote :

Marking incomplete -

Soren, could you verify that you can repro the issue here and put any additional details in about how to do so?

Changed in keystone:
status: Confirmed → Incomplete
Revision history for this message
Soren Hansen (soren) wrote :

I'm trying to reproduce it now. It was on a standard devstack deploy (on Ubuntu Precise).

Revision history for this message
Soren Hansen (soren) wrote :

Very odd. I can't reproduce now, because I'm hitting another problem (bug 960977). I don't know if they're a symptom of the same wonkiness, so filed as a separate bug for now. We can always mark them dupes later.

Revision history for this message
Joseph Heck (heckj) wrote :

removing RC1 link from this, but leaving tagged as candidate for a back port if we hit this.

Changed in keystone:
milestone: essex-rc1 → none
Revision history for this message
Adam Gandelman (gandelman-a) wrote :

Running milestone-proposed branch of keystone, hitting this on Ubuntu precise when trying to delete a user that was created /w --tenant_id. Here's a log of steps used to reproduce: http://paste.ubuntu.com/911906/

Revision history for this message
Adam Gandelman (gandelman-a) wrote :

This is a bug, and is not surfacing in devstack testing because devstack hasn't yet moved to precise. Oneiric MySQL is configured with MyISAM storage engine by default, which means FK constraints are non-existent in the backend and are essentially meaningless. MySQL on Precise uses InnoDB as the default, and FK constraints described in the models exist and are enforced. Using Precise or configuring an Oneiric devstack MySQL install to use InnoDB by default (prior to syncing the database) will reproduce this issue.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/6109

Changed in keystone:
assignee: Dolph Mathews (dolph) → Adam Gandelman (gandelman-a)
status: Incomplete → In Progress
Thierry Carrez (ttx)
Changed in keystone:
milestone: none → essex-rc2
tags: removed: essex-rc-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (master)

Reviewed: https://review.openstack.org/6109
Committed: http://github.com/openstack/keystone/commit/75a8dfef51f3566cd5d4cacee41f34bbbf9d15bd
Submitter: Jenkins
Branch: master

commit 75a8dfef51f3566cd5d4cacee41f34bbbf9d15bd
Author: Adam Gandelman <email address hidden>
Date: Mon Apr 2 14:21:43 2012 -0700

    Remove tenant membership during user deletion

    Remove users' tenant membership on user deletion. Resolves a FK constraint
    issue that previously went unnoticed due to testing against database
    configurations that do not support FK constraints (MyISAM).

    Fixes LP bug 959294.

    Update: * Move tenant membership cleanup to the sql identity backend
            * Add a test case to test_backend_sql

    Change-Id: Ib4f5da03033f7886b36d1ab3b8b4ac37f08b2e0e

Changed in keystone:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (milestone-proposed)

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/6207

Devin Carlen (devcamcar)
Changed in keystone:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (milestone-proposed)

Reviewed: https://review.openstack.org/6207
Committed: http://github.com/openstack/keystone/commit/7d08d12cea96910145f05499ba7d124603d7c4f6
Submitter: Jenkins
Branch: milestone-proposed

commit 7d08d12cea96910145f05499ba7d124603d7c4f6
Author: Adam Gandelman <email address hidden>
Date: Mon Apr 2 14:21:43 2012 -0700

    Remove tenant membership during user deletion

    Remove users' tenant membership on user deletion. Resolves a FK constraint
    issue that previously went unnoticed due to testing against database
    configurations that do not support FK constraints (MyISAM).

    Fixes LP bug 959294.

    Update: * Move tenant membership cleanup to the sql identity backend
            * Add a test case to test_backend_sql

    Change-Id: Ib4f5da03033f7886b36d1ab3b8b4ac37f08b2e0e

Revision history for this message
Launchpad Janitor (janitor) wrote : Re: Can't delete users

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in keystone (Ubuntu):
status: New → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/6216

Changed in keystone:
assignee: Adam Gandelman (gandelman-a) → J. Daniel Schmidt (jdsn)
status: Fix Released → In Progress
Revision history for this message
Thierry Carrez (ttx) wrote : Re: Can't delete users

jdsn: please open a separate bug for this fix

Changed in keystone:
assignee: J. Daniel Schmidt (jdsn) → nobody
milestone: essex-rc2 → none
status: In Progress → Fix Released
milestone: none → 2012.1
Revision history for this message
J. Daniel Schmidt (jdsn) wrote :

opened new bug 974199 to track the open issues

Revision history for this message
J. Daniel Schmidt (jdsn) wrote :

FYI:

remove tenant
* does not delete user association: in bug 974199
* does not delete tenant metadata: in bug 973243

remove user
* does not delete tenant association: in bug 959294
* does not delete user metadata: in bug 973243

Changed in keystone (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
bastichelaar (bas-t) wrote :

I was running Keystone with a Postgresql backend. It seems the patch above doesn't solve the problem for Postgresql, I get lots of IntegrityErrors still.

Revision history for this message
Debo~ Dutta (debo) wrote :

I see the same issue

+ keystone tenant-delete 6e491a65755745aba0b3062952481d0f
No handlers could be found for logger "keystoneclient.client"
An unexpected error prevented the server from fulfilling your request. (IntegrityError) (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`keystone`.`user_tenant_membership`, CONSTRAINT `user_tenant_membership_ibfk_2` FOREIGN KEY (`tenant_id`) REFERENCES `tenant` (`id`))') 'DELETE FROM tenant WHERE tenant.id = %s' ('6e491a65755745aba0b3062952481d0f',) (HTTP 500)

Revision history for this message
David Kranz (david-kranz) wrote :

We just saw this running a new tempest keystone test submitted as https://review.openstack.org/#/c/7231/

(keystone.policy.backends.rules): 2012-05-10 11:06:00,254 DEBUG enforce admin_required: {'tenant_id': u'758e5226e8a542c385ad447383f51b44', 'user_id': u'533cb3d7ca7f4f36ab6bbc4547b445c5', u'roles': [u'admin']}
(root): 2012-05-10 11:06:00,299 ERROR (IntegrityError) (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`keystone`.`user_tenant_membership`, CONSTRAINT `user_tenant_membership_ibfk_1` FOREIGN KEY (`user_id`) REFERENCES `user` (`id`))') 'DELETE FROM user WHERE user.id = %s' ('50c9e4fd7af947a2895d8c1bcb17d51c',)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 184, in __call__
    result = method(context, **params)
  File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 415, in delete_user
    self.identity_api.delete_user(context, user_id)
  File "/usr/lib/python2.7/dist-packages/keystone/common/manager.py", line 48, in _wrapper
    return f(*args, **kw)
  File "/usr/lib/python2.7/dist-packages/keystone/identity/backends/sql.py", line 340, in delete_user
    session.flush()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1547, in flush
    self._flush(objects)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1616, in _flush
    flush_context.execute()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 328, in execute
    rec.execute(self)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 495, in execute
    uow
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/mapper.py", line 2499, in _delete_obj
    c = connection.execute(statement, del_objects)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1405, in execute
    params)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1538, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1646, in _execute_context
    context)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1639, in _execute_context
    context)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 330, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
IntegrityError: (IntegrityError) (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`keystone`.`user_tenant_membership`, CONSTRAINT `user_tenant_membership_ibfk_1` FOREIGN KEY (`user_id`) REFERENCES `user` (`id`))') 'DELETE FROM user WHERE user.id = %s' ('50c9e4fd7af947a2895d8c1bcb17d51c',)
(eventlet.wsgi.server): 2012-05-10 11:06:00,300 DEBUG 172.17.1.170 - - [10/May/2012 11:06:00] "DELETE /v2.0/users/50c9e4fd7af947a2895d8c1bcb17d51c HTTP/1.1" 500 613 0.053749

Revision history for this message
Adam Gandelman (gandelman-a) wrote :

Something strange is going on. Tested on a fresh folsom devstack: http://paste.ubuntu.com/980372/ It looks like after the 4th failure. After trying this a few times it seems to succeed after a varying number of failed attempts

Revision history for this message
Adam Gandelman (gandelman-a) wrote :

Seems the membership removal is not atomic. Opened Bug #998137 to track there.

Chuck Short (zulcss)
summary: - Can't delete users
+ [SRU] Can't delete users
Chuck Short (zulcss)
Changed in keystone (Ubuntu Precise):
milestone: none → ubuntu-12.04.1
assignee: nobody → Chuck Short (zulcss)
Revision history for this message
Chuck Short (zulcss) wrote :

** Impact **

Remove users' tenant membership on user deletion. Resolves a FK constraint issue that previously went unnoticed due to testing against database configurations that do not support FK constraints (MyISAM).

** Development Fix **

This has been resolved in the development trunk at https://review.openstack.org/6216 and in quantal.

** Stable Fix **

This has been resolved in the stable essex/trunk at https://review.openstack.org/#/c/7353/

**Test Case **

http://paste.ubuntu.com/980372/

** Regression Possiblity **

Minimal, removing a user shouldnt result in a regression.

Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Soren, or anyone else affected,

Accepted keystone into precise-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in keystone (Ubuntu Precise):
status: New → Fix Committed
tags: added: verification-needed
Revision history for this message
Sebastien Bacher (seb128) wrote :

This update is in proposed for 26 days, could somebody using that software verify the fix so the SRU can be moved to updates?

Revision history for this message
Adam Gandelman (gandelman-a) wrote : Verification report.

Please find the attached Jenkins job results from the Ubuntu Server Team's CI infrastructure. As part of the verification process for this bug, Keystone has been deployed and configured across multiple nodes using precise-proposed as an installation source. After successful bring-up and configuration of the cluster, a number of exercises and smoke tests have be invoked to ensure the updated package did not introduce any regressions. A number of test iterations were carried out to catch any possible transient errors.

Please Note the list of installed packages at the top and bottom of the report.

For records of upstream test coverage of this update, please see the Jenkins links in the comments of the relevant upstream code-review(s):

Trunk review: https://review.openstack.org/6109

As per the provisional Micro Release Exception granted to this package by the Technical Board, we hope this contributes toward verification of this update.

Revision history for this message
Adam Gandelman (gandelman-a) wrote :

Test coverage log.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package keystone - 2012.1+stable~20120608-aff45d6-0ubuntu1

---------------
keystone (2012.1+stable~20120608-aff45d6-0ubuntu1) precise-proposed; urgency=low

  * New usptream snapshot. (LP: #1010473)
  * Resynchronize with stable/essex:
    - aff45d6 - Make import_nova_auth only create roles which don't already exist
      (LP: #959294)
  * debian/patches/0013-Flush-tenant-membership-deletion-before-user.patch: Backported
    fix for "Flush tenant membership deletion before user." (LP: #998137)
 -- Chuck Short <email address hidden> Tue, 05 Jun 2012 11:24:02 -0400

Changed in keystone (Ubuntu Precise):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.