Performance issue when delete tokens for users

Bug #1178063 reported by David Geng on 2013-05-09
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Medium
David Geng
Grizzly
Medium
David Geng

Bug Description

There is performance issue when we delete a user in our long run test environment:
- 5 tenants
- 300 users associated with tenants as follow:
- Tenant 1 : 100 users
- Tenant 2 : 80 users
- Tenant 3 : 60 users
- Tenant 4 : 40 users
- Tenant 5 : 20 users

And there are more than 144 thousand tokens, with more than 19000 non expired after about 2 days.

David Geng (genggjh) wrote :

I think the root cause is in the keystone/keystone/identity/controllers.py, we delete the token for user one by one in the for loop instead of do it in one session.

    def _delete_tokens_for_user(self, context, user_id, project_id=None):
        #First delete tokens that could get other tokens.
        for token_id in self.token_api.list_tokens(context,
                                                   user_id,
                                                   tenant_id=project_id):
            try:
                self.token_api.delete_token(context, token_id)
            except exception.NotFound:
                pass

The better solution is we provide a new api like token_api.delete_token_by_user(context, user_id) to batch delete tokens.

Changed in keystone:
assignee: nobody → David Geng (genggjh)

Please help me understand why there are so many tokens.
In your test scenario, total 300 users, why not just 300 tokens, one per user? Not introducing new token till earlier one for the same user expires. I was recently examining db tables in my dev environment and wondered about it.

Regards
Malini

David Geng (genggjh) wrote :

Our test cases are trying to simulate many users login/logout concurrency, disable/enable user, create/delete user in long run env.
Each time you login will generate a new token recorder in db. The token will not be deleted even it expires before the user is deleted or disabled.

I understand your need to stress test, many user environment. I would basically create random users and logins associated from each .. 100, 200, 500, 1000 etc and see when the system breaks, where etc, response times.
But for each user a single token for scalability. After they login, and are validated, link to the same token, if anything extend the expire time of the token by whatever default lifetime interval.

Here is another bug stemming from a token explosion.
https://bugs.launchpad.net/keystone/+bug/1168399

Kurt Seifried (kseifried) wrote :

So it looks like for every user action (e.g. login/logout) tokens are generated, how fast can a user log in and out, I'm guessing through the API and a high speed network connection they can do this pretty fast (especially since they can run it concurrently?). This issue may be security related if an attacker can cause enough tokens to be created to trigger a denial of service condition. Also how do we flag public bugs with the "security" tag (I don't want to make it private, I just want to make sure the right people see it)?

David Geng (genggjh) wrote :

For the security issue I think it is another story should not be covered in this bug, I believe we already have some proposal regarding it, such as use OAuth to generate token, reduce the token expiration date, renew token and etc.

Here, I just want to improve the performance of token deletion and make it more efficiency.

Fix proposed to branch: master
Review: https://review.openstack.org/28866

Changed in keystone:
status: New → In Progress

Reviewed: https://review.openstack.org/28866
Committed: http://github.com/openstack/keystone/commit/d6cfe4f2e2c7c1b3eb23bf9f864779940a645555
Submitter: Jenkins
Branch: master

commit d6cfe4f2e2c7c1b3eb23bf9f864779940a645555
Author: gengjh <email address hidden>
Date: Sat May 11 17:20:10 2013 +0800

    Improve the performance of tokens deletion for user

    Provide new delete the tokens api 'delete_tokens' to support
    delete all the tokens for user in one session in the sql backend. For
    the kvs and memcache, I also provide the corresponding implementation.

    Fix bug 1178063

    Change-Id: I986a583e5900ea04e26cbdb7c49638a33818bca7

Changed in keystone:
status: In Progress → Fix Committed
David Geng (genggjh) on 2013-06-06
tags: added: grizzly-backport-potential
Dolph Mathews (dolph) on 2013-06-06
Changed in keystone:
importance: Undecided → Medium
tags: removed: grizzly-backport-potential

Reviewed: https://review.openstack.org/31907
Committed: http://github.com/openstack/keystone/commit/ef467b2281ba40c858faa76ec8315c637481f8cf
Submitter: Jenkins
Branch: stable/grizzly

commit ef467b2281ba40c858faa76ec8315c637481f8cf
Author: gengjh <email address hidden>
Date: Sat May 11 17:20:10 2013 +0800

    Improve the performance of tokens deletion for user

    Provide new delete the tokens api 'delete_tokens' to support
    delete all the tokens for user in one session in the sql backend. For
    the kvs and memcache, I also provide the corresponding implementation.

    Fix bug 1178063

    Change-Id: I986a583e5900ea04e26cbdb7c49638a33818bca7
    (cherry picked from commit d6cfe4f2e2c7c1b3eb23bf9f864779940a645555)

Thierry Carrez (ttx) on 2013-07-17
Changed in keystone:
milestone: none → havana-2
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2013-10-17
Changed in keystone:
milestone: havana-2 → 2013.2
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers