Performance issue when delete tokens for users

[David Geng]

There is performance issue when we delete a user in our long run test environment:
- 5 tenants
- 300 users associated with tenants as follow:
- Tenant 1 : 100 users
- Tenant 2 : 80 users
- Tenant 3 : 60 users
- Tenant 4 : 40 users
- Tenant 5 : 20 users

And there are more than 144 thousand tokens, with more than 19000 non expired after about 2 days.

[David Geng (genggjh)]

I think the root cause is in the keystone/keystone/identity/controllers.py, we delete the token for user one by one in the for loop instead of do it in one session.

    def _delete_tokens_for_user(self, context, user_id, project_id=None):
        #First delete tokens that could get other tokens.
        for token_id in self.token_api.list_tokens(context,
                                                   user_id,
                                                   tenant_id=project_id):
            try:
                self.token_api.delete_token(context, token_id)
            except exception.NotFound:
                pass

The better solution is we provide a new api like token_api.delete_token_by_user(context, user_id) to batch delete tokens.

[Malini Bhandaru (malini-k-bhandaru)]

Please help me understand why there are so many tokens.
In your test scenario, total 300 users, why not just 300 tokens, one per user? Not introducing new token till earlier one for the same user expires. I was recently examining db tables in my dev environment and wondered about it.

Regards
Malini

[David Geng (genggjh)]

Our test cases are trying to simulate many users login/logout concurrency, disable/enable user, create/delete user in long run env.
Each time you login will generate a new token recorder in db. The token will not be deleted even it expires before the user is deleted or disabled.

[Malini Bhandaru (malini-k-bhandaru)]

I understand your need to stress test, many user environment. I would basically create random users and logins associated from each .. 100, 200, 500, 1000 etc and see when the system breaks, where etc, response times.
But for each user a single token for scalability. After they login, and are validated, link to the same token, if anything extend the expire time of the token by whatever default lifetime interval.

Here is another bug stemming from a token explosion.
https://bugs.launchpad.net/keystone/+bug/1168399

[Kurt Seifried (kseifried)]

So it looks like for every user action (e.g. login/logout) tokens are generated, how fast can a user log in and out, I'm guessing through the API and a high speed network connection they can do this pretty fast (especially since they can run it concurrently?). This issue may be security related if an attacker can cause enough tokens to be created to trigger a denial of service condition. Also how do we flag public bugs with the "security" tag (I don't want to make it private, I just want to make sure the right people see it)?

[David Geng (genggjh)]

For the security issue I think it is another story should not be covered in this bug, I believe we already have some proposal regarding it, such as use OAuth to generate token, reduce the token expiration date, renew token and etc.

Here, I just want to improve the performance of token deletion and make it more efficiency.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/28866

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/28866
Committed: http://github.com/openstack/keystone/commit/d6cfe4f2e2c7c1b3eb23bf9f864779940a645555
Submitter: Jenkins
Branch: master

commit d6cfe4f2e2c7c1b3eb23bf9f864779940a645555
Author: gengjh <email address hidden>
Date: Sat May 11 17:20:10 2013 +0800

    Improve the performance of tokens deletion for user

    Provide new delete the tokens api 'delete_tokens' to support
    delete all the tokens for user in one session in the sql backend. For
    the kvs and memcache, I also provide the corresponding implementation.

    Fix bug 1178063

    Change-Id: I986a583e5900ea04e26cbdb7c49638a33818bca7

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/31907

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/31907
Committed: http://github.com/openstack/keystone/commit/ef467b2281ba40c858faa76ec8315c637481f8cf
Submitter: Jenkins
Branch: stable/grizzly

commit ef467b2281ba40c858faa76ec8315c637481f8cf
Author: gengjh <email address hidden>
Date: Sat May 11 17:20:10 2013 +0800

    Improve the performance of tokens deletion for user

    Provide new delete the tokens api 'delete_tokens' to support
    delete all the tokens for user in one session in the sql backend. For
    the kvs and memcache, I also provide the corresponding implementation.

    Fix bug 1178063

    Change-Id: I986a583e5900ea04e26cbdb7c49638a33818bca7
    (cherry picked from commit d6cfe4f2e2c7c1b3eb23bf9f864779940a645555)