Keystone's token table grows unconditionally when using SQL backend.

Bug #1032633 reported by Andrew Glen-Young on 2012-08-03
82
This bug affects 19 people
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Medium
Jamie Lennox
openstack-manuals
Undecided
Unassigned
keystone (Ubuntu)
Medium
David Höppner

Bug Description

Keystone's `token` table grows unconditionally with expired tokens when using the SQL backend.

Keystone should provide a backend-agnostic method to find and delete these tokens. This could be run via a periodic task or supplied as a script to run as a cron job.

An example SQL statement (if you're using a SQL backend) to workaround this problem:

    sql> DELETE FROM token WHERE expired <= NOW();

It may be ideal to allow a date smear to allow older tokens to persist if needed.

Choosing the `memcache` backend may workaround this issue, but SQL is the package default.

System Information:

$ dpkg-query --show keystone
keystone 2012.1+stable~20120608-aff45d6-0ubuntu1

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04 LTS"

description: updated
summary: - Keystone's token table grows unconditionally.
+ Keystone's token table grows unconditionally when using SQL backend.
description: updated
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in keystone (Ubuntu):
status: New → Confirmed
Dolph Mathews (dolph) on 2012-08-31
Changed in keystone:
status: New → Confirmed
James Page (james-page) on 2012-11-25
Changed in keystone (Ubuntu):
importance: Undecided → Medium
Dolph Mathews (dolph) wrote :
Changed in keystone:
status: Confirmed → Invalid
David Höppner (0xffea) on 2013-03-17
Changed in keystone (Ubuntu):
assignee: nobody → David Höppner (0xffea)

Fix proposed to branch: master
Review: https://review.openstack.org/28133

Changed in keystone:
assignee: nobody → Jamie Lennox (jamielennox)
status: Invalid → In Progress

Reviewed: https://review.openstack.org/28133
Committed: http://github.com/openstack/keystone/commit/ff76a1b5cd3308cfb0ce936800364e27413ed946
Submitter: Jenkins
Branch: master

commit ff76a1b5cd3308cfb0ce936800364e27413ed946
Author: Jamie Lennox <email address hidden>
Date: Fri May 3 14:04:09 2013 +1000

    Implement Token Flush via keystone-manage.

    Creates a cli entry 'token_flush' which removes all expired tokens.

    Fixes: bug 1032633
    Implements: blueprint keystone-manage-token-flush
    Change-Id: I47eab99b577ff9e9ee74fee08e18fd07c4af5aad

Changed in keystone:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2013-05-29
Changed in keystone:
milestone: none → havana-1
status: Fix Committed → Fix Released
Dolph Mathews (dolph) on 2013-08-28
Changed in keystone:
importance: Undecided → Medium
Thierry Carrez (ttx) on 2013-10-17
Changed in keystone:
milestone: havana-1 → 2013.2
James Page (james-page) on 2014-03-07
Changed in keystone (Ubuntu):
status: Confirmed → Fix Released
Andrew Mann (o-andrew-v) wrote :

A CLI command is an interesting stopgap, but on a heavily utilized OpenStack installation with automated tools operating against OpenStack, this has a high manual maintenance cost. Surely there is some better default that lies in the middle ground between keeping tokens for ever and ever and requiring a manual removal of tokens?

As a reference point, I wasn't even aware this was an issue, until one of our test deployments of grizzly using a limited IO system started acting horribly (30 second response times). After tracing the problem from nova to keystone to mysql, I found a 442,000 row token table with >440,000 expired tokens. I went and checked our havana test on a somewhat beefier system and found > 1M rows.

This issue is a timebomb for any production OS install.

On 04/05/2014 01:43 PM, Andrew Mann wrote:
> A CLI command is an interesting stopgap, but on a heavily utilized
> OpenStack installation with automated tools operating against OpenStack,
> this has a high manual maintenance cost. Surely there is some better
> default that lies in the middle ground between keeping tokens for ever
> and ever and requiring a manual removal of tokens?
>
> As a reference point, I wasn't even aware this was an issue, until one
> of our test deployments of grizzly using a limited IO system started
> acting horribly (30 second response times). After tracing the problem
> from nova to keystone to mysql, I found a 442,000 row token table with
>> 440,000 expired tokens. I went and checked our havana test on a
> somewhat beefier system and found > 1M rows.
>
> This issue is a timebomb for any production OS install.
>
CRON the CLI job. There is no reason to try and integrate a scheduler
into Keystone.

Andrew Mann (o-andrew-v) wrote :

That seems reasonable. I would suggest adding that as a step to the OpenStack setup documentation. No one has infinite database space, eventually all used OpenStack installations will suffer unless this is done. Regardless of what component (or person) has to complete the task, they'll need to be aware of it.

I would prefer to see some kind of "automatically expire old tokens" configuration option so that the maintenance of keystone stays in keystone - whether that be implemented as a expire-on-access query or cron-derived mechanism, but re-implementing cron is not great and a purge tagged on to other operations could cause a slowdown on those operations.

Dolph Mathews (dolph) wrote :

Since it's not already mentioned in this bug, the long term solution here is to simply not persist tokens at all:

  https://blueprints.launchpad.net/keystone/+spec/ephemeral-pki-tokens

Matt Kassawara (ionosphere80) wrote :

Addressed by the following patch:

https://review.openstack.org/#/c/79105/

Changed in openstack-manuals:
status: New → Fix Released
milestone: none → icehouse
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers