keystone kvs driver causes process to grow indefinitely and spin on CPU with thousands of keys in a single python dict

[Clint Byrum]

After about 10 days of being up, things got sluggish in our proof of concept rack.

The kvs middleware for tokens is basically just for testing. It is in-process and there is no persistence, and no garbage collection or memory limit at all.

We should use sql if we want to persist tokens, or memcache if we are fine with losing some.

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 6955 keystone 20 0 3255m 3.0g 5640 S 92 3.2 6532:10 keystone-all

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/32970
Committed: http://github.com/stackforge/tripleo-image-elements/commit/8fa5669f4984503f62c87ab5f0b797dbf8e0f416
Submitter: Jenkins
Branch: master

commit 8fa5669f4984503f62c87ab5f0b797dbf8e0f416
Author: Clint Byrum <email address hidden>
Date: Thu Jun 13 15:30:30 2013 -0700

    Set keystone token backend to sql.

    Because the table could grow forever, also add a cron job to flush tokens
    every two hours.

    Due to bug #1188378 we need to use percona-toolkit's pt-archiver to
    clean out the table on mysql.

    Fixes bug #1188301

    Change-Id: I90aac347993ee741231fbfb99a32dc6e3f3a75ec

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/34650

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/34650
Committed: http://github.com/openstack/keystone/commit/3f51d2addd38c418a55c01ca2f37e8bbe94319b5
Submitter: Jenkins
Branch: master

commit 3f51d2addd38c418a55c01ca2f37e8bbe94319b5
Author: Joe Gordon <email address hidden>
Date: Wed Jun 26 18:09:13 2013 -0700

    Deprecate kvs token backend

    This backend is not usable in any production environment. All OpenStack
    environments will already have a SQL DB, and if someone does not want to
    use the DB they can use the memcache backend.

    Fixes bug 1188301 and bug 1188370

    DocImpact This backend should not be mentioned in documentation, as it
    is not production grade and is deprecated.

    Change-Id: I41b147bcc70b79b4fc6df50b242a73cfcad33114