keystone kvs driver causes process to grow indefinitely and spin on CPU with thousands of keys in a single python dict

Bug #1188301 reported by Clint Byrum on 2013-06-06
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Medium
Joe Gordon
tripleo
Critical
Clint Byrum

Bug Description

After about 10 days of being up, things got sluggish in our proof of concept rack.

The kvs middleware for tokens is basically just for testing. It is in-process and there is no persistence, and no garbage collection or memory limit at all.

We should use sql if we want to persist tokens, or memcache if we are fine with losing some.

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 6955 keystone 20 0 3255m 3.0g 5640 S 92 3.2 6532:10 keystone-all

Changed in tripleo:
status: Triaged → Fix Released
assignee: nobody → Clint Byrum (clint-fewbar)
status: Fix Released → In Progress

Reviewed: https://review.openstack.org/32970
Committed: http://github.com/stackforge/tripleo-image-elements/commit/8fa5669f4984503f62c87ab5f0b797dbf8e0f416
Submitter: Jenkins
Branch: master

commit 8fa5669f4984503f62c87ab5f0b797dbf8e0f416
Author: Clint Byrum <email address hidden>
Date: Thu Jun 13 15:30:30 2013 -0700

    Set keystone token backend to sql.

    Because the table could grow forever, also add a cron job to flush tokens
    every two hours.

    Due to bug #1188378 we need to use percona-toolkit's pt-archiver to
    clean out the table on mysql.

    Fixes bug #1188301

    Change-Id: I90aac347993ee741231fbfb99a32dc6e3f3a75ec

Changed in tripleo:
status: In Progress → Fix Released

Fix proposed to branch: master
Review: https://review.openstack.org/34650

Changed in keystone:
assignee: nobody → Joe Gordon (jogo)
status: New → In Progress
Dolph Mathews (dolph) on 2013-06-27
Changed in keystone:
importance: Undecided → Medium

Reviewed: https://review.openstack.org/34650
Committed: http://github.com/openstack/keystone/commit/3f51d2addd38c418a55c01ca2f37e8bbe94319b5
Submitter: Jenkins
Branch: master

commit 3f51d2addd38c418a55c01ca2f37e8bbe94319b5
Author: Joe Gordon <email address hidden>
Date: Wed Jun 26 18:09:13 2013 -0700

    Deprecate kvs token backend

    This backend is not usable in any production environment. All OpenStack
    environments will already have a SQL DB, and if someone does not want to
    use the DB they can use the memcache backend.

    Fixes bug 1188301 and bug 1188370

    DocImpact This backend should not be mentioned in documentation, as it
    is not production grade and is deprecated.

    Change-Id: I41b147bcc70b79b4fc6df50b242a73cfcad33114

Changed in keystone:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2013-09-05
Changed in keystone:
milestone: none → havana-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2013-10-17
Changed in keystone:
milestone: havana-3 → 2013.2
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers