OpenStack Identity (keystone)

token_flush can hang if lots of tokens

Bug #1387401 reported by Brant Knudson on 2014-10-29

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Identity (keystone)	Fix Released	Medium	Brant Knudson	OpenStack Identity (keystone) 2015.1.0 "kilo"
	Juno	Fix Released	Medium	Brant Knudson	OpenStack Identity (keystone) 2014.2.1

Bug Description

If you've got a system that can generate lots of tokens, token_flush can hang. For DB2, this happens if you create > 100 tokens in a second (for mysql it's 1000 tokens in a second). The query to get the time to delete returns the 100th timestamp which is the same as the min timestamp, and then it goes to delete < min timestamp and none match, so none are deleted, then it gets stuck in a loop since the function always returns the min timestamp.

This could be fixed easily by using <= rather than < for the deletion comparison.

Brant Knudson (blk-u) on 2014-10-29

Changed in keystone:
assignee:	nobody → Brant Knudson (blk-u)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-10-30: Fix proposed to keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/131899

Changed in keystone:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-11-11: Fix merged to keystone (master)

Reviewed: https://review.openstack.org/131899
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=6bf78255b2529fd3a83e749101130cf595f1544e
Submitter: Jenkins
Branch: master

commit 6bf78255b2529fd3a83e749101130cf595f1544e
Author: Brant Knudson <email address hidden>
Date: Wed Oct 29 19:17:45 2014 -0500

Prevent infinite loop in token_flush

    If there were > batch_size tokens with the same expiration time then
    token_flush could hang because it picks the expiration time as the
    target to delete with a < check.

The check is changed to a <= so that the tokens in the batch are
deleted so that token_flush always makes progress.

Change-Id: Id0b6f7fcdf7ebb02a6f38fbb9a4e25bf714a614c
Closes-Bug: #1387401

Changed in keystone:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-11-17: Fix proposed to keystone (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/135118

Brant Knudson (blk-u) on 2014-11-17

tags:

added: juno-backport-potential

Dolph Mathews (dolph) on 2014-11-18

Changed in keystone:
importance:	Undecided → Medium

Alan Pevec (apevec) on 2014-11-20

tags:

removed: juno-backport-potential

Morgan Fainberg (mdrnstm) on 2014-11-20

Changed in keystone:
milestone:	none → kilo-1

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-11-22: Fix merged to keystone (stable/juno)

Reviewed: https://review.openstack.org/135118
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=f54fa8fa56d3708d5ef5b0173d9f6e4dce7130f2
Submitter: Jenkins
Branch: stable/juno

commit f54fa8fa56d3708d5ef5b0173d9f6e4dce7130f2
Author: Brant Knudson <email address hidden>
Date: Wed Oct 29 19:17:45 2014 -0500

Prevent infinite loop in token_flush

    If there were > batch_size tokens with the same expiration time then
    token_flush could hang because it picks the expiration time as the
    target to delete with a < check.

The check is changed to a <= so that the tokens in the batch are
deleted so that token_flush always makes progress.

    Change-Id: Id0b6f7fcdf7ebb02a6f38fbb9a4e25bf714a614c
    Closes-Bug: #1387401
    (cherry picked from commit 6bf78255b2529fd3a83e749101130cf595f1544e)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-12-01: Fix proposed to keystone (feature/hierarchical-multitenancy)

Fix proposed to branch: feature/hierarchical-multitenancy
Review: https://review.openstack.org/138182

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-12-01: Change abandoned on keystone (feature/hierarchical-multitenancy)

Change abandoned by Morgan Fainberg (<email address hidden>) on branch: feature/hierarchical-multitenancy
Review: https://review.openstack.org/138182

Thierry Carrez (ttx) on 2014-12-17

Changed in keystone:
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2015-04-30

Changed in keystone:
milestone:	kilo-1 → 2015.1.0

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.