Use memcached for keystone tokens

[Boris Pavlovic]

Keystone sql storage for storing tokens doesn't work at scale at all.

Issue:
1) We never delete any token from tokens tables
(And amount of tokens is pretty big millions)
2) We are doing a lot of checks does token exist (so a big load on DB)

As a result it works super slow. + Will die after N days (because there will be too much tokens in DB)

It will be better to use memcached as a backend for tokens. Because it has:
1) built in mechanism for purging expired records
2) stores all data in memory (not on disk) => works much faster
3) handle bigger load
4) reduce load on mysql server of OpenStack installation

As a result much better performance of whole system

[Sergey Vasilenko (xenolog)]

Related bugs and links:
https://bugs.launchpad.net/keystone/+bug/1247703
https://mirantis.jira.com/browse/OSCI-1005

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/67496

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/67496
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=b8b954caa4d26f0a4cfe8043535ec8d2bb7c49d1
Submitter: Jenkins
Branch: master

commit b8b954caa4d26f0a4cfe8043535ec8d2bb7c49d1
Author: Sergey Vasilenko <email address hidden>
Date: Fri Jan 17 19:29:56 2014 +0400

    Keystone should store tokens in memcache if it present.

    Without it we have decrease mysql perfomance day by day at big clesters.

    Change-Id: Ib683574411c8e68ac18f7d55c9ce13fd0729d1ea
    Closes-bug: #1269819

[Bogdan Dobrelya (bogdando)]

Please make sure the memcache sync issue https://bugs.launchpad.net/fuel/+bug/1251190 won't affect this one

[Roman Alekseenkov (ralekseenkov)]

It looks like implementation with memcached has a lot of issues. As we discussed with Dmitry Borodaenko, we can just use MySQL and create a cron job (or modify keystone code) to periodically purge expired tokens from the DB. with LIMIT 1000 in the DELETE clause, to avoid putting long locks on the database

[Matthew Mosesohn (raytrac3r)]

It really impacts performance, so updating to Medium.

[Tatyanka (tatyana-leontovich)]

Reproduced on 235 iso on simple deployment after ostf run.
Ubuntu, netutron vlan with untagged networks and murano.
1 controller + 4 computes - deploy env and run ostf - as result http://paste.openstack.org/show/73128/

[Tatyanka (tatyana-leontovich)]

seems this is a diff problem - will create a separate isssue

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: master
Review: https://review.openstack.org/91303

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/91303
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=0a3cdce60389ea0a6adc8f97afc7e2e9bfd25210
Submitter: Jenkins
Branch: master

commit 0a3cdce60389ea0a6adc8f97afc7e2e9bfd25210
Author: Sergii Golovatiuk <email address hidden>
Date: Wed Apr 30 08:56:37 2014 +0000

    Implements memcache storage for keystone tokens

    Switching keystone token storage to memcache with dogpile cache implementation.
    This allows to use memcache as non persistence storage in HA mode.

    Change-Id: I511f5849206455d65ca11ecbb8e8c1c3f639fbcb
    Closes-Bug: 1269819

[Meg McRoberts (dreidellhasa)]

Documented as Fixed Issue in 5.0 Release Notes, with link to section in Operations Guide about how to use Percona to clean expired tokens from the Keystone database.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/4.1
Review: https://review.openstack.org/100173

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sergii Golovatiuk (<email address hidden>) on branch: stable/4.1
Review: https://review.openstack.org/100173