fernet token validation is slow

[Matt Fischer]

keystone token validation operations are much slower than uuid operations. The performance is up to 4x slower which makes other openstack API calls slower too.

Numbers from Dolph:

Token validation performance

Response time Requests per second
UUID 18.8 ms (baseline) 256.7 (baseline)
Fernet 93.8 ms (400% slower) 48.3 (81% slower)

My numbers running on a basic setup running keystone in a VM without a load balancer:

Tokens per second (serial):
UUID: 14.97
Fernet: 3.66

Tokens per second (concurrent 20 threads):
UUID: 46.18
Fernet: 12.92

Our numbers are similarly bad in production and its impacting OpenStack performance when we're under load.

[Lance Bragstad (lbragstad)]

related fix - https://review.openstack.org/#/c/215212/

[Dolph Mathews (dolph)]

Lance: if we can benchmark a performance improvement with that patch, it should be a Partial fix, not just Related.

[Lance Bragstad (lbragstad)]

I'll see if I can find some time to stand up an environment to test against.

[Lance Bragstad (lbragstad)]

More related/partial fixes - https://review.openstack.org/#/q/status:open+project:openstack/keystone+branch:master+topic:bug/1489061,n,z

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/215212
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=0bbf4c5b73c4a49a9c8d574b3326d0a16cbf6c2c
Submitter: Jenkins
Branch: master

commit 0bbf4c5b73c4a49a9c8d574b3326d0a16cbf6c2c
Author: Lance Bragstad <email address hidden>
Date: Fri Oct 16 16:51:21 2015 +0000

    Add caching to get_catalog

    This commit adds caching to the get_catalog call, per user ID and
    project ID. Both the v2 and v3 catalogs are built based on the result of
    this method, so this provides caching for both APIs.

    As a result, this should provide a performance benefit to Fernet-based
    deployments, which must otherwise rebuild the catalog on each token
    validation.

    Tested on top of the current stable/liberty, this patch improves the
    time of cacheable Fernet token creation requests by 16.7% and of
    Fernet token validation requests by 7.5%:

      https://gist.github.com/dolph/3bf24039b83a147eeb5c

    Change-Id: I147c5b8e7dcfe1786fe8a534182dbeef33925eac
    Co-Authored-By: Dolph Mathews <email address hidden>
    Related-Bug: 1489061

[Lance Bragstad (lbragstad)]

We are also in the process of adding caching to role assignments - https://review.openstack.org/#/c/215715/

Which is related to this review.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/215715
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=cd906266e67c8a8a8ae9676d2fa6dd1bd2f59bd8
Submitter: Jenkins
Branch: master

commit cd906266e67c8a8a8ae9676d2fa6dd1bd2f59bd8
Author: Morgan Fainberg <email address hidden>
Date: Fri Jan 22 13:13:46 2016 -0800

    Add caching to role assignments

    This patch aims to optimize the token creation
    time by caching role assignments used for
    populating user roles in the token data.

    Two methods in the assignment manager are cached:

    - get_roles_for_user_and_domain
    - get_roles_for_user_and_project

    This change, in addition to I147c5b8e7dcfe1786fe8a534182dbeef33925eac,
    reduces the Fernet token creation time by 28%.

    The full performance report can be found here:

      https://gist.github.com/dolph/3bf24039b83a147eeb5c

    Partial-Bug: 1489061

    Co-Authored-By: Samuel de Medeiros Queiroz <email address hidden>
    Co-Authored-By: Morgan Fainberg <email address hidden>

    Change-Id: Id6ee76f896fe738b4c4de432eb57cc0437ffdb6b

[Steve Martinelli (stevemar)]

Lance, should we close this up?

No other recent fixes and I'm not sure on the numbers, but caching the catalog had to have helped quite a bit,

[Matt Fischer (mfisch)]

Lets close it out. I will generate new numbers are we Dockerize and upgrade keystone hopefully in March.

[Dolph Mathews (dolph)]

This patch should address my last few concerns revealed from profiling a synthetic benchmark run:

  https://review.openstack.org/#/c/272007/

[Morgan Fainberg (mdrnstm)]

Marked incomplete awaiting more data. There has been a lot of varied updates to address token performance.

This change of status is based on convo w/ mfisch in IRC on March 2, 2016

[Launchpad Janitor (janitor)]

[Expired for OpenStack Identity (keystone) because there has been no activity for 60 days.]

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/298748
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=54da44bd41f956b43e018b69df2eb65a53e62fc1
Submitter: Jenkins
Branch: master

commit 54da44bd41f956b43e018b69df2eb65a53e62fc1
Author: Lance Bragstad <email address hidden>
Date: Fri Jun 3 04:19:45 2016 +0000

    Cache service providers on token validation

    On fernet token validation we rebuild the entire token catalog. This includes
    rebuilding the service providers references in the token if using a federated
    token. In the mitaka release, patches were added to cache user catalogs and
    role assignments. This allows us to do the same thing for service providers,
    which decreases the time spent in get_token_data() by 34%.

    Co-Authored-By: Dolph Mathews <email address hidden>

    Related-Bug: 1489061
    Change-Id: I3dfe66880295ef36a18ca3e384191106b74be795