Keystone client token cache doesn't respect revoked tokens

[Alexei Kornienko]

If we'll enable caching for keystoneclient tokens we'll be able to use tokens that are already revoked if they are present in cache:

https://github.com/openstack/python-keystoneclient/blob/0.6.0/keystoneclient/middleware/auth_token.py#L831

steps to recreate:
1) get a token
2) use it to make a request via keystoneclient using default properties (thus it will be cached)
3) delete the token
4) use the token to make another request via keystoneclient

expected result: the token should not work (HTTP 401)
actual result: the token still works

[Alexei Kornienko (alexei-kornienko)]

There is a fix available - https://review.openstack.org/#/c/78241/

[Tristan Cacqueray (tristan-cacqueray)]

Draft impact description #1:

Title: Token revocation does not revoke cached tokens
Reporter: Alexei Kornienko (Mirantis)
Products: python-keystoneclient
Affects: All versions up to 0.6.0

Description:
Alexei Kornienko from Mirantis reported a vulnerability in Keystone auth_token middleware (shipped in python-keystoneclient). Once an user is authenticated to a service, issuing a token revocation for this user won't prevent him from using that service with the same token until it is expired. Only Keystone middleware setups using auth_token with PKI token and cache enabled are affected.

[Tristan Cacqueray (tristan-cacqueray)]

Affects line may be too wide, it seems cache was introduced in version 0.2.0. So:

Affects: 0.2.0 versions up to 0.6.0

[Dolph Mathews (dolph)]

After attempting to reproduce, this doesn't appear to be nearly as severe as it initially seemed to me. In fact, everything appears to behave as designed and I'd qualify the issue here as an opportunity for better documentation, better default values and security hardening, rather than any sort of vulnerability.

Per-token caching behavior is configurable via token_cache_time (defaults to 300 seconds, reasonable for use in production).

Revocation list caching is configurable via revocation_cache_time (defaults to 1 second, not particularly reasonable for use in production).

The large gap between the default values creates a poor expectation that revocations will take effect within 1 second (that'd be great if it were true, thus the opportunity for security hardening), where they're actually only *guaranteed* to take effect within max([token_cache_time, revocation_cache_time]).

This is certainly poorly documented, especially given the help text for "revocation_cache_time" which reads "Value only used for unit testing." This unfortunately refers to the default value, not the configuration option itself (no help is actually provided for the configuration option itself).

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/78650

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/78700

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/78650
Committed: https://git.openstack.org/cgit/openstack/python-keystoneclient/commit/?id=813f80cab84b436d42f3d7909957348b25bc34c2
Submitter: Jenkins
Branch: master

commit 813f80cab84b436d42f3d7909957348b25bc34c2
Author: Dolph Mathews <email address hidden>
Date: Thu Mar 6 10:25:26 2014 -0600

    improve configuration help text in auth_token

    Related-Bug: 1287301
    Change-Id: I3499fdf87b49b7499c525465ccd947edfa5cec6d

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/78700
Committed: https://git.openstack.org/cgit/openstack/python-keystoneclient/commit/?id=c59f0ab2634fa844f7e1fa02abcb145b3f14e7f7
Submitter: Jenkins
Branch: master

commit c59f0ab2634fa844f7e1fa02abcb145b3f14e7f7
Author: Dolph Mathews <email address hidden>
Date: Thu Mar 6 10:26:58 2014 -0600

    increase default revocation_cache_time

    1 second was originally intended as a useful value for testing purposes,
    but was never intended to be a production-friendly default. Raising the
    default value to match the default token_cache_time results in better
    production performance.

    Change-Id: I03939d41bc50b32a6aa5ebf93eb71103e16e988f
    Related-Bug: 1287301

[Jeremy Stanley (fungi)]

Tagging security. The OSSG may decide this is worth drafting a note about, for broader visibility within the community.

[Matthew Edmonds (edmondsw)]

caching tokens for 5 minutes by default may be all well and good for performance, but not so much for security. Consider the following cases:
1) If an admin detects that someone is using a token maliciously, they'll delete it and expect that to stop the usage immediately. But it won't.
2) If someone deletes the token they were using and then walks away, they should not have to worry about someone else stepping up and continuing to use that token.

Is token caching really something we should be doing at all? By default?

If so, should the default really be as high as 5 minutes? How did we settle on such a large value?

Should we implement a notification mechanism for token revokation which would cause listening clients to update their cache immediately? (Note: someone may find a way to block the notification, so this isn't perfect...)

[David Chadwick (d-w-chadwick)]

Implementing security controls is always a balance between cost, usability, effectiveness etc. So there are no right answers. It all depends upon the risk aversion (or willingness) of the organisation. Therefore having configuration values for token cache time and revocation cache time seems to be the best way to deal with this (including allowing a zero cache time). The risk averse organisation can set low or zero values and take the cost and performance consequences of this, whereas the risk willing organisation can set much higher values and have lower costs, higher performance and a greater risk of revoked tokens being wrongly used. I dont really see what Matthew's problem is providing this is clearly documented (as Dolph agrees it should be)

[Alexei Kornienko (alexei-kornienko)]

This particular issue has a simple solution and I proposed a fix to keystone client - https://review.openstack.org/#/c/78241/
With such fix we won't have to choose between cache efficiency and security for the cost of some additional computation

[Robert Clark (robert-clark)]

Very good points raised here.

I think this is going to come down to a decision to be made by the deployer - do I use token caching and for how long?

Secure deployments will likely not use caching, deployments with moderate requirements might want to use a shorter cache life and isolated or low risk clouds may even use longer life caches.

Personally I think there's good grounds here for not only an OSSN but also an entry in the OpenStack Security Guide, discussing the tradeoff and possible compensating controls/procedures.

[Matthew Edmonds (edmondsw)]

I don't believe this only affects PKI tokens, as stated in comment 2. I hit this with UUID tokens.

[Brant Knudson (blk-u)]

d-w-chadwick , robert-clark : Based on the fix in https://review.openstack.org/#/c/78241/ , it looks like if the token is validated once and put in the cache, then after that the token is valid for the entire cache period (token_cache_time, which defaults to 5 mins) even if the token is revoked.

With PKI tokens, we've now got the revocation list... should auth_token be checking the revocation list for both UUID and PKI tokens?

With the latest change to set the revocation_cache_time to 5 minutes, then if a deployment is using the defaults there's nothing to gain by checking the revocation list... it'll be just as out of date as the cache. If someone was willing to set a lower time for revocation_cache_time then they'd have the shorter valid period for tokens.

What I'm trying to figure out is -- if we take the fix in https://review.openstack.org/#/c/78241/ will that provide more flexibility for the deployer to pick how they want revocations to work with caching? They can set revocation_cache_time to 10 seconds and tokens will be valid for 10 seconds after they're revoked... but they can already do the same thing by setting a short cache time, and then the code would be using the normal flow for UUID tokens which validates them against keystone rather than getting the revocation list.

[Matthew Edmonds (edmondsw)]

setting a higher value for token_cache_time and a lower value for revocation_cache_time (assuming we start using the revocation list here as proposed by https://review.openstack.org/#/c/78241/) would allow you to gain the performance improvement of not having to re-request tokens as often while satisfying the security requirement that revocation take effect in a timely manner. Yes, the revocation list is being requested more frequently, and may offset some of the performance gains from caching tokens. But the revocation list can be used to validate any token, so multiple tokens could be validated over the life of the cached revocation list, instead of each token validation requiring a call back to keystone should token_cache_time be similarly reduced.

[David Chadwick (d-w-chadwick)]

can I make a few assertions that might help in these deliberations.
1. If you dont cache tokens you dont need revocation lists. You get a fresh token each time.
2. There is no point in having a revocation cache time longer than a token cache time, because once a token cache time expires a new token has to be obtained. So if the token has been revoked you wont learn about it.
3. There is no point in having a revocation cache time equal to a token cache time because they will both expire at the same time, so even if the token has been revoked you still wont learn about it (until you get the next revocation list).
4. So the only sensible solution is to have a token cache time significantly longer than the revocation cache time (I would say at least twice as long, since the probability that a token has been revoked in the second half of its life is 50%. Using longer revocation times you decrease the probability that you will learn about the revocation.
5. If you have short token cache times, you probably dont need revocation lists.

[Alexei Kornienko (alexei-kornienko)]

d-w-chadwick I'm sorry but you are wrong in some of your assertions:

1. If you dont cache tokens you dont need revocation lists. You get a fresh token each time.
Cache is needed to speed up validation. For UUID tokens validation requires HTTP request to keystone and for PKI tokens it requires a subprocess call to openssl.
If you disable cache you are still able to use the same token until it's *expired*.

Rest of the assertions has to be updated to separate token exparation and validation cache.

[Dolph Mathews (dolph)]

Revocation lists are currently only required to prematurely invalidate PKI tokens, regardless of caching.

[Abu Shohel Ahmed (shohel-csdu)]

Some pros and cons analysis on this topic. What are the related threats and
their impacts.

…shohel

On 12 Mar 2014, at 18:35, Matthew Edmonds <email address hidden> wrote:

> setting a higher value for token_cache_time and a lower value for
> revocation_cache_time (assuming we start using the revocation list here
> as proposed by https://review.openstack.org/#/c/78241/) would allow you
> to gain the performance improvement of not having to re-request tokens
> as often while satisfying the security requirement that revocation take
> effect in a timely manner. Yes, the revocation list is being requested
> more frequently, and may offset some of the performance gains from
> caching tokens. But the revocation list can be used to validate any
> token, so multiple tokens could be validated over the life of the cached
> revocation list, instead of each token validation requiring a call back
> to keystone should token_cache_time be similarly reduced.
>
> --
> You received this bug notification because you are a member of OpenStack
> Security Group, which is subscribed to OpenStack.
> https://bugs.launchpad.net/bugs/1287301
>
> Title:
> Keystone client token cache doesn't respect revoked tokens
>
> Status in OpenStack Security Advisories:
> Invalid
> Status in Python client library for Keystone:
> In Progress
>
> Bug description:
> If we'll enable caching for keystoneclient tokens we'll be able to use
> tokens that are already revoked if they are present in cache:
>
> https://github.com/openstack/python-
> keystoneclient/blob/0.6.0/keystoneclient/middleware/auth_token.py#L831
>
> steps to recreate:
> 1) get a token
> 2) use it to make a request via keystoneclient using default properties (thus it will be cached)
> 3) delete the token
> 4) use the token to make another request via keystoneclient
>
> expected result: the token should not work (HTTP 401)
> actual result: the token still works
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ossa/+bug/1287301/+subscriptions
>
> _______________________________________________
> Openstack-security mailing list
> <email address hidden>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-security

[Abu Shohel Ahmed (shohel-csdu)]

The earlier link somehow did not showed in the email. Some analysis i have done
on this topic: Threats Vs Performanace

https://drive.google.com/file/d/0B1aEVfmQtqnoM0luMFpMMXh4RkE/edit?usp=sharing

…shohel

On 13 Mar 2014, at 14:29, Abu Shohel Ahmed <email address hidden> wrote:

> Some pros and cons analysis on this topic. What are the related threats and
> their impacts.
>
>
>
>
> …shohel
>
>
> On 12 Mar 2014, at 18:35, Matthew Edmonds <email address hidden> wrote:
>
>> setting a higher value for token_cache_time and a lower value for
>> revocation_cache_time (assuming we start using the revocation list here
>> as proposed by https://review.openstack.org/#/c/78241/) would allow you
>> to gain the performance improvement of not having to re-request tokens
>> as often while satisfying the security requirement that revocation take
>> effect in a timely manner. Yes, the revocation list is being requested
>> more frequently, and may offset some of the performance gains from
>> caching tokens. But the revocation list can be used to validate any
>> token, so multiple tokens could be validated over the life of the cached
>> revocation list, instead of each token validation requiring a call back
>> to keystone should token_cache_time be similarly reduced.
>>
>> --
>> You received this bug notification because you are a member of OpenStack
>> Security Group, which is subscribed to OpenStack.
>> https://bugs.launchpad.net/bugs/1287301
>>
>> Title:
>> Keystone client token cache doesn't respect revoked tokens
>>
>> Status in OpenStack Security Advisories:
>> Invalid
>> Status in Python client library for Keystone:
>> In Progress
>>
>> Bug description:
>> If we'll enable caching for keystoneclient tokens we'll be able to use
>> tokens that are already revoked if they are present in cache:
>>
>> https://github.com/openstack/python-
>> keystoneclient/blob/0.6.0/keystoneclient/middleware/auth_token.py#L831
>>
>> steps to recreate:
>> 1) get a token
>> 2) use it to make a request via keystoneclient using default properties (thus it will be cached)
>> 3) delete the token
>> 4) use the token to make another request via keystoneclient
>>
>> expected result: the token should not work (HTTP 401)
>> actual result: the token still works
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/ossa/+bug/1287301/+subscriptions
>>
>> _______________________________________________
>> Openstack-security mailing list
>> <email address hidden>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-security
>
>
> ** Attachment added: "Token_Access_scenario_CACHE Sheet1.pdf"
> https://bugs.launchpad.net/bugs/1287301/+attachment/4022028/+files/Token_Access_scenario_CACHE%20Sheet1.pdf
>
> --
> You received this bug notification because you are a member of OpenStack
> Security Group, which is subscribed to OpenStack.
> https://bugs.launchpad.net/bugs/1287301
>
> Title:
> Keystone client token cache doesn't respect revoked tokens
>
> Status in OpenStack Security Advisories:
> Invalid
> Status in Python client library for Keystone:
> In Progress
>
> Bug description:
> If we'll enable cachi...

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/78241
Committed: https://git.openstack.org/cgit/openstack/python-keystoneclient/commit/?id=8574256f9342faeba2ce64080ab5190023524e0a
Submitter: Jenkins
Branch: master

commit 8574256f9342faeba2ce64080ab5190023524e0a
Author: Alexei Kornienko <email address hidden>
Date: Wed Mar 5 16:50:37 2014 +0200

    Ensure that cached token is not revoked

    We need to ensure that tokens won't stay in cache after they have been
    revoked.

    Changed default revocation_cache_time 300 -> 10 seconds.
    revocation_cache_time has to be << than token_cache_time to make token
    cache efficient.

    Fixes bug #1287301

    Change-Id: I14c0eacac3b431c06e40385c891a6636736e5b4a