Disabling users & groups may not invalidate previously-issued tokens

[Yukihiro KAWADA]

Even if the user is disabled, can use the last token is validated.

0. user foo is enable
1. get token (a)
2. user foo is disabled
3. foo can still use any APIs by token(a)

that's all.
This issue is not cache process.

[Tristan Cacqueray (tristan-cacqueray)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

[Morgan Fainberg (mdrnstm)]

Is this using V2 validation, V3 validation? What version (release name) are you running [master? Juno? Icehouse? earlier?]

Is this using Keystone's APIs? or other services APIs?

[Morgan Fainberg (mdrnstm)]

Is this PKI Tokens, UUID, PKIZ, Fernet?

[Morgan Fainberg (mdrnstm)]

Is this using LDAP Identity or SQL Identity backends? Is this a federated identity user?

If this is using LDAP, was the user disabled via an update to the keystone API or disabled directly in the backend (e.g. disabled in Active Directory/LDAP not via the Keystone API)?

[Yukihiro KAWADA (warp-kawada)]

I'm using juno, uuid, sql, v2 and v3.
But these things would not matter.

Please try like this.
1. get foo's token (a) using keystone api.
2. user foo is disabled. (direct operation to mysql : keystone.user.enabled=0)
3. get new token using keystone api. then can't get new token. this is ok.
4. but foo CAN nova list api by token(a)

This issue is because token/provider.py does not check the status of the user or tenant.

[Morgan Fainberg (mdrnstm)]

After doing a bunch of research, this is *not* a bug in Keystone. Keystone properly invalidates the token(s) assuming you are not doing a direct update of the database (e.g. with SQL) and using the keystone API to disable the user (such as with the CLI or Curl).

What is happening is that keystonemiddleware defaults to caching for ~300 second in-memory. This means that once the token has been used against an endpoint, there is 300 second window where the token will be valid even if the user has been disabled.

If the option in nova.conf the [keystone_authtoken] section, the cache time is controlled by the 'token_cache_time' option.

I performed the following tests:

1) Get token (for this example, token is 2ee5b4d696da4364800f9a703a316c2e)

2) Get server list from nova (either by curl or by nova cli) succeeds.
* curl -i 'http://10.0.2.15:8774/v2/5cef269803424bdca931da8a06190f62/servers/detail' -X GET -H "Accept: application/json" -H "X-Auth-Project-Id: demo" -H "X-Auth-Token: 2ee5b4d696da4364800f9a703a316c2e"
* nova --os-auth-token a58ce46932784f569432054a68be86af list

3) Disable user via either keystone cli or curl:
* curl -g -i --cacert "/opt/stack/data/CA/int-ca/ca-chain.pem" -X PUT http://10.0.2.15:35357/v2.0/users/5d9ca3899f0f4db3a813990739dcbb52 -H "User-Agent: python-keystoneclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: c1ebb3c3300841d7b63745b3a233e99d" -d '{"user": {"enabled": false, "id": "5d9ca3899f0f4db3a813990739dcbb52"}}'
* keystone update-user --enabled false 5d9ca3899f0f4db3a813990739dcbb52

4) Get server list from nova (same as above) works.
5) After 300 seconds the the command now appropriately fails with a 401.

If the 'token_cache_time' option is set to 1 in the nova.conf, the above scenario acts more as expected.

The token/provider.py explicitly does not check user/tenant enabled as it relies on either the Token Revocation List (a list of invalidated tokens) or the Revocation Event System (a list of events that can invalidate tokens, such as 'invalidate all tokens for user x'). The intent is that Keystone relies on a consistent way to determine if tokens are valid vs. needing to examine the DB for all the details, since there are cases that we need to produce other invalidations (e.g. password change).

=====================

If a direct update of the SQL DB is performed, there is currently no direct method for invalidation of all tokens for a user. This could be generally improved to validate where users are query-able. Alternatively we could provide an API that allows for issuing the revocation events that would be expected for disabled users (etc). This is also an issue with LDAP Identity in read-only mode.

=====================

In short there are two issues:

1. The default caching in keystonemiddleware produces a sub-optimial experience with cached tokens.

2. The provider validation line should be improved to assert enabled, or at the very least provide an API to indicate a change in state including disabling identity resources. Assignment resources are currently assumed to be managed by keystone (there is no read-only mode) and therefore it is expected that the Key...

[Morgan Fainberg (mdrnstm)]

Icehouse will need to be evaluated independently. This is confirmed to affect both Juno and Kilo.

[Yukihiro KAWADA (warp-kawada)]

First, I doubted the cache.
So, of course I checked set to nova.token_cache_time=-1.
Actually keystonemiddleware did not use cache.

# configurable duration (in seconds). Set to -1 to disable
# caching completely. (integer value)
#token_cache_time=300
# Y.Kawada
token_cache_time=-1

And I re-checked.
token_cache_time=1 is same result.

I was confirmed using keystone client not operate the database directly.

keystone --debug user-update --enabled false 568cdf76c85a457bae3a7f8bc15fd72d
Then, result was same.
I can use nova list after 1sec or 300 sec.

keystone token-get;date
+-----------+----------------------------------+
| Property | Value |
+-----------+----------------------------------+
| expires | 2015-03-24T04:19:45Z |
| id | 83242b13093544b5bf1ef629cc372485 |
| tenant_id | 26b0778356a343739578dcebbf48c486 |
| user_id | 568cdf76c85a457bae3a7f8bc15fd72d |
Mon, 23 Mar 2015 04:20:11 GMT

keystone --debug user-update --enabled false 568cdf76c85a457bae3a7f8bc15fd72d; date
User has been updated.
Mon Mar 23 13:20:43 JST 2015

curl -i 'http://nova-host:8774/v2/xxxxxxxxxxxxxxxxx/servers/detail' -X GET -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Project-Id: vps_kawada" -H "X-Auth-Token: 83242b13093544b5bf1ef629cc372485";date
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 1667
X-Compute-Request-Id: req-d0aeb86a-471c-4b5e-b830-74d5893af02a
Date: Mon, 23 Mar 2015 04:44:19 GMT
{"servers": [{"status": "ACTIVE", "updated": "2015-03-19T01:22:30Z", "hostId": "
:
 "metadata": {}}]}
Mon Mar 23 13:44:19 JST 2015

[Yukihiro KAWADA (warp-kawada)]

my solution codes:

diff --git a/keystone/token/provider.py b/keystone/token/provider.py
index fb41d4b..e4cea63 100644
--- a/keystone/token/provider.py
+++ b/keystone/token/provider.py
@@ -284,6 +284,24 @@ class Manager(manager.Manager):
             # Get the data we need from the correct location (V2 and V3 tokens
             # differ in structure, Try V3 first, fall back to V2 second)
             token_data = token.get('token', token.get('access'))
+
+ user_data = token_data['user']
+ user_ref = self.identity_api.get_user(user_data['id']) ## Y.Kawada
+
+ if not user_ref.get('enabled', True):
+ msg = _('User is disabled: %s') % user_ref['id']
+ LOG.warning(msg)
+ raise exception.Unauthorized(msg)
+
+ _token_data = token_data.get('token', token_data)
+ project_data = _token_data.get('tenant', _token_data.get('project'))
+ if project_data:
+ project_ref = self.assignment_api.get_project(project_data['id'])
+ if not project_ref.get('enabled', True):
+ msg = _('Project is disabled: %s') % project_ref['id']
+ LOG.warning(msg)
+ raise exception.Unauthorized(msg)
+
             expires_at = token_data.get('expires_at',
                                         token_data.get('expires'))

But run_test.sh reports v3_federation FAILs.

[Adam Young (ayoung)]

DIsabling the user should revoke the token. Why is this not happening?

[Morgan Fainberg (mdrnstm)]

@Adam,

This is the issue with LDAP / Read-Only backends *and* a secondary issue with default middleware behavior:

===== Read-Only Backend ======
We don't know a user was disabled. We should check this when live-validating a token. PKI(Z) nothing we can do. The request to do a .user_is_enabled check is not unreasonable (for Federated users we obviously can't do this, but that is a separate concern wrt to the notification from IDP on user disable, this is an expected gap that we should be addressing in Liberty).

===== Keystonemiddleware w/o TRL checking (default) =====
KSM caches for ~300s, we will keep tokens valid once they've been validated. TRL is a PKI-ism, and can (is?) disabled by default with UUID since we don't have signing infrastructure.

[Morgan Fainberg (mdrnstm)]

I'll split the cache bit out into it's own bug so we can issue an OSSA/OSSN [not sure which is correct], because the behavior should be *expected*, but it is by design. I want to revisit that design decision.

[Brant Knudson (blk-u)]

Is this the same as https://bugs.launchpad.net/python-keystoneclient/+bug/1287301 ?

[Morgan Fainberg (mdrnstm)]

@Brant: I split the ksm part to: https://bugs.launchpad.net/keystonemiddleware/+bug/1435530, this is not directly related.

[Morgan Fainberg (mdrnstm)]

Attached is a fix for this issue against Master (Kilo). This is loosely based on the ideas from Y.Kawada.

@Y.Kawada, if you have an email (and signed the appropriate CLA/under a CCLA) I'd be happy to put you down as a co-author for this fix. Let me know either here or on IRC (I'm "morganfainberg" on FreeNode).

---

This will need to be backported to Juno and Icehouse (if Icehouse has not hit actual EOL by the time it's ready)

[Guang Yee (guang-yee)]

Can we do the fix at the driver?

https://github.com/openstack/keystone/blob/master/keystone/identity/core.py#L754

Like

if enabled_change or user.get('password') is not None or user['enabled'] is False:
    self.emit_invalidate_user_token_persistence(user_id)

[Yukihiro KAWADA (warp-kawada)]

I already have singed ICLA.
*送信者 :* OpenStack LLC (OpenStack LLC)
*宛先 :* Yukihiro Kawada
*日付 :* 2012-09-11 5:38 PM
*ステータス :* 署名済み
This email is "<email address hidden>".

But we have not singed CCLA yet.

Thank you.

2015-03-27 7:54 GMT+09:00 Morgan Fainberg <email address hidden>:

> Attached is a fix for this issue against Master (Kilo). This is loosely
> based on the ideas from Y.Kawada.
>
> @Y.Kawada, if you have an email (and signed the appropriate CLA/under a
> CCLA) I'd be happy to put you down as a co-author for this fix. Let me
> know either here or on IRC (I'm "morganfainberg" on FreeNode).
>
> ---
>
> This will need to be backported to Juno and Icehouse (if Icehouse has
> not hit actual EOL by the time it's ready)
>
> ** Patch added: "Patch to solve #1434034 for Keystone Master (Kilo)"
>
> https://bugs.launchpad.net/keystone/+bug/1434034/+attachment/4357437/+files/0001-Assert-the-user-and-trustor-are-enabled-when-validat.master.patch
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1434034
>
> Title:
> Even if the user is disabled, can use the last token is validated
>
> Status in OpenStack Identity (Keystone):
> Triaged
> Status in Keystone juno series:
> Triaged
> Status in OpenStack Security Advisories:
> Incomplete
>
> Bug description:
> Even if the user is disabled, can use the last token is validated.
>
> 0. user foo is enable
> 1. get token (a)
> 2. user foo is disabled
> 3. foo can still use any APIs by token(a)
>
> that's all.
> This issue is not cache process.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/keystone/+bug/1434034/+subscriptions
>

--
===
Yukihiro KAWADA

[Adam Young (ayoung)]

Need to handle groups, too. Basically, we need to recreate the whole token from the identity assertion on forward. Otherwise we will have the same issue when a user is removed from a group, the token will have a role on it that is no longer valid.

This is never going to work for Federation, as we will not be able to check at token validation time.

[Morgan Fainberg (mdrnstm)]

@Guang,

This is when the state of the user changes outside of the API, think read-only LDAP (AD).

@Adam,

Yep, I'll include groups. Federation has another mechanism to communicate when things are disabled/deleted. There is an open spec/bp/bug for it and explicitly avoided in this case.

@Yukihiro Kawada,

I will add you as co-author in the next proposed fix.

[Morgan Fainberg (mdrnstm)]

The more I am digging into this the more I realize that the only clear option to handle all the cases is to leverage some of the Fernet code to build the token from a basic set of data and ensure that the roles, etc exist as expected.

The general workflow to fix this bug is to make UUID tokens reconstruct the payload based upon a reduced set of data similar to what is stored for Fernet tokens (basically uuid tokens become a similar method to fernet, but with out the fernet encapsulation).

PKI tokens cannot be fixed due to signed datasets.

This is looking like it is not something that is easily fixible in kilo. The patch proposed can address the user-disable case but cannot easily address the group conveying a role to the user disabled case or even inheritance cases.

I am thinking that this is something that will need to restructure a bunch of code to be solved. The real question is if addressing this in Kilo is worth the large volume of code restructuring, even knowing PKI tokens cannot be "solved".

The options are:

1) solve the user-disabled case, knowing that group disable does not work (tokens must natually expire if you disable a group in a read-only backend). This can be backported. OSSA/OSSN can be issued indicating groups cannot be handled. (I have minor corrections to make in the patch currently proposed before it is ready).

2) solve the case by restructuring UUID tokens to work like fernet and reconstruct the full token data from a very very limited data set (easy to add compatibility code in for this to support a transition if we want to all outstanding tokens to continue to work when upgrading) - this would *not* be a token table migration. This is possibly backportable, but would be a very large/scary backport. OSSA/OSSN can be issued indicating that previous releases are affected *if* the scary backport is not warranted.

3) Defer this until Liberty, knowing a backport will be nearly impossible (same as 2), and really focus on the provider cleanup that can directly address the issue. PKI(Z) tokens will not be guaranteed to see this type of revocation working. Same idea on OSSA/OSSN for option 2.

--- I'd like some feedback on the preferred approach.

[Thierry Carrez (ttx)]

If I understand correctly the worse case scenario here is a slightly deferred token invalidation, which I would not consider OSSA (advisory) material (could be considered "working as designed"). It is certainly OSSN (security note / documentation) material though.

Given the impact and in order to facilitate work on this, I propose we open this bug publicly.

I think I have a slight preference for option 1 + OSSN in the immediate future.

[Morgan Fainberg (mdrnstm)]

@thierry,

Unfortunately, in this case that is incorrect. The part split out to keystone middleware bug is a deferred invalidation. The bug here is that in the case of a read-only ldap backend (e.g. Active directory), if a user or group is disabled/deleted/etc, the tokens for those users (or authorization for the users in the group) will not be revoked at all.

This means that interactions with APIs that are not keystone (nova, glance, etc) will continue to work for the life of the token. The interaction with the keystone api should be prevented in some cases (some Apis check if user from the context exists).

The only token provider unaffected is the new fernet tokens because keystone must lookup the user / groups to validate the token.

Even if this is fixed for UUID tokens, PKI tokens will still see this gap for non-Keystone APIs when the pki token is validated at the endpoint, since there is no active way for keystone to communicate that a user was disabled directly in the backend. Read-only ldap idenity is a common deployment mode.

[Morgan Fainberg (mdrnstm)]

I guess the deferred invalidation would be the life of the token, which for many environments is set to 86400s or higher. (Many deployers increase the TTL)

[Jeremy Stanley (fungi)]

Yes, still deferred invalidation, just via expiration rather than revocation in this regard. It can't be leveraged to gain access to something the user never had access to though, was I think the point of the distinction.

[Morgan Fainberg (mdrnstm)]

Right. I'll respin he patch to cover the minor fixes today so we can move with option 1 as needed.

[Morgan Fainberg (mdrnstm)]

Respin with commit message/co-author added for the master patch:

[Morgan Fainberg (mdrnstm)]

Patch in #26 does not address group issues.

[Morgan Fainberg (mdrnstm)]

This version addresses adding the identity_api to the token_provider manager.

[Morgan Fainberg (mdrnstm)]

Fixes for master [including removing a regression i introduced when fixing the provider having identity_api on it.

[Morgan Fainberg (mdrnstm)]

Fixes for Juno, based on master. The code base has changed a bunch so it wasn't even possible to show a "conflicts" easily. It is a straight port.

[Morgan Fainberg (mdrnstm)]

Both #29, and #30 pass pep8 and py27 tests locally.

[Thierry Carrez (ttx)]

This one is a bit borderline, but I would err on the side of issuing an advisory here.

[Morgan Fainberg (mdrnstm)]

Added Samuel de Medeiros Queiroz as he reported a bug that is specifically related to this bug.

[Tristan Cacqueray (tristan-cacqueray)]

Morgan, Icehouse EOL is 15 month after its release, so it won't happen until somewhere in July... So we'll need a fix there too.

Title: User token revocation does not work with read-only LDAP backend
Reporter: Yukihiro KAWADA (GMO Internet, Inc)
Products: Keystone
Affects: up to 2014.1.4 and 2014.2 versions through 2014.2.2

Description:
Yukihiro KAWADA from GMO Internet, Inc reported a vulnerability in Keystone read-only LDAP backend. When a user or group is disabled/deleted, the tokens for those users (or authorization for the users in the group) will not be revoked at all and will only expire according to the tokens expiration date. Only setups using a read-only LDAP backend in Keystone are affected.

[Morgan Fainberg (mdrnstm)]

We may want to add information that the PKI(Z) tokens will continue (with no plans for fixing) to be affected, as keystone does not perform the validation of the users. The endpoint validates pki tokens and the endpoint cannot know the state of user enabled/disabled.

Federated users are are also not controlled by keystone and keystone cannot directly query the enabled status of these users. It is expected that the Identity Provider must notify the service provider that a user has been disabled or like many SSO type technologies, the session (token ttl this case) must expire.

[Guang Yee (guang-yee)]

For PKI(Z) tokens, if we update the revocation events or TRLs whenever a user is disabled, middleware token validation logic would still take that into consideration, though there could still be a small gap between when it actually update its local revocation events and the event actually occurred at the server side. Also, older releases only support token revocation list, not revocation events.

Morgan's right about external IdPs, it would be difficult for Keystone to know when the user was actually disabled. Or whether a user was disabled in real time or on a scheduled update. Keystone have absolutely no control over that.

We only have control over local identities. Though for LDAP identities, we also have a locally mapping table so in theory we can disable LDAP users locally with minimal code changes.

To me, security is about risk management. And this is no different then assessing the risk associated with caching the token validation results (i.e. memcache) at auth_token middleware. If there's a highly sensitive deployment out there which request real time token invalidation, then use locally managed identities and no caching. Also, calibrate the token TTL according to your risk tolerance.

[Morgan Fainberg (mdrnstm)]

@Guang.

I think it is unreasonable to ask an AD admin to issue a api call when disabling a user (let's be fair, they disable the user in AD and expect the user to stop working everywhere). I am ok with expanding and allowing a revocation event to be manually issued in these cases, but that is a feature and beyond the scope of the bug fix.

[Morgan Fainberg (mdrnstm)]

This is specific to the case when (for example) AD admin disabled a user. Keystone never knows the user was disabled because no api call was made to disable the user via keystone.

[Brant Knudson (blk-u)]

I'm concerned about issuing an advisory for something we're not going to fix. If we can't fix this for PKI tokens or federated users then we're saying these features are inherently insecure.

[Morgan Fainberg (mdrnstm)]

There is no good way to solve this for PKI. I would actually say pki is more insecure today than uuid/Fernet. I actively recommend that PkI tokens are not used.

Federated is more of an accepted risk: we have an open BP / RFE to handle the standard ways of disable/invalidating from the IdP.

[Morgan Fainberg (mdrnstm)]

Also you can end up with SSO sessions that need to timeout after access is disabled. Depending on cookie/impl.

[Steve Martinelli (stevemar)]

I somewhat agree with Brant's comment. If we do issue an advisory, we have to explain each case sufficiently. The fix itself seems good.

[Guang Yee (guang-yee)]

I would think an advisory is also about educating users the potential risks associated with a particular deployment so they can make the proper tradeoffs.

[Jeremy Stanley (fungi)]

If there is no patch which mitigates or fixes the issue on its own, the VMT doesn't issue an OpenStack Security Advisory (OSSA). Instead the Security Group (OSSG) may choose to publish a Note (OSSN) describing the risks and explaining how impacted parties can protect themselves.

[Dolph Mathews (dolph)]

Impacted parties can tune Keystone's behavior in this scenario by reducing their token lifespan (keystone.conf [token] expiration) to match the perceived risk of disabled users & groups continuing to have valid access to OpenStack APIs. This is true regardless of the token format or identity backend in use.

For example, an organization using LDAP concerned that a terminated employee will continue to have access to OpenStack APIs beyond their effective termination date should use a token lifespan of somewhere in the range of 2-24 hours, rather than 7 days, for example.

Keystone already uses a default token expiration value of 3600 seconds (1 hour), which I imagine would be sufficient to mitigate most real world security concerns.

I do not consider this behavior to be a vulnerability and would have assumed it was "well-known tribal knowledge" (sadly undocumented, AFAIK), so +1 for OSSN and a public patch to improve documentation.

[Morgan Fainberg (mdrnstm)]

Most orgs I've talked to have retuned ttl back up to 86400 because 3600 is insufficient for glance/cinder actions.

[Morgan Fainberg (mdrnstm)]

@dolph,

While it is tribal knowledge and undocumented, unfortunately it presents as a vulnerability due to impact and unexpected behavior. I honestly don't care if it is OSSN or OSSA. What I care about is that the knowledge is made clear and we close the gap to the best of our ability so that behavior is more aligned with what would be expected.

[Morgan Fainberg (mdrnstm)]

I targeted from rc1. We can backport / rc2 as needed

[Tristan Cacqueray (tristan-cacqueray)]

Thanks for the quick feedback. Considering PKI token will remain vulnerable, I suggest we subscribe ossg-coresec member so they can work on a security note.

Also if we go the OSSN way, then we can remove the privacy settings and patches could be submitted directly to gerrit.

[Thierry Carrez (ttx)]

Adding OSSN for discussion

[Morgan Fainberg (mdrnstm)]

For the record, I'm fine with this being either OSSN or OSSA. This should be clearly communicated.

[Robert Clark (robert-clark)]

Is this caching similar to OSSN-0032? https://wiki.openstack.org/wiki/OSSN/OSSN-0032

[Nathan Kinder (nkinder)]

I think this should be an OSSN (not an OSSA). I do agree that this is more of a "tribal knowledge" item at this point, but we need to get the word out abotu how it bahaves more broadly. I honestly don't think that this is a solvable problem. The token validity period needs to be a choice of acceptable risk (just as it is for things like a Kerberos deployment).

[Jeremy Stanley (fungi)]

I agree, precedent is for this to be documented as current behavior with known workarounds.

[Thierry Carrez (ttx)]

Agreed, let's go the OSSN route

[Tristan Cacqueray (tristan-cacqueray)]

This seems to be discussed in the public bug 1461095. Is there a reason this is still private ?

[Morgan Fainberg (mdrnstm)]

Tristan: no I was waiting for the VMT team to mark it public.

[Dolph Mathews (dolph)]

Based on today's keystone meeting and the above comments, I've reduced the priority of this to Medium across the board and marked this as Won't Fix in Keystone.

Although this is working as intended, we acknowledge that that intended behavior is poorly documented, and it seems an OSSN is the best route to rectify that.

I'd be happy to work with whoever wants to write the OSSN - ping me in IRC (dolphm) or leave a comment here.