Potential token revocation abuse via group membership

[Adam Young]

If a group is deleted, all tokens for all users that are a member of that group are revoked. This leads to potential abuse:

1. A group admin adds a user to a group without users knowledge
2. User creates token
3. Admin deletes group.
4. All of the users tokens are revoked.

Admittedly, this abuse must be instigated by a group admin, which is the global admin in the default policy file, but an alternative policy file could allow for the delegation of "add user to group" behavior. In such a system, this could act as a denial of service attack for a set of users.

[Dolph Mathews (dolph)]

I'd suggest opening publicly as this possibility was discussed/discovered publicly in #opensatck-dev prior to this bug report. Given that this requires a very specific custom policy configuration to provide a viable means of exposure, I suspect that the real-world impact is also minimal.

[Jeremy Stanley (fungi)]

Not only discussed in public, but archived for posterity... http://eavesdrop.openstack.org/irclogs/%23openstack-dev/%23openstack-dev.2014-01-13.log @21:32.

[Jeremy Stanley (fungi)]

What releases of Keystone are affected by this? And is it something you're likely to backport fixes for if it's more than just Icehouse impacted?

[Adam Young (ayoung)]

All currently support releases are affected.

It does have backport potential

This probably can be made public.

[Jeremy Stanley (fungi)]

Since we're on the same page about the cat being out of the bag, I've switched it to public-security.

[Thierry Carrez (ttx)]

Please push patches directly on Gerrit.

[Tristan Cacqueray (tristan-cacqueray)]

Assuming user's token created before group membership are also affected.
(ie: step 2 of the description could be step 0)

Draft impact description -

Title: Potential token revocation abuse via group membership
Reporter: Adam Young
Products: Keystone
Affects: All supported versions

Description:
Adam Young reported a vulnerability in Keystone revocation process.
If a group is deleted, all tokens for all users that are a member of that
group are revoked. By adding users to a group without users knowledge then
deleting that group, a group admin can revoke all of the users tokens.
While the default policy file give the group admin role to global admin, an
alternative policy could delegate the "create_group", "add_user_to_group",
"delete_group" capabilities to a set of users. In such a system, those users
will also get a token revocation capability.

Only setups using a custom policy file in Keystone are affected.

[Thierry Carrez (ttx)]

@Adam; please check proposed impact desc for technical correctness

Grammar typo: While the default policy file *gives*. Also do not break description into multiple paragraphs, keep it in one block.
Otherwise sounds great!

[Grant Murphy (gmurphy)]

Also -

s/$REPORTER/$REPORTER from $COMPANY/

where applicable?

[Dolph Mathews (dolph)]

Adam Young works for Red Hat, if you want to include that.

+1 for rest of impact description.

[Tristan Cacqueray (tristan-cacqueray)]

Thanks for the feedback, this is:
Draft impact description #2 -

Title: Potential token revocation abuse via group membership
Reporter: Adam Young (Red Hat)
Products: Keystone
Affects: All supported versions

Description:
Adam Young from Red Hat reported a vulnerability in Keystone revocation
process. If a group is deleted, all tokens for all users that are a
member of that group are revoked. By adding users to a group without
users knowledge then deleting that group, a group admin can revoke all
of the users tokens. While the default policy file gives the group
admin role to global admin, an alternative policy could delegate the
"create_group", "add_user_to_group", "delete_group" capabilities to a
set of users. In such a system, those users will also get a token
revocation capability. Only setups using a custom policy file in
Keystone are affected.

[Jeremy Stanley (fungi)]

Looks good! Here are a few more minor grammar and punctuation nits...

"reported a vulnerability in the Keystone revocation process"

"all users that are members of that group"

"By adding users to a group without those users' knowledge and then deleting that group, a group admin can revoke all of the users' tokens."

[Tristan Cacqueray (tristan-cacqueray)]

Alright, thank you Jeremy, so this is:
Draft impact description #3 -

Title: Potential token revocation abuse via group membership
Reporter: Adam Young (Red Hat)
Products: Keystone
Affects: All supported versions

Description:
Adam Young from Red Hat reported a vulnerability in the Keystone revocation process. If a group is deleted, all tokens for all users that are members of that group are revoked. By adding users to a group without those users' knowledge and then deleting that group, a group admin can revoke all of the users' tokens. While the default policy file gives the group admin role to global admin, an alternative policy could delegate the "create_group", "add_user_to_group", "delete_group" capabilities to a set of users. In such a system, those users will also get a token revocation capability. Only setups using a custom policy file in Keystone are affected.

[Jeremy Stanley (fungi)]

That latest impact description (comment #13) looks great--thanks!

[Dolph Mathews (dolph)]

+1 for impact description in #13

[Adam Young (ayoung)]

Impact desc looks Correct/

[Thierry Carrez (ttx)]

+1 for Impact desc

[Thierry Carrez (ttx)]

@Adam: do you have a patch for that ?

[Adam Young (ayoung)]

Not yet. I've been working on the replacement mechanism for revocations, which don't have this problem. I can expedite a fix if necessary.

[Thierry Carrez (ttx)]

@Adam: we'll need a basic fix/workaround for grizzly and havana, so we could first work around the issue and then you can rip it off in master if you need to.

[Thierry Carrez (ttx)]

Adam confirmed he is working on the master solution, not a backportable patch. So we need to find someone else to work on that fix.

[Thierry Carrez (ttx)]

Hrm. not sure there is a way to fix that that would fit backporting guidelines (i.e. not changing the behavior).
We may have to document this as a known issue, attract attention to the combination of factors that make it exploitable, and then "fix" it in future versions by changing the revocation system entirely.

Thoughts ?

[Thierry Carrez (ttx)]

So it looks like this is not really fisable in stable branches... it should rather be documented as a known issue when you set up specific policies, so that you know what to expect if you do enable them this way. That would make it OSSN territory.

The whole situation will be avoided in the future with the new design that Adam is pushing, hopefully in time for Icehouse release.

[Jeremy Stanley (fungi)]

Agreed. In this case the unusual circumstances under which it can be exploited outweigh the risk/effort involved in trying to backport such a redesign. At least with a documented workaround, the few deployments which might be affected have some clear means to mitigate it.

[Thierry Carrez (ttx)]

Adding OSSG contacts so that they are aware of the OSSN route for this one

[Robert Clark (robert-clark)]

Thanks Thierry, we've added it to the queue :)

[Nathan Kinder (nkinder)]

An OSSN on this issue has been published to the wiki, openstack-dev, and openstack mailing lists:

http://git.openstack.org/cgit/openstack/openstack-security-notes/commit/?id=5380798f052eaebc023271c90d65b8f6d6fa6331

https://wiki.openstack.org/w/index.php?title=OSSN/OSSN-0009&action=edit&redlink=1

[Adam Young (ayoung)]

With revocation events, we have a potential fix. Revocation events should not revoke tokens based on group membership changes. Instead, when a token is validated, it should expand the roles related to that token at validation time. If group membership means that the user no longer has the role assignment, then they should rightfully lost the role. But if the role assignment has nothing to do with the token, no revocation should take place.

Fixing this for UUID tokens means reworking them to use the same mechanism as Fernet tokens for validation: Only the comparable section of the token that would be in the signed body of the Fernet token should be persisted in the database, and the Access Info returned during a validation response should be rebuilt at that time, not based on cached data.

[Steve Martinelli (stevemar)]

target mitaka-2 or mitaka-3 if work is going to land

[Lance Bragstad (lbragstad)]

I believe what Adam just described is actually a FIXME from gyee [0].

[0] https://github.com/openstack/keystone/blob/ed2ff17290959cc873a8295249833738e0bd6256/keystone/token/providers/common.py#L724-L731

[Adam Young (ayoung)]

This is only a problem when using revoke by ID. It will get cleaned up as a side effect of going to Fernet and using revocation events. Since it has hit no-one in the wild, lowering the priority.

[Steve Martinelli (stevemar)]

Automatically unassigning due to inactivity.

[Richard (csravelar)]

The way tokens are revoked via group deletions is how keystone fundamentally operates and is more an issue of giving the wrong user the authority to create groups, add users, delete groups repeatedly. Again, you probably have bigger problems if you've given authority to someone who is intentionally filling up the revocation table with entries. However, if you have a user who legitimately is trying to delete a few groups with thousands of users in each group, this can be an issue. When you delete a group it creates an entry for each user_id and if the group has thousands of users then you will get thousands of revocations. Because a user's tokens are revoked after group deletion, they must issue themselves another token. The improvements in revocation performance make it so that we only look at revocation entries that were issued after the token which would ignore all the entries that got created by the group deletion anyway.
https://github.com/openstack/keystone/blob/master/keystone/revoke/backends/sql.py#L95-L96

This still leaves the issue of needing to create a ton of revokes if the group has a ton of users which could be improved as ayoung suggested from
http://eavesdrop.openstack.org/irclogs/%23openstack-dev/%23openstack-dev.2014-01-13.log
by adding a group_id to the column if this issue is really a concern.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/399728

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/399728
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=df721d05bfa4c69f8540d6051912d2430ed06213
Submitter: Jenkins
Branch: master

commit df721d05bfa4c69f8540d6051912d2430ed06213
Author: “Richard <email address hidden>
Date: Fri Nov 18 18:25:07 2016 +0000

    Don't invalidate all user tokens of roleless group

    As discussed in [1], deleting a group invalidates all user tokens
    which can flood the revocation event table if the deleted group
    contained thousands of users in the group. This happens regardless
    of whether the group had any role assignment or not. This patch makes
    it so that only groups that had role assignments to a project or
    domain can then invalidate user tokens, otherwise there is no need
    to revoke each user token because the group was not assigned any form
    of authorization to begin with.

    [1]: https://bugs.launchpad.net/keystone/+bug/1268751

    Related-Bug: #1268751

    Change-Id: I22ad364cb4737df3ed086f78310f75f3099ab4c1