[OSSA 2014-006] Trustee token revocations with memcache backend (CVE-2014-2237)

Bug #1260080 reported by Morgan Fainberg
262
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Fix Released
Medium
Morgan Fainberg
Grizzly
Fix Released
Medium
Morgan Fainberg
Havana
Fix Released
Medium
Morgan Fainberg
OpenStack Security Advisory
Fix Released
High
Tristan Cacqueray

Bug Description

When using the memcache token backend, In cases where token revocations (bulk revocations) that rely on the trustee's token-index-list to perform the revocations - the revocations cannot occur. This is because trustee tokens are added only to the trustor's token list. This appears to only be an issue for trusts with impersonation enabled. This is most noticeable when the trustee user is disabled or the trustee changes a password. I am sure there are other related scenarios.

Example reproduction -

Create the trust:

curl -H "X-Auth-Token: $TOKEN" -X POST http://127.0.0.1:35357/v3/OS-TRUST/trusts -d '{"trust":{"impersonation": true, "project_id": "9b2c18a72c8a477b891b48931c26cebe", "roles": [{"name":"admin"}],"trustee_user_id":"36240e5f6ed04857bf15d08a519b6e37","trustor_user_id":"46c9e36c50c14dfab36b6511d6ebedd4"}}' -H 'Content-Type: application/json'

RESPONSE:
{
    "trust": {
        "expires_at": null,
        "id": "0d2dc361043d4f9d8a84a7f253e20924",
        "impersonation": true,
        "links": {
            "self": "http://172.16.30.195:5000/v3/OS-TRUST/trusts/0d2dc361043d4f9d8a84a7f253e20924"
        },
        "project_id": "9b2c18a72c8a477b891b48931c26cebe",
        "roles": [
            {
                "id": "0bd1d61badd742bebc044dc246a43513",
                "links": {
                    "self": "http://172.16.30.195:5000/v3/roles/0bd1d61badd742bebc044dc246a43513"
                },
                "name": "admin"
            }
        ],
        "roles_links": {
            "next": null,
            "previous": null,
            "self": "http://172.16.30.195:5000/v3/OS-TRUST/trusts/0d2dc361043d4f9d8a84a7f253e20924/roles"
        },
        "trustee_user_id": "36240e5f6ed04857bf15d08a519b6e37",
        "trustor_user_id": "46c9e36c50c14dfab36b6511d6ebedd4"
    }
}

Consume the trust:

vagrant@precise64:~/devstack$ cat trust_token.json
{
    "auth": {
        "identity": {
            "methods": [
                "token"
            ],
            "token": {
                "id": "<PKI TOKEN ID>"
            }
        },
        "scope": {
            "OS-TRUST:trust": {
                "id": "0d2dc361043d4f9d8a84a7f253e20924"
            }
        }
    }
}

curl -si -d @trust_token.json -H 'Content-Type: application/json' -X POST http://localhost:35357/v3/auth/tokens

RESPONSE:

{
    "token": {
        "OS-TRUST:trust": {
            "id": "0d2dc361043d4f9d8a84a7f253e20924",
            "impersonation": true,
            "trustee_user": {
                "id": "36240e5f6ed04857bf15d08a519b6e37"
            },
            "trustor_user": {
                "id": "46c9e36c50c14dfab36b6511d6ebedd4"
            }
        },
        "catalog": [
           ... <SNIP FOR BREVITY>
        ],
        "expires_at": "2013-12-12T20:06:00.239812Z",
        "extras": {},
        "issued_at": "2013-12-11T20:10:22.224381Z",
        "methods": [
            "token",
            "password"
        ],
        "project": {
            "domain": {
                "id": "default",
                "name": "Default"
            },
            "id": "9b2c18a72c8a477b891b48931c26cebe",
            "name": "admin"
        },
        "roles": [
            {
                "id": "0bd1d61badd742bebc044dc246a43513",
                "name": "admin"
            }
        ],
        "user": {
            "domain": {
                "id": "default",
                "name": "Default"
            },
            "id": "46c9e36c50c14dfab36b6511d6ebedd4",
            "name": "admin"
        }
    }
}

Check the memcache token lists, the admin user should have 1 token (used to create the trust), the demo user should have 2 tokens (initial token, and trust token)

ADMIN-ID: 46c9e36c50c14dfab36b6511d6ebedd4
DEMO-ID: 36240e5f6ed04857bf15d08a519b6e37

vagrant@precise64:~/devstack$ telnet localhost 11211
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get usertokens-46c9e36c50c14dfab36b6511d6ebedd4
VALUE usertokens-46c9e36c50c14dfab36b6511d6ebedd4 0 104
"1bb6a8abc42e1d1a944e08a20de20a91","ee512379a66027733001648799083349"
END
get usertokens-36240e5f6ed04857bf15d08a519b6e37
VALUE usertokens-36240e5f6ed04857bf15d08a519b6e37 0 34
"36df008ec5b5d3dd6983c8fe84d407f6"

This does not affect the SQL or KVS backends. This will affect Master (until KVS refactor is complete), Havana, and Grizzly. The reason for this occurring is a lack of testing against the memcache backend itself (and it’s logic). With the refactor of KVS (keyvaluestore) it will ensure the testing is the same against all KVS (memcache, in-memory, redis, etc) backends since the interfaces work in the same manner/mechanism.

The root of this issue stems from the old KVS and SQL systems being less strict on matching trust scoped tokens:
(KVS) http://git.openstack.org/cgit/openstack/keystone/tree/keystone/token/backends/kvs.py?id=cc0c6565e3fe6ed60543e6ebd5e54cba51e59fc6#n86
(SQL) http://git.openstack.org/cgit/openstack/keystone/tree/keystone/token/backends/sql.py?id=cc0c6565e3fe6ed60543e6ebd5e54cba51e59fc6#n126

Whereas memcache needs to use the user-index to determine tokens for a given user before matching against the trust scope:
http://git.openstack.org/cgit/openstack/keystone/tree/keystone/token/backends/memcache.py?id=cc0c6565e3fe6ed60543e6ebd5e54cba51e59fc6#n190

CVE References

description: updated
Revision history for this message
Thierry Carrez (ttx) wrote :

Sounds legit... Could anyone else in Keystone confirm ?

Changed in ossa:
status: New → Incomplete
Revision history for this message
Adam Young (ayoung) wrote :

I discussed with Morgan when he discovered it. It is legitimate.

Changed in ossa:
status: Incomplete → Confirmed
Revision history for this message
Morgan Fainberg (mdrnstm) wrote :

Review that solves the issue in Master: https://review.openstack.org/#/c/60742/
I'll post up a quick couple havana/grizzly patchsets (hopefully) this weekend.

Revision history for this message
Morgan Fainberg (mdrnstm) wrote :

Sorry, wrong review in #3,

https://review.openstack.org/#/c/60743/

That review is what migrates Memcache token driver to the new KVS system that does not have this issue.

Revision history for this message
Jeremy Stanley (fungi) wrote :

With the patch already public (though it doesn't seem to call itself out as a vulnerability fix specifically), is there any further benefit to keeping this report embargoed?

Revision history for this message
Morgan Fainberg (mdrnstm) wrote :

That is up to the Security team. This does affect Havana and Grizzly and the patchset that will fix it for Icehouse is a drastic refactor that basically solves the issue by changing how the functionality of the KVS/Memcache backend works completely.

I am fine publishing the fixes directly and to the public for Havana and Grizzly if that is desired. I was erring on the side of caution and figured the Security team can make the call on embargo or not.

Revision history for this message
Thierry Carrez (ttx) wrote :

My vote would be on opening it, since this cannot really be triggered by an attacker, if I understand correctly. So we are not really making anyone more vulnerable by opening it ?

Changed in ossa:
importance: Undecided → High
Revision history for this message
Morgan Fainberg (mdrnstm) wrote :

It doesn't appear to be directly trigger-able by an attacker. The attacker would need a trust-scoped-token or acccount and issue a trust-scoped token, and the events that normally would revoke these trust-scoped-tokens based upon the trustee (password change, disable, etc) will not occur.

It's a narrow-scope, but still a security risk.

Revision history for this message
Thierry Carrez (ttx) wrote :

Agree it's a security risk. But keeping it embargoed isn't making anyone safer since it's not manually triggerable. Better fix it quickly.

Revision history for this message
Morgan Fainberg (mdrnstm) wrote :

So, the general consensus is to post the patches publicly vs on this report? I'll get the patches posted in the next couple days in either case.

Revision history for this message
Thierry Carrez (ttx) wrote :

Yes, post patches publicly and reference this bug.

information type: Private Security → Public Security
Revision history for this message
Thierry Carrez (ttx) wrote :

@Morgan: any progress on patches ?

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Draft impact description -

Title: Trustee token revocations does not work with memcache backend
Reporter: Morgan Fainberg
Products: Keystone
Affects: All supported versions

Description:
Morgan Fainberg reported a vulnerability in Keystone memcache token backend.
When a trustor issue a trust token with impersonation enabled, the token is
only added to the trustor's token list and not to the trustee's token list.
This results in the trust token not being invalidated by trustee's token
revocation (bulk revocation). This is most noticeable when the trustee user
is disabled or the trustee change a password.
Only setups using memcache backend for token in Keystone are affected.

Changed in ossa:
assignee: nobody → Tristan Cacqueray (tristan-cacqueray)
Revision history for this message
Thierry Carrez (ttx) wrote :

A few grammar typos: When a trustor *issues* ... or the trustee *changes*.
I would also say *tokens* instead of token in that last sentence...

Otherwise, great stuff.

Thierry Carrez (ttx)
Changed in ossa:
status: Confirmed → Triaged
Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Thank you for the feedback Thierry. This is:
Draft impact description #2 -

Title: Trustee token revocations does not work with memcache backend
Reporter: Morgan Fainberg (Metacloud)
Products: Keystone
Affects: All supported versions

Description:
Morgan Fainberg from Metacloud reported a vulnerability in Keystone memcache
token backend. When a trustor issues a trust token with impersonation enabled,
the token is only added to the trustor's token list and not to the trustee's
token list. This results in the trust token not being invalidated by trustee's
token revocation (bulk revocation). This is most noticeable when the trustee
user is disabled or the trustee changes a password. Only setups using memcache
backend for tokens in Keystone are affected.

Revision history for this message
Jeremy Stanley (fungi) wrote :

This is great! If you want to address a few other minor grammar errors...

"token revocation does not work"

"a vulnerability in the Keystone memcache token backend"

"invalidated by the trustee's token revocation"

"setups using the memcache backend for tokens"

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Thank you again Jeremy, here is the corrected:
Draft impact description #3 -

Title: Trustee token revocation does not work with memcache backend
Reporter: Morgan Fainberg (Metacloud)
Products: Keystone
Affects: All supported versions

Description:
Morgan Fainberg from Metacloud reported a vulnerability in the Keystone memcache token backend. When a trustor issues a trust token with impersonation enabled, the token is only added to the trustor's token list and not to the trustee's token list. This results in the trust token not being invalidated by the trustee's token revocation (bulk revocation). This is most noticeable when the trustee user is disabled or the trustee changes a password. Only setups using the memcache backend for tokens in Keystone are affected.

Revision history for this message
Jeremy Stanley (fungi) wrote :

That latest impact description (comment #17) looks great--thanks!

Revision history for this message
Thierry Carrez (ttx) wrote :

+1 on impact desc. Still missing patches

Revision history for this message
Dolph Mathews (dolph) wrote :
Changed in keystone:
status: New → In Progress
assignee: nobody → Morgan Fainberg (mdrnstm)
importance: Undecided → Medium
Changed in keystone:
milestone: none → icehouse-3
Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

In the impact desc, the new affect line should be:

Versions: 2013.1 up to 2013.1.4 and 2013.2 versions up to 2013.2.2

The keystone/token/backends/memcache.py only mention trust starting from 2013.1 and I assumed all version since then are vulnerable.

@Morgan, do you know when fix for Havana and Grizzly will be ready ?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (master)

Reviewed: https://review.openstack.org/60743
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=813d1254eb4f7a7d40009b23bbadbc4c5cc5daac
Submitter: Jenkins
Branch: master

commit 813d1254eb4f7a7d40009b23bbadbc4c5cc5daac
Author: Morgan Fainberg <email address hidden>
Date: Sun Dec 8 21:28:33 2013 -0800

    Convert Token Memcache backend to new KeyValueStore Impl

    This patchset converts the memcache backend for tokens to the new
    KeyValueStore implementation. This implementation provides basic
    configuration choices for utilizing memcache as a token backend.

    The test_backend_memcache test suite has been removed, as there is
    no extra functionality added on top of the KeyValueStore base
    besides the basic configuration choices. The test_backend_memcache
    test suite was only used for testing token memcache backend. The
    important unicode tests have been moved into the base token test
    suite and should now validate against all token backends.

    Closes-Bug: #1260080
    bp: dogpile-kvs-backends
    Change-Id: I8f0bca409a6393176e77f7a27ba15959756da06f

Changed in keystone:
status: In Progress → Fix Committed
Revision history for this message
Thierry Carrez (ttx) wrote : Re: Trustee token revocations with memcache backend

Anyone up for the grizzly/havana backports ?

Revision history for this message
Thierry Carrez (ttx) wrote :

Hmm, sounds tricky to backport the "fix" that was merged in master, as you can't really remove or substantially change functionality. What do you think ?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (stable/havana)

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/75521

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (stable/grizzly)

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/75526

Revision history for this message
Morgan Fainberg (mdrnstm) wrote : Re: Trustee token revocations with memcache backend

With another set of recent changes to Havana the backport was much easier. It took some doing wrt other changes in master to see the easiest approach to solve the issue.

summary: - Trustee token revocations with memcache backend
+ Trustee token revocations with memcache backend (CVE-2014-2237)
Changed in ossa:
status: Triaged → Fix Committed
Thierry Carrez (ttx)
Changed in keystone:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (stable/grizzly)

Reviewed: https://review.openstack.org/75526
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=a411c944af78c36f2fdb87d305ba452dc52d7ed3
Submitter: Jenkins
Branch: stable/grizzly

commit a411c944af78c36f2fdb87d305ba452dc52d7ed3
Author: Morgan Fainberg <email address hidden>
Date: Fri Feb 21 14:09:04 2014 -0800

    Ensure tokens are added to both Trustor and Trustee indexes

    Tokens are now added to both the Trustor and Trustee user-token-index
    so that bulk token revocations (e.g. password change) of the trustee
    will work as expected. This is a backport of the basic code that was
    used in the Icehouse-vintage Dogpile Token KVS backend that resolves
    this issue by merging the handling of memcache and KVS backends into
    the same logic.

    Change-Id: I3e19e4a8fc1e11cef6db51d364e80061e97befa7
    Closes-Bug: #1260080

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (stable/havana)

Reviewed: https://review.openstack.org/75521
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=b6f0e26da0e2ab0892a5658da281a065e668637b
Submitter: Jenkins
Branch: stable/havana

commit b6f0e26da0e2ab0892a5658da281a065e668637b
Author: Morgan Fainberg <email address hidden>
Date: Fri Feb 21 13:33:25 2014 -0800

    Ensure tokens are added to both Trustor and Trustee indexes

    Tokens are now added to both the Trustor and Trustee user-token-index
    so that bulk token revocations (e.g. password change) of the trustee
    will work as expected. This is a backport of the basic code that was
    used in the Icehouse-vintage Dogpile Token KVS backend that resolves
    this issue by merging the handling of memcache and KVS backends into
    the same logic.

    Change-Id: I3e19e4a8fc1e11cef6db51d364e80061e97befa7
    Closes-Bug: #1260080

Revision history for this message
Thierry Carrez (ttx) wrote : Re: Trustee token revocations with memcache backend (CVE-2014-2237)

All fixes merged.

Changed in ossa:
status: Fix Committed → Fix Released
Alan Pevec (apevec)
summary: - Trustee token revocations with memcache backend (CVE-2014-2237)
+ [OSSA 2014-006] Trustee token revocations with memcache backend
+ (CVE-2014-2237)
Thierry Carrez (ttx)
Changed in keystone:
milestone: icehouse-3 → 2014.1
To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.