[OSSA 2014-026] Revocation events are broken with mysql (CVE-2014-5251)

[Brant Knudson]

Since mysql only stores timestamps with an accuracy of seconds rather than microseconds, doing comparisons of token expiration times will fail and tokens will not show up as being revoked.

[Brant Knudson (blk-u)]

A breakpoint in keystone/contrib/revoke/model.py, is_revoked():

mysql:

(Pdb) p self.revoke_map
{'trust_id=*': {'consumer_id=*': {'access_token_id=*': {'expires_at=2014-07-22 22:55:53': {'domain_id=*': {'project_id=*': {u'user_id=949c28307de74cafb4ab07c6ada75d6c': {'role_id=*': {'issued_before': datetime.datetime(2014, 7, 22, 21, 55, 59, 610579)}}}}}}}}}

DB2:

(Pdb) p self.revoke_map
{'trust_id=*': {'consumer_id=*': {'access_token_id=*': {'expires_at=2014-07-22 22:58:44.322976': {'domain_id=*': {'project_id=*': {u'user_id=c4ed3fa9ee5f4e02b580389400a817e0': {'role_id=*': {'issued_before': datetime.datetime(2014, 7, 22, 21, 58, 49, 390556)}}}}}}}}}

[Brant Knudson (blk-u)]

This is in add_event in model.py:

mysql:

(Pdb) event.expires_at
datetime.datetime(2014, 7, 22, 23, 28, 2)

db2:

(Pdb) event.expires_at
datetime.datetime(2014, 7, 22, 23, 22, 40, 481090)

[Morgan Fainberg (mdrnstm)]

This looks like something we might be able to fix with a SQL migration to make the column a DATETIME(6) instead of DATETIME(0) (0 is the default in MySQL, where the SQL standard is datetime(6) http://dev.mysql.com/doc/refman/5.6/en/fractional-seconds.html )

[Morgan Fainberg (mdrnstm)]

Marking this as a high priority as we have some large features that will rely on revocation events during the Juno cycle.

[Brant Knudson (blk-u)]

This is a security vulnerability since tokens aren't being revoked as they should be.

[Tristan Cacqueray (tristan-cacqueray)]

@Brant, just to clarify, with a mysql token backend, token never expired ?

Also, do you know if icehouse/havana are also impacted ?

[Brant Knudson (blk-u)]

@Tristan - I have to look into this more to be able to answer the question. All I can say is that when working with mysql the data isn't being returned on the query in the expected format.

This would only impact icehouse since revocation events were added in icehouse.

[Brant Knudson (blk-u)]

OK. I tried a test and I think this is legitimate and that tokens aren't getting expired.

To test this I modified the existing tempest test[1], to add a self.client.delete_token(token_id) right after the scoped token is revoked. This should revoke the unscoped token.

Then I changed the Keystone configuration so that it uses revocation events only, by setting revoke_by_id=false in keystone.conf

When running the test, the unscoped token IS NOT revoked. Looking at the debugger, after the revocations, the revocation event's self.revoke_map has ...'expires_at=2014-07-31 20:13:06'...

And the token data has 'expires_at': datetime.datetime(2014, 7, 31, 20, 13, 6, 126800). So is_revoked tries to match 'expires_at=2014-07-31 20:13:06.126800' against the revoke_map and it doesn't match because it's an exact match.

[1] http://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/identity/admin/v3/test_tokens.py?id=973c6cd98e4b226bec2f1b40943afaa910e18047#n140

Essentially, revoking tokens individually doesn't work with mysql and revocation events.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/111106

[Tristan Cacqueray (tristan-cacqueray)]

Title: Token does not expire or revoke with Mysql backend
Reporter: Brant Knudson (IBM)
Products: Keystone
Versions: 2014.1.1

Description:
Brant Knudson from IBM reported a vulnerability in Keystone revocation events. Mysql does not store expiration date correctly resulting in tokens to never expire and breaking manual revocation. Only Keystone setups configured to use revocation events and Mysql tokens backend are affected.

[Brant Knudson (blk-u)]

The impact statement isn't correct since it says that tokens won't expire using MySQL, but they will. It just that revocations don't work with revocation events at all. Here's my attempt:

Brant Knudson from IBM reported a vulnerability in Keystone revocation events. The Keystone revocation events code expects the database to store expiration timestamps with subsecond accuracy, which Mysql does not do. This causes tokens that are manually revoked to remain valid. Only Keystone setups configured to use revocation events and the SQL token driver with MySQL are affected.

[Tristan Cacqueray (tristan-cacqueray)]

@Brant thanks again for those reviews, here is a more simplified description considering revocations events are very recent and that no keystone middleware supporting those have been released yet...

So here is the combined impact description for bugs #1348820, #1347961, #1349597:

Title: Multiple vulnerabilities in Keystone revocation events
Reporter: Lance Bragstad (Rackspace) and Brant Knudson (IBM)
Products: Keystone
Versions: 2014.1.1

Description:
Lance Bragstad from Rackspace and Brant Knudson from IBM reported 3 vulnerabilities in Keystone revocations events. Lance Bragstad discovered that UUID v2 tokens processed by the V3 API are incorectly updated and get their "issued_at" time regenerated. Brant Knudson discovered that Mysql token driver stores expiration date incorrectly which prevent manual revocation and that domain-scoped tokens don't get revoked when the domain is disabled. Tokens impacted by one of those bug may allow a user to evade token revocation. Only Keystone setups configured to use revocations events are affected.

[Brant Knudson (blk-u)]

Comments on the impact statement in comment #12:

change "revocations events" to "revocation events" (remove s, this is at the beginning and at the end of the text)

change "that Mysql token" to "that the MySQL token" (add the)

change "expiration date incorrectly" to "expiration dates incorrectly" (add s)

change "which prevent manual" to "which prevents manual" (add s)

change "those bug may" to "these bugs may"

[Tristan Cacqueray (tristan-cacqueray)]

Thanks Brant!
Here is the revised version:

Title: Multiple vulnerabilities in Keystone revocation events
Reporter: Lance Bragstad (Rackspace) and Brant Knudson (IBM)
Products: Keystone
Versions: 2014.1.1

Description:
Lance Bragstad from Rackspace and Brant Knudson from IBM reported 3 vulnerabilities in Keystone revocation events. Lance Bragstad discovered that UUID v2 tokens processed by the V3 API are incorectly updated and get their "issued_at" time regenerated. Brant Knudson discovered that the MySQL token driver stores expiration dates incorrectly which prevents manual revocation and that domain-scoped tokens don't get revoked when the domain is disabled. Tokens impacted by one of these bugs may allow a user to evade token revocation. Only Keystone setups configured to use revocation events are affected.

[Brant Knudson (blk-u)]

change "incorectly" to "incorrectly"

[Brant Knudson (blk-u)]

should say other than that the impact statement looks good to me.

[Thierry Carrez (ttx)]

Impact desc looks good; except "Versions" should probably read "2014.1 versions up to 2014.1.1"

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/111106
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=7aee6304f653475a4130dc3e5be602e91481f108
Submitter: Jenkins
Branch: master

commit 7aee6304f653475a4130dc3e5be602e91481f108
Author: Brant Knudson <email address hidden>
Date: Thu Jul 31 17:47:00 2014 -0500

    Fix revocation event handling with MySQL

    When MySQL is used to store revocation events, events are returned
    from the database with the timestamps truncated to the second. This
    causes a revocation event for a token (which has the issued_at
    timestamp to the microsecond) to not match the revocation event and
    therefore the token is not considered to be revoked.

    The fix is to have the revocation events and token timestamps both
    always be truncated to the second. This will cause all tokens for a
    user that are issued within a second to be revoked when any of those
    tokens are revoked, which shouldn't be a problem.

    Change-Id: Ibd82b4ce910206dfd504c396614ae2ebed025e9b
    Closes-Bug: #1347961

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/112087

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/112087
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=6cbf835542d62e6e5db4b4aef7141b1731cad9dc
Submitter: Jenkins
Branch: stable/icehouse

commit 6cbf835542d62e6e5db4b4aef7141b1731cad9dc
Author: Brant Knudson <email address hidden>
Date: Thu Jul 31 17:47:00 2014 -0500

    Fix revocation event handling with MySQL

    When MySQL is used to store revocation events, events are returned
    from the database with the timestamps truncated to the second. This
    causes a revocation event for a token (which has the issued_at
    timestamp to the microsecond) to not match the revocation event and
    therefore the token is not considered to be revoked.

    The fix is to have the revocation events and token timestamps both
    always be truncated to the second. This will cause all tokens for a
    user that are issued within a second to be revoked when any of those
    tokens are revoked, which shouldn't be a problem.

    Conflicts:

     keystone/tests/test_v3_os_revoke.py

    Change-Id: Ibd82b4ce910206dfd504c396614ae2ebed025e9b
    Closes-Bug: #1347961
    (cherry picked from commit 7aee6304f653475a4130dc3e5be602e91481f108)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/121711

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ryan Hsu (<email address hidden>) on branch: master
Review: https://review.openstack.org/121711
Reason: Testing