Timing vulnerabilities throughout openstack

Bug #942644 reported by Paul McMillan
264
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Russell Bryant
OpenStack Identity (keystone)
Fix Released
Medium
Russell Bryant
OpenStack Object Storage (swift)
Fix Released
Medium
Russell Bryant

Bug Description

I filed this bug against keystone because it is the most practically exploitable. Everywhere else that uses a tokenized API or signatures should be audited.

For those who haven't encountered them before, the timing attacks I'm reporting here allow an attacker to deduce the correct value for authentication tokens in much less than brute force time when those tokens are validated with a simple equality comparison.

A quick and easy introduction:
http://codahale.com/a-lesson-in-timing-attacks/

Some practical recent research:
http://rdist.root.org/2010/07/19/exploiting-remote-timing-attacks/
(the actual presentation)
http://www.youtube.com/results?search_query=nate+lawson+timing

Keystone uses passlib to hash and compare user passwords. This is good, because it includes an opaque password validity check which is hardened against timing attacks.

Unfortunately, most of the other token validation routines in openstack use equality comparisons for this, rather than a constant time compare function like this one:
https://code.djangoproject.com/browser/django/trunk/django/utils/crypto.py#L78

Here are a few examples of brokenness. There are probably quite a few more, I haven't done a full audit.

https://github.com/openstack/keystone/blob/master/keystone/contrib/s3/core.py#L50
https://github.com/openstack/keystone/blob/master/keystone/contrib/ec2/core.py#L108
https://github.com/openstack/nova/blob/master/nova/auth/manager.py#L331,
https://github.com/openstack/nova/blob/master/nova/auth/manager.py#L343
https://github.com/openstack/nova/blob/master/nova/auth/manager.py#L352
https://github.com/openstack/swift/blob/master/swift/common/middleware/formpost.py#L445

This one is more subtle - given a valid token, it allows an attacker to deduce the correct username
https://github.com/openstack/nova/blob/master/nova/api/openstack/auth.py#L245

While these vulnerabilities may be difficult to exploit over the general internet, an openstack installation can expect an attack from a compromised node, which may be as far away as gigabit switched ethernet, or as close as a vm container on the same physical host. In the latter case, timing attacks even against string comparisons are eminently practical.

Timing attacks are one of the topics I'll be discussing at Pycon on March 9. It would be nice to have these fixed before then.

Revision history for this message
Russell Bryant (russellb) wrote :

Thanks for the report. I have marked this issue as confirmed and subscribed the project leads to the bug for the projects you reported this against.

I can write patches for the instances pointed out here, but will probably need some help with the audit to make sure all the appropriate places get patched.

Changed in keystone:
status: New → Confirmed
Changed in nova:
status: New → Confirmed
Changed in swift:
status: New → Confirmed
Revision history for this message
Thierry Carrez (ttx) wrote :

Should that one be kept private ? Though it definitely needs to be fixed, it doesn't sound directly exploitable, so there is no reason for an information embargo ?

Revision history for this message
Russell Bryant (russellb) wrote :

I would be fine with classifying this as a "hardening" issue and opening this up now. That makes sense to me.

I'm attaching a first pass at patches for this.

Revision history for this message
Russell Bryant (russellb) wrote :
Revision history for this message
Russell Bryant (russellb) wrote :
Revision history for this message
Russell Bryant (russellb) wrote :

Paul, are you ok with opening this up and fixing it in the open as opposed to the embargoed disclosure process?

description: updated
Revision history for this message
Paul McMillan (paul-mcmillan) wrote :

I'm ok with opening it up. It's exploitable, but the exploit is noisy, slow, and requires specialized knowledge and some amount of per-system tuning. It's definitely not something that's off-the-shelf publicly remotely exploitable.

I'll probably pick one of these and use it to demo the issue at Pycon, but that will be in a controlled environment, and still require hours worth of data collection.

description: updated
visibility: private → public
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/4642

Changed in nova:
assignee: nobody → Russell Bryant (russellb)
status: Confirmed → In Progress
Changed in swift:
assignee: nobody → Russell Bryant (russellb)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to swift (master)

Fix proposed to branch: master
Review: https://review.openstack.org/4643

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/4642
Committed: http://github.com/openstack/nova/commit/1ea998649058f78f63a167dd697cf5f6732596ff
Submitter: Jenkins
Branch: master

commit 1ea998649058f78f63a167dd697cf5f6732596ff
Author: Russell Bryant <email address hidden>
Date: Tue Feb 28 10:55:38 2012 -0500

    Use constant time string comparisons for auth.

    Fix bug 942644.

    Use constant time string comparisons when doing authentication to help
    guard against timing attacks.

    Change-Id: Iaaefb13f7618b06834630d9ccb97aff056b4bf4c

Changed in nova:
status: In Progress → Fix Committed
Joseph Heck (heckj)
Changed in keystone:
assignee: nobody → Russell Bryant (russellb)
milestone: none → essex-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to swift (master)

Reviewed: https://review.openstack.org/4643
Committed: http://github.com/openstack/swift/commit/1f0798f4fbab62661a5560697b541fcaad7f8232
Submitter: Jenkins
Branch: master

commit 1f0798f4fbab62661a5560697b541fcaad7f8232
Author: Russell Bryant <email address hidden>
Date: Tue Feb 28 11:17:21 2012 -0500

    Use constant time string comparisons for auth.

    Fix bug 942644.

    Use constant time string comparisons when doing authentication to help
    guard against timing attacks.

    Change-Id: I88c4c5cd9edd9e5d60db07b6ae2638b74a2a2e17

Changed in swift:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → essex-4
status: Fix Committed → Fix Released
Changed in keystone:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (master)

Reviewed: https://review.openstack.org/4644
Committed: http://github.com/openstack/keystone/commit/1c2419114702865fcec97ea957dc1086f29893bd
Submitter: Jenkins
Branch: master

commit 1c2419114702865fcec97ea957dc1086f29893bd
Author: Russell Bryant <email address hidden>
Date: Tue Feb 28 11:42:19 2012 -0500

    Use constant time string comparisons for auth.

    Fix bug 942644.

    Use constant time string comparisons when doing authentication to help
    guard against timing attacks.

    Change-Id: I5fa5c8f07e57201e129903f71b3dea19071cac5e

Changed in keystone:
status: In Progress → Fix Committed
Revision history for this message
Thierry Carrez (ttx) wrote :

Great work, Russell !

Changed in keystone:
importance: Undecided → Medium
Changed in nova:
importance: Undecided → Medium
Changed in swift:
importance: Undecided → Medium
Thierry Carrez (ttx)
Changed in swift:
milestone: none → 1.4.7
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in keystone:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in keystone:
milestone: essex-rc1 → 2012.1
Thierry Carrez (ttx)
Changed in nova:
milestone: essex-4 → 2012.1
To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.