container-sync IP-based authentication is broken

Bug #1068420 reported by Faidon Liambotis
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
Fix Released
Medium
Samuel Merritt

Bug Description

Swift's container-sync uses a combination of a static key authentication, mutually agreed and set between the two containers ("X-Container-Sync-Key") and IP-based authantication for the whole cluster, as set in the "allowed_sync_hosts" configuration directive.

Unfortunately, the code that validates the IP-based authentication has this:
            and (req.remote_addr in self.allowed_sync_hosts
                 or swift_utils.get_remote_client(req)
                 in self.allowed_sync_hosts)):
(from keystoneauth, but is the same in tempauth & swauth)

And, in turn, get_remote_client() is:

def get_remote_client(req):
    # remote host for zeus
    client = req.headers.get('x-cluster-client-ip')
    if not client and 'x-forwarded-for' in req.headers:
        # remote host for other lbs
        client = req.headers['x-forwarded-for'].split(',')[0].strip()
    if not client:
        client = req.remote_addr
    return client

In other words, it unconditionally trusts X-Cluster-Client-IP or X-Forwarded-For as sent by the remote end. This is on the path to the proxy servers, which are exposed and on the public side of the cluster.

This isn't very serious, since the attacker would have to know the key too; but it essentially renders the IP-based authentication part almost useless and the two-factor authentication breaks down to just one factor.

Tags: security
Revision history for this message
Thierry Carrez (ttx) wrote :

@John: could you (or someone elese in swift-core) attach a patch to fix that to this bug (no public fix at the moment please) ?

We still need to decide if we'll consider this a vulnerability that would force us to embargo the fix (not directly exploitable but definitely a false sense of security)

@Russell, Steve: opinion on whether this warrants an OSSA ?

Changed in swift:
importance: Undecided → Medium
status: New → Confirmed
Revision history for this message
Russell Bryant (russellb) wrote :

My opinion is that this does not warrant an OSSA. While this is certainly something that should be fixed, since it's not an exploitable vulnerability, I don't think we need to go through embargo. The key is what is providing the real security here.

Revision history for this message
Thierry Carrez (ttx) wrote :

Yes, looks like allowed_sync_hosts should not be considered a security feature (the shared key is what provides security, IP source is pretty spoofable anyway), at best it should be considered a way to catch misconfigurations (among the trusted that know the shared secret).

I agree that it should be documented, though.

Revision history for this message
Thierry Carrez (ttx) wrote :

Opening the bug per previous comments.

tags: added: security
information type: Private Security → Public
Revision history for this message
Samuel Merritt (torgomatic) wrote :

Well, we can't simply ask the underlying TCP connection what its peer is, since there could be an external load balancer in front of the proxies. Therefore, we're stuck looking at something in the request to determine the remote IP, which is obviously trivially spoofable by any attacker with two brain cells to rub together.

I think the answer is to take out the IP authentication stuff entirely. It can't be made to work, and it provides a false sense of security for operators that don't look under the hood.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to swift (master)

Fix proposed to branch: master
Review: https://review.openstack.org/16358

Changed in swift:
assignee: nobody → Samuel Merritt (torgomatic)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to swift (master)

Reviewed: https://review.openstack.org/16358
Committed: http://github.com/openstack/swift/commit/357b12dc2ba7b19c66196a573ccb2489d2104b93
Submitter: Jenkins
Branch: master

commit 357b12dc2ba7b19c66196a573ccb2489d2104b93
Author: Samuel Merritt <email address hidden>
Date: Fri Nov 16 17:05:37 2012 -0800

    Remove IP-based container-sync ACLs from auth middlewares.

    The determination of the client IP looked at the X-Cluster-Client-Ip
    and X-Forwarded-For headers in the incoming HTTP request. This is
    trivially spoofable by a malicious client, so there's no security
    gained by having the check there.

    Worse, having the check there provides a false sense of security to
    cluster operators. It sounds like it's based on the client IP, so an
    attacker would have to do IP spoofing to defeat it. However, it's
    really just a shared secret, and there's already a secret key set
    up. Basically, it looks like 2-factor auth (IP+key), but it's really
    1-factor (key).

    Now, the one case where this might provide some security is where the
    Swift cluster is behind an external load balancer that strips off the
    X-Cluster-Client-Ip and X-Forwarded-For headers and substitutes its
    own. I don't think it's worth the tradeoff, hence this commit.

    Fixes bug 1068420 for very small values of "fixes".

    DocImpact

    Change-Id: I2bef64c2e1e4df8a612a5531a35721202deb6964

Changed in swift:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in swift:
milestone: none → 1.7.6
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.