SLO / DLO auth bug

[David Goetz]

When an SLO is being downloaded and the auth token expires in the middle of the download the download fails because the sub request to the segment gets a 401.

[clayg (clay-gerrard)]

what if we let the authorize callback do it's thing with container info timeouts all that, but we only let each request do the get_groups dance once? i.e. what if we just don't send x-auth-token down with subrequests and leave the already set remote_user and swift.authorize in the request environ to do it's thing?

[clayg (clay-gerrard)]

I have a few thoughts on this.

since the token validation that actually triggers the the re-authn for keystone happens in keystoneclient.middleware.authtoken it does not seem reasonable to push swift specific log such as 'swift.cache_authentication' into that layer to prevent an attempt to re-populate identity information extracted from the (possibly expired) pki token. Rather, removing the token from subrequests and re-using cached identity/authn and authz hooks from the pre-flight manifest request for xlo seems to in fact be the best (and possibly only compatible) way to prevent re-authn of a token for each sub-req.

I attached a patch that forces keystone_auth middleware to re-use cached identity on subrequests, since this code lives in swift I don't think there's a concern for upgrades except that we may want to back port this fix to Icehouse.

Another equally gross option would be to fix keystone so that (like other auth middlewares) it will reattach it's authorize call back for all requests made to it's reseller prefix'd accounts.

Both of the above solutions depend on a change to common.wsgi.make_env so that cached identity information *is* actually provided in subrequests as REMOTE_USER isn't apparently the only way to do this.

If make_env was less restrictive on the keys copied for subrequests, and the authoriziation callback could somehow be captured for re-use in subrequests (maybe in server.py) it would probably be possible to craft a solution that would work with all middlewares regardless of the underlying implementation... well assuming that cache identity in the environ and look for that before deciding if it's anonymous request. What a drag...

[clayg (clay-gerrard)]

dfg thinks we can fix this by moving xlo to the right of auth and I think it's a pretty good idea.

It solves caching authn and leaves us in a good place to reuse swift.authorize on subrequests before it get's popped out of the env in proxy.server

I needed something like:

https://gist.github.com/clayg/8378436d6772ac6fae3c

to make common.wsgi and keystone_auth support cached identity information though.

Option B is let make_env just copy over everything and then keystone auth can just pull out the headers it needs to rebuild keystone.identity every request.

[Alistair Coles (alistair-coles)]

Seems reasonable. I had it on my list of 'silly questions' to ask someone why xLO had to be to left of auth :)

I tried moving xLO to the right of tempauth and seems to work fine (as in func tests pass). With keystoneauth I got timeouts due to keystoneauth exceptions when the subrequests env wasn't as expected but Clay's patch sorted that.

Putting paranoia hat on - is there a risk that a client could take advantage of this change to deliberately extend the lifetime of a token/avoid revocation? e.g. client starts a GET just before revocation, reads slowly... owner revokes token and PUTs new xLO segments, client reads new content. Do we/should we check that segment put timestamps are < original GET request timestamp?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/92165
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=ab510952efdd5333e197d23042816ba6ecfa3339
Submitter: Jenkins
Branch: master

commit ab510952efdd5333e197d23042816ba6ecfa3339
Author: David Goetz <email address hidden>
Date: Tue May 27 09:52:39 2014 -0700

    xLO bug with auth tokens expiring during download.

    Just put SLO and DLO after any auth middleware. This works because when
    the request goes through that middleware in the pipeline the
    authentication takes place: validation of the token, setting up who the
    user is, and setting the authorization call back. Each subrequest made
    for the segments will be subjected to that authorization call back which
    verifies the user has access to the individual segments.

    To get this to work with keystone, the keystone identity is set up
    during __call__ and applied to the authorize function using a
    functools.partial. When the authorize function is later called from the
    environ by the proxy server the idenity that was set up when the request
    passed through the auth middleware is used, not what can be pulled out
    of the possibly altered state of the request's environment.

    DocImpact
    fixes bug: 1315133

    Change-Id: I7827dd2d9dfbb3c6424773fb2891355d47e372ba