S3 requires seekable file. webob versions 0.9.8 through 1.0.7 make_body_seekable() method broken for chunked transfer requests

[Jay Pipes]

This is a fairly complicated bug in the upstream webob library. Different versions of the webob library have different problems with handling chunked transfer requests in the webob.Request object:

webob 0.9.8

Does not have a webob.Request.is_body_seekable attribute. However, it *does* have a webob.Request.make_body_seekable() method. Unfortunately, calling this method when the request has no Content-Length header -- as is the case for chunked transfer requests --- results in the following code being run:

    def make_body_seekable(self):
        """
        This forces ``environ['wsgi.input']`` to be seekable. That
        is, if it doesn't have a `seek` method already, the content is
        copied into a StringIO or temporary file.

        The choice to copy to StringIO is made from
        ``self.request_body_tempfile_limit``
        """
        input = self.body_file
        if hasattr(input, 'seek'):
            # It has a seek method, so we don't need to do anything
            return
        self.copy_body()

    def copy_body(self):
        """
        Copies the body, in cases where it might be shared with
        another request object and that is not desired.

        This copies the body in-place, either into a StringIO object
        or a temporary file.
        """
        length = self.content_length
        if length == 0:
            # No real need to copy this, but of course it is free
            self.body_file = StringIO('')
            return

This unfortunately unsets the body of the request, which means a content-length of 0 and a body of '' is passed along to the backend.

The S3 backend requires a *seekable* body_file attribute, and in 0.9.8, the only way to provide this is to call make_body_seekable().

webob 1.0.3 through 1.0.6:

See http://groups.google.com/group/paste-users/browse_thread/thread/10157caa893a82e7

Unfortunately, in this particular version of webob, the Request's body_file object is not readable in a secure way

webob 1.0.7:

See https://bitbucket.org/ianb/webob/issue/12/fix-for-issue-6-broke-chunked-transfer

In this version of webob.Request, for chunked transfer requests, the webob.Request.content_length and body_file attributes are not properly handled (the body_file is unset if there is no content-length header)

The solution here is to set Request.is_body_readable = True manually.

So, we unfortunately need to standardize on webob 1.0.7+ if we are to support the S3 backend (which requires a seekable body_file) and support Swift backends properly for chunked transfer encoding.

Swift has already standardized on 1.0.7+ for similar reasons, so I think we should do the same.