Comment 8 for bug 1412802

I did some digging to try to understand the issue a little better for myself.

I checked out the Essex code base which contained the copy-from functionality (including the ResponseIndexable class) but did not include the CooperativeReader yet.

Running the reproducer gave:

 2015-02-06 16:01:36 23295 ERROR [glance.api.v1.images] Traceback (most recent call last):
   File "/mnt/ubuntu/git/glance-essex/glance/glance/api/v1/images.py", line 398, in _upload
     image_size)
   File "/mnt/ubuntu/git/glance-essex/glance/glance/store/swift.py", line 418, in add
     content_length=content_length)
   File "/mnt/ubuntu/git/glance-essex/glance/.venv/local/lib/python2.7/site-packages/swift/common/client.py", line 940, in put_object
     content_type=content_type, headers=headers)
   File "/mnt/ubuntu/git/glance-essex/glance/.venv/local/lib/python2.7/site-packages/swift/common/client.py", line 844, in _retry
     rv = func(self.url, self.token, *args, **kwargs)
   File "/mnt/ubuntu/git/glance-essex/glance/.venv/local/lib/python2.7/site-packages/swift/common/client.py", line 709, in put_object
     chunk = contents.read(size)
   File "/mnt/ubuntu/git/glance-essex/glance/glance/store/swift.py", line 532, in read
     result = self.fd.read(i)
 AttributeError: 'ResponseIndexable' object has no attribute 'read'

So, as far as I can tell, copy-from *never* worked when the image was larger than 'swift_store_large_object_size'.

With CooperativeReader present we just get a more subtle error.

We supply an iterator to wsgi for a normal download (it's all it needs). Hence _get_from_store doesn't supply a 'read' method if the underlying file descriptor/iterator doesn't have one, and downloads work fine.

response.app_iter = checked_iter(image_id, expected_size, image_iter)

So the original iterator:

 def http_response_iterator(conn, response, size):
    """
    Return an iterator for a file-like object.

    :param conn: HTTP(S) Connection
    :param response: httplib.HTTPResponse object
    :param size: Chunk size to iterate with
    """
    chunk = response.read(size)
    while chunk:
        yield chunk
        chunk = response.read(size)
    conn.close()

worked fine for standard downloads.

There seems to be a bit of a catch-22, with evenlet's greenio when reading in the data over the network. It requires more than a typical iterator/generator provides, eg it also needs read() and len. ResponseIndexable adds these, but doesn't work for images with size > swift_store_large_object_size. (See the stack trace above).

Alexander's patch makes the read behaviour of CooperativeReader respect the read length passed in. This effectively masks the lack of a working read method in 'ResponseIndexable'. (But it works! Yay!)

I *think* we could get rid of a lot of the iterator complexity (eg even http_response_iterator) by moving to 'requests', but I'd need to try it out a bit more to be sure.