Comment 46 for bug 1411704

Reviewed: https://review.fuel-infra.org/9279
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-5.1-updates/2014.1.1

Commit: d3d83d98a770ba69666d4f3d4998ede2021c97cd
Author: Alexander Tivelkov <email address hidden>
Date: Mon Jul 13 15:32:03 2015

Fix for CooperativeReader to process read length

CooperativeReader, being an eventlet-friendly wrapper around the
generator- based reader of image data, actually transforms
chunk-by-chunk iteration into the readable stream. It is used when the
image is being copied from the remote source: some generator-based
image data representing the remote source acts as its underlying
object, and the instance of CooperativeReader is passed as a data
stream to the backend client which uses it to read the data.

Before this patch, the CooperativeReader was ignoring the "length"
parameter of the read method, always returning the whole chunk returned
by the underlying generator (in case of HTTP source the size of this
chunk is 16 M). This was causing problems for the clients attempting to
read data from it, and - under some circumstances - the loss of data.

For chunked storage of files in Swift a special class (ChunkReader,
declared in the swift store driver) is used to reduce the requested
read length so no extra data is read and transferred. However, this was
not working as the CooperativeReader (which was the underlying stream
for the ChunkReader) was ignoring the requested size. This was causing
the data to be lost when reading behind the boundaries of the Chunks.

This patchset introduces a buffer in the CooperativeReader to store the
most recently fetched iterator chunk. The reads are independent from
requests to iterator, so the CooperativeReader is able to return the
exact requested amount of bytes and no data is lost due to extra-reads.

SecurityImpact

Change-Id: Ief37d1e29487bb03e612320f5cc06910cfd1c23a
Closes-bug: #1411704
Closes-bug: #1405386
(cherry picked from commit 270ec44a890f4bd2917310e2fcab9ad2bb9413b7)
(cherry picked from commit de45f294dccaeb8e2d933ab17f9f001600da0b5e)
(cherry picked from commit 9cc6b55e6e5432de48d2e207b8e2dae0a1478172)