Upload starves greenthreads

[Samuel Matzek]

The upload of files can starve other greenthreads in the same process. When glanceclient is used in Nova for snapshot upload, this can lead to RPC connection timeouts and connection aborts causing concurrent operations to fail.

In our KVM libvirt environment, 5 concurrent snapshot uploads from the same host was able to cause significant starvation on the other threads.

The root cause is that file I/O goes not yield to other threads with greenthreads. This causes the greenthreads that are running the upload to capitalize on the processor time and starve the other threads.

The suggested fix for this would be to take the CooperativeReader class from glance/common/utils.py which is used to fix this very issue on the Glance API side, and put it in glanceclient/common/utils.py. The data part of the body in v1/images.py ImageManager, upload method can be wrappered in the CooperativeReader like this:
        resp, body_iter = self.api.raw_request(
            'PUT', url, headers=hdrs,
            body=CooperativeReader(image_data))

In our environment this has been shown to alleviate the Nova greenthread starvation issues when doing snapshots.

[Ian Cordasco (icordasc)]

Is this still a problem you can replicate with the latest versions of Nova and glanceclient?