Upload starves greenthreads

Bug #1327248 reported by Samuel Matzek
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Glance Client
Invalid
Undecided
Unassigned

Bug Description

The upload of files can starve other greenthreads in the same process. When glanceclient is used in Nova for snapshot upload, this can lead to RPC connection timeouts and connection aborts causing concurrent operations to fail.

In our KVM libvirt environment, 5 concurrent snapshot uploads from the same host was able to cause significant starvation on the other threads.

The root cause is that file I/O goes not yield to other threads with greenthreads. This causes the greenthreads that are running the upload to capitalize on the processor time and starve the other threads.

The suggested fix for this would be to take the CooperativeReader class from glance/common/utils.py which is used to fix this very issue on the Glance API side, and put it in glanceclient/common/utils.py. The data part of the body in v1/images.py ImageManager, upload method can be wrappered in the CooperativeReader like this:
        resp, body_iter = self.api.raw_request(
            'PUT', url, headers=hdrs,
            body=CooperativeReader(image_data))

In our environment this has been shown to alleviate the Nova greenthread starvation issues when doing snapshots.

Revision history for this message
Ian Cordasco (icordasc) wrote :

Is this still a problem you can replicate with the latest versions of Nova and glanceclient?

Changed in python-glanceclient:
status: New → Incomplete
Ian Cordasco (icordasc)
Changed in python-glanceclient:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers