Volume to Image upload very slow due to use of tpool.Proxy()

[Robert Franzke]

In our environment, we observed that the speed of a volume upload to a Glance image decreases very quickly over the course of an upload.

The upload starts at around 300mb/s but drops below 50mb/s within the first gigabyte and becomes even slower as it progresses. Using the patch in the Glance client, which disables chunked endcoding, the performance could be improved a bit. However, it was still too slow. (https://review.opendev.org/c/openstack/python-glanceclient/+/733984)

In cinder/image/image_utils.py, the FileHandler is wrapped in a tpool.Proxy(). However, if you instead run the entire image_service.update call with tpool.execute() and remove the tpool.Proxy(), you get very good performance. Like constant 700mb/s and more. But this works only in combination with the above mentioned patch for the Glance client. Otherwise you see again the behavior described above, that the speed decreases a lot during the upload.

So replace this:
image_service.update(context, image_id, {},
                     tpool.Proxy(image_file),
                     store_id=store_id,
                     base_image_ref=base_image_ref)

with this:
tpool.execute(image_service.update, context, image_id, {},
              image_file, store_id=store_id,
              base_image_ref=base_image_ref)

We use cinder and glance with the Yoga release. For Cinder we use the Netapp NFS backend and Glance with the S3 backend.

[Robert Franzke (robertfranzke)]

We were able to recreate the behavior of it being slower due to chuncked encoding outside of cinder volume. However, only with the python-glanceclient and not with the python-openstackclient.

This is because the python-openstackclient does not use chunked encoding when uploading. Removing this from the python-glanceclient with the above patch we have seen the same constant performance as the python-openstack client. In our environment this is about 700mb/s, as with cinder in combination with the tpool patch.

[Robert Franzke (robertfranzke)]

Proposed fix: https://review.opendev.org/c/openstack/cinder/+/888676
(I don't know why it wasn't automatically linked.)