glance backend is in crazy resize when an image is uploading

[Liang Fang]

When uploading a volume to glance as an image, the glance server don't know the image size, so the backend storage server(such as ceph) need to resize the image every time it received new chunk of data(by default 64K). So there will be huge times of resize operations that will impact the performance.

- regarding cinder, it has not calculate the image size and pass the correct size to glance
- regarding glance, it should allow the client to set image size.

This is an known issue which can be found in driver files of all kinds of backend storage system:
In file: glance_store/_drivers/rbd.py, function: add
In file: glance_store/_drivers/cinder.py, function: add
In file: glance_store/_drivers/sheepdog.py, function: add
In all these files, there're comments like below:
    # If the image size provided is zero we need to do
    # a resize for the amount we are writing. This will
    # be slower so setting a higher chunk size may
    # speed things up a bit.

[Sean McGinnis (sean-mcginnis)]

What is the Cinder issue here?

[Liang Fang (liangfang)]

cinder need to get the image size and send the size to glance at the beginning of the uploading. I will point the review to this bug soon.

[Sean McGinnis (sean-mcginnis)]

Glance and glanceclient take an optional size as input. What is the bug there? This needs better detail.

[Liang Fang (liangfang)]

@Sean, more information provided in "Bug Description", thanks

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/609994

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/609997

[Erno Kuvaja (jokke)]

PTG May 2nd Cinder/Glance session:

Lets not make the size user settable and instead just resize bigger chunks at the time and shrink back after EOF.

[Liang Fang (liangfang)]

Thanks Jokke. I know ceph support shrink, but not sure if other storage backend support shrink or not. It's a good idea, but may need to check if backends support shrink when doing resize. if not support shrink, then may need to keep current way - resize in a small step.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/746579

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/746579
Committed: https://git.openstack.org/cgit/openstack/glance_store/commit/?id=c43f19e8456b9e20f03709773fb2ffdb94807a0a
Submitter: Zuul
Branch: master

commit c43f19e8456b9e20f03709773fb2ffdb94807a0a
Author: Erno Kuvaja <email address hidden>
Date: Mon Aug 17 16:35:03 2020 +0100

    Ramp up rbd resize to avoid excessive calls

    Change the RBD store to resize the image by up to 8GiB
    at the time to not resize on every write.

    Trim the image in Ceph after all data has been written
    to the actual size in case we overshot the resize.

    Partial-Bug: #1792710
    Related-to: spec-lite-Ceph-Store-Optimization

    Change-Id: I7f0bffda222b663d4316c5d6c03fdbd0d3337035

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/762498

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/762498
Committed: https://opendev.org/openstack/glance_store/commit/ca0c58b52756058b6d51bf6a47aeac3d525c1e16
Submitter: Zuul
Branch: stable/ussuri

commit ca0c58b52756058b6d51bf6a47aeac3d525c1e16
Author: Erno Kuvaja <email address hidden>
Date: Mon Aug 17 16:35:03 2020 +0100

    Ramp up rbd resize to avoid excessive calls

    Change the RBD store to resize the image by up to 8GiB
    at the time to not resize on every write.

    Trim the image in Ceph after all data has been written
    to the actual size in case we overshot the resize.

    Partial-Bug: #1792710
    Related-to: spec-lite-Ceph-Store-Optimization

    Change-Id: I7f0bffda222b663d4316c5d6c03fdbd0d3337035
    (cherry picked from commit c43f19e8456b9e20f03709773fb2ffdb94807a0a)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/763570