glance backend is in crazy resize when an image is uploading

Bug #1792710 reported by Liang Fang on 2018-09-15
20
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Cinder
Medium
Liang Fang
Glance
Undecided
Unassigned
glance_store
High
Erno Kuvaja

Bug Description

When uploading a volume to glance as an image, the glance server don't know the image size, so the backend storage server(such as ceph) need to resize the image every time it received new chunk of data(by default 64K). So there will be huge times of resize operations that will impact the performance.

- regarding cinder, it has not calculate the image size and pass the correct size to glance
- regarding glance, it should allow the client to set image size.

This is an known issue which can be found in driver files of all kinds of backend storage system:
In file: glance_store/_drivers/rbd.py, function: add
In file: glance_store/_drivers/cinder.py, function: add
In file: glance_store/_drivers/sheepdog.py, function: add
In all these files, there're comments like below:
    # If the image size provided is zero we need to do
    # a resize for the amount we are writing. This will
    # be slower so setting a higher chunk size may
    # speed things up a bit.

Revision history for this message
Sean McGinnis (sean-mcginnis) wrote :

What is the Cinder issue here?

Changed in cinder:
status: New → Incomplete
Liang Fang (liangfang) on 2018-09-15
Changed in cinder:
assignee: nobody → Liang Fang (liangfang)
Changed in glance:
assignee: nobody → Liang Fang (liangfang)
Changed in python-glanceclient:
assignee: nobody → Liang Fang (liangfang)
Revision history for this message
Liang Fang (liangfang) wrote :

cinder need to get the image size and send the size to glance at the beginning of the uploading. I will point the review to this bug soon.

description: updated
Changed in cinder:
status: Incomplete → In Progress
Revision history for this message
Sean McGinnis (sean-mcginnis) wrote :

Glance and glanceclient take an optional size as input. What is the bug there? This needs better detail.

Changed in glance:
status: New → In Progress
Changed in python-glanceclient:
status: New → In Progress
Revision history for this message
Liang Fang (liangfang) wrote :

@Sean, more information provided in "Bug Description", thanks

description: updated
Jay Bryant (jsbryant) on 2018-09-20
Changed in cinder:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/609994

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (master)

Fix proposed to branch: master
Review: https://review.openstack.org/609997

Liang Fang (liangfang) on 2019-04-18
no longer affects: python-glanceclient
description: updated
Revision history for this message
Erno Kuvaja (jokke) wrote :

PTG May 2nd Cinder/Glance session:

Lets not make the size user settable and instead just resize bigger chunks at the time and shrink back after EOF.

Changed in glance:
importance: Undecided → High
milestone: none → next
Changed in glance-store:
importance: Undecided → High
no longer affects: glance
Revision history for this message
Liang Fang (liangfang) wrote :

Thanks Jokke. I know ceph support shrink, but not sure if other storage backend support shrink or not. It's a good idea, but may need to check if backends support shrink when doing resize. if not support shrink, then may need to keep current way - resize in a small step.

Erno Kuvaja (jokke) on 2020-08-17
Changed in glance-store:
assignee: nobody → Erno Kuvaja (jokke)
status: New → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance_store (master)

Fix proposed to branch: master
Review: https://review.opendev.org/746579

Changed in glance-store:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance_store (master)

Reviewed: https://review.opendev.org/746579
Committed: https://git.openstack.org/cgit/openstack/glance_store/commit/?id=c43f19e8456b9e20f03709773fb2ffdb94807a0a
Submitter: Zuul
Branch: master

commit c43f19e8456b9e20f03709773fb2ffdb94807a0a
Author: Erno Kuvaja <email address hidden>
Date: Mon Aug 17 16:35:03 2020 +0100

    Ramp up rbd resize to avoid excessive calls

    Change the RBD store to resize the image by up to 8GiB
    at the time to not resize on every write.

    Trim the image in Ceph after all data has been written
    to the actual size in case we overshot the resize.

    Partial-Bug: #1792710
    Related-to: spec-lite-Ceph-Store-Optimization

    Change-Id: I7f0bffda222b663d4316c5d6c03fdbd0d3337035

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance_store (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/762498

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance_store (stable/ussuri)

Reviewed: https://review.opendev.org/762498
Committed: https://opendev.org/openstack/glance_store/commit/ca0c58b52756058b6d51bf6a47aeac3d525c1e16
Submitter: Zuul
Branch: stable/ussuri

commit ca0c58b52756058b6d51bf6a47aeac3d525c1e16
Author: Erno Kuvaja <email address hidden>
Date: Mon Aug 17 16:35:03 2020 +0100

    Ramp up rbd resize to avoid excessive calls

    Change the RBD store to resize the image by up to 8GiB
    at the time to not resize on every write.

    Trim the image in Ceph after all data has been written
    to the actual size in case we overshot the resize.

    Partial-Bug: #1792710
    Related-to: spec-lite-Ceph-Store-Optimization

    Change-Id: I7f0bffda222b663d4316c5d6c03fdbd0d3337035
    (cherry picked from commit c43f19e8456b9e20f03709773fb2ffdb94807a0a)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance_store (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/763570

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers