s3 backend takes time exponentially

[Nobuto Murata]

[Impact]

Glance with S3 backend cannot accept image uploads in a realistic time
frame. For example, an 1GB image upload takes ~60 minutes although other
backends such as swift can complete it with 10 seconds.

[Test Plan]

1. Deploy a partial OpenStack with multiple Glance backends including S3
  (zaza test bundles can be used with "ceph" which will set up "rbd", "swift", and "s3" backends - https://opendev.org/openstack/charm-glance/src/branch/master/tests/tests.yaml)
2. Upload multiple images with variety of sizes
3. confirm the duration of uploading images are shorter in general after applying the updated package
  (expected duration of 1GB is from ~60 minutes to 1-3 minutes)

for backend in ceph swift s3; do
    echo "[$backend]"
    for i in {0,3,5,9,10,128,512,1024}; do
        dd if=/dev/zero of=my-image.img bs=1M count=$i oflag=sync
        echo "${i}MiB"
        time glance image-create \
            --store $backend \
            --file my-image.img --name "my-image-${backend}-${i}MiB" \
            --disk-format raw --container-format bare \
            --progress
    done
done

[Where problems could occur]

Since we bump WRITE_CHUNKSIZE from 64KiB to 5MiB, there might be a case where image uploads fail if the size of the image is less than WRITE_CHUNKSIZE. Or there might be an unexpected latency in the worst case scenario. We will try to address the concerns by testing multiple images uploads with multiple sizes including some corner cases as follows:
- 0 - zero
- 3MiB - less than the new WRITE_CHUNKSIZE(5MiB)
- 5MiB - exactly same as the new WRITE_CHUNKSIZE(5MiB)
- 9MiB - bigger than new WRITE_CHUNKSIZE(5MiB) but less than twice
- 10MiB - exactly twice as the new WRITE_CHUNKSIZE(5MiB)
- 128MiB, 512MiB, 1024MiB - some large images

====

I have a test Ceph cluster as an object storage with both Swift and S3
protocols enabled for Glance (Ussuri). When I use Swift backend with
Glance, an image upload completes quickly enough. But with S3 backend
Glance, it takes much more time to upload an image and it seems to rise
exponentially.

It's worth noting that when uploading an image with S3 backend, a single
core is consumed 100% by glance-api process.

[swift]
8MB - 2.4s
16MB - 2.8s
32MB - 2.6s
64MB - 2.7s
128MB - 3.1s
...
512MB - 5.9s

[s3]
8MB - 2.2s
16MB - 2.9s
32MB - 5.5s
64MB - 16.3s
128MB - 54.9s
...
512MB - 14m26s

Btw, downloading of 512MB image with S3 backend can complete with less
than 10 seconds.

$ time openstack image save --file downloaded.img 917c5424-4350-4bc5-98ca-66d40e101843
real 0m5.673s

$ du -h downloaded.img
512M downloaded.img

[/etc/glance/glance-api.conf]

enabled_backends = local:file, ceph:rbd, swift:swift, s3:s3

[swift]
auth_version = 3
auth_address = http://192.168.151.131:5000/v3
...
container = glance
large_object_size = 5120
large_object_chunk_size = 200

[s3]
s3_store_host = http://192.168.151.137:80/
...
s3_store_bucket = zaza-glance-s3-test
s3_store_large_object_size = 5120
s3_store_large_object_chunk_size = 200

ProblemType: Bug
DistroRelease: Ubuntu 20.04
ProblemType: BugDistroRelease: Ubuntu 20.04
Package: python3-glance-store 2.0.0-0ubuntu1
ProcVersionSignature: Ubuntu 5.4.0-77.86-generic 5.4.119
Uname: Linux 5.4.0-77-generic x86_64
ApportVersion: 2.20.11-0ubuntu27.18
Architecture: amd64
CasperMD5CheckResult: skip
Date: Wed Jul 7 04:46:05 2021
PackageArchitecture: all
ProcEnviron:
 TERM=screen-256color
 PATH=(custom, no user)
 LANG=C.UTF-8
 SHELL=/bin/bash
SourcePackage: python-glance-store
 SHELL=/bin/bashSourcePackage: python-glance-store
UpgradeStatus: No upgrade log present (probably fresh install)

[Nobuto Murata (nobuto)]

python3-boto3 1.9.253-1

[Nobuto Murata (nobuto)]

Debug log of when uploading a 512MB image with S3 backend.

[Nobuto Murata (nobuto)]

S3 performance itself is not bad. Uploading 512MB object can complete within a few seconds. So I suppose it's on how Glance S3 driver is using boto3.

$ time python3 upload.py

real 0m3.644s
user 0m3.124s
sys 0m1.835s

$ cat upload.py
import boto3

s3 = boto3.client(
    "s3",
    endpoint_url="http://192.168.151.137:80/",
    aws_access_key_id="Z99Y0ST1T8F66RUZXSJT",
    aws_secret_access_key="S5sYaE6pa8btV1HQLtz99RG2DNvX4CpzjOfDMjMZ",
)

with open("my-image.img", "rb") as f:
    s3.upload_fileobj(f, "zaza-glance-s3-test", "my-test-object")

[Nobuto Murata (nobuto)]

The code part in question is this for loop:
https://opendev.org/openstack/glance_store/src/branch/stable/ussuri/glance_store/_drivers/s3.py#L638-L644

2021-07-07 11:50:06.735 - def _add_singlepart
2021-07-07 11:50:06.736 - getting into utils.chunkreadable loop
2021-07-07 11:50:06.736 - loop invoked
2021-07-07 11:50:06.737 - loop invoked
2021-07-07 11:50:06.737 - loop invoked
...
2021-07-07 11:50:22.514 - loop invoked (1024 times in total)
2021-07-07 11:50:22.544 - put_object

So it's taking time before data is passed to boto3 using put_object.

[Nobuto Murata (nobuto)]

Okay, as the utils.chunkreadable loop is taking time I've tried a larger WRITE_CHUNKSIZE by hand. It can decrease the amount of time of uploading a 512MB image from 14 minutes to 60 seconds.

$ git diff
diff --git a/glance_store/_drivers/s3.py b/glance_store/_drivers/s3.py
index 1c18531..576c573 100644
--- a/glance_store/_drivers/s3.py
+++ b/glance_store/_drivers/s3.py
@@ -361,7 +361,7 @@ class Store(glance_store.driver.Store):
     EXAMPLE_URL = "s3://<ACCESS_KEY>:<SECRET_KEY>@<S3_URL>/<BUCKET>/<OBJ>"

     READ_CHUNKSIZE = 64 * units.Ki
- WRITE_CHUNKSIZE = READ_CHUNKSIZE
+ WRITE_CHUNKSIZE = 1024 * units.Ki

     @staticmethod
     def get_schemes():

[Nobuto Murata (nobuto)]

And by using "4 * units.Mi" it can be 20s.

[James Page (james-page)]

I think the swift driver uses 200M:

DEFAULT_LARGE_OBJECT_CHUNK_SIZE = 200 # 200M

On Wed, Jul 7, 2021 at 1:50 PM Nobuto Murata <email address hidden>
wrote:

> And by using "4 * units.Mi" it can be 20s.
>
> --
> You received this bug notification because you are a member of Ubuntu
> OpenStack, which is subscribed to python-glance-store in Ubuntu.
> https://bugs.launchpad.net/bugs/1934849
>
> Title:
> s3 backend takes time exponentially
>
> Status in glance_store:
> New
> Status in python-glance-store package in Ubuntu:
> New
>
> Bug description:
> I have a test Ceph cluster as an object storage with both Swift and S3
> protocols enabled for Glance (Ussuri). When I use Swift backend with
> Glance, an image upload completes quickly enough. But with S3 backend
> Glance, it takes much more time to upload an image and it seems to
> rise exponentially.
>
> It's worth noting that when uploading an image with S3 backend, a
> single core is consumed 100% by glance-api process.
>
> for backend in swift s3; do
> for i in {8,16,32,64,128,512}; do
> dd if=/dev/zero of=my-image.img bs=1M count=$i oflag=sync
> time glance image-create \
> --store $backend \
> --file my-image.img --name my-image \
> --disk-format raw --container-format bare \
> --progress
> done
> done
>
> [swift]
> 8MB - 2.4s
> 16MB - 2.8s
> 32MB - 2.6s
> 64MB - 2.7s
> 128MB - 3.1s
> ...
> 512MB - 5.9s
>
> [s3]
> 8MB - 2.2s
> 16MB - 2.9s
> 32MB - 5.5s
> 64MB - 16.3s
> 128MB - 54.9s
> ...
> 512MB - 14m26s
>
> Btw, downloading of 512MB image with S3 backend can complete with less
> than 10 seconds.
>
> $ time openstack image save --file downloaded.img
> 917c5424-4350-4bc5-98ca-66d40e101843
> real 0m5.673s
>
> $ du -h downloaded.img
> 512M downloaded.img
>
>
> [/etc/glance/glance-api.conf]
>
> enabled_backends = local:file, ceph:rbd, swift:swift, s3:s3
>
> [swift]
> auth_version = 3
> auth_address = http://192.168.151.131:5000/v3
> ...
> container = glance
> large_object_size = 5120
> large_object_chunk_size = 200
>
>
> [s3]
> s3_store_host = http://192.168.151.137:80/
> ...
> s3_store_bucket = zaza-glance-s3-test
> s3_store_large_object_size = 5120
> s3_store_large_object_chunk_size = 200
>
> ProblemType: Bug
> DistroRelease: Ubuntu 20.04
> Package: python3-glance-store 2.0.0-0ubuntu1
> ProcVersionSignature: Ubuntu 5.4.0-77.86-generic 5.4.119
> Uname: Linux 5.4.0-77-generic x86_64
> NonfreeKernelModules: bluetooth ecdh_generic ecc tcp_diag inet_diag
> binfmt_misc veth zfs zunicode zlua zavl icp zcommon znvpair spl unix_diag
> nft_masq nft_chain_nat bridge stp llc vhost_vsock
> vmw_vsock_virtio_transport_common vhost vsock ebtable_filter ebtables
> ip6table_raw ip6table_mangle ip6table_nat ip6table_filter ip6_tables
> iptable_raw iptable_mangle iptable_nat nf_nat nf_conntrack nf_defrag_ipv6
> nf_defrag_ipv4 iptable_filter bpfilter nf_tables nfnetlink dm_multipath
> scsi_dh_rdac scsi_dh_emc scsi_dh_alua kvm_amd ccp input_leds kvm joydev
> mac_hid serio_raw qemu_fw_cfg...

[James Page (james-page)]

Actually that's only used for images over 5GB

On Wed, Jul 7, 2021 at 1:59 PM James Page <email address hidden> wrote:

> I think the swift driver uses 200M:
>
> DEFAULT_LARGE_OBJECT_CHUNK_SIZE = 200 # 200M
>
> On Wed, Jul 7, 2021 at 1:50 PM Nobuto Murata <email address hidden>
> wrote:
>
>> And by using "4 * units.Mi" it can be 20s.
>>
>> --
>> You received this bug notification because you are a member of Ubuntu
>> OpenStack, which is subscribed to python-glance-store in Ubuntu.
>> https://bugs.launchpad.net/bugs/1934849
>>
>> Title:
>> s3 backend takes time exponentially
>>
>> Status in glance_store:
>> New
>> Status in python-glance-store package in Ubuntu:
>> New
>>
>> Bug description:
>> I have a test Ceph cluster as an object storage with both Swift and S3
>> protocols enabled for Glance (Ussuri). When I use Swift backend with
>> Glance, an image upload completes quickly enough. But with S3 backend
>> Glance, it takes much more time to upload an image and it seems to
>> rise exponentially.
>>
>> It's worth noting that when uploading an image with S3 backend, a
>> single core is consumed 100% by glance-api process.
>>
>> for backend in swift s3; do
>> for i in {8,16,32,64,128,512}; do
>> dd if=/dev/zero of=my-image.img bs=1M count=$i oflag=sync
>> time glance image-create \
>> --store $backend \
>> --file my-image.img --name my-image \
>> --disk-format raw --container-format bare \
>> --progress
>> done
>> done
>>
>> [swift]
>> 8MB - 2.4s
>> 16MB - 2.8s
>> 32MB - 2.6s
>> 64MB - 2.7s
>> 128MB - 3.1s
>> ...
>> 512MB - 5.9s
>>
>> [s3]
>> 8MB - 2.2s
>> 16MB - 2.9s
>> 32MB - 5.5s
>> 64MB - 16.3s
>> 128MB - 54.9s
>> ...
>> 512MB - 14m26s
>>
>> Btw, downloading of 512MB image with S3 backend can complete with less
>> than 10 seconds.
>>
>> $ time openstack image save --file downloaded.img
>> 917c5424-4350-4bc5-98ca-66d40e101843
>> real 0m5.673s
>>
>> $ du -h downloaded.img
>> 512M downloaded.img
>>
>>
>> [/etc/glance/glance-api.conf]
>>
>> enabled_backends = local:file, ceph:rbd, swift:swift, s3:s3
>>
>> [swift]
>> auth_version = 3
>> auth_address = http://192.168.151.131:5000/v3
>> ...
>> container = glance
>> large_object_size = 5120
>> large_object_chunk_size = 200
>>
>>
>> [s3]
>> s3_store_host = http://192.168.151.137:80/
>> ...
>> s3_store_bucket = zaza-glance-s3-test
>> s3_store_large_object_size = 5120
>> s3_store_large_object_chunk_size = 200
>>
>> ProblemType: Bug
>> DistroRelease: Ubuntu 20.04
>> Package: python3-glance-store 2.0.0-0ubuntu1
>> ProcVersionSignature: Ubuntu 5.4.0-77.86-generic 5.4.119
>> Uname: Linux 5.4.0-77-generic x86_64
>> NonfreeKernelModules: bluetooth ecdh_generic ecc tcp_diag inet_diag
>> binfmt_misc veth zfs zunicode zlua zavl icp zcommon znvpair spl unix_diag
>> nft_masq nft_chain_nat bridge stp llc vhost_vsock
>> vmw_vsock_virtio_transport_common vhost vsock ebtable_filter ebtables
>> ip6table_raw ip6table_mangle ip6table_nat ip6table_filter ip6_tables
>> iptable_raw iptable_...

[Nobuto Murata (nobuto)]

Yeah, I put the same config on purpose for both s3 and swift. But tweaking large_object_size didn't make any difference.

[swift]
large_object_size = 5120
large_object_chunk_size = 200

[s3]
s3_store_large_object_size = 5120
s3_store_large_object_chunk_size = 200

After digging into the actual environment, it seems image_size is always zero so this condition never goes to multipart upload, but singlepart upload always.
https://opendev.org/openstack/glance_store/src/branch/stable/ussuri/glance_store/_drivers/s3.py#L597

I guess Glance will never know the actual image_size at the beginning of getting image data from a client until getting all of the data. I just don't know what kind of cases we will get non-null value in "size".
https://docs.openstack.org/api-ref/image/v2/index.html?expanded=create-image-detail#create-image

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Nobuto Murata (nobuto)]

I *think* hash calculation and verifier have to be outside of the loop to avoid the overhead. I will confirm it with a manual testing.

        for chunk in utils.chunkreadable(image_file, self.WRITE_CHUNKSIZE):
            image_data += chunk
            image_size += len(chunk)
            os_hash_value.update(chunk)
            checksum.update(chunk)
            if verifier:
                verifier.update(chunk)

[Nobuto Murata (nobuto)]

> I *think* hash calculation and verifier have to be outside of the loop to avoid the overhead. I will confirm it with a manual testing.

This hypothesis wasn't true, it was really about the chunk size.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/glance_store/+/799870

[Vladimir Grevtsev (vlgrevtsev)]

I'm facing the same issue and can confirm the suggested fix improves the upload speed dramatically (without it, I had ~3.5min for 200mb image, now the same time is consumed for 10GB image upload).

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/glance_store/+/799870
Committed: https://opendev.org/openstack/glance_store/commit/32f9a1509bc94baa6acaf508c922f3b7edf5b65f
Submitter: "Zuul (22348)"
Branch: master

commit 32f9a1509bc94baa6acaf508c922f3b7edf5b65f
Author: Nobuto Murata <email address hidden>
Date: Thu Jul 8 01:36:38 2021 +0900

    s3: Optimize WRITE_CHUNKSIZE to minimize an overhead

    When processing the image data from a client, the previous and small
    chunk size had a significant overhead. Let's be aligned with other chunk
    size related parameters such as DEFAULT_LARGE_OBJECT_MIN_CHUNK_SIZE
    (=5MiB) and DEFAULT_LARGE_OBJECT_CHUNK_SIZE (=10MiB).

    The performance difference should be tangible especially with a
    singlepart upload. And the upload time can be shortened for example as
    follows:

    100 MiB: 36.5s -> 4.0s
    200 MiB: 2m16.8s -> 6.1s
    300 MiB: 4m50.4s -> 9.1s

    Closes-Bug: 1934849
    Change-Id: Icecac80dd9e4e9f7ffa76bb7ca63d8d112036b70

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/glance_store/+/800183

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/glance_store/+/800184

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/glance_store/+/800185

[Nobuto Murata (nobuto)]

Subscribing Canonical's ~field-high to initiate the Ubuntu package's SRU process in a timely manner.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance_store 2.6.0 release.

[Corey Bryant (corey.bryant)]

Uploaded to hirsute unapproved queue, focal unapproved queue, and victoria-staging PPA:

https://launchpad.net/ubuntu/hirsute/+queue?queue_state=1&queue_text=python-glance-store
https://launchpad.net/ubuntu/focal/+queue?queue_state=1&queue_text=python-glance-store
https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/victoria-staging/+packages?field.name_filter=glance-store&field.status_filter=published&field.series_filter=

[Nobuto Murata (nobuto)]

My update in the bug description was somehow rolled back (by me in the record), trying again.

[Robie Basak (racb)]

Hello Nobuto, or anyone else affected,

Accepted python-glance-store into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-glance-store/2.5.0-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Robie Basak (racb)]

Hello Nobuto, or anyone else affected,

Accepted python-glance-store into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-glance-store/2.0.0-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Corey Bryant (corey.bryant)]

Hello Nobuto, or anyone else affected,

Accepted python-glance-store into victoria-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:victoria-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-victoria-needed to verification-victoria-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-victoria-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Nobuto, or anyone else affected,

Accepted python-glance-store into wallaby-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:wallaby-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-wallaby-needed to verification-wallaby-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-wallaby-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Nobuto, or anyone else affected,

Accepted python-glance-store into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Nobuto Murata (nobuto)]

[hirsute]

All of the uploads succeeded. And -proposed shortened time for the larger sizes.

$ sudo apt-get install python3-glance-store/hirsute-proposed
$ sudo systemctl restart glance-api

$ apt-cache policy python3-glance-store
python3-glance-store:
  Installed: 2.5.0-0ubuntu2
  Candidate: 2.5.0-0ubuntu2
  Version table:
 *** 2.5.0-0ubuntu2 400
        400 http://archive.ubuntu.com/ubuntu hirsute-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2.5.0-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu hirsute/main amd64 Packages

size ceph swift s3 s3 -proposed
0MiB 0m2.495s 0m2.394s 0m2.262s 0m2.352s
3MiB 0m2.721s 0m2.270s 0m2.144s 0m2.430s
5MiB 0m2.726s 0m2.314s 0m2.200s 0m2.177s
9MiB 0m2.737s 0m2.377s 0m2.664s 0m2.238s
10MiB 0m2.772s 0m2.350s 0m2.756s 0m2.160s
128MiB 0m4.261s 0m3.381s 0m57.001s 0m5.542s
512MiB 0m11.619s 0m11.734s 14m13.448s 0m24.012s
1024MiB 0m23.787s 0m20.756s 56m37.906s 1m3.716s

[Nobuto Murata (nobuto)]

[focal]

All of the uploads succeeded. And -proposed shortened time for the larger sizes.

$ sudo apt-get install python3-glance-store/focal-proposed
$ sudo systemctl restart glance-api

$ apt-cache policy python3-glance-store
python3-glance-store:
  Installed: 2.0.0-0ubuntu2
  Candidate: 2.0.0-0ubuntu2
  Version table:
 *** 2.0.0-0ubuntu2 400
        400 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2.0.0-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

size ceph swift s3 s3 -proposed
0MiB 0m3.571s 0m2.249s 0m2.185s 0m1.959s
3MiB 0m2.724s 0m2.285s 0m1.986s 0m2.118s
5MiB 0m2.717s 0m2.694s 0m2.093s 0m2.213s
9MiB 0m2.749s 0m2.342s 0m2.357s 0m2.107s
10MiB 0m3.415s 0m2.342s 0m2.289s 0m2.139s
128MiB 0m4.644s 0m3.109s 0m54.343s 0m4.287s
512MiB 0m11.121s 0m6.246s 13m35.948s 0m16.853s
1024MiB 0m16.292s 0m11.441s 54m24.339s 0m52.231s

[Nobuto Murata (nobuto)]

[focal-wallaby]

All of the uploads succeeded. And -proposed shortened time for the larger sizes.

$ sudo apt-get install python3-glance-store/focal-proposed
$ sudo systemctl restart glance-api

$ apt-cache policy python3-glance-store
python3-glance-store:
  Installed: 2.5.0-0ubuntu2~cloud0
  Candidate: 2.5.0-0ubuntu2~cloud0
  Version table:
 *** 2.5.0-0ubuntu2~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-proposed/wallaby/main amd64 Packages
        100 /var/lib/dpkg/status
     2.5.0-0ubuntu1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/wallaby/main amd64 Packages
     2.0.0-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

size ceph swift s3 s3 -proposed
0MiB 0m3.389s 0m2.876s 0m2.389s 0m4.459s
3MiB 0m2.788s 0m2.278s 0m2.076s 0m3.138s
5MiB 0m2.716s 0m2.337s 0m2.122s 0m3.246s
9MiB 0m3.506s 0m2.396s 0m2.274s 0m2.782s
10MiB 0m2.859s 0m2.366s 0m2.324s 0m2.829s
128MiB 0m4.514s 0m3.306s 0m55.145s 0m4.868s
512MiB 0m10.862s 0m9.692s 13m31.848s 0m21.378s
1024MiB 0m21.965s 0m27.575s 54m27.784s 1m4.299s

[Nobuto Murata (nobuto)]

@Corey,

Somehow the binary package for cloud-archive:victoria-proposed is not published yet. Can you please double-check the build status of the package? I just don't know where to look.

cloud1 in the source vs cloud0 in the binary.

$ curl -s http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/focal-proposed/victoria/main/source/Sources.gz | zcat | grep -A2 'Package: python-glance-store'
Package: python-glance-store
Binary: python-glance-store-doc, python3-glance-store
Version: 2.3.0-0ubuntu1~cloud1

$ curl -s http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/focal-proposed/victoria/main/binary-amd64/Packages | grep -A7 'Package: python3-glance-store'
Package: python3-glance-store
Source: python-glance-store
Priority: optional
Section: python
Installed-Size: 983
Maintainer: Ubuntu Developers <email address hidden>
Architecture: all
Version: 2.3.0-0ubuntu1~cloud0

[Nobuto Murata (nobuto)]

Just for the record, this is the current status with focal-victoria. No diff between -updates and -proposed.

$ apt-cache policy python3-glance-store
python3-glance-store:
  Installed: 2.3.0-0ubuntu1~cloud0
  Candidate: 2.3.0-0ubuntu1~cloud0
  Version table:
 *** 2.3.0-0ubuntu1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/victoria/main amd64 Packages
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-proposed/victoria/main amd64 Packages
        100 /var/lib/dpkg/status
     2.0.0-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

[Nobuto Murata (nobuto)]

[bionic-ussuri]

All of the uploads succeeded. And -proposed shortened time for the larger sizes.

$ sudo apt-get install python3-glance-store/bionic-proposed
$ sudo systemctl restart glance-api

$ apt-cache policy python3-glance-store
python3-glance-store:
  Installed: 2.0.0-0ubuntu2~cloud0
  Candidate: 2.0.0-0ubuntu2~cloud0
  Version table:
 *** 2.0.0-0ubuntu2~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/ussuri/main amd64 Packages
        100 /var/lib/dpkg/status
     2.0.0-0ubuntu1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/ussuri/main amd64 Packages
     0.23.0-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages

size ceph swift s3 s3 -proposed
0MiB 0m2.985s 0m2.832s 0m2.487s 0m2.293s
3MiB 0m2.761s 0m2.175s 0m1.970s 0m2.367s
5MiB 0m2.729s 0m2.224s 0m2.031s 0m1.981s
9MiB 0m2.783s 0m2.265s 0m2.168s 0m2.087s
10MiB 0m2.824s 0m2.256s 0m2.122s 0m2.037s
128MiB 0m4.703s 0m2.966s 0m47.824s 0m3.869s
512MiB 0m8.979s 0m5.110s 12m27.668s 0m16.269s
1024MiB 0m15.121s 0m15.742s 50m56.406s 0m48.282s

[Nobuto Murata (nobuto)]

[focal-victoria]

All of the uploads succeeded. And -proposed shortened time for the larger sizes.

$ sudo apt-get install python3-glance-store/focal-proposed
$ sudo systemctl restart glance-api

$ apt-cache policy python3-glance-store
python3-glance-store:
  Installed: 2.3.0-0ubuntu1~cloud1
  Candidate: 2.3.0-0ubuntu1~cloud1
  Version table:
 *** 2.3.0-0ubuntu1~cloud1 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-proposed/victoria/main amd64 Packages
        100 /var/lib/dpkg/status
     2.3.0-0ubuntu1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/victoria/main amd64 Packages
     2.0.0-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

size ceph swift s3 s3 -proposed
0MiB 0m10.780s 0m2.762s 0m3.819s 0m3.235s
3MiB 0m3.334s 0m2.541s 0m3.019s 0m2.389s
5MiB 0m2.731s 0m2.244s 0m2.687s 0m2.417s
9MiB 0m2.725s 0m2.334s 0m2.243s 0m2.069s
10MiB 0m2.752s 0m2.315s 0m3.055s 0m2.120s
128MiB 0m4.404s 0m3.604s 0m56.566s 0m4.728s
512MiB 0m8.411s 0m6.021s 13m45.246s 0m17.453s
1024MiB 0m14.707s 0m11.430s 54m22.978s 0m55.937s

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-glance-store - 2.5.0-0ubuntu2

---------------
python-glance-store (2.5.0-0ubuntu2) hirsute; urgency=medium

  * d/gbp.conf: Create stable/wallaby branch.
  * d/p/s3-optimize-write-chunksize.patch: Cherry-picked from upstream
    to fix S3 write performance (LP: #1934849).

 -- Corey Bryant <email address hidden> Mon, 19 Jul 2021 14:07:21 -0400

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for python-glance-store has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-glance-store - 2.0.0-0ubuntu2

---------------
python-glance-store (2.0.0-0ubuntu2) focal; urgency=medium

  [ Corey Bryant ]
  * d/gbp.conf: Create stable/ussuri branch.

  [ Chris MacNaughton ]
  * d/control: Update VCS paths for move to lp:~ubuntu-openstack-dev.
  * d/p/s3-optimize-write-chunksize.patch: Cherry-picked from upstream
    to fix S3 write performance (LP: #1934849).

 -- Corey Bryant <email address hidden> Mon, 19 Jul 2021 14:30:58 -0400

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for python-glance-store has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package python-glance-store - 2.5.0-0ubuntu2~cloud0
---------------

 python-glance-store (2.5.0-0ubuntu2~cloud0) focal-wallaby; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 python-glance-store (2.5.0-0ubuntu2) hirsute; urgency=medium
 .
   * d/gbp.conf: Create stable/wallaby branch.
   * d/p/s3-optimize-write-chunksize.patch: Cherry-picked from upstream
     to fix S3 write performance (LP: #1934849).

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for python-glance-store has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package python-glance-store - 2.3.0-0ubuntu1~cloud1
---------------

 python-glance-store (2.3.0-0ubuntu1~cloud1) focal-victoria; urgency=medium
 .
   [ Chris MacNaughton ]
   * d/control: Update VCS paths for move to lp:~ubuntu-openstack-dev.
 .
   [ Corey Bryant ]
   * d/gbp.conf: Create stable/victoria branch.
   * d/p/s3-optimize-write-chunksize.patch: Cherry-picked from upstream
     to fix S3 write performance (LP: #1934849).

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for python-glance-store has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package python-glance-store - 2.0.0-0ubuntu2~cloud0
---------------

 python-glance-store (2.0.0-0ubuntu2~cloud0) bionic-ussuri; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 python-glance-store (2.0.0-0ubuntu2) focal; urgency=medium
 .
   [ Corey Bryant ]
   * d/gbp.conf: Create stable/ussuri branch.
 .
   [ Chris MacNaughton ]
   * d/control: Update VCS paths for move to lp:~ubuntu-openstack-dev.
   * d/p/s3-optimize-write-chunksize.patch: Cherry-picked from upstream
     to fix S3 write performance (LP: #1934849).

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Abhishek Kekane <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/glance_store/+/800184
Reason: No update since last commit

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Abhishek Kekane <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/glance_store/+/800183
Reason: This does not fits with upstream stable branch policy (current branch - 2), feel free to reach back to glance team on #openstack-glance irc channel

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Abhishek Kekane <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/glance_store/+/800185
Reason: This does not fits with upstream stable branch policy (current branch - 2), feel free to reach back to glance team on #openstack-glance irc channel