Actually that's only used for images over 5GB



On Wed, Jul 7, 2021 at 1:59 PM James Page <email address hidden> wrote:

> I think the swift driver uses 200M:
>
> DEFAULT_LARGE_OBJECT_CHUNK_SIZE = 200  # 200M
>
> On Wed, Jul 7, 2021 at 1:50 PM Nobuto Murata <email address hidden>
> wrote:
>
>> And by using "4 * units.Mi" it can be 20s.
>>
>> --
>> You received this bug notification because you are a member of Ubuntu
>> OpenStack, which is subscribed to python-glance-store in Ubuntu.
>> https://bugs.launchpad.net/bugs/1934849
>>
>> Title:
>>   s3 backend takes time exponentially
>>
>> Status in glance_store:
>>   New
>> Status in python-glance-store package in Ubuntu:
>>   New
>>
>> Bug description:
>>   I have a test Ceph cluster as an object storage with both Swift and S3
>>   protocols enabled for Glance (Ussuri). When I use Swift backend with
>>   Glance, an image upload completes quickly enough. But with S3 backend
>>   Glance, it takes much more time to upload an image and it seems to
>>   rise exponentially.
>>
>>   It's worth noting that when uploading an image with S3 backend, a
>>   single core is consumed 100% by glance-api process.
>>
>>   for backend in swift s3; do
>>       for i in {8,16,32,64,128,512}; do
>>           dd if=/dev/zero of=my-image.img bs=1M count=$i oflag=sync
>>           time glance image-create \
>>               --store $backend \
>>               --file my-image.img --name my-image \
>>               --disk-format raw --container-format bare \
>>               --progress
>>       done
>>   done
>>
>>   [swift]
>>   8MB   -  2.4s
>>   16MB  -  2.8s
>>   32MB  -  2.6s
>>   64MB  -  2.7s
>>   128MB -  3.1s
>>   ...
>>   512MB -  5.9s
>>
>>   [s3]
>>   8MB   -  2.2s
>>   16MB  -  2.9s
>>   32MB  -  5.5s
>>   64MB  - 16.3s
>>   128MB - 54.9s
>>   ...
>>   512MB - 14m26s
>>
>>   Btw, downloading of 512MB image with S3 backend can complete with less
>>   than 10 seconds.
>>
>>   $ time openstack image save --file downloaded.img
>> 917c5424-4350-4bc5-98ca-66d40e101843
>>   real    0m5.673s
>>
>>   $ du -h downloaded.img
>>   512M    downloaded.img
>>
>>
>>   [/etc/glance/glance-api.conf]
>>
>>   enabled_backends = local:file, ceph:rbd, swift:swift, s3:s3
>>
>>   [swift]
>>   auth_version = 3
>>   auth_address = http://192.168.151.131:5000/v3
>>   ...
>>   container = glance
>>   large_object_size = 5120
>>   large_object_chunk_size = 200
>>
>>
>>   [s3]
>>   s3_store_host = http://192.168.151.137:80/
>>   ...
>>   s3_store_bucket = zaza-glance-s3-test
>>   s3_store_large_object_size = 5120
>>   s3_store_large_object_chunk_size = 200
>>
>>   ProblemType: Bug
>>   DistroRelease: Ubuntu 20.04
>>   Package: python3-glance-store 2.0.0-0ubuntu1
>>   ProcVersionSignature: Ubuntu 5.4.0-77.86-generic 5.4.119
>>   Uname: Linux 5.4.0-77-generic x86_64
>>   NonfreeKernelModules: bluetooth ecdh_generic ecc tcp_diag inet_diag
>> binfmt_misc veth zfs zunicode zlua zavl icp zcommon znvpair spl unix_diag
>> nft_masq nft_chain_nat bridge stp llc vhost_vsock
>> vmw_vsock_virtio_transport_common vhost vsock ebtable_filter ebtables
>> ip6table_raw ip6table_mangle ip6table_nat ip6table_filter ip6_tables
>> iptable_raw iptable_mangle iptable_nat nf_nat nf_conntrack nf_defrag_ipv6
>> nf_defrag_ipv4 iptable_filter bpfilter nf_tables nfnetlink dm_multipath
>> scsi_dh_rdac scsi_dh_emc scsi_dh_alua kvm_amd ccp input_leds kvm joydev
>> mac_hid serio_raw qemu_fw_cfg sch_fq_codel ip_tables x_tables autofs4 btrfs
>> zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq
>> async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear
>> crct10dif_pclmul crc32_pclmul cirrus ghash_clmulni_intel drm_kms_helper
>> virtio_net syscopyarea aesni_intel sysfillrect sysimgblt fb_sys_fops
>> crypto_simd cryptd drm virtio_blk glue_helper net_failover psmouse failover
>> floppy i2c_piix4 pata_acpi
>>   ApportVersion: 2.20.11-0ubuntu27.18
>>   Architecture: amd64
>>   CasperMD5CheckResult: skip
>>   Date: Wed Jul  7 04:46:05 2021
>>   PackageArchitecture: all
>>   ProcEnviron:
>>    TERM=screen-256color
>>    PATH=(custom, no user)
>>    LANG=C.UTF-8
>>    SHELL=/bin/bash
>>   SourcePackage: python-glance-store
>>   UpgradeStatus: No upgrade log present (probably fresh install)
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/glance-store/+bug/1934849/+subscriptions
>>
>> --
>> Ubuntu-openstack-bugs mailing list
>> <email address hidden>
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/ubuntu-openstack-bugs
>>
>