Glance

utils.chunkreadable is not returning the correct chunk

Bug #1360151 reported by Masashi Ozawa
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Glance
Incomplete
Undecided
Unassigned

Bug Description

During the testing with the master glance with S3, we found that "utils.chunkreadable" is not returning the correct chunk if the image location URL is specified.

Because "Implemented S3 multi-part upload functionality" (https://review.openstack.org/#/c/49316/) is merged into master, if the image to be created is specified as URL like below and its size is greater than 100MB(as default), it will always fail.

$ glance image-create --name="Ubuntu1404 Cloud" --is-public=true --container-format=ovf --disk-format=qcow2 --copy-from https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img

Here is an excerpt from api.log.

2014-08-22 08:45:59.601 13656 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.184.145.84
2014-08-22 08:46:01.188 13656 INFO glance.api.v1.images [-] Triggering asynchronous copy from external source
2014-08-22 08:46:01.192 13656 INFO glance.wsgi.server [-] 10.184.145.84 - - [22/Aug/2014 08:46:01] "POST /v1/images HTTP/1.1" 201 722 1.592873
2014-08-22 08:46:03.257 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=1, size=65536, key=5f35132b-18e0-47a9-8d59-0370afd99230, UploadId=MmI0ZDY2NTQ5MTQwODY5NzE2MjI2Mw
2014-08-22 08:46:03.765 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=2, size=65536, key=5f35132b-18e0-47a9-8d59-0370afd99230, UploadId=MmI0ZDY2NTQ5MTQwODY5NzE2MjI2Mw
2014-08-22 08:46:04.273 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=3, size=65536, key=5f35132b-18e0-47a9-8d59-0370afd99230, UploadId=MmI0ZDY2NTQ5MTQwODY5NzE2MjI2Mw
2014-08-22 08:46:04.528 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=4, size=65536, key=5f35132b-18e0-47a9-8d59-0370afd99230, UploadId=MmI0ZDY2NTQ5MTQwODY5NzE2MjI2Mw
2014-08-22 08:46:05.036 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=5, size=65536, key=5f35132b-18e0-47a9-8d59-0370afd99230, UploadId=MmI0ZDY2NTQ5MTQwODY5NzE2MjI2Mw
2014-08-22 08:46:05.291 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=6, size=65536, key=5f35132b-18e0-47a9-8d59-0370afd99230, UploadId=MmI0ZDY2NTQ5MTQwODY5NzE2MjI2Mw
2014-08-22 08:46:05.545 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=7, size=65536, key=5f35132b-18e0-47a9-8d59-0370afd99230, UploadId=MmI0ZDY2NTQ5MTQwODY5NzE2MjI2Mw
2014-08-22 08:46:05.799 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=8, size=65536, key=5f35132b-18e0-47a9-8d59-0370afd99230, UploadId=MmI0ZDY2NTQ5MTQwODY5NzE2MjI2Mw
2014-08-22 08:46:05.802 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=9, size=65536, key=5f35132b-18e0-47a9-8d59-0370afd99230, UploadId=MmI0ZDY2NTQ5MTQwODY5NzE2MjI2Mw
2014-08-22 08:46:06.054 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=10, size=65536, key=5f35132b-18e0-47a9-8d59-0370afd99230, UploadId=MmI0ZDY2NTQ5MTQwODY5NzE2MjI2Mw
2014-08-22 08:46:06.308 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=11, size=65536, key=5f35132b-18e0-47a9-8d59-0370afd99230, UploadId=MmI0ZDY2NTQ5MTQwODY5NzE2MjI2Mw
2014-08-22 08:46:06.319 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=12, size=65536, key=5f35132b-18e0-47a9-8d59-0370afd99230, UploadId=MmI0ZDY2NTQ5MTQwODY5NzE2MjI2Mw
2014-08-22 08:46:06.563 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=13, size=65536, key=5f35132b-18e0-47a9-8d59-0370afd99230, UploadId=MmI0ZDY2NTQ5MTQwODY5NzE2MjI2Mw
2014-08-22 08:46:06.573 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=14, size=65536, key=5f35132b-18e0-47a9-8d59-0370afd99230, UploadId=MmI0ZDY2NTQ5MTQwODY5NzE2MjI2Mw
2014-08-22 08:46:06.817 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=15, size=65536, key=5f35132b-18e0-47a9-8d59-0370afd99230, UploadId=MmI0ZDY2NTQ5MTQwODY5NzE2MjI2Mw
2014-08-22 08:46:06.827 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=16, size=65536, key=5f35132b-18e0-47a9-8d59-0370afd99230, UploadId=MmI0ZDY2NTQ5MTQwODY5NzE2MjI2Mw
2014-08-22 08:46:07.072 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=17, size=65536, key=5f35132b-18e0-47a9-8d59-0370afd99230, UploadId=MmI0ZDY2NTQ5MTQwODY5NzE2MjI2Mw
2014-08-22 08:46:07.082 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=18, size=65536, key=5f35132b-18e0-47a9-8d59-0370afd99230, UploadId=MmI0ZDY2NTQ5MTQwODY5NzE2MjI2Mw
2014-08-22 08:46:07.326 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=19, size=65536, key=5f35132b-18e0-47a9-8d59-0370afd99230, UploadId=MmI0ZDY2NTQ5MTQwODY5NzE2MjI2Mw
2014-08-22 08:46:07.335 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=20, size=65536, key=5f35132b-18e0-47a9-8d59-0370afd99230, UploadId=MmI0ZDY2NTQ5MTQwODY5NzE2MjI2Mw
2014-08-22 08:46:07.364 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=21, size=65536, key=5f35132b-18e0-47a9-8d59-0370afd99230, UploadId=MmI0ZDY2NTQ5MTQwODY5NzE2MjI2Mw
2014-08-22 08:46:07.588 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=22, size=65536, key=5f35132b-18e0-47a9-8d59-0370afd99230, UploadId=MmI0ZDY2NTQ5MTQwODY5NzE2MjI2Mw
2014-08-22 08:46:07.590 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=23, size=65536, key=5f35132b-18e0-47a9-8d59-0370afd99230, UploadId=MmI0ZDY2NTQ5MTQwODY5NzE2MjI2Mw
2014-08-22 08:46:07.618 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=24, size=65536, key=5f35132b-18e0-47a9-8d59-0370afd99230, UploadId=MmI0ZDY2NTQ5MTQwODY5NzE2MjI2Mw
2014-08-22 08:46:07.842 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=25, size=65536, key=5f35132b-18e0-47a9-8d59-0370afd99230, UploadId=MmI0ZDY2NTQ5MTQwODY5NzE2MjI2Mw
2014-08-22 08:46:07.918 13656 INFO glance.store.s3 [-] Multipart complete key=5f35132b-18e0-47a9-8d59-0370afd99230 UploadId=MmI0ZDY2NTQ5MTQwODY5NzE2MjI2Mw Wrote 1638400 bytes to S3 keynamed 5f35132b-18e0-47a9-8d59-0370afd99230 with checksum ce5344ca799c564cc356d6dd05d09059
2014-08-22 08:46:07.919 13656 ERROR glance.api.v1.upload_utils [-] Supplied size (255066624) and size generated from uploaded image (1638400) did not match. Setting image status to 'killed'.
2014-08-22 08:46:08.184 13656 ERROR glance.api.v1.upload_utils [-] Received HTTP error while uploading image 5f35132b-18e0-47a9-8d59-0370afd99230
2014-08-22 08:46:08.184 13656 TRACE glance.api.v1.upload_utils Traceback (most recent call last):
2014-08-22 08:46:08.184 13656 TRACE glance.api.v1.upload_utils File "/opt/stack/glance/glance/api/v1/upload_utils.py", line 141, in upload_data_to_store
2014-08-22 08:46:08.184 13656 TRACE glance.api.v1.upload_utils _kill_mismatched(image_meta, 'size', size)
2014-08-22 08:46:08.184 13656 TRACE glance.api.v1.upload_utils File "/opt/stack/glance/glance/api/v1/upload_utils.py", line 137, in _kill_mismatched
2014-08-22 08:46:08.184 13656 TRACE glance.api.v1.upload_utils request=req)
2014-08-22 08:46:08.184 13656 TRACE glance.api.v1.upload_utils HTTPBadRequest: Supplied size (255066624) and size generated from uploaded image (1638400) did not match. Setting image status to 'killed'.
2014-08-22 08:46:08.184 13656 TRACE glance.api.v1.upload_utils
2014-08-22 08:46:08.219 13656 INFO glance.registry.client.v1.client [-] Registry client request PUT /images/5f35132b-18e0-47a9-8d59-0370afd99230 raised Duplicate
2014-08-22 08:46:08.219 13656 ERROR glance.api.v1.upload_utils [-] Unable to kill image 5f35132b-18e0-47a9-8d59-0370afd99230:
2014-08-22 08:46:08.219 13656 TRACE glance.api.v1.upload_utils Traceback (most recent call last):
2014-08-22 08:46:08.219 13656 TRACE glance.api.v1.upload_utils File "/opt/stack/glance/glance/api/v1/upload_utils.py", line 76, in safe_kill
2014-08-22 08:46:08.219 13656 TRACE glance.api.v1.upload_utils _kill(req, image_id, from_state)
2014-08-22 08:46:08.219 13656 TRACE glance.api.v1.upload_utils File "/opt/stack/glance/glance/api/v1/upload_utils.py", line 61, in _kill
2014-08-22 08:46:08.219 13656 TRACE glance.api.v1.upload_utils from_state=from_state)
2014-08-22 08:46:08.219 13656 TRACE glance.api.v1.upload_utils File "/opt/stack/glance/glance/registry/client/v1/api.py", line 169, in update_image_metadata
2014-08-22 08:46:08.219 13656 TRACE glance.api.v1.upload_utils from_state=from_state)
2014-08-22 08:46:08.219 13656 TRACE glance.api.v1.upload_utils File "/opt/stack/glance/glance/registry/client/v1/client.py", line 198, in update_image
2014-08-22 08:46:08.219 13656 TRACE glance.api.v1.upload_utils headers=headers)
2014-08-22 08:46:08.219 13656 TRACE glance.api.v1.upload_utils File "/opt/stack/glance/glance/registry/client/v1/client.py", line 130, in do_request
2014-08-22 08:46:08.219 13656 TRACE glance.api.v1.upload_utils 'exc_name': exc_name})
2014-08-22 08:46:08.219 13656 TRACE glance.api.v1.upload_utils File "/opt/stack/glance/glance/openstack/common/excutils.py", line 82, in __exit__
2014-08-22 08:46:08.219 13656 TRACE glance.api.v1.upload_utils six.reraise(self.type_, self.value, self.tb)
2014-08-22 08:46:08.219 13656 TRACE glance.api.v1.upload_utils File "/opt/stack/glance/glance/registry/client/v1/client.py", line 115, in do_request
2014-08-22 08:46:08.219 13656 TRACE glance.api.v1.upload_utils **kwargs)
2014-08-22 08:46:08.219 13656 TRACE glance.api.v1.upload_utils File "/opt/stack/glance/glance/common/client.py", line 68, in wrapped
2014-08-22 08:46:08.219 13656 TRACE glance.api.v1.upload_utils return func(self, *args, **kwargs)
2014-08-22 08:46:08.219 13656 TRACE glance.api.v1.upload_utils File "/opt/stack/glance/glance/common/client.py", line 388, in do_request
2014-08-22 08:46:08.219 13656 TRACE glance.api.v1.upload_utils headers=copy.deepcopy(headers))
2014-08-22 08:46:08.219 13656 TRACE glance.api.v1.upload_utils File "/opt/stack/glance/glance/common/client.py", line 85, in wrapped
2014-08-22 08:46:08.219 13656 TRACE glance.api.v1.upload_utils return func(self, method, url, body, headers)
2014-08-22 08:46:08.219 13656 TRACE glance.api.v1.upload_utils File "/opt/stack/glance/glance/common/client.py", line 537, in _do_request
2014-08-22 08:46:08.219 13656 TRACE glance.api.v1.upload_utils raise exception.Duplicate(res.read())
2014-08-22 08:46:08.219 13656 TRACE glance.api.v1.upload_utils Duplicate: Image operation conflicts
2014-08-22 08:46:08.219 13656 TRACE glance.api.v1.upload_utils

If the image(same size) is specified the local file like below, it works fine.

$ ls -l trusty-server-cloudimg-amd64-disk1.img
-rw-rw-r-- 1 ubuntu ubuntu 255066624 Aug 20 04:44 trusty-server-cloudimg-amd64-disk1.img
$

$ glance image-create --name="Ubuntu1404 Local" --is-public=true --container-format=ovf --disk-format=qcow2 < ./trusty-server-cloudimg-amd64-disk1.img

2014-08-22 08:45:22.653 13656 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.184.145.84
2014-08-22 08:45:22.870 13656 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.184.145.84
2014-08-22 08:45:23.443 13656 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.184.145.84:5672
2014-08-22 08:45:23.462 13656 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.184.145.84:5672
2014-08-22 08:45:23.907 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=1, size=10485760, key=e6b01d04-82fe-46b2-86ea-3e9e66a5123b, UploadId=NGU5NDY2NTgyMTQwODY5NzEyMzU3OA
2014-08-22 08:45:24.087 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=2, size=10485760, key=e6b01d04-82fe-46b2-86ea-3e9e66a5123b, UploadId=NGU5NDY2NTgyMTQwODY5NzEyMzU3OA
2014-08-22 08:45:24.261 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=3, size=10485760, key=e6b01d04-82fe-46b2-86ea-3e9e66a5123b, UploadId=NGU5NDY2NTgyMTQwODY5NzEyMzU3OA
2014-08-22 08:45:24.464 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=4, size=10485760, key=e6b01d04-82fe-46b2-86ea-3e9e66a5123b, UploadId=NGU5NDY2NTgyMTQwODY5NzEyMzU3OA
2014-08-22 08:45:24.645 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=5, size=10485760, key=e6b01d04-82fe-46b2-86ea-3e9e66a5123b, UploadId=NGU5NDY2NTgyMTQwODY5NzEyMzU3OA
2014-08-22 08:45:24.849 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=6, size=10485760, key=e6b01d04-82fe-46b2-86ea-3e9e66a5123b, UploadId=NGU5NDY2NTgyMTQwODY5NzEyMzU3OA
2014-08-22 08:45:25.050 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=7, size=10485760, key=e6b01d04-82fe-46b2-86ea-3e9e66a5123b, UploadId=NGU5NDY2NTgyMTQwODY5NzEyMzU3OA
2014-08-22 08:45:25.246 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=8, size=10485760, key=e6b01d04-82fe-46b2-86ea-3e9e66a5123b, UploadId=NGU5NDY2NTgyMTQwODY5NzEyMzU3OA
2014-08-22 08:45:25.446 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=9, size=10485760, key=e6b01d04-82fe-46b2-86ea-3e9e66a5123b, UploadId=NGU5NDY2NTgyMTQwODY5NzEyMzU3OA
2014-08-22 08:45:25.648 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=10, size=10485760, key=e6b01d04-82fe-46b2-86ea-3e9e66a5123b, UploadId=NGU5NDY2NTgyMTQwODY5NzEyMzU3OA
2014-08-22 08:45:27.172 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=11, size=10485760, key=e6b01d04-82fe-46b2-86ea-3e9e66a5123b, UploadId=NGU5NDY2NTgyMTQwODY5NzEyMzU3OA
2014-08-22 08:45:27.329 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=12, size=10485760, key=e6b01d04-82fe-46b2-86ea-3e9e66a5123b, UploadId=NGU5NDY2NTgyMTQwODY5NzEyMzU3OA
2014-08-22 08:45:27.532 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=13, size=10485760, key=e6b01d04-82fe-46b2-86ea-3e9e66a5123b, UploadId=NGU5NDY2NTgyMTQwODY5NzEyMzU3OA
2014-08-22 08:45:27.666 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=14, size=10485760, key=e6b01d04-82fe-46b2-86ea-3e9e66a5123b, UploadId=NGU5NDY2NTgyMTQwODY5NzEyMzU3OA
2014-08-22 08:45:27.931 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=15, size=10485760, key=e6b01d04-82fe-46b2-86ea-3e9e66a5123b, UploadId=NGU5NDY2NTgyMTQwODY5NzEyMzU3OA
2014-08-22 08:45:28.139 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=16, size=10485760, key=e6b01d04-82fe-46b2-86ea-3e9e66a5123b, UploadId=NGU5NDY2NTgyMTQwODY5NzEyMzU3OA
2014-08-22 08:45:28.330 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=17, size=10485760, key=e6b01d04-82fe-46b2-86ea-3e9e66a5123b, UploadId=NGU5NDY2NTgyMTQwODY5NzEyMzU3OA
2014-08-22 08:45:28.533 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=18, size=10485760, key=e6b01d04-82fe-46b2-86ea-3e9e66a5123b, UploadId=NGU5NDY2NTgyMTQwODY5NzEyMzU3OA
2014-08-22 08:45:28.731 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=19, size=10485760, key=e6b01d04-82fe-46b2-86ea-3e9e66a5123b, UploadId=NGU5NDY2NTgyMTQwODY5NzEyMzU3OA
2014-08-22 08:45:28.950 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=20, size=10485760, key=e6b01d04-82fe-46b2-86ea-3e9e66a5123b, UploadId=NGU5NDY2NTgyMTQwODY5NzEyMzU3OA
2014-08-22 08:45:29.741 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=21, size=10485760, key=e6b01d04-82fe-46b2-86ea-3e9e66a5123b, UploadId=NGU5NDY2NTgyMTQwODY5NzEyMzU3OA
2014-08-22 08:45:29.958 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=22, size=10485760, key=e6b01d04-82fe-46b2-86ea-3e9e66a5123b, UploadId=NGU5NDY2NTgyMTQwODY5NzEyMzU3OA
2014-08-22 08:45:30.153 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=23, size=10485760, key=e6b01d04-82fe-46b2-86ea-3e9e66a5123b, UploadId=NGU5NDY2NTgyMTQwODY5NzEyMzU3OA
2014-08-22 08:45:30.636 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=24, size=10485760, key=e6b01d04-82fe-46b2-86ea-3e9e66a5123b, UploadId=NGU5NDY2NTgyMTQwODY5NzEyMzU3OA
2014-08-22 08:45:30.729 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=25, size=3408384, key=e6b01d04-82fe-46b2-86ea-3e9e66a5123b, UploadId=NGU5NDY2NTgyMTQwODY5NzEyMzU3OA
2014-08-22 08:45:32.064 13656 INFO glance.store.s3 [-] Multipart complete key=e6b01d04-82fe-46b2-86ea-3e9e66a5123b UploadId=NGU5NDY2NTgyMTQwODY5NzEyMzU3OA Wrote 255066624 bytes to S3 keynamed e6b01d04-82fe-46b2-86ea-3e9e66a5123b with checksum 64ed4d0a696f6aae29068e03a606e1b7

Revision history for this message
Masashi Ozawa (mozawa) wrote :

Here is a snippet from s3.py

[s3.py]
:
:
            it = utils.chunkreadable(image_file,
                                     self.s3_store_large_object_chunk_size)

            for p in range(threads):
                chunk = next(it)
                clen = len(chunk)
                checksum.update(chunk)
                fp = six.BytesIO(chunk)
                fp.seek(0)
                part = UploadPart(mpu, fp, cstart + 1, clen)
                pool.spawn_n(run_upload, part)
                plist.append(part)
                cstart += 1

Revision history for this message
Masashi Ozawa (mozawa) wrote :

Tried with Icehouse and it's also incorrect.

$ glance image-create ....--copy-from https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img

- master

  it = utils.chunkreadable(image_file, 10485760)

  "it" will be a chunk of 65536 bytes.

- icehouse

  it = utils.chunkreadable(image_file, 10485760)

  "it" will be a chunk of 16777216 bytes.

Both should be a chunk of 10485760 bytes.

Revision history for this message
Masashi Ozawa (mozawa) wrote :

OK. It's because the fixed chunk size(16M) is used for "--copy-from" case.

[http.py]
class Store(glance.store.base.Store):

    """An implementation of the HTTP(S) Backend Adapter"""

    def get(self, location):
        """
        Takes a `glance.store.location.Location` object that indicates
        where to find the image file, and returns a tuple of generator
        (for reading the image file) and image_size

        :param location `glance.store.location.Location` object, supplied
                        from glance.store.location.get_location_from_uri()
        """
        conn, resp, content_length = self._query(location, 'GET')

        iterator = http_response_iterator(conn, resp, self.CHUNKSIZE)

[base.py]
class Store(object):

    CHUNKSIZE = (16 * 1024 * 1024) # 16M

    def __init__(self, context=None, location=None):

Revision history for this message
Masashi Ozawa (mozawa) wrote :

This issue is not only S3 but also Swift and others.

If I configure 10M to swift_store_large_object_chunk_size and tried to create an image.

For local file specified, the chunk size is 10M(10240000 ) which is the configured size.
For url specified, the chunk size is 16M(16777216 ) which is the hard coded inside the code.

[glance-api.conf]
swift_store_large_object_chunk_size = 10

[local file specified]
glance image-create --name="Ubuntu1404 Local" --is-public=true --container-format=ovf --disk-format=qcow2 < ./trusty-server-cloudimg-amd64-disk1.img

2014-08-28 03:27:20.095 16078 DEBUG glance.store.swift [-] Wrote chunk 96706771-4228-4a25-8ca6-0a05c41c3a9d-00022 (22/25) of length 10240000 to Swift returning MD5 of content: 7ad5204577447e5aab255b02acc26b64 add /opt/stack/glance/glance/store/swift.py:515

[url specified]
glance image-create --name="Ubuntu1404 Cloud" --is-public=true --container-format=ovf --disk-format=qcow2 --copy-from https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img

2014-08-28 03:39:43.306 16078 DEBUG glance.store.swift [-] Wrote chunk 726d28f8-9054-4857-8de0-d005ce8980a8-00002 (2/25) of length 16777216 to Swift returning MD5 of content: aa5d01c638642615f19dd0dcb0696200 add /opt/stack/glance/glance/store/swift.py:515

Revision history for this message
Mark Washenberger (markwash) wrote :

Masashi,

It looks like maybe the recent change to adopt glance_store as a separate project may have included a fix for this. Specifically, I see:

https://github.com/openstack/glance/blob/f6e7992a68f6ab36f87604fffbe5349f2a12b25e/glance/api/v1/images.py#L454

Can you attempt to reproduce the bug with master again?

As a bonus, it might make sense for glance.common.utils.chunkiter to not be so trusting of length of chunks it reads. It could buffer multiple calls to fp.read() until it actually gets the correct size.

Changed in glance:
status: New → Incomplete
importance: Undecided → High
importance: High → Undecided
Revision history for this message
Masashi Ozawa (mozawa) wrote :

Yes, it's still a same behavior and maybe this is by design. So now this bug is confusing. An actual problem that I'm now seeing is that if copy-from is used and s3 is used for storage and its size is big(>100M), the multi-part upload will be used.(I'm now backporting this feature to glance_store in 120714) However it will be always done per the chunk_size which is 64k. That is the problem. Because each part must be at least 5 MB in size in Amazon S3.(http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html) So eventually the operation will fail. I think it's better to file a new BUG for this and close this BUG.

[Implemented S3 multi-part upload functionality]
https://review.openstack.org/#/c/49316/

[Backporting S3 multi-part upload functionality to glace_store]
https://review.openstack.org/#/c/120714/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on glance (master)

Change abandoned by Masashi Ozawa (<email address hidden>) on branch: master
Review: https://review.openstack.org/117411
Reason: This will be handled by https://review.openstack.org/#/c/120714/

Revision history for this message
Zap Chang (zapchang) wrote :

I see similar failure in 2013.2.2 when using swift as backend. If user specifies `--copy-from` option when creating a new image, the uploading procedure to swift fails because the reader always return 16 MB chunks while swiftclient only asks 65536 bytes to write by default. ChunkReader might not be ideal reader in this situation.

Revision history for this message
Zhi Yan Liu (lzy-dev) wrote :

https://review.openstack.org/#/c/120714/ could fix this defect for s3 store.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.