utils.chunkreadable is not returning the correct chunk
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Glance |
Incomplete
|
Undecided
|
Unassigned |
Bug Description
During the testing with the master glance with S3, we found that "utils.
Because "Implemented S3 multi-part upload functionality" (https:/
$ glance image-create --name="Ubuntu1404 Cloud" --is-public=true --container-
Here is an excerpt from api.log.
2014-08-22 08:45:59.601 13656 INFO urllib3.
2014-08-22 08:46:01.188 13656 INFO glance.
2014-08-22 08:46:01.192 13656 INFO glance.wsgi.server [-] 10.184.145.84 - - [22/Aug/2014 08:46:01] "POST /v1/images HTTP/1.1" 201 722 1.592873
2014-08-22 08:46:03.257 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=1, size=65536, key=5f35132b-
2014-08-22 08:46:03.765 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=2, size=65536, key=5f35132b-
2014-08-22 08:46:04.273 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=3, size=65536, key=5f35132b-
2014-08-22 08:46:04.528 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=4, size=65536, key=5f35132b-
2014-08-22 08:46:05.036 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=5, size=65536, key=5f35132b-
2014-08-22 08:46:05.291 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=6, size=65536, key=5f35132b-
2014-08-22 08:46:05.545 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=7, size=65536, key=5f35132b-
2014-08-22 08:46:05.799 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=8, size=65536, key=5f35132b-
2014-08-22 08:46:05.802 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=9, size=65536, key=5f35132b-
2014-08-22 08:46:06.054 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=10, size=65536, key=5f35132b-
2014-08-22 08:46:06.308 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=11, size=65536, key=5f35132b-
2014-08-22 08:46:06.319 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=12, size=65536, key=5f35132b-
2014-08-22 08:46:06.563 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=13, size=65536, key=5f35132b-
2014-08-22 08:46:06.573 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=14, size=65536, key=5f35132b-
2014-08-22 08:46:06.817 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=15, size=65536, key=5f35132b-
2014-08-22 08:46:06.827 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=16, size=65536, key=5f35132b-
2014-08-22 08:46:07.072 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=17, size=65536, key=5f35132b-
2014-08-22 08:46:07.082 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=18, size=65536, key=5f35132b-
2014-08-22 08:46:07.326 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=19, size=65536, key=5f35132b-
2014-08-22 08:46:07.335 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=20, size=65536, key=5f35132b-
2014-08-22 08:46:07.364 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=21, size=65536, key=5f35132b-
2014-08-22 08:46:07.588 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=22, size=65536, key=5f35132b-
2014-08-22 08:46:07.590 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=23, size=65536, key=5f35132b-
2014-08-22 08:46:07.618 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=24, size=65536, key=5f35132b-
2014-08-22 08:46:07.842 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=25, size=65536, key=5f35132b-
2014-08-22 08:46:07.918 13656 INFO glance.store.s3 [-] Multipart complete key=5f35132b-
2014-08-22 08:46:07.919 13656 ERROR glance.
2014-08-22 08:46:08.184 13656 ERROR glance.
2014-08-22 08:46:08.184 13656 TRACE glance.
2014-08-22 08:46:08.184 13656 TRACE glance.
2014-08-22 08:46:08.184 13656 TRACE glance.
2014-08-22 08:46:08.184 13656 TRACE glance.
2014-08-22 08:46:08.184 13656 TRACE glance.
2014-08-22 08:46:08.184 13656 TRACE glance.
2014-08-22 08:46:08.184 13656 TRACE glance.
2014-08-22 08:46:08.219 13656 INFO glance.
2014-08-22 08:46:08.219 13656 ERROR glance.
2014-08-22 08:46:08.219 13656 TRACE glance.
2014-08-22 08:46:08.219 13656 TRACE glance.
2014-08-22 08:46:08.219 13656 TRACE glance.
2014-08-22 08:46:08.219 13656 TRACE glance.
2014-08-22 08:46:08.219 13656 TRACE glance.
2014-08-22 08:46:08.219 13656 TRACE glance.
2014-08-22 08:46:08.219 13656 TRACE glance.
2014-08-22 08:46:08.219 13656 TRACE glance.
2014-08-22 08:46:08.219 13656 TRACE glance.
2014-08-22 08:46:08.219 13656 TRACE glance.
2014-08-22 08:46:08.219 13656 TRACE glance.
2014-08-22 08:46:08.219 13656 TRACE glance.
2014-08-22 08:46:08.219 13656 TRACE glance.
2014-08-22 08:46:08.219 13656 TRACE glance.
2014-08-22 08:46:08.219 13656 TRACE glance.
2014-08-22 08:46:08.219 13656 TRACE glance.
2014-08-22 08:46:08.219 13656 TRACE glance.
2014-08-22 08:46:08.219 13656 TRACE glance.
2014-08-22 08:46:08.219 13656 TRACE glance.
2014-08-22 08:46:08.219 13656 TRACE glance.
2014-08-22 08:46:08.219 13656 TRACE glance.
2014-08-22 08:46:08.219 13656 TRACE glance.
2014-08-22 08:46:08.219 13656 TRACE glance.
2014-08-22 08:46:08.219 13656 TRACE glance.
2014-08-22 08:46:08.219 13656 TRACE glance.
If the image(same size) is specified the local file like below, it works fine.
$ ls -l trusty-
-rw-rw-r-- 1 ubuntu ubuntu 255066624 Aug 20 04:44 trusty-
$
$ glance image-create --name="Ubuntu1404 Local" --is-public=true --container-
2014-08-22 08:45:22.653 13656 INFO urllib3.
2014-08-22 08:45:22.870 13656 INFO urllib3.
2014-08-22 08:45:23.443 13656 INFO oslo.messaging.
2014-08-22 08:45:23.462 13656 INFO oslo.messaging.
2014-08-22 08:45:23.907 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=1, size=10485760, key=e6b01d04-
2014-08-22 08:45:24.087 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=2, size=10485760, key=e6b01d04-
2014-08-22 08:45:24.261 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=3, size=10485760, key=e6b01d04-
2014-08-22 08:45:24.464 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=4, size=10485760, key=e6b01d04-
2014-08-22 08:45:24.645 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=5, size=10485760, key=e6b01d04-
2014-08-22 08:45:24.849 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=6, size=10485760, key=e6b01d04-
2014-08-22 08:45:25.050 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=7, size=10485760, key=e6b01d04-
2014-08-22 08:45:25.246 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=8, size=10485760, key=e6b01d04-
2014-08-22 08:45:25.446 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=9, size=10485760, key=e6b01d04-
2014-08-22 08:45:25.648 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=10, size=10485760, key=e6b01d04-
2014-08-22 08:45:27.172 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=11, size=10485760, key=e6b01d04-
2014-08-22 08:45:27.329 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=12, size=10485760, key=e6b01d04-
2014-08-22 08:45:27.532 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=13, size=10485760, key=e6b01d04-
2014-08-22 08:45:27.666 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=14, size=10485760, key=e6b01d04-
2014-08-22 08:45:27.931 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=15, size=10485760, key=e6b01d04-
2014-08-22 08:45:28.139 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=16, size=10485760, key=e6b01d04-
2014-08-22 08:45:28.330 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=17, size=10485760, key=e6b01d04-
2014-08-22 08:45:28.533 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=18, size=10485760, key=e6b01d04-
2014-08-22 08:45:28.731 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=19, size=10485760, key=e6b01d04-
2014-08-22 08:45:28.950 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=20, size=10485760, key=e6b01d04-
2014-08-22 08:45:29.741 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=21, size=10485760, key=e6b01d04-
2014-08-22 08:45:29.958 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=22, size=10485760, key=e6b01d04-
2014-08-22 08:45:30.153 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=23, size=10485760, key=e6b01d04-
2014-08-22 08:45:30.636 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=24, size=10485760, key=e6b01d04-
2014-08-22 08:45:30.729 13656 INFO glance.store.s3 [-] Uploading upload part in S3 partnum=25, size=3408384, key=e6b01d04-
2014-08-22 08:45:32.064 13656 INFO glance.store.s3 [-] Multipart complete key=e6b01d04-
Here is a snippet from s3.py
[s3.py] able(image_ file,
self. s3_store_ large_object_ chunk_size)
:
:
it = utils.chunkread
for p in range(threads):
chunk = next(it)
clen = len(chunk)
checksum. update( chunk)
fp.seek( 0)
part = UploadPart(mpu, fp, cstart + 1, clen)
pool. spawn_n( run_upload, part)
plist. append( part)
cstart += 1
fp = six.BytesIO(chunk)