a recent bug in the cloud-images service caused the metadata to have an incorrect path, which meant that simplestreams got a 403 when requesting an image. However it swallowed that error and apparently uploaded 384 bytes of something, possibly an error HTML page, as the new image to a user's glance store, when used as part of the glance-simplestreams-sync charm.
Here's an edited bit of log from http://paste.ubuntu.com/12112343/ (this snippet starts at line 2303 in there)
First we see they have a real 20150810 image, size=258343424.
DEBUG * 08-15 06:41:20 [PID:2747] * glanceclient.common.http * curl -i -X GET -H 'X-Auth-Token: fe1ca44f89f24174a9ccf3418fddf9d3' -H 'Content-Type: application/json' -H 'User-Agent: python-glanceclient' http://10.16.100.108:9292/v1/images/detail?limit=20
DEBUG * 08-15 06:41:20 [PID:2747] * glanceclient.common.http *
HTTP/1.1 200 OK
date: Sat, 15 Aug 2015 06:41:20 GMT
content-length: 3746
content-type: application/json; charset=UTF-8
x-openstack-request-id: req-req-1c273480-d084-46d9-b951-64ba5714d605
{"images": [{"status": "active", "deleted_at": null, "name": "auto-sync/ubuntu-trusty-14.04-amd64-server-20150810-disk1.img", "deleted": false, "container_format": "bare", "created_at": "2015-08-12T06:46:58.000000", "disk_format": "qcow2", "updated_at": "2015-08-12T06:47:00.000000", "min_disk": 0, "protected": false, "id": "016346a7-8b29-4f2d-8391-e7ad85968ce0", "min_ram": 0, "checksum": "9cf87eeb3c6595fc3965ccf7d9941360", "owner": "f011a026b2fc4165b99ae4740f467d1d", "is_public": true, "virtual_size": null, "properties": {"item_name": "disk1.img", "architecture": "x86_64", "version_name": "20150810", "content_id": "auto.sync", "product_name": "com.ubuntu.cloud:server:14.04:amd64", "source_content_id": "com.ubuntu.cloud:released:download"}, "size": 258343424},
.... snip, edited out a bunch of non-ubuntu images in their glance
]}
## then we see that it got new metadata and decided to delete 0810 and get 0813.
INFO * 08-15 06:41:20 [PID:2747] * sstreams * com.ubuntu.cloud:released:download/com.ubuntu.cloud:server:15.04:arm64: to_add=[u'20150729'] to_remove=[]
INFO * 08-15 06:41:20 [PID:2747] * sstreams * com.ubuntu.cloud:released:download/com.ubuntu.cloud:server:13.04:amd64: to_add=[u'20140111'] to_remove=[]
INFO * 08-15 06:41:20 [PID:2747] * sstreams * com.ubuntu.cloud:released:download/com.ubuntu.cloud:server:15.04:ppc64el: to_add=[u'20150729'] to_remove=[]
INFO * 08-15 06:41:20 [PID:2747] * sstreams * com.ubuntu.cloud:released:download/com.ubuntu.cloud:server:15.04:armhf: to_add=[u'20150729'] to_remove=[]
INFO * 08-15 06:41:20 [PID:2747] * sstreams * com.ubuntu.cloud:released:download/com.ubuntu.cloud:server:15.04:amd64: to_add=[u'20150729'] to_remove=[]
INFO * 08-15 06:41:20 [PID:2747] * sstreams * com.ubuntu.cloud:released:download/com.ubuntu.cloud:server:10.04:amd64: to_add=[u'20150427'] to_remove=[]
INFO * 08-15 06:41:20 [PID:2747] * sstreams * com.ubuntu.cloud:released:download/com.ubuntu.cloud:server:14.10:arm64: to_add=[u'20150723'] to_remove=[]
INFO * 08-15 06:41:20 [PID:2747] * sstreams * com.ubuntu.cloud:released:download/com.ubuntu.cloud:server:14.04:amd64: to_add=[u'20150813'] to_remove=[u'20150810']
DEBUG * 08-15 06:41:20 [PID:2747] * sstreams * getting local copy of http://cloud-images.ubuntu.com/releases/server/releases/trusty/release-20150813/ubuntu-14.04-server-cloudimg-amd64-disk1.img
INFO * 08-15 06:41:20 [PID:2747] * urllib3.connectionpool * Starting new HTTP connection (1): cloud-images.ubuntu.com
DEBUG * 08-15 06:41:20 [PID:2747] * urllib3.connectionpool * Setting read timeout to <object object at 0x7fc83ec93170>
DEBUG * 08-15 06:41:20 [PID:2747] * urllib3.connectionpool * "GET /releases/server/releases/trusty/release-20150813/ubuntu-14.04-server-cloudimg-amd64-disk1.img HTTP/1.1" 403 289
## uh oh, that path is bad and we got a 403 with 289 bytes left. we should bail here, but instead we upload to glance:
DEBUG * 08-15 06:41:20 [PID:2747] * glanceclient.common.http * curl -i -X POST -H 'x-image-meta-property-source_content_id: com.ubuntu.cloud:released:download' -H 'X-Auth-Token: fe1ca44f89f24174a9ccf3418fddf9d3' -H 'x-image-meta-property-version_name: 20150813' -H 'x-image-meta-property-product_name: com.ubuntu.cloud:server:14.04:amd64' -H 'x-image-meta-checksum: f655dd7ce96dd5b76b663edcec931b8e' -H 'Transfer-Encoding: chunked' -H 'x-image-meta-container_format: bare' -H 'x-image-meta-property-content_id: auto.sync' -H 'User-Agent: python-glanceclient' -H 'x-image-meta-property-item_name: disk1.img' -H 'x-image-meta-property-architecture: x86_64' -H 'x-image-meta-is_public: True' -H 'x-image-meta-size: 384' -H 'Content-Type: application/octet-stream' -H 'x-image-meta-disk_format: qcow2' -H 'x-image-meta-name: auto-sync/ubuntu-trusty-14.04-amd64-server-20150813-disk1.img' -d '<open file '/tmp/tmpcb3szh', mode 'rb' at 0x7fc8390ac300>' http://10.16.100.108:9292/v1/images
## note the image-meta-size: 384
DEBUG * 08-15 06:41:21 [PID:2747] * glanceclient.common.http *
HTTP/1.1 201 Created
content-length: 761
etag: f655dd7ce96dd5b76b663edcec931b8e
location: http://10.16.100.108:9292/v1/images/1c22ab11-168e-42b1-a772-16a761c75f32
date: Sat, 15 Aug 2015 06:41:21 GMT
content-type: application/json
x-openstack-request-id: req-req-d31a8004-c158-432c-b70c-13cc7811eaac
{"image": {"status": "active", "deleted": false, "container_format": "bare", "min_ram": 0, "updated_at": "2015-08-15T06:41:21.000000", "owner": "f011a026b2fc4165b99ae4740f467d1d", "min_disk": 0, "is_public": true, "deleted_at": null, "id": "1c22ab11-168e-42b1-a772-16a761c75f32", "size": 384, "virtual_size": null, "name": "auto-sync/ubuntu-trusty-14.04-amd64-server-20150813-disk1.img", "checksum": "f655dd7ce96dd5b76b663edcec931b8e", "created_at": "2015-08-15T06:41:21.000000", "disk_format": "qcow2", "properties": {"item_name": "disk1.img", "architecture": "x86_64", "version_name": "20150813", "content_id": "auto.sync", "product_name": "com.ubuntu.cloud:server:14.04:amd64", "source_content_id": "com.ubuntu.cloud:released:download"}, "protected": false}}
## note the size: 384. at this point, glance has a broken image but further runs of the sync charm don't know that so they don't get a working image automatically until they either delete that broken image or we release a new image.
Additional info - the version of simplestreams in use at the external site is from a PPA, and is just rev 354: http://
That's the rev that adds progress callbacks.
I looked through the commit messages since then and it doesn't look like this particular bug was squashed yet.