glance mirror sync should ignore images in 'deleted' state

[Felipe Reyes]

We have a system where for some unknown reason there is an image in 'deleted' status, it appears in the list of images (glance image-list), but can't be deleted. So everytime glance-simplestreams-sync charm tries to create a mirror, this image is identified to be deleted (because it's too old), but glance returns a 404 error and simplestreams crashes.

I think simplestreams.mirrors.glance.GlanceMirror.remove_item() should catch the 404 exception and return None, the other option is that when the list of images is retrieved from glance, the images in 'deleted' status are discarded immediately.

$ glance image-list | grep b1e00628-a1ee-4241-b05e-bea5e965d91c
| b1e00628-a1ee-4241-b05e-bea5e965d91c | auto-sync/ubuntu-trusty-14.04-amd64-server-20160627-disk1.img |
$ glance image-show b1e00628-a1ee-4241-b05e-bea5e965d91c
+-------------------+---------------------------------------------------------------+
| Property | Value |
+-------------------+---------------------------------------------------------------+
| architecture | x86_64 |
| checksum | 272bc4cde0803923b73746ac17894fc1 |
| container_format | bare |
| content_id | auto.sync |
| created_at | 2016-09-30T16:37:11Z |
| disk_format | qcow2 |
| id | b1e00628-a1ee-4241-b05e-bea5e965d91c |
| item_name | disk1.img |
| locations | [] |
| min_disk | 0 |
| min_ram | 0 |
| name | auto-sync/ubuntu-trusty-14.04-amd64-server-20160627-disk1.img |
| owner | 53d1e365300b485b91049ad270e3f2cf |
| product_name | com.ubuntu.cloud:server:14.04:amd64 |
| protected | False |
| size | 260440576 |
| source_content_id | com.ubuntu.cloud:released:download |
| status | deleted |
| tags | [] |
| updated_at | 2016-09-30T16:59:46Z |
| version_name | 20160627 |
| virtual_size | None |
| visibility | public |
+-------------------+---------------------------------------------------------------+
$ glance image-delete b1e00628-a1ee-4241-b05e-bea5e965d91c
'400 Bad Request: Image status transition from deleted to deleted is not allowed (HTTP 400)': Unable to delete image 'b1e00628-a1ee-4241-b05e-bea5e965d91c'

Stacktrace found in the glance-simplestreams-sync charm:

INFO * 10-03 18:29:56 [PID:12923] * sstreams * com.ubuntu.cloud:released:download/com.ubuntu.cloud:server:15.04:arm64: to_add=[u'20160203', u'20160119', u'20160114.1'] to_remove=[]
INFO * 10-03 18:29:56 [PID:12923] * sstreams * com.ubuntu.cloud:released:download/com.ubuntu.cloud:server:13.04:amd64: to_add=[u'20140111', u'20131214', u'20131203'] to_remove=[]
INFO * 10-03 18:29:56 [PID:12923] * sstreams * com.ubuntu.cloud:released:download/com.ubuntu.cloud:server:15.04:ppc64el: to_add=[u'20160203', u'20160119', u'20160114.1'] to_remove=[]
INFO * 10-03 18:29:56 [PID:12923] * sstreams * com.ubuntu.cloud:released:download/com.ubuntu.cloud:server:14.04:i386: to_add=[] to_remove=[]
INFO * 10-03 18:29:56 [PID:12923] * sstreams * com.ubuntu.cloud:released:download/com.ubuntu.cloud:server:15.04:amd64: to_add=[u'20160203', u'20160119', u'20160114.1'] to_remove=[]
INFO * 10-03 18:29:56 [PID:12923] * sstreams * com.ubuntu.cloud:released:download/com.ubuntu.cloud:server:10.04:amd64: to_add=[u'20150427', u'20150127', u'20150109'] to_remove=[]
INFO * 10-03 18:29:56 [PID:12923] * sstreams * com.ubuntu.cloud:released:download/com.ubuntu.cloud:server:14.10:arm64: to_add=[u'20150723', u'20150708', u'20150620'] to_remove=[]
INFO * 10-03 18:29:56 [PID:12923] * sstreams * com.ubuntu.cloud:released:download/com.ubuntu.cloud:server:14.04:amd64: to_add=[] to_remove=[u'20160627']
DEBUG * 10-03 18:29:56 [PID:12923] * glanceclient.common.http * curl -i -X DELETE -H 'X-Auth-Token: XXXXXX' -H 'Content-Type: application/octet-stream' -H 'User-Agent: python-glanceclient' http://1.2.3.4:9292/v1/images/b1e00628-a1ee-4241-b05e-bea5e965d91c
DEBUG * 10-03 18:29:57 [PID:12923] * glanceclient.common.http *
HTTP/1.1 404 Not Found
date: Mon, 03 Oct 2016 18:29:57 GMT
content-length: 73
content-type: text/plain; charset=UTF-8
x-openstack-request-id: req-93dee12b-0236-4da6-837f-8e394c5f143d

404 Not Found

Image b1e00628-a1ee-4241-b05e-bea5e965d91c not found.

ERROR * 10-03 18:29:57 [PID:12923] * glanceclient.common.http * Request returned failure status.
ERROR * 10-03 18:29:57 [PID:12923] * root * Glance Client exception during do_sync. will continue polling.
Traceback (most recent call last):
  File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 459, in main
    do_sync(charm_conf, status_exchange)
  File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 211, in do_sync
    tmirror.sync(smirror, path=initial_path)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 91, in sync
    return self.sync_index(reader, path, data, content)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 244, in sync_index
    self.sync(reader, path=epath)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 89, in sync
    return self.sync_products(reader, path, data, content)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 355, in sync_products
    (prodname, vername, itemname))
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/glance.py", line 231, in remove_item
    self.gclient.images.delete(data['id'])
  File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 226, in delete
    self._delete("/v1/images/%s" % base.getid(image))
  File "/usr/lib/python2.7/dist-packages/glanceclient/common/base.py", line 62, in _delete
    self.api.raw_request('DELETE', url)
  File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 289, in raw_request
    return self._http_request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 249, in _http_request
    raise exc.from_response(resp, body_str)
HTTPNotFound: 404 Not Found
Image b1e00628-a1ee-4241-b05e-bea5e965d91c not found.
    (HTTP 404)
INFO * 10-03 18:29:57 [PID:12923] * root * sync done.

[Felipe Reyes (freyes)]

Details about why glance doesn't consider valid to remove an image with status='deleted' can be found at https://bugs.launchpad.net/glance/+bug/1452788