When delete a synced image from the slave, the sync stops working

Bug #1865045 reported by Giuseppe Petralia
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
glance-sync-layer
In Progress
High
Xav Paice

Bug Description

If an image synchronized by the glance-sync-slave is deleted using glance api, sync stop working and the following exceptions are logged on the glance-sync-slave cronjob:

====================================================================================
2020-02-27 14:54:12 starting glance sync
2020-02-27 14:54:12 getting metadata from master
b''
2020-02-27 14:54:13 getting image list from slave
2020-02-27 14:54:13 INFO: is_latest_metadata :: 1dde7ef7-ae25-4ce0-bb1e-ac3ee7e216e2 up to date
2020-02-27 14:54:13 DEBUG: images pending to be deleted: set()
2020-02-27 14:54:13 WARNING: precautionary halt. No glance images found to be deleted. noop.
2020-02-27 14:54:13 DEBUG: processed images (to skip while parsing metadata files): {'1dde7ef7-ae25-4ce0-bb1e-ac3ee7e216e2'}
2020-02-27 14:54:13 INFO: download_from_master (e69b8354-7bf7-498e-b5aa-13187732217f - 443b7623e27ecf03dc9e01ee93f67afe):: checksum OK
2020-02-27 14:54:13 INFO: creating image e69b8354-7bf7-498e-b5aa-13187732217f
2020-02-27 14:54:13 EXCEPTION: upload_to_slave :: 409 Conflict: Image ID e69b8354-7bf7-498e-b5aa-13187732217f already exists! (HTTP 409)
Traceback (most recent call last):
  File "/srv/glance_sync/scripts/glance_sync_slave.py", line 223, in upload_to_slave
    self.glance_slave.images.create(**clean_metadata)
  File "/usr/lib/python3/dist-packages/glanceclient/common/utils.py", line 545, in inner
    return RequestIdProxy(wrapped(*args, **kwargs))
  File "/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 283, in create
    resp, body = self.http_client.post(url, data=image)
  File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 310, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python3/dist-packages/glanceclient/common/http.py", line 349, in request
    return self._handle_response(resp)
  File "/usr/lib/python3/dist-packages/glanceclient/common/http.py", line 98, in _handle_response
    raise exc.from_response(resp, resp.content)
glanceclient.exc.HTTPConflict: 409 Conflict: Image ID e69b8354-7bf7-498e-b5aa-13187732217f already exists! (HTTP 409)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/glance_sync/scripts/glance_sync_slave.py", line 231, in upload_to_slave
    **clean_metadata)
  File "/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 310, in update
    unvalidated_image = self.get(image_id)
  File "/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 197, in get
    return self._get(image_id)
  File "/usr/lib/python3/dist-packages/glanceclient/common/utils.py", line 545, in inner
    return RequestIdProxy(wrapped(*args, **kwargs))
  File "/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 190, in _get
    resp, body = self.http_client.get(url, headers=header)
  File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 304, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python3/dist-packages/glanceclient/common/http.py", line 349, in request
    return self._handle_response(resp)
  File "/usr/lib/python3/dist-packages/glanceclient/common/http.py", line 98, in _handle_response
    raise exc.from_response(resp, resp.content)
glanceclient.exc.HTTPNotFound: 404 Not Found: No image found with ID e69b8354-7bf7-498e-b5aa-13187732217f (HTTP 404)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/glance_sync/scripts/glance_sync_slave.py", line 642, in <module>
    slave.main()
  File "/srv/glance_sync/scripts/glance_sync_slave.py", line 614, in main
    self.create_missing_slave_images(processed_images_ids)
  File "/srv/glance_sync/scripts/glance_sync_slave.py", line 359, in create_missing_slave_images
    self.upload_to_slave(metadata_local)
  File "/srv/glance_sync/scripts/glance_sync_slave.py", line 235, in upload_to_slave
    if "HTTPNotFound" in e:
TypeError: argument of type 'HTTPNotFound' is not iterable
=================================================================================================

Steps to reproduce.

Syncronized an image:

$ openstack image list
+--------------------------------------+----------+--------+
| ID | Name | Status |
+--------------------------------------+----------+--------+
| 1dde7ef7-ae25-4ce0-bb1e-ac3ee7e216e2 | cirros-1 | active |
+--------------------------------------+----------+--------+

Delete it:

$ openstack image delete 1dde7ef7-ae25-4ce0-bb1e-ac3ee7e216e2

Run the syncronization script:

juju ssh gss/0 " source /srv/glance_sync/config/novarc && /srv/glance_sync/scripts/glance_sync_slave.py -d /srv/glance_sync/data/ -s ubuntu@10.0.8.155:/srv/glance_sync/data/"

Changed in charm-glance-sync:
importance: Undecided → High
Changed in charm-glance-sync:
status: New → Triaged
Xav Paice (xavpaice)
Changed in charm-glance-sync:
assignee: nobody → Xav Paice (xavpaice)
status: Triaged → In Progress
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers