scrubber edge cases orphan objects and records

[Jesse J. Cook]

The scrubber can leave orphaned objects and db records in error / edge cases. This is because the order in which it updates the DB and object store. Recommended solution:

For each image that has status pending_delete:
    For each image location that has status pending_delete:
        Delete the object in the object store
        If error other than object not found, continue
        Mark image location status as deleted
    If all image locations are deleted, mark image as deleted
    Else if no image locations are marked as pending_delete, change status to something else??? # I suppose it's possible an image_location would still be active or some other non-deleted status. I don't think we want to orphan the image_location by marking the image deleted in this case.

[Jesse J. Cook (jesse-j-cook)]

I suppose examples are warranted:

I had a case where the scheme wasn't supported as configured and all the image statuses were marked as deleted and all the image_location statuses were still pending_delete with objects still in object store.

I had another case where the image statuses were marked as deleted, image location statuses were still pending_delete and there were no objects in the object store.

[Nikhil Komawar (nikhil-komawar)]

https://review.openstack.org/#/c/217370/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/217370
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=21debbd5bfd0641166a36f6bf5f2a3e189163de1
Submitter: Jenkins
Branch: master

commit 21debbd5bfd0641166a36f6bf5f2a3e189163de1
Author: Hemanth Makkapati <email address hidden>
Date: Wed Aug 26 14:23:41 2015 -0500

    Fixes the possibility of leaving orphaned data

    When Glance scrubber encounters an error while scrubbing image data
    from the backend, it puts the image in deleted status while leaving
    the image locations in pending_delete status with data still in the
    backend. This results in dark data.

    This change attempts to fix this by leaving the image in
    pending_delete status if the scrubber fails to delete the data from
    the backend. This presents the scrubber with an opportunity to
    retry the scrubbing in the next run.

    Change-Id: Ia6fa0008317cc0d619e5f212c0f043e078fbea55
    Closes-bug: #1444767