Glance cannot remove image if Nova boots instance from image with incorrect signature.

[Mitya Eremeev]

Description
===========
Nova is configured to verify glance images:
[glance]
verify_glance_signatures=true

Glance backend is Ceph.

Steps to reproduce
==================
1. create glance image with proper signature
2. update glance image with incorrect signature
3. try to boot instance from the glance image with incorrect signature.
Boot fails because Nova checks signature and verification fails.
It's correct behavior.

barbican_tempest_plugin.tests.scenario.test_image_signing.ImageSigningTest.test_signed_image_upload_boot_failure[compute,id-74f022d6-a6ef-4458-96b7-541deadacf99,image,smoke]
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/var/lib/openstack/lib/python3.6/site-packages/tempest/lib/services/image/v2/images_client.py", line 103, in delete_image
    resp, _ = self.delete(url)

      File "/var/lib/openstack/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 330, in delete
    return self.request('DELETE', url, extra_headers, headers, body)

      File "/var/lib/openstack/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 710, in request
    self._error_checker(resp, resp_body)

      File "/var/lib/openstack/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 831, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)

    tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'message': 'Image c321f6be-a4d3-42d2-bc3f-f0ea913b83b7 could not be deleted because it is in use: The image cannot be deleted because it is in use through the backend store outside of Glance.<br /><br />\n\n\n', 'code': '409 Conflict', 'title': 'Conflict'}

4. Delete the glance image right after failed instance boot.

Expected result
===============
Glance image was deleted successfully.

Actual result
=============
Glance cannot be deleted.
In Glance backend we see that there are watchers that protect glance image from deletion:

# rbd rm --pool images-hdd c321f6be-a4d3-42d2-bc3f-f0ea913b83b7
2021-10-15T13:25:03.862+0000 7f36b98c8700 -1 librbd::image::PreRemoveRequest: 0x562785d77a50 check_image_watchers: image has watchers - not removing
Removing image: 0% complete...failed.
rbd: error: image still has watchers
This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout.

# rbd status --pool images-hdd c321f6be-a4d3-42d2-bc3f-f0ea913b83b7
Watchers:
        watcher=10.10.0.89:0/729945307 client.374098 cookie=140684808072160

The behavior is reproduced by tempest test:
https://github.com/openstack/barbican-tempest-plugin/blob/master/barbican_tempest_plugin/tests/scenario/test_image_signing.py#L67

Environment
===========
1. Openstack version: Victoria
2. Hypervisor: KVM + libvirt
3. Glance storage: Ceph, Nova storage: local.
4. Networking: Neutron with OVS

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/815347

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/815347
Committed: https://opendev.org/openstack/nova/commit/43bca185fe2d00bb70d7486fa6c6a0b9eda1fc17
Submitter: "Zuul (22348)"
Branch: master

commit 43bca185fe2d00bb70d7486fa6c6a0b9eda1fc17
Author: Mitya_Eremeev <email address hidden>
Date: Thu Nov 11 18:52:11 2021 +0300

    Close Glance image if downloading failed.

    If downloding of Glance image failed we should
    close iterator of image body.
    Otherwise Glance is unable to delete the image.

    Change-Id: I193df2fcbf2588c10be953eb4e9eef4609b6286f
    Closes-Bug: 1948706

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 25.0.0.0rc1 release candidate.