Activity log for bug #1896617

Date Who What changed Old value New value Message
2020-09-22 12:43:10 Vladimir Grevtsev bug added bug
2020-09-22 12:45:58 Vladimir Grevtsev summary Creation of image from the existing VM fails if libvirt-image-backend is configured to qcow2 Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2
2020-09-22 12:49:09 Vladimir Grevtsev description tl;dr creating of the image from the existing VM fails if qcow2 image backend is used, but everything is fine if using rbd image backend in nova-compute. Two Tempest tests ([1] and [2]) from the 2018.02 Refstack test lists [0] are failing with the following exception: 49701867-bedc-4d7d-aa71-7383d877d90c Traceback (most recent call last): File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/api/compute/base.py", line 369, in create_image_from_server waiters.wait_for_image_status(client, image_id, wait_until) File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/common/waiters.py", line 161, in wait_for_image_status image = show_image(image_id) File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/services/compute/images_client.py", line 74, in show_image resp, body = self.get("images/%s" % image_id) File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/common/rest_client.py", line 298, in get return self.request('GET', url, extra_headers, headers) File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/services/compute/base_compute_client.py", line 48, in request method, url, extra_headers, headers, body, chunked) File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/common/rest_client.py", line 687, in request self._error_checker(resp, resp_body) File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/common/rest_client.py", line 793, in _error_checker raise exceptions.NotFound(resp_body, resp=resp) tempest.lib.exceptions.NotFound: Object not found Details: {'code': 404, 'message': 'Image not found.'} During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/api/compute/images/test_images_oneserver.py", line 69, in test_create_delete_image wait_until='ACTIVE') File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/api/compute/base.py", line 384, in create_image_from_server image_id=image_id) tempest.exceptions.SnapshotNotFoundException: Server snapshot image d82e95b0-9c62-492d-a08c-5bb118d3bf56 not found. So far I was able to identify the following: 1) https://github.com/openstack/tempest/blob/master/tempest/api/compute/images/test_images_oneserver.py#L69 invokes a "create image from server" 2) It fails with the following error message in the nova-compute logs: https://pastebin.canonical.com/p/h6ZXdqjRRm/ The same occurs if the "openstack server image create --wait" will be executed; however, according to https://docs.openstack.org/nova/ussuri/admin/migrate-instance-with-snapshot.html the VM has to be shut down before the image creation: "Shut down the source VM before you take the snapshot to ensure that all data is flushed to disk. If necessary, list the instances to view the instance name. Use the openstack server stop command to shut down the instance:" This step is definitely being skipped by the test (e.g it's trying to perform the snapshot on top of the live VM). FWIW, I'm using libvirt-image-backend: qcow2 in my nova-compute application params; and I was able to confirm that if the above parameter will be changed to "libvirt-image-backend: rbd", the tests will pass successfully. Also, there is similar issue I was able to find: https://bugs.launchpad.net/nova/+bug/1885418 but it doesn't have any useful information rather then confirmation of the fact that OpenStack Ussuri + libvirt backend has some problem with the live snapshotting. [0] https://refstack.openstack.org/api/v1/guidelines/2018.02/tests?target=platform&type=required&alias=true&flag=false [1] tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_delete_image[id-3731d080-d4c5-4872-b41a-64d0d0021314] [2] tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_image_specify_multibyte_character_image_name[id-3b7c6fe4-dfe7-477c-9243-b06359db51e6] tl;dr 1) creating of the image from the existing VM fails if qcow2 image backend is used, but everything is fine if using rbd image backend in nova-compute. 2) openstack server image create --name <name of the new image> <instance name or uuid> fails with some unrelated error: $ openstack server image create --wait 842fa12c-19ee-44cb-bb31-36d27ec9d8fc HTTP 404 Not Found: No image found with ID f4693860-cd8d-4088-91b9-56b2f173ffc7 == Details == Two Tempest tests ([1] and [2]) from the 2018.02 Refstack test lists [0] are failing with the following exception: 49701867-bedc-4d7d-aa71-7383d877d90c Traceback (most recent call last):   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/api/compute/base.py", line 369, in create_image_from_server     waiters.wait_for_image_status(client, image_id, wait_until)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/common/waiters.py", line 161, in wait_for_image_status     image = show_image(image_id)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/services/compute/images_client.py", line 74, in show_image     resp, body = self.get("images/%s" % image_id)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/common/rest_client.py", line 298, in get     return self.request('GET', url, extra_headers, headers)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/services/compute/base_compute_client.py", line 48, in request     method, url, extra_headers, headers, body, chunked)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/common/rest_client.py", line 687, in request     self._error_checker(resp, resp_body)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/common/rest_client.py", line 793, in _error_checker     raise exceptions.NotFound(resp_body, resp=resp) tempest.lib.exceptions.NotFound: Object not found Details: {'code': 404, 'message': 'Image not found.'} During handling of the above exception, another exception occurred: Traceback (most recent call last):   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/api/compute/images/test_images_oneserver.py", line 69, in test_create_delete_image     wait_until='ACTIVE')   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/api/compute/base.py", line 384, in create_image_from_server     image_id=image_id) tempest.exceptions.SnapshotNotFoundException: Server snapshot image d82e95b0-9c62-492d-a08c-5bb118d3bf56 not found. So far I was able to identify the following: 1) https://github.com/openstack/tempest/blob/master/tempest/api/compute/images/test_images_oneserver.py#L69 invokes a "create image from server" 2) It fails with the following error message in the nova-compute logs: https://pastebin.canonical.com/p/h6ZXdqjRRm/ The same occurs if the "openstack server image create --wait" will be executed; however, according to https://docs.openstack.org/nova/ussuri/admin/migrate-instance-with-snapshot.html the VM has to be shut down before the image creation: "Shut down the source VM before you take the snapshot to ensure that all data is flushed to disk. If necessary, list the instances to view the instance name. Use the openstack server stop command to shut down the instance:" This step is definitely being skipped by the test (e.g it's trying to perform the snapshot on top of the live VM). FWIW, I'm using libvirt-image-backend: qcow2 in my nova-compute application params; and I was able to confirm that if the above parameter will be changed to "libvirt-image-backend: rbd", the tests will pass successfully. Also, there is similar issue I was able to find: https://bugs.launchpad.net/nova/+bug/1885418 but it doesn't have any useful information rather then confirmation of the fact that OpenStack Ussuri + libvirt backend has some problem with the live snapshotting. [0] https://refstack.openstack.org/api/v1/guidelines/2018.02/tests?target=platform&type=required&alias=true&flag=false [1] tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_delete_image[id-3731d080-d4c5-4872-b41a-64d0d0021314] [2] tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_image_specify_multibyte_character_image_name[id-3b7c6fe4-dfe7-477c-9243-b06359db51e6]
2020-09-22 12:49:22 Vladimir Grevtsev description tl;dr 1) creating of the image from the existing VM fails if qcow2 image backend is used, but everything is fine if using rbd image backend in nova-compute. 2) openstack server image create --name <name of the new image> <instance name or uuid> fails with some unrelated error: $ openstack server image create --wait 842fa12c-19ee-44cb-bb31-36d27ec9d8fc HTTP 404 Not Found: No image found with ID f4693860-cd8d-4088-91b9-56b2f173ffc7 == Details == Two Tempest tests ([1] and [2]) from the 2018.02 Refstack test lists [0] are failing with the following exception: 49701867-bedc-4d7d-aa71-7383d877d90c Traceback (most recent call last):   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/api/compute/base.py", line 369, in create_image_from_server     waiters.wait_for_image_status(client, image_id, wait_until)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/common/waiters.py", line 161, in wait_for_image_status     image = show_image(image_id)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/services/compute/images_client.py", line 74, in show_image     resp, body = self.get("images/%s" % image_id)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/common/rest_client.py", line 298, in get     return self.request('GET', url, extra_headers, headers)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/services/compute/base_compute_client.py", line 48, in request     method, url, extra_headers, headers, body, chunked)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/common/rest_client.py", line 687, in request     self._error_checker(resp, resp_body)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/common/rest_client.py", line 793, in _error_checker     raise exceptions.NotFound(resp_body, resp=resp) tempest.lib.exceptions.NotFound: Object not found Details: {'code': 404, 'message': 'Image not found.'} During handling of the above exception, another exception occurred: Traceback (most recent call last):   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/api/compute/images/test_images_oneserver.py", line 69, in test_create_delete_image     wait_until='ACTIVE')   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/api/compute/base.py", line 384, in create_image_from_server     image_id=image_id) tempest.exceptions.SnapshotNotFoundException: Server snapshot image d82e95b0-9c62-492d-a08c-5bb118d3bf56 not found. So far I was able to identify the following: 1) https://github.com/openstack/tempest/blob/master/tempest/api/compute/images/test_images_oneserver.py#L69 invokes a "create image from server" 2) It fails with the following error message in the nova-compute logs: https://pastebin.canonical.com/p/h6ZXdqjRRm/ The same occurs if the "openstack server image create --wait" will be executed; however, according to https://docs.openstack.org/nova/ussuri/admin/migrate-instance-with-snapshot.html the VM has to be shut down before the image creation: "Shut down the source VM before you take the snapshot to ensure that all data is flushed to disk. If necessary, list the instances to view the instance name. Use the openstack server stop command to shut down the instance:" This step is definitely being skipped by the test (e.g it's trying to perform the snapshot on top of the live VM). FWIW, I'm using libvirt-image-backend: qcow2 in my nova-compute application params; and I was able to confirm that if the above parameter will be changed to "libvirt-image-backend: rbd", the tests will pass successfully. Also, there is similar issue I was able to find: https://bugs.launchpad.net/nova/+bug/1885418 but it doesn't have any useful information rather then confirmation of the fact that OpenStack Ussuri + libvirt backend has some problem with the live snapshotting. [0] https://refstack.openstack.org/api/v1/guidelines/2018.02/tests?target=platform&type=required&alias=true&flag=false [1] tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_delete_image[id-3731d080-d4c5-4872-b41a-64d0d0021314] [2] tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_image_specify_multibyte_character_image_name[id-3b7c6fe4-dfe7-477c-9243-b06359db51e6] tl;dr 1) creating the image from the existing VM fails if qcow2 image backend is used, but everything is fine if using rbd image backend in nova-compute. 2) openstack server image create --name <name of the new image> <instance name or uuid> fails with some unrelated error: $ openstack server image create --wait 842fa12c-19ee-44cb-bb31-36d27ec9d8fc HTTP 404 Not Found: No image found with ID f4693860-cd8d-4088-91b9-56b2f173ffc7 == Details == Two Tempest tests ([1] and [2]) from the 2018.02 Refstack test lists [0] are failing with the following exception: 49701867-bedc-4d7d-aa71-7383d877d90c Traceback (most recent call last):   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/api/compute/base.py", line 369, in create_image_from_server     waiters.wait_for_image_status(client, image_id, wait_until)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/common/waiters.py", line 161, in wait_for_image_status     image = show_image(image_id)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/services/compute/images_client.py", line 74, in show_image     resp, body = self.get("images/%s" % image_id)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/common/rest_client.py", line 298, in get     return self.request('GET', url, extra_headers, headers)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/services/compute/base_compute_client.py", line 48, in request     method, url, extra_headers, headers, body, chunked)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/common/rest_client.py", line 687, in request     self._error_checker(resp, resp_body)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/common/rest_client.py", line 793, in _error_checker     raise exceptions.NotFound(resp_body, resp=resp) tempest.lib.exceptions.NotFound: Object not found Details: {'code': 404, 'message': 'Image not found.'} During handling of the above exception, another exception occurred: Traceback (most recent call last):   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/api/compute/images/test_images_oneserver.py", line 69, in test_create_delete_image     wait_until='ACTIVE')   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/api/compute/base.py", line 384, in create_image_from_server     image_id=image_id) tempest.exceptions.SnapshotNotFoundException: Server snapshot image d82e95b0-9c62-492d-a08c-5bb118d3bf56 not found. So far I was able to identify the following: 1) https://github.com/openstack/tempest/blob/master/tempest/api/compute/images/test_images_oneserver.py#L69 invokes a "create image from server" 2) It fails with the following error message in the nova-compute logs: https://pastebin.canonical.com/p/h6ZXdqjRRm/ The same occurs if the "openstack server image create --wait" will be executed; however, according to https://docs.openstack.org/nova/ussuri/admin/migrate-instance-with-snapshot.html the VM has to be shut down before the image creation: "Shut down the source VM before you take the snapshot to ensure that all data is flushed to disk. If necessary, list the instances to view the instance name. Use the openstack server stop command to shut down the instance:" This step is definitely being skipped by the test (e.g it's trying to perform the snapshot on top of the live VM). FWIW, I'm using libvirt-image-backend: qcow2 in my nova-compute application params; and I was able to confirm that if the above parameter will be changed to "libvirt-image-backend: rbd", the tests will pass successfully. Also, there is similar issue I was able to find: https://bugs.launchpad.net/nova/+bug/1885418 but it doesn't have any useful information rather then confirmation of the fact that OpenStack Ussuri + libvirt backend has some problem with the live snapshotting. [0] https://refstack.openstack.org/api/v1/guidelines/2018.02/tests?target=platform&type=required&alias=true&flag=false [1] tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_delete_image[id-3731d080-d4c5-4872-b41a-64d0d0021314] [2] tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_image_specify_multibyte_character_image_name[id-3b7c6fe4-dfe7-477c-9243-b06359db51e6]
2020-09-22 15:51:48 Vladimir Grevtsev summary Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2 Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2 starting from Ussuri
2020-09-22 15:52:00 Vladimir Grevtsev bug added subscriber Canonical Field Critical
2020-09-23 09:30:23 Vladimir Grevtsev bug task added nova
2020-09-23 14:37:48 Lee Yarwood nova: status New Incomplete
2020-09-23 20:15:24 Billy Olsen charm-nova-compute: assignee Corey Bryant (corey.bryant)
2020-09-23 20:16:43 Corey Bryant bug task added nova (Ubuntu)
2020-09-23 20:16:50 Corey Bryant nova (Ubuntu): status New Triaged
2020-09-23 20:17:47 Corey Bryant nova (Ubuntu): importance Undecided Critical
2020-09-23 20:17:54 Corey Bryant nova (Ubuntu): assignee Corey Bryant (corey.bryant)
2020-09-23 20:17:57 Corey Bryant charm-nova-compute: assignee Corey Bryant (corey.bryant)
2020-09-23 20:18:05 Corey Bryant charm-nova-compute: status New Invalid
2020-09-23 22:55:39 Lee Yarwood nova: status Incomplete Invalid
2020-09-24 15:32:01 Corey Bryant nova: status Invalid New
2020-09-24 17:46:41 Maysam Fazeli bug added subscriber Maysam Fazeli
2020-09-24 20:26:12 Corey Bryant nova: status New Invalid
2020-09-24 20:29:31 Corey Bryant summary Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2 starting from Ussuri [SRU] Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2 starting from Ussuri
2020-09-24 20:29:41 Corey Bryant nominated for series Ubuntu Groovy
2020-09-24 20:29:41 Corey Bryant bug task added nova (Ubuntu Groovy)
2020-09-24 20:29:41 Corey Bryant nominated for series Ubuntu Focal
2020-09-24 20:29:41 Corey Bryant bug task added nova (Ubuntu Focal)
2020-09-24 20:29:51 Corey Bryant nova (Ubuntu Focal): status New Triaged
2020-09-24 20:29:57 Corey Bryant nova (Ubuntu Focal): importance Undecided Critical
2020-09-24 20:30:00 Corey Bryant nova (Ubuntu Focal): assignee Corey Bryant (corey.bryant)
2020-09-24 20:30:22 Corey Bryant bug task added cloud-archive
2020-09-24 20:30:33 Corey Bryant nominated for series cloud-archive/ussuri
2020-09-24 20:30:33 Corey Bryant bug task added cloud-archive/ussuri
2020-09-24 20:30:33 Corey Bryant nominated for series cloud-archive/victoria
2020-09-24 20:30:33 Corey Bryant bug task added cloud-archive/victoria
2020-09-24 20:30:57 Corey Bryant cloud-archive/ussuri: status New Triaged
2020-09-24 20:31:01 Corey Bryant cloud-archive/victoria: status New Triaged
2020-09-24 20:31:04 Corey Bryant cloud-archive/victoria: importance Undecided Critical
2020-09-24 20:31:08 Corey Bryant cloud-archive/ussuri: importance Undecided Critical
2020-09-24 20:31:13 Corey Bryant cloud-archive/victoria: assignee Corey Bryant (corey.bryant)
2020-09-24 20:31:16 Corey Bryant cloud-archive/ussuri: assignee Corey Bryant (corey.bryant)
2020-09-24 20:35:28 Corey Bryant description tl;dr 1) creating the image from the existing VM fails if qcow2 image backend is used, but everything is fine if using rbd image backend in nova-compute. 2) openstack server image create --name <name of the new image> <instance name or uuid> fails with some unrelated error: $ openstack server image create --wait 842fa12c-19ee-44cb-bb31-36d27ec9d8fc HTTP 404 Not Found: No image found with ID f4693860-cd8d-4088-91b9-56b2f173ffc7 == Details == Two Tempest tests ([1] and [2]) from the 2018.02 Refstack test lists [0] are failing with the following exception: 49701867-bedc-4d7d-aa71-7383d877d90c Traceback (most recent call last):   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/api/compute/base.py", line 369, in create_image_from_server     waiters.wait_for_image_status(client, image_id, wait_until)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/common/waiters.py", line 161, in wait_for_image_status     image = show_image(image_id)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/services/compute/images_client.py", line 74, in show_image     resp, body = self.get("images/%s" % image_id)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/common/rest_client.py", line 298, in get     return self.request('GET', url, extra_headers, headers)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/services/compute/base_compute_client.py", line 48, in request     method, url, extra_headers, headers, body, chunked)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/common/rest_client.py", line 687, in request     self._error_checker(resp, resp_body)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/common/rest_client.py", line 793, in _error_checker     raise exceptions.NotFound(resp_body, resp=resp) tempest.lib.exceptions.NotFound: Object not found Details: {'code': 404, 'message': 'Image not found.'} During handling of the above exception, another exception occurred: Traceback (most recent call last):   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/api/compute/images/test_images_oneserver.py", line 69, in test_create_delete_image     wait_until='ACTIVE')   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/api/compute/base.py", line 384, in create_image_from_server     image_id=image_id) tempest.exceptions.SnapshotNotFoundException: Server snapshot image d82e95b0-9c62-492d-a08c-5bb118d3bf56 not found. So far I was able to identify the following: 1) https://github.com/openstack/tempest/blob/master/tempest/api/compute/images/test_images_oneserver.py#L69 invokes a "create image from server" 2) It fails with the following error message in the nova-compute logs: https://pastebin.canonical.com/p/h6ZXdqjRRm/ The same occurs if the "openstack server image create --wait" will be executed; however, according to https://docs.openstack.org/nova/ussuri/admin/migrate-instance-with-snapshot.html the VM has to be shut down before the image creation: "Shut down the source VM before you take the snapshot to ensure that all data is flushed to disk. If necessary, list the instances to view the instance name. Use the openstack server stop command to shut down the instance:" This step is definitely being skipped by the test (e.g it's trying to perform the snapshot on top of the live VM). FWIW, I'm using libvirt-image-backend: qcow2 in my nova-compute application params; and I was able to confirm that if the above parameter will be changed to "libvirt-image-backend: rbd", the tests will pass successfully. Also, there is similar issue I was able to find: https://bugs.launchpad.net/nova/+bug/1885418 but it doesn't have any useful information rather then confirmation of the fact that OpenStack Ussuri + libvirt backend has some problem with the live snapshotting. [0] https://refstack.openstack.org/api/v1/guidelines/2018.02/tests?target=platform&type=required&alias=true&flag=false [1] tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_delete_image[id-3731d080-d4c5-4872-b41a-64d0d0021314] [2] tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_image_specify_multibyte_character_image_name[id-3b7c6fe4-dfe7-477c-9243-b06359db51e6] [Impact] tl;dr 1) creating the image from the existing VM fails if qcow2 image backend is used, but everything is fine if using rbd image backend in nova-compute. 2) openstack server image create --name <name of the new image> <instance name or uuid> fails with some unrelated error: $ openstack server image create --wait 842fa12c-19ee-44cb-bb31-36d27ec9d8fc HTTP 404 Not Found: No image found with ID f4693860-cd8d-4088-91b9-56b2f173ffc7 == Details == Two Tempest tests ([1] and [2]) from the 2018.02 Refstack test lists [0] are failing with the following exception: 49701867-bedc-4d7d-aa71-7383d877d90c Traceback (most recent call last):   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/api/compute/base.py", line 369, in create_image_from_server     waiters.wait_for_image_status(client, image_id, wait_until)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/common/waiters.py", line 161, in wait_for_image_status     image = show_image(image_id)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/services/compute/images_client.py", line 74, in show_image     resp, body = self.get("images/%s" % image_id)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/common/rest_client.py", line 298, in get     return self.request('GET', url, extra_headers, headers)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/services/compute/base_compute_client.py", line 48, in request     method, url, extra_headers, headers, body, chunked)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/common/rest_client.py", line 687, in request     self._error_checker(resp, resp_body)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/common/rest_client.py", line 793, in _error_checker     raise exceptions.NotFound(resp_body, resp=resp) tempest.lib.exceptions.NotFound: Object not found Details: {'code': 404, 'message': 'Image not found.'} During handling of the above exception, another exception occurred: Traceback (most recent call last):   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/api/compute/images/test_images_oneserver.py", line 69, in test_create_delete_image     wait_until='ACTIVE')   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/api/compute/base.py", line 384, in create_image_from_server     image_id=image_id) tempest.exceptions.SnapshotNotFoundException: Server snapshot image d82e95b0-9c62-492d-a08c-5bb118d3bf56 not found. So far I was able to identify the following: 1) https://github.com/openstack/tempest/blob/master/tempest/api/compute/images/test_images_oneserver.py#L69 invokes a "create image from server" 2) It fails with the following error message in the nova-compute logs: https://pastebin.canonical.com/p/h6ZXdqjRRm/ The same occurs if the "openstack server image create --wait" will be executed; however, according to https://docs.openstack.org/nova/ussuri/admin/migrate-instance-with-snapshot.html the VM has to be shut down before the image creation: "Shut down the source VM before you take the snapshot to ensure that all data is flushed to disk. If necessary, list the instances to view the instance name. Use the openstack server stop command to shut down the instance:" This step is definitely being skipped by the test (e.g it's trying to perform the snapshot on top of the live VM). FWIW, I'm using libvirt-image-backend: qcow2 in my nova-compute application params; and I was able to confirm that if the above parameter will be changed to "libvirt-image-backend: rbd", the tests will pass successfully. Also, there is similar issue I was able to find: https://bugs.launchpad.net/nova/+bug/1885418 but it doesn't have any useful information rather then confirmation of the fact that OpenStack Ussuri + libvirt backend has some problem with the live snapshotting. [0] https://refstack.openstack.org/api/v1/guidelines/2018.02/tests?target=platform&type=required&alias=true&flag=false [1] tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_delete_image[id-3731d080-d4c5-4872-b41a-64d0d0021314] [2] tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_image_specify_multibyte_character_image_name[id-3b7c6fe4-dfe7-477c-9243-b06359db51e6] [Test Case] deploy/configure openstack, using juju here create openstack instance openstack server image create --wait <instance-uuid> successful if fixed; fails with permissions error if not fixed [Regression Potential] This actually reverts the nova group members to what they used to be prior to the focal version of the packages. If there is a regression in this fix it would likely result in a permissions issue.
2020-09-25 14:00:31 Corey Bryant description [Impact] tl;dr 1) creating the image from the existing VM fails if qcow2 image backend is used, but everything is fine if using rbd image backend in nova-compute. 2) openstack server image create --name <name of the new image> <instance name or uuid> fails with some unrelated error: $ openstack server image create --wait 842fa12c-19ee-44cb-bb31-36d27ec9d8fc HTTP 404 Not Found: No image found with ID f4693860-cd8d-4088-91b9-56b2f173ffc7 == Details == Two Tempest tests ([1] and [2]) from the 2018.02 Refstack test lists [0] are failing with the following exception: 49701867-bedc-4d7d-aa71-7383d877d90c Traceback (most recent call last):   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/api/compute/base.py", line 369, in create_image_from_server     waiters.wait_for_image_status(client, image_id, wait_until)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/common/waiters.py", line 161, in wait_for_image_status     image = show_image(image_id)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/services/compute/images_client.py", line 74, in show_image     resp, body = self.get("images/%s" % image_id)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/common/rest_client.py", line 298, in get     return self.request('GET', url, extra_headers, headers)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/services/compute/base_compute_client.py", line 48, in request     method, url, extra_headers, headers, body, chunked)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/common/rest_client.py", line 687, in request     self._error_checker(resp, resp_body)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/common/rest_client.py", line 793, in _error_checker     raise exceptions.NotFound(resp_body, resp=resp) tempest.lib.exceptions.NotFound: Object not found Details: {'code': 404, 'message': 'Image not found.'} During handling of the above exception, another exception occurred: Traceback (most recent call last):   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/api/compute/images/test_images_oneserver.py", line 69, in test_create_delete_image     wait_until='ACTIVE')   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/api/compute/base.py", line 384, in create_image_from_server     image_id=image_id) tempest.exceptions.SnapshotNotFoundException: Server snapshot image d82e95b0-9c62-492d-a08c-5bb118d3bf56 not found. So far I was able to identify the following: 1) https://github.com/openstack/tempest/blob/master/tempest/api/compute/images/test_images_oneserver.py#L69 invokes a "create image from server" 2) It fails with the following error message in the nova-compute logs: https://pastebin.canonical.com/p/h6ZXdqjRRm/ The same occurs if the "openstack server image create --wait" will be executed; however, according to https://docs.openstack.org/nova/ussuri/admin/migrate-instance-with-snapshot.html the VM has to be shut down before the image creation: "Shut down the source VM before you take the snapshot to ensure that all data is flushed to disk. If necessary, list the instances to view the instance name. Use the openstack server stop command to shut down the instance:" This step is definitely being skipped by the test (e.g it's trying to perform the snapshot on top of the live VM). FWIW, I'm using libvirt-image-backend: qcow2 in my nova-compute application params; and I was able to confirm that if the above parameter will be changed to "libvirt-image-backend: rbd", the tests will pass successfully. Also, there is similar issue I was able to find: https://bugs.launchpad.net/nova/+bug/1885418 but it doesn't have any useful information rather then confirmation of the fact that OpenStack Ussuri + libvirt backend has some problem with the live snapshotting. [0] https://refstack.openstack.org/api/v1/guidelines/2018.02/tests?target=platform&type=required&alias=true&flag=false [1] tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_delete_image[id-3731d080-d4c5-4872-b41a-64d0d0021314] [2] tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_image_specify_multibyte_character_image_name[id-3b7c6fe4-dfe7-477c-9243-b06359db51e6] [Test Case] deploy/configure openstack, using juju here create openstack instance openstack server image create --wait <instance-uuid> successful if fixed; fails with permissions error if not fixed [Regression Potential] This actually reverts the nova group members to what they used to be prior to the focal version of the packages. If there is a regression in this fix it would likely result in a permissions issue. [Impact] tl;dr 1) creating the image from the existing VM fails if qcow2 image backend is used, but everything is fine if using rbd image backend in nova-compute. 2) openstack server image create --name <name of the new image> <instance name or uuid> fails with some unrelated error: $ openstack server image create --wait 842fa12c-19ee-44cb-bb31-36d27ec9d8fc HTTP 404 Not Found: No image found with ID f4693860-cd8d-4088-91b9-56b2f173ffc7 == Details == Two Tempest tests ([1] and [2]) from the 2018.02 Refstack test lists [0] are failing with the following exception: 49701867-bedc-4d7d-aa71-7383d877d90c Traceback (most recent call last):   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/api/compute/base.py", line 369, in create_image_from_server     waiters.wait_for_image_status(client, image_id, wait_until)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/common/waiters.py", line 161, in wait_for_image_status     image = show_image(image_id)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/services/compute/images_client.py", line 74, in show_image     resp, body = self.get("images/%s" % image_id)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/common/rest_client.py", line 298, in get     return self.request('GET', url, extra_headers, headers)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/services/compute/base_compute_client.py", line 48, in request     method, url, extra_headers, headers, body, chunked)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/common/rest_client.py", line 687, in request     self._error_checker(resp, resp_body)   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/common/rest_client.py", line 793, in _error_checker     raise exceptions.NotFound(resp_body, resp=resp) tempest.lib.exceptions.NotFound: Object not found Details: {'code': 404, 'message': 'Image not found.'} During handling of the above exception, another exception occurred: Traceback (most recent call last):   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/api/compute/images/test_images_oneserver.py", line 69, in test_create_delete_image     wait_until='ACTIVE')   File "/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/api/compute/base.py", line 384, in create_image_from_server     image_id=image_id) tempest.exceptions.SnapshotNotFoundException: Server snapshot image d82e95b0-9c62-492d-a08c-5bb118d3bf56 not found. So far I was able to identify the following: 1) https://github.com/openstack/tempest/blob/master/tempest/api/compute/images/test_images_oneserver.py#L69 invokes a "create image from server" 2) It fails with the following error message in the nova-compute logs: https://pastebin.canonical.com/p/h6ZXdqjRRm/ The same occurs if the "openstack server image create --wait" will be executed; however, according to https://docs.openstack.org/nova/ussuri/admin/migrate-instance-with-snapshot.html the VM has to be shut down before the image creation: "Shut down the source VM before you take the snapshot to ensure that all data is flushed to disk. If necessary, list the instances to view the instance name. Use the openstack server stop command to shut down the instance:" This step is definitely being skipped by the test (e.g it's trying to perform the snapshot on top of the live VM). FWIW, I'm using libvirt-image-backend: qcow2 in my nova-compute application params; and I was able to confirm that if the above parameter will be changed to "libvirt-image-backend: rbd", the tests will pass successfully. Also, there is similar issue I was able to find: https://bugs.launchpad.net/nova/+bug/1885418 but it doesn't have any useful information rather then confirmation of the fact that OpenStack Ussuri + libvirt backend has some problem with the live snapshotting. [0] https://refstack.openstack.org/api/v1/guidelines/2018.02/tests?target=platform&type=required&alias=true&flag=false [1] tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_delete_image[id-3731d080-d4c5-4872-b41a-64d0d0021314] [2] tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_image_specify_multibyte_character_image_name[id-3b7c6fe4-dfe7-477c-9243-b06359db51e6] [Test Case] deploy/configure openstack, using juju here if upgrading to the fixed package, libvirt-guests will require restart: sudo systemctl restart libvirt-guests create openstack instance openstack server image create --wait <instance-uuid> successful if fixed; fails with permissions error if not fixed [Regression Potential] This actually reverts the nova group members to what they used to be prior to the focal version of the packages. If there is a regression in this fix it would likely result in a permissions issue.
2020-09-25 14:47:07 Corey Bryant bug added subscriber Ubuntu Stable Release Updates Team
2020-09-25 14:54:48 Timo Aaltonen nova (Ubuntu Focal): status Triaged Fix Committed
2020-09-25 14:54:53 Timo Aaltonen bug added subscriber SRU Verification
2020-09-25 14:54:57 Timo Aaltonen tags verification-needed verification-needed-focal
2020-09-25 17:11:33 Corey Bryant cloud-archive/ussuri: status Triaged Fix Committed
2020-09-25 17:11:35 Corey Bryant tags verification-needed verification-needed-focal verification-needed verification-needed-focal verification-ussuri-needed
2020-09-25 17:11:49 Corey Bryant cloud-archive: status Triaged Fix Committed
2020-09-25 18:28:39 Launchpad Janitor nova (Ubuntu Groovy): status Triaged Fix Released
2020-09-29 12:29:13 Michael Skalka removed subscriber Canonical Field Critical
2020-09-29 12:29:19 Michael Skalka bug added subscriber Canonical Field High
2020-10-01 12:27:46 Corey Bryant tags verification-needed verification-needed-focal verification-ussuri-needed verification-failed verification-failed-focal verification-ussuri-failed
2020-10-02 09:38:05 Timo Aaltonen tags verification-failed verification-failed-focal verification-ussuri-failed verification-needed verification-needed-focal verification-ussuri-failed
2020-10-02 12:37:49 Corey Bryant tags verification-needed verification-needed-focal verification-ussuri-failed verification-needed verification-needed-focal verification-ussuri-needed
2020-10-05 17:09:41 Corey Bryant cloud-archive: status Fix Committed Fix Released
2020-10-07 18:55:32 Corey Bryant attachment added 1896617-focal-testing.txt https://bugs.launchpad.net/cloud-archive/+bug/1896617/+attachment/5419085/+files/1896617-focal-testing.txt
2020-10-07 18:56:08 Corey Bryant tags verification-needed verification-needed-focal verification-ussuri-needed verification-done verification-done-focal verification-ussuri-needed
2020-10-07 21:25:40 Corey Bryant attachment added 1896617-ussuri-testing.txt https://bugs.launchpad.net/cloud-archive/+bug/1896617/+attachment/5419127/+files/1896617-ussuri-testing.txt
2020-10-07 21:25:55 Corey Bryant tags verification-done verification-done-focal verification-ussuri-needed verification-done verification-done-focal verification-ussuri-done
2020-10-13 17:39:45 Launchpad Janitor nova (Ubuntu Focal): status Fix Committed Fix Released
2020-10-13 17:39:59 Brian Murray removed subscriber Ubuntu Stable Release Updates Team
2020-10-14 13:04:03 Corey Bryant cloud-archive/ussuri: status Fix Committed Fix Released
2022-09-08 14:11:28 Corey Bryant nova: status Invalid New
2022-09-13 13:15:23 Uggla nova: status New Triaged
2022-09-13 16:29:50 Sylvain Bauza nova: status Triaged Opinion
2022-09-13 16:31:03 Sylvain Bauza nova: importance Undecided Wishlist