[2023.1ev 254] Tempest "test create delete image" fails on nova server fault

[Alexander Balderson]

Testing Sunbeam rev 254 with 6 nodes, 3 as compute + storage and 3 as compute + storage + control the tempest test "test_create_delete_image" failed on a nova server fault:

Traceback (most recent call last):
  File "/home/ubuntu/snap/fcbtest/50/.rally/verification/verifier-5695bd71-3864-495a-b6ad-6fa1b0ae6481/repo/tempest/api/compute/images/test_images_oneserver.py", line 93, in test_create_delete_image
    self.client.delete_image(image['id'])
  File "/home/ubuntu/snap/fcbtest/50/.rally/verification/verifier-5695bd71-3864-495a-b6ad-6fa1b0ae6481/repo/tempest/lib/services/compute/images_client.py", line 82, in delete_image
    resp, body = self.delete("images/%s" % image_id)
  File "/home/ubuntu/snap/fcbtest/50/.rally/verification/verifier-5695bd71-3864-495a-b6ad-6fa1b0ae6481/repo/tempest/lib/common/rest_client.py", line 339, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/home/ubuntu/snap/fcbtest/50/.rally/verification/verifier-5695bd71-3864-495a-b6ad-6fa1b0ae6481/repo/tempest/lib/services/compute/base_compute_client.py", line 47, in request
    resp, resp_body = super(BaseComputeClient, self).request(
  File "/home/ubuntu/snap/fcbtest/50/.rally/verification/verifier-5695bd71-3864-495a-b6ad-6fa1b0ae6481/repo/tempest/lib/common/rest_client.py", line 742, in request
    self._error_checker(resp, resp_body)
  File "/home/ubuntu/snap/fcbtest/50/.rally/verification/verifier-5695bd71-3864-495a-b6ad-6fa1b0ae6481/repo/tempest/lib/common/rest_client.py", line 922, in _error_checker
    raise exceptions.ServerFault(resp_body, resp=resp,
tempest.lib.exceptions.ServerFault: Got server fault
Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'glanceclient.exc.HTTPInternalServerError'>

Nothing in the Nova logs stands out, but the pod logs for glance-2 we got this error:

2023-09-03T16:11:09.950Z [glance-api] 2023-09-03 16:11:09.946 2303 ERROR glance.common.wsgi [None req-abc4319a-fbb2-4800-8e96-ced69cc38e78 3032407ff2e44e108c84c37d4aefdc1b e0ff68bea50542968545e80a4f5726fd - - c1eedb1065ba47559a77f3085f56f3f2 c1eedb1065ba47559a77f3085f56f3f2] Caught error: Deleting images from this store is not supported.: glance_store.exceptions.StoreDeleteNotSupported: Deleting images from this store is not supported.
2023-09-03T16:11:09.950Z [glance-api] 2023-09-03 16:11:09.946 2303 ERROR glance.common.wsgi Traceback (most recent call last):
2023-09-03T16:11:09.950Z [glance-api] 2023-09-03 16:11:09.946 2303 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/common/wsgi.py", line 1286, in __call__
2023-09-03T16:11:09.950Z [glance-api] 2023-09-03 16:11:09.946 2303 ERROR glance.common.wsgi action_result = self.dispatch(self.controller, action,
2023-09-03T16:11:09.950Z [glance-api] 2023-09-03 16:11:09.946 2303 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/common/wsgi.py", line 1329, in dispatch
2023-09-03T16:11:09.950Z [glance-api] 2023-09-03 16:11:09.946 2303 ERROR glance.common.wsgi return method(*args, **kwargs)
2023-09-03T16:11:09.950Z [glance-api] 2023-09-03 16:11:09.946 2303 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/common/utils.py", line 414, in wrapped
2023-09-03T16:11:09.950Z [glance-api] 2023-09-03 16:11:09.946 2303 ERROR glance.common.wsgi return func(self, req, *args, **kwargs)
2023-09-03T16:11:09.950Z [glance-api] 2023-09-03 16:11:09.946 2303 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/api/v2/images.py", line 882, in delete
2023-09-03T16:11:09.950Z [glance-api] 2023-09-03 16:11:09.946 2303 ERROR glance.common.wsgi staging_store.delete(loc)
2023-09-03T16:11:09.950Z [glance-api] 2023-09-03 16:11:09.946 2303 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance_store/capabilities.py", line 174, in op_checker
2023-09-03T16:11:09.950Z [glance-api] 2023-09-03 16:11:09.946 2303 ERROR glance.common.wsgi raise op_exec_map[op](**kwargs)
2023-09-03T16:11:09.950Z [glance-api] 2023-09-03 16:11:09.946 2303 ERROR glance.common.wsgi glance_store.exceptions.StoreDeleteNotSupported: Deleting images from this store is not supported.

The testrun can be found at https://solutions.qa.canonical.com/testruns/94df2198-91ff-49e1-a9fe-9f3f9e32b14c

the pod logs can be found at https://oil-jenkins.canonical.com/artifacts/94df2198-91ff-49e1-a9fe-9f3f9e32b14c/generated/generated/sunbeam/pods_openstack_logs.tgz but should also be attached.

[James Page (james-page)]

Another clue -

2023-09-03T16:01:30.409Z [glance-api] Traceback (most recent call last):
2023-09-03T16:01:30.409Z [glance-api] File "/usr/lib/python3/dist-packages/eventlet/greenpool.py", line 88, in _spawn_n_impl
2023-09-03T16:01:30.409Z [glance-api] func(*args, **kwargs)
2023-09-03T16:01:30.409Z [glance-api] File "/usr/lib/python3/dist-packages/glance/housekeeping.py", line 87, in clean_orphaned_staging_residue
2023-09-03T16:01:30.409Z [glance-api] files = os.listdir(staging_store_path())
2023-09-03T16:01:30.409Z [glance-api] PermissionError: [Errno 13] Permission denied: '/var/lib/glance/os_glance_staging_store'

that's thrown no startup of the service.