Testing Sunbeam rev 254 with 6 nodes, 3 as compute + storage and 3 as compute + storage + control the tempest test "test_create_delete_image" failed on a nova server fault:
Traceback (most recent call last):
File "/home/ubuntu/snap/fcbtest/50/.rally/verification/verifier-5695bd71-3864-495a-b6ad-6fa1b0ae6481/repo/tempest/api/compute/images/test_images_oneserver.py", line 93, in test_create_delete_image
self.client.delete_image(image['id'])
File "/home/ubuntu/snap/fcbtest/50/.rally/verification/verifier-5695bd71-3864-495a-b6ad-6fa1b0ae6481/repo/tempest/lib/services/compute/images_client.py", line 82, in delete_image
resp, body = self.delete("images/%s" % image_id)
File "/home/ubuntu/snap/fcbtest/50/.rally/verification/verifier-5695bd71-3864-495a-b6ad-6fa1b0ae6481/repo/tempest/lib/common/rest_client.py", line 339, in delete
return self.request('DELETE', url, extra_headers, headers, body)
File "/home/ubuntu/snap/fcbtest/50/.rally/verification/verifier-5695bd71-3864-495a-b6ad-6fa1b0ae6481/repo/tempest/lib/services/compute/base_compute_client.py", line 47, in request
resp, resp_body = super(BaseComputeClient, self).request(
File "/home/ubuntu/snap/fcbtest/50/.rally/verification/verifier-5695bd71-3864-495a-b6ad-6fa1b0ae6481/repo/tempest/lib/common/rest_client.py", line 742, in request
self._error_checker(resp, resp_body)
File "/home/ubuntu/snap/fcbtest/50/.rally/verification/verifier-5695bd71-3864-495a-b6ad-6fa1b0ae6481/repo/tempest/lib/common/rest_client.py", line 922, in _error_checker
raise exceptions.ServerFault(resp_body, resp=resp,
tempest.lib.exceptions.ServerFault: Got server fault
Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'glanceclient.exc.HTTPInternalServerError'>
Nothing in the Nova logs stands out, but the pod logs for glance-2 we got this error:
2023-09-03T16:11:09.950Z [glance-api] 2023-09-03 16:11:09.946 2303 ERROR glance.common.wsgi [None req-abc4319a-fbb2-4800-8e96-ced69cc38e78 3032407ff2e44e108c84c37d4aefdc1b e0ff68bea50542968545e80a4f5726fd - - c1eedb1065ba47559a77f3085f56f3f2 c1eedb1065ba47559a77f3085f56f3f2] Caught error: Deleting images from this store is not supported.: glance_store.exceptions.StoreDeleteNotSupported: Deleting images from this store is not supported.
2023-09-03T16:11:09.950Z [glance-api] 2023-09-03 16:11:09.946 2303 ERROR glance.common.wsgi Traceback (most recent call last):
2023-09-03T16:11:09.950Z [glance-api] 2023-09-03 16:11:09.946 2303 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/common/wsgi.py", line 1286, in __call__
2023-09-03T16:11:09.950Z [glance-api] 2023-09-03 16:11:09.946 2303 ERROR glance.common.wsgi action_result = self.dispatch(self.controller, action,
2023-09-03T16:11:09.950Z [glance-api] 2023-09-03 16:11:09.946 2303 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/common/wsgi.py", line 1329, in dispatch
2023-09-03T16:11:09.950Z [glance-api] 2023-09-03 16:11:09.946 2303 ERROR glance.common.wsgi return method(*args, **kwargs)
2023-09-03T16:11:09.950Z [glance-api] 2023-09-03 16:11:09.946 2303 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/common/utils.py", line 414, in wrapped
2023-09-03T16:11:09.950Z [glance-api] 2023-09-03 16:11:09.946 2303 ERROR glance.common.wsgi return func(self, req, *args, **kwargs)
2023-09-03T16:11:09.950Z [glance-api] 2023-09-03 16:11:09.946 2303 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance/api/v2/images.py", line 882, in delete
2023-09-03T16:11:09.950Z [glance-api] 2023-09-03 16:11:09.946 2303 ERROR glance.common.wsgi staging_store.delete(loc)
2023-09-03T16:11:09.950Z [glance-api] 2023-09-03 16:11:09.946 2303 ERROR glance.common.wsgi File "/usr/lib/python3/dist-packages/glance_store/capabilities.py", line 174, in op_checker
2023-09-03T16:11:09.950Z [glance-api] 2023-09-03 16:11:09.946 2303 ERROR glance.common.wsgi raise op_exec_map[op](**kwargs)
2023-09-03T16:11:09.950Z [glance-api] 2023-09-03 16:11:09.946 2303 ERROR glance.common.wsgi glance_store.exceptions.StoreDeleteNotSupported: Deleting images from this store is not supported.
The testrun can be found at https://solutions.qa.canonical.com/testruns/94df2198-91ff-49e1-a9fe-9f3f9e32b14c
the pod logs can be found at https://oil-jenkins.canonical.com/artifacts/94df2198-91ff-49e1-a9fe-9f3f9e32b14c/generated/generated/sunbeam/pods_openstack_logs.tgz but should also be attached.
Another clue -
2023-09- 03T16:01: 30.409Z [glance-api] Traceback (most recent call last): 03T16:01: 30.409Z [glance-api] File "/usr/lib/ python3/ dist-packages/ eventlet/ greenpool. py", line 88, in _spawn_n_impl 03T16:01: 30.409Z [glance-api] func(*args, **kwargs) 03T16:01: 30.409Z [glance-api] File "/usr/lib/ python3/ dist-packages/ glance/ housekeeping. py", line 87, in clean_orphaned_ staging_ residue 03T16:01: 30.409Z [glance-api] files = os.listdir( staging_ store_path( )) 03T16:01: 30.409Z [glance-api] PermissionError: [Errno 13] Permission denied: '/var/lib/ glance/ os_glance_ staging_ store'
2023-09-
2023-09-
2023-09-
2023-09-
2023-09-
that's thrown no startup of the service.