Tempest swift tests fail randomly. get_object_delete function throws BucketAlreadyExits exception which should be handled

[Sampat Ponnaganti]

Tempest Swift tests fail randomly with b'BucketAlreadyExists errors:

Issue:
wait_for_resource_deletion function waits for the resource to be deleted.
update is_resource_deleted function here and handle exception.Conflict and return False. (https://github.com/openstack/tempest/blob/7e96c8e854386f43604ad098a6ec7606ee676145/tempest/lib/services/object_storage/object_client.py#L28)

tempest.api.object_storage.test_container_acl.ObjectTestACLs.test_read_object_with_rights[id-a3270f3f-7640-4944-8448-c7ea783ea5b6]

class ObjectClient(rest_client.RestClient):

    def is_resource_deleted(self, object_name, container):
        try:
            self.get_object(container, object_name)
        except exceptions.NotFound:
            return True
        return False

Traceback:
Traceback (most recent call last):
File " /tempest/tempest/api/object_storage/test_container_acl.py", line 35, in tearDown
self.delete_containers()
File " /tempest/tempest/api/object_storage/base.py", line 135, in delete_containers
delete_containers(cls.containers, container_client, object_client)
File " /tempest/tempest/api/object_storage/base.py", line 51, in delete_containers
object_client.wait_for_resource_deletion(obj['name'], cont)
File " /tempest/tempest/lib/common/rest_client.py", line 905, in wait_for_resource_deletion
if self.is_resource_deleted(id, *args, **kwargs):
File " /tempest/tempest/lib/services/object_storage/object_client.py", line 28, in is_resource_deleted
self.get_object(container, object_name)
File " /tempest/tempest/lib/services/object_storage/object_client.py", line 93, in get_object
resp, body = self.get(url, headers=headers)
File " /tempest/tempest/lib/common/rest_client.py", line 314, in get
return self.request('GET', url, extra_headers, headers)
File " /tempest/tempest/lib/common/rest_client.py", line 703, in request
self._error_checker(resp, resp_body)
File " /tempest/tempest/lib/common/rest_client.py", line 824, in _error_checker
raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: b'BucketAlreadyExists'

HTTP Calls made:
2021-07-29 05:55:26,315 226 INFO [tempest.lib.common.rest_client] Request (ObjectTestACLs:tearDown): 409 GET https://xxx/swift/v1/KEY_8e25785e30aa4f1796d150c284b0fd75/aqua-35c75ad3-TestContainer-227439978/aqua-35c75ad3-Object-2104822729 0.860s
2021-07-29 05:55:26,315 226 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': ''}
        Body: None
    Response - Headers: {'date': 'Thu, 29 Jul 2021 05:55:26 GMT', 'content-type': 'text/plain; charset=utf-8', 'content-length': '19', 'connection': 'close', 'x-trans-id': 'tx0000000000000000009f9-00610242cd-8943-default', 'x-openstack-request-id': 'tx0000000000000000009f9-00610242cd-8943-default', 'accept-ranges': 'bytes', 'strict-transport-security': 'max-age=15724800; includeSubDomains', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'x-frame-options': 'deny', 'status': '409', 'content-location': 'https://xxx/swift/v1/KEY_8e25785e30aa4f1796d150c284b0fd75/aqua-35c75ad3-TestContainer-227439978/aqua-35c75ad3-Object-2104822729'}
        Body: b'BucketAlreadyExists'

To reproduce:
run the test such that the object/container is not deleted within the build_interval:
https://github.com/openstack/tempest/blob/7e96c8e854386f43604ad098a6ec7606ee676145/tempest/lib/common/rest_client.py#L917

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tempest/+/803058

[Sampat Ponnaganti (sp810x)]

Created patchset: https://review.opendev.org/c/openstack/tempest/+/803058

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tempest/+/803058
Committed: https://opendev.org/openstack/tempest/commit/717cc793dba2aa8f5da5d212f6e0118b92dcd1d3
Submitter: "Zuul (22348)"
Branch: master

commit 717cc793dba2aa8f5da5d212f6e0118b92dcd1d3
Author: Sam Kumar <email address hidden>
Date: Fri Jul 30 12:03:22 2021 -0400

    Adds exception.Conflict handler to is_resource_delete function

    Swift API get_objects returns 409 when object delete in progress.
    HTTP 409 is converted to exception.Conflict and raised
    Updated is_resource_deleted check to handle exception.Conflict
      and return False (Object is not deleted)
    If the exception is not handled the test cases fails with the error
    b'BucketAlreadyExists which is not correct status for a bucket
    which is being deleted by the tempest tearDown call.

    Closes-Bug: #1938583
    Change-Id: I6957c3ee1f3d60e7b8cd34d3db89bb27347d08b7

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tempest 28.1.0 release.