tempest

Tempest tests failed randomly in tempest.api.object_storage.test_container_services.ContainerTest

Bug #1938593 reported by Huy Tran
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tempest
Fix Released
Medium
Huy Tran

Bug Description

Container names for created containers by tests in the class are kept in the internal list 'containers'. The container names are not being removed from the list when associated containers are deleted. This causes extra delete requests for same containers in tearDown of all subsequent tests/methods. Possible race condition can also caused by this issue. Changes have been proposed via https://review.opendev.org/c/openstack/tempest/+/802388.

In following logs, it can be seen test_create_container_overwrite is deleting container (aqua-ea4b5c86-TestContainer-897342407) which was created by test_create_container. Such delete requests grow exponentially by the time the last test in the class is executed.

FAIL 0:00:02.805269 tempest.api.object_storage.test_container_services.ContainerTest.test_create_container[id-92139d73-7819-4db1-85f8-3f2f22a8d91f,smoke]
KibanaLog(Right click and Open in New Tab)
Traceback:
Traceback (most recent call last):
  File "/home/aqua/tempest/tempest/api/object_storage/test_container_services.py", line 25, in tearDown
    self.delete_containers()
  File "/home/aqua/tempest/tempest/api/object_storage/base.py", line 135, in delete_containers
    delete_containers(cls.containers, container_client, object_client)
  File "/home/aqua/tempest/tempest/api/object_storage/base.py", line 53, in delete_containers
    container_client.delete_container(cont)
  File "/home/aqua/tempest/tempest/lib/services/object_storage/container_client.py", line 57, in delete_container
    resp, body = self.delete(url)
  File "/home/aqua/tempest/tempest/lib/common/rest_client.py", line 330, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/home/aqua/tempest/tempest/lib/common/rest_client.py", line 703, in request
    self._error_checker(resp, resp_body)
  File "/home/aqua/tempest/tempest/lib/common/rest_client.py", line 824, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'Code': 'BucketAlreadyExists', 'RequestId': 'tx0000000000000000002e1-0060fa2dfa-4e48-default', 'HostId': '4e48-default-default'}
Logging:
2021-07-23 02:48:25,118 222 INFO [tempest.lib.common.rest_client] Request (ContainerTest:test_create_container): 201 POST https://identity-somewhere.com/v3/auth/tokens 0.640s
2021-07-23 02:48:25,118 222 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json'}
        Body:
    Response - Headers: {'date': 'Fri, 23 Jul 2021 02:48:25 GMT', 'content-type': 'application/json', 'content-length': '8152', 'connection': 'close', 'x-subject-token': '', 'x-content-type-options': 'nosniff', 'strict-transport-security': 'max-age=15724800; includeSubDomains', 'content-security-policy': "script-src 'self'; object-src 'self'", 'x-frame-options': 'deny', 'x-permitted-cross-domain-policies': 'none', 'x-xss-protection': '1; mode=block', 'vary': 'X-Auth-Token', 'x-openstack-request-id': 'req-f2400050-4ad8-46ea-a310-4755ac3533e3', 'status': '201', 'content-location': 'https://identity-somewhere.com/v3/auth/tokens'}
        Body: b'{"token": {"methods": ["password"], "user": {"domain": {"id": "ae6e834987fa4fd98160a0a8bc03359d", "name": "qa"}, "id": "1a34677a9247465fb2a321a364ad0f2e", "name": "pinocchio", "password_expires_at": null}, "audit_ids": ["-Z4DzNXfShWx_R2QfkwfZw"], "expires_at": "2021-07-24T10:48:25.000000Z", "issued_at": "2021-07-23T02:48:25.000000Z", "project": {"domain": {"id": "ae6e834987fa4fd98160a0a8bc03359d", "name": "qa"}, "id": "6287d937b3254bb681ce7f84671ec31c", "name": "z-aqua-ea4b5c86_NONADMIN2_4_pinocchio"}, "is_domain": false, "roles": [{"id": "f47a160c756c4528b6e2703a9eee220f", "name": "member"}, {"id": "f6423146158243cd8405cf80a47530ea", "name": "reader"}, {"id": "e04874a87b48428bafd2cb3e31fcbcf2", "name": "admin"}, {"id": "a0bb72c2879742a09137de2d022c388c", "name": "heat_stack_owner"}], "catalog": [{"endpoints": [{"id": "119e07f9449948169ef856b6b4fa2e08", "interface": "admin", "region_id": "mtn57c", "url": "https://neutron-server.openstack.svc.cluster.local:9696/", "region": "mtn57c"}, {"id": "9fce7a802b724416b68dd17835f49216", "interface": "internal", "region_id": "mtn57c", "url": "https://neutron-server.openstack.svc.cluster.local:9696/", "region": "mtn57c"}, {"id": "a1b01b9fa5744435a726fc3f67f66cef", "interface": "public", "region_id": "mtn57c", "url": "https://network-somewhere.com/", "region": "mtn57c"}], "id": "2b108410f27542a5a9347a5a2d15f485", "type": "network", "name": "neutron"}, {"endpoints": [{"id": "037d5a67f6aa453586f8b7496d29343e", "interface": "admin", "region_id": "mtn57c", "url": "https://cinder-api.openstack.svc.cluster.local:8776/v3/6287d937b3254bb681ce7f84671ec31c", "region": "mtn57c"}, {"id": "8cf6521af14e44b399e66f77c672d0fb", "interface": "internal", "region_id": "mtn57c", "url": "https://cinder-api.openstack.svc.cluster.local:8776/v3/6287d937b3254bb681ce7f84671ec31c", "region": "mtn57c"}, {"id": "a96b4944e04342fda08e47f8f83899cd", "interface": "public", "region_id": "mtn57c", "url": "https://volume-somewhere.com/v3/6287d937b3254bb681ce7f84671ec31c", "region": "mtn57c"}], "id": "3ca6e000ada44fd5a013adf9b4a59790", "type": "volumev3", "name": "cinderv3"}, {"endpoints": [{"id": "6167db1d702b46599092843b34b24670", "interface": "public", "region_id": "mtn57c", "url": "https://compute-somewhere.com/v2.1/6287d937b3254bb681ce7f84671ec31c", "region": "mtn57c"}, {"id": "9e97ffb0c2da42b8b382ce3425c133dc", "interface": "internal", "region_id": "mtn57c", "url": "https://nova-api.openstack.svc.cluster.local:8774/v2.1/6287d937b3254bb681ce7f84671ec31c", "region": "mtn57c"}, {"id": "ef9b2f418bb64894b16026a2cf9166b1", "interface": "admin", "region_id": "mtn57c", "url": "https://nova-api.openstack.svc.cluster.local:8774/v2.1/6287d937b3254bb681ce7f84671ec31c", "region": "mtn57c"}], "id": "74a9e266859847f4aac852f86e22d5b2", "type": "compute", "name": "nova"}, {"endpoints": [{"id": "2855c818f0e8420899998088086dc1d6", "interface": "public", "region_id": "mtn57c", "url": "https://volume-somewhere.com/v2/6287d937b3254bb681ce7f84671ec31c", "region": "mtn57c"}, {"id": "4c4769a1cb8f4e8a9c26ae2a65ff24a1", "interface": "admin", "region_id": "mtn57c", "url": "https://cinder-api.openstack.svc.cluster.local:8776/v2/6287d937b3254bb681ce7f84671ec31c", "region": "mtn57c"}, {"id": "a3f40b28d4544bfeb743c0414c40ee80", "interface": "internal", "region_id": "mtn57c", "url": "https://cinder-api.openstack.svc.cluster.local:8776/v2/6287d937b3254bb681ce7f84671ec31c", "region": "mtn57c"}], "id": "853847505ea74849b34c48d91efe6211", "type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"id": "5701869052fa420ca5500b9dbb3b2b4d", "interface": "internal", "region_id": "mtn57c", "url": "http://ceph-rgw.openstack.svc.cluster.local:8088/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c", "region": "mtn57c"}, {"id": "76559585162b41ca8161d1f3c095b054", "interface": "admin", "region_id": "mtn57c", "url": "http://ceph-rgw.openstack.svc.cluster.local:8088/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c", "region": "mtn57c"}, {"id": "d90bdb26547144298c8ca25ab1685bc5", "interface": "public", "region_id": "mtn57c", "url": "https:/
2021-07-23 02:48:25,870 222 INFO [tempest.lib.common.rest_client] Request (ContainerTest:test_create_container): 201 PUT https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-897342407 0.751s
2021-07-23 02:48:25,870 222 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': ''}
        Body: None
    Response - Headers: {'date': 'Fri, 23 Jul 2021 02:48:25 GMT', 'content-type': 'text/plain; charset=utf-8', 'content-length': '0', 'connection': 'close', 'x-trans-id': 'tx000000000000000000302-0060fa2df9-4dea-default', 'x-openstack-request-id': 'tx000000000000000000302-0060fa2df9-4dea-default', 'accept-ranges': 'bytes', 'strict-transport-security': 'max-age=15724800; includeSubDomains', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'x-frame-options': 'deny', 'status': '201', 'content-location': 'https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-897342407'}
        Body: b''
2021-07-23 02:48:26,549 222 INFO [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 200 GET https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-897342407?&limit=9999&format=json 0.679s
2021-07-23 02:48:26,549 222 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': ''}
        Body: None
    Response - Headers: {'date': 'Fri, 23 Jul 2021 02:48:26 GMT', 'content-type': 'application/json; charset=utf-8', 'content-length': '2', 'connection': 'close', 'x-timestamp': '1627008505.80754', 'x-container-object-count': '0', 'x-container-bytes-used': '0', 'x-container-bytes-used-actual': '0', 'x-storage-policy': 'default-placement', 'x-storage-class': 'STANDARD', 'last-modified': 'Fri, 23 Jul 2021 02:48:25 GMT', 'x-trans-id': 'tx0000000000000000002fb-0060fa2dfa-4ec8-default', 'x-openstack-request-id': 'tx0000000000000000002fb-0060fa2dfa-4ec8-default', 'accept-ranges': 'bytes', 'strict-transport-security': 'max-age=15724800; includeSubDomains', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'x-frame-options': 'deny', 'status': '200', 'content-location': 'https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-897342407?&limit=9999&format=json'}
        Body: b'[]'
2021-07-23 02:48:27,281 222 INFO [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 409 DELETE https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-897342407 0.731s
2021-07-23 02:48:27,281 222 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': ''}
        Body: None
    Response - Headers: {'date': 'Fri, 23 Jul 2021 02:48:27 GMT', 'content-type': 'application/json; charset=utf-8', 'content-length': '124', 'connection': 'close', 'x-trans-id': 'tx0000000000000000002e1-0060fa2dfa-4e48-default', 'x-openstack-request-id': 'tx0000000000000000002e1-0060fa2dfa-4e48-default', 'accept-ranges': 'bytes', 'strict-transport-security': 'max-age=15724800; includeSubDomains', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'x-frame-options': 'deny', 'status': '409', 'content-location': 'https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-897342407'}
        Body: b'{"Code":"BucketAlreadyExists","RequestId":"tx0000000000000000002e1-0060fa2dfa-4e48-default","HostId":"4e48-default-default"}'

===============
SUCCESS 0:00:06.279329 tempest.api.object_storage.test_container_services.ContainerTest.test_create_container_overwrite[id-49f866ed-d6af-4395-93e7-4187eb56d322]
KibanaLog(Right click and Open in New Tab)
Logging:
2021-07-23 02:48:28,298 222 INFO [tempest.lib.common.rest_client] Request (ContainerTest:test_create_container_overwrite): 201 PUT https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-1988003183 1.014s
2021-07-23 02:48:28,298 222 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': ''}
        Body: None
    Response - Headers: {'date': 'Fri, 23 Jul 2021 02:48:28 GMT', 'content-type': 'text/plain; charset=utf-8', 'content-length': '0', 'connection': 'close', 'x-trans-id': 'tx0000000000000000002e2-0060fa2dfb-4e48-default', 'x-openstack-request-id': 'tx0000000000000000002e2-0060fa2dfb-4e48-default', 'accept-ranges': 'bytes', 'strict-transport-security': 'max-age=15724800; includeSubDomains', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'x-frame-options': 'deny', 'status': '201', 'content-location': 'https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-1988003183'}
        Body: b''
2021-07-23 02:48:29,142 222 INFO [tempest.lib.common.rest_client] Request (ContainerTest:test_create_container_overwrite): 202 PUT https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-1988003183 0.844s
2021-07-23 02:48:29,143 222 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': ''}
        Body: None
    Response - Headers: {'date': 'Fri, 23 Jul 2021 02:48:29 GMT', 'content-type': 'text/plain; charset=utf-8', 'content-length': '0', 'connection': 'close', 'x-trans-id': 'tx000000000000000000300-0060fa2dfc-4ec8-default', 'x-openstack-request-id': 'tx000000000000000000300-0060fa2dfc-4ec8-default', 'accept-ranges': 'bytes', 'strict-transport-security': 'max-age=15724800; includeSubDomains', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'x-frame-options': 'deny', 'status': '202', 'content-location': 'https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-1988003183'}
        Body: b''
2021-07-23 02:48:29,825 222 INFO [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 200 GET https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-897342407?&limit=9999&format=json 0.682s
2021-07-23 02:48:29,825 222 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': ''}
        Body: None
    Response - Headers: {'date': 'Fri, 23 Jul 2021 02:48:29 GMT', 'content-type': 'application/json; charset=utf-8', 'content-length': '2', 'connection': 'close', 'x-timestamp': '1627008505.80754', 'x-container-object-count': '0', 'x-container-bytes-used': '0', 'x-container-bytes-used-actual': '0', 'x-storage-policy': 'default-placement', 'x-storage-class': 'STANDARD', 'last-modified': 'Fri, 23 Jul 2021 02:48:25 GMT', 'x-trans-id': 'tx000000000000000000306-0060fa2dfd-4dea-default', 'x-openstack-request-id': 'tx000000000000000000306-0060fa2dfd-4dea-default', 'accept-ranges': 'bytes', 'strict-transport-security': 'max-age=15724800; includeSubDomains', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'x-frame-options': 'deny', 'status': '200', 'content-location': 'https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-897342407?&limit=9999&format=json'}
        Body: b'[]'
2021-07-23 02:48:30,547 222 INFO [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 204 DELETE https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-897342407 0.721s
2021-07-23 02:48:30,547 222 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': ''}
        Body: None
    Response - Headers: {'date': 'Fri, 23 Jul 2021 02:48:30 GMT', 'content-type': 'application/json; charset=utf-8', 'connection': 'close', 'x-trans-id': 'tx0000000000000000002e3-0060fa2dfe-4e48-default', 'x-openstack-request-id': 'tx0000000000000000002e3-0060fa2dfe-4e48-default', 'accept-ranges': 'bytes', 'strict-transport-security': 'max-age=15724800; includeSubDomains', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'x-frame-options': 'deny', 'status': '204', 'content-location': 'https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-897342407'}
        Body: b''
2021-07-23 02:48:31,230 222 INFO [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 404 HEAD https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-897342407 0.682s
2021-07-23 02:48:31,230 222 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': ''}
        Body: None
    Response - Headers: {'date': 'Fri, 23 Jul 2021 02:48:31 GMT', 'content-type': 'application/json; charset=utf-8', 'content-length': '170', 'connection': 'close', 'x-trans-id': 'tx0000000000000000002e4-0060fa2dfe-4e48-default', 'x-openstack-request-id': 'tx0000000000000000002e4-0060fa2dfe-4e48-default', 'accept-ranges': 'bytes', 'strict-transport-security': 'max-age=15724800; includeSubDomains', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'x-frame-options': 'deny', 'status': '404', 'content-location': 'https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-897342407'}
        Body: b''
2021-07-23 02:48:32,046 222 INFO [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 200 GET https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-1988003183?&limit=9999&format=json 0.816s
2021-07-23 02:48:32,047 222 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': ''}
        Body: None
    Response - Headers: {'date': 'Fri, 23 Jul 2021 02:48:32 GMT', 'content-type': 'application/json; charset=utf-8', 'content-length': '2', 'connection': 'close', 'x-timestamp': '1627008508.24041', 'x-container-object-count': '0', 'x-container-bytes-used': '0', 'x-container-bytes-used-actual': '0', 'x-storage-policy': 'default-placement', 'x-storage-class': 'STANDARD', 'last-modified': 'Fri, 23 Jul 2021 02:48:29 GMT', 'x-trans-id': 'tx000000000000000000308-0060fa2dff-4dea-default', 'x-openstack-request-id': 'tx000000000000000000308-0060fa2dff-4dea-default', 'accept-ranges': 'bytes', 'strict-transport-security': 'max-age=15724800; includeSubDomains', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'x-frame-options': 'deny', 'status': '200', 'content-location': 'https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-1988003183?&limit=9999&format=json'}
        Body: b'[]'
2021-07-23 02:48:32,888 222 INFO [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 204 DELETE https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-1988003183 0.841s
2021-07-23 02:48:32,888 222 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': ''}
        Body: None
    Response - Headers: {'date': 'Fri, 23 Jul 2021 02:48:32 GMT', 'content-type': 'application/json; charset=utf-8', 'connection': 'close', 'x-trans-id': 'tx000000000000000000308-0060fa2e00-4ec8-default', 'x-openstack-request-id': 'tx000000000000000000308-0060fa2e00-4ec8-default', 'accept-ranges': 'bytes', 'strict-transport-security': 'max-age=15724800; includeSubDomains', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'x-frame-options': 'deny', 'status': '204', 'content-location': 'https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-1988003183'}
        Body: b''
2021-07-23 02:48:33,562 222 INFO [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 404 HEAD https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-1988003183 0.673s
2021-07-23 02:48:33,562 222 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': ''}
        Body: None
    Response - Headers: {'date': 'Fri, 23 Jul 2021 02:48:33 GMT', 'content-type': 'application/json; charset=utf-8', 'content-length': '171', 'connection': 'close', 'x-trans-id': 'tx00000000000000000030a-0060fa2e01-4ec8-default', 'x-openstack-request-id': 'tx00000000000000000030a-0060fa2e01-4ec8-default', 'accept-ranges': 'bytes', 'strict-transport-security': 'max-age=15724800; includeSubDomains', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'x-frame-options': 'deny', 'status': '404', 'content-location': 'https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-1988003183'}
        Body: b''

==================
SUCCESS 0:00:05.271855 tempest.api.object_storage.test_container_services.ContainerTest.test_create_container_with_metadata_key[id-c2ac4d59-d0f5-40d5-ba19-0635056d48cd]
KibanaLog(Right click and Open in New Tab)
Logging:
2021-07-23 02:48:34,259 222 INFO [tempest.lib.common.rest_client] Request (ContainerTest:test_create_container_with_metadata_key): 201 PUT https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-805112835 0.696s
2021-07-23 02:48:34,259 222 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Container-Meta-test-container-meta': '', 'X-Auth-Token': ''}
        Body: None
    Response - Headers: {'date': 'Fri, 23 Jul 2021 02:48:34 GMT', 'content-type': 'text/plain; charset=utf-8', 'content-length': '0', 'connection': 'close', 'x-trans-id': 'tx00000000000000000030c-0060fa2e01-4dea-default', 'x-openstack-request-id': 'tx00000000000000000030c-0060fa2e01-4dea-default', 'accept-ranges': 'bytes', 'strict-transport-security': 'max-age=15724800; includeSubDomains', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'x-frame-options': 'deny', 'status': '201', 'content-location': 'https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-805112835'}
        Body: b''
2021-07-23 02:48:34,948 222 INFO [tempest.lib.common.rest_client] Request (ContainerTest:test_create_container_with_metadata_key): 204 HEAD https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-805112835 0.688s
2021-07-23 02:48:34,948 222 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': ''}
        Body: None
    Response - Headers: {'date': 'Fri, 23 Jul 2021 02:48:34 GMT', 'content-type': 'application/json; charset=utf-8', 'connection': 'close', 'x-timestamp': '1627008514.20251', 'x-container-object-count': '0', 'x-container-bytes-used': '0', 'x-container-bytes-used-actual': '0', 'x-storage-policy': 'default-placement', 'x-storage-class': 'STANDARD', 'last-modified': 'Fri, 23 Jul 2021 02:48:34 GMT', 'x-trans-id': 'tx00000000000000000030d-0060fa2e02-4ec8-default', 'x-openstack-request-id': 'tx00000000000000000030d-0060fa2e02-4ec8-default', 'accept-ranges': 'bytes', 'strict-transport-security': 'max-age=15724800; includeSubDomains', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'x-frame-options': 'deny', 'status': '204', 'content-location': 'https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-805112835'}
        Body: b''
2021-07-23 02:48:35,634 222 INFO [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 404 GET https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-897342407?&limit=9999&format=json 0.685s
2021-07-23 02:48:35,635 222 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': ''}
        Body: None
    Response - Headers: {'date': 'Fri, 23 Jul 2021 02:48:35 GMT', 'content-type': 'application/json; charset=utf-8', 'content-length': '170', 'connection': 'close', 'x-trans-id': 'tx00000000000000000030f-0060fa2e03-4ec8-default', 'x-openstack-request-id': 'tx00000000000000000030f-0060fa2e03-4ec8-default', 'accept-ranges': 'bytes', 'strict-transport-security': 'max-age=15724800; includeSubDomains', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'x-frame-options': 'deny', 'status': '404', 'content-location': 'https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-897342407?&limit=9999&format=json'}
        Body: b'{"Code":"NoSuchBucket","BucketName":"aqua-ea4b5c86-TestContainer-897342407","RequestId":"tx00000000000000000030f-0060fa2e03-4ec8-default","HostId":"4ec8-default-default"}'
2021-07-23 02:48:36,338 222 INFO [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 404 GET https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-1988003183?&limit=9999&format=json 0.703s
2021-07-23 02:48:36,338 222 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': ''}
        Body: None
    Response - Headers: {'date': 'Fri, 23 Jul 2021 02:48:36 GMT', 'content-type': 'application/json; charset=utf-8', 'content-length': '171', 'connection': 'close', 'x-trans-id': 'tx00000000000000000030e-0060fa2e03-4dea-default', 'x-openstack-request-id': 'tx00000000000000000030e-0060fa2e03-4dea-default', 'accept-ranges': 'bytes', 'strict-transport-security': 'max-age=15724800; includeSubDomains', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'x-frame-options': 'deny', 'status': '404', 'content-location': 'https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-1988003183?&limit=9999&format=json'}
        Body: b'{"Code":"NoSuchBucket","BucketName":"aqua-ea4b5c86-TestContainer-1988003183","RequestId":"tx00000000000000000030e-0060fa2e03-4dea-default","HostId":"4dea-default-default"}'
2021-07-23 02:48:37,281 222 INFO [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 200 GET https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-805112835?&limit=9999&format=json 0.942s
2021-07-23 02:48:37,281 222 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': ''}
        Body: None
    Response - Headers: {'date': 'Fri, 23 Jul 2021 02:48:37 GMT', 'content-type': 'application/json; charset=utf-8', 'content-length': '2', 'connection': 'close', 'x-timestamp': '1627008514.20251', 'x-container-object-count': '0', 'x-container-bytes-used': '0', 'x-container-bytes-used-actual': '0', 'x-storage-policy': 'default-placement', 'x-storage-class': 'STANDARD', 'last-modified': 'Fri, 23 Jul 2021 02:48:34 GMT', 'x-trans-id': 'tx000000000000000000313-0060fa2e04-4ec8-default', 'x-openstack-request-id': 'tx000000000000000000313-0060fa2e04-4ec8-default', 'accept-ranges': 'bytes', 'strict-transport-security': 'max-age=15724800; includeSubDomains', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'x-frame-options': 'deny', 'status': '200', 'content-location': 'https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-805112835?&limit=9999&format=json'}
        Body: b'[]'
2021-07-23 02:48:38,000 222 INFO [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 204 DELETE https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-805112835 0.718s
2021-07-23 02:48:38,000 222 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': ''}
        Body: None
    Response - Headers: {'date': 'Fri, 23 Jul 2021 02:48:37 GMT', 'content-type': 'application/json; charset=utf-8', 'connection': 'close', 'x-trans-id': 'tx000000000000000000315-0060fa2e05-4ec8-default', 'x-openstack-request-id': 'tx000000000000000000315-0060fa2e05-4ec8-default', 'accept-ranges': 'bytes', 'strict-transport-security': 'max-age=15724800; includeSubDomains', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'x-frame-options': 'deny', 'status': '204', 'content-location': 'https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-805112835'}
        Body: b''
2021-07-23 02:48:38,834 222 INFO [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 404 HEAD https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-805112835 0.833s
2021-07-23 02:48:38,834 222 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': ''}
        Body: None
    Response - Headers: {'date': 'Fri, 23 Jul 2021 02:48:38 GMT', 'content-type': 'application/json; charset=utf-8', 'content-length': '170', 'connection': 'close', 'x-trans-id': 'tx0000000000000000002e9-0060fa2e06-4e48-default', 'x-openstack-request-id': 'tx0000000000000000002e9-0060fa2e06-4e48-default', 'accept-ranges': 'bytes', 'strict-transport-security': 'max-age=15724800; includeSubDomains', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'x-frame-options': 'deny', 'status': '404', 'content-location': 'https://object-store-somewhere.com/swift/v1/KEY_6287d937b3254bb681ce7f84671ec31c/aqua-ea4b5c86-TestContainer-805112835'}
        Body: b''

Revision history for this message
Huy Tran (ht095u) wrote :
OpenStack Infra (hudson-openstack)
Changed in tempest:
status: New → In Progress
Revision history for this message
Ghanshyam Mann (ghanshyammann) wrote :

I understand now. so basically we are not deleting the created container from cls.containers list and end up try to delete all the containers created by self.create_containers() method or any test adding created container in cls.containers list.

I will say we should cleanup the list in class method delete_containers().

Changed in tempest:
status: In Progress → Confirmed
importance: Undecided → Medium
Revision history for this message
Huy Tran (ht095u) wrote :
Download full text (19.7 KiB)

As mentioned in IRC channel, clearing the container names list after delete_containers() in method delete_containers won't solve the race condition. Here are the log to show tearDown for second method/test can delete the container which was created by first test/method.

FAIL 0:00:03.006318 tempest.api.object_storage.test_container_services.ContainerTest.test_create_container[id-92139d73-7819-4db1-85f8-3f2f22a8d91f,smoke]
KibanaLog(Right click and Open in New Tab)
Traceback:
Traceback (most recent call last):
  File "/home/aqua/tempest/tempest/api/object_storage/test_container_services.py", line 25, in tearDown
    self.delete_containers()
  File "/home/aqua/tempest/tempest/api/object_storage/base.py", line 137, in delete_containers
    delete_containers(cls.containers, container_client, object_client)
  File "/home/aqua/tempest/tempest/api/object_storage/base.py", line 53, in delete_containers
    container_client.delete_container(cont)
  File "/home/aqua/tempest/tempest/lib/services/object_storage/container_client.py", line 57, in delete_container
    resp, body = self.delete(url)
  File "/home/aqua/tempest/tempest/lib/common/rest_client.py", line 330, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/home/aqua/tempest/tempest/lib/common/rest_client.py", line 703, in request
    self._error_checker(resp, resp_body)
  File "/home/aqua/tempest/tempest/lib/common/rest_client.py", line 824, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'Code': 'BucketAlreadyExists', 'RequestId': 'tx000000000000000008bfe-00610954ef-2e51-default', 'HostId': '2e51-default-default'}
Logging:
2021-08-03 14:38:37,077 214 INFO [tempest.lib.common.rest_client] Request (ContainerTest:test_create_container): 201 POST https://identity-nc.mtn52b.somewhere.com/v3/auth/tokens 0.672s
2021-08-03 14:38:37,077 214 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json'}
        Body:
    Response - Headers: {'date': 'Tue, 03 Aug 2021 14:38:37 GMT', 'content-type': 'application/json', 'content-length': '8154', 'connection': 'close', 'x-subject-token': '', 'x-content-type-options': 'nosniff', 'strict-transport-security': 'max-age=15724800; includeSubDomains', 'content-security-policy': "script-src 'self'; object-src 'self'", 'x-frame-options': 'deny', 'x-permitted-cross-domain-policies': 'none', 'x-xss-protection': '1; mode=block', 'vary': 'X-Auth-Token', 'x-openstack-request-id': 'req-ece7db31-b7d4-480d-8f47-b72738e7943e', 'status': '201', 'content-location': 'https://identity-nc.mtn52b.somewhere.com/v3/auth/tokens'}
        Body: b'{"token": {"methods": ["password"], "user": {"domain": {"id": "c47b968450f84a17bc63a40128461fa7", "name": "qa"}, "id": "1887aaeb88644628b13fa9b23576f957", "name": "pinocchio", "password_expires_at": null}, "audit_ids": ["-nszMgoeRK29h4HREPGsXA"], "expires_at": "2021-08-04T22:38:37.000000Z", "issued_at": "2021-08-03T14:38:37.000000Z", "project": {"domain": {"id": "c47b968450f84a17bc63a40128461fa7", "name": "qa"}, "id": "95453694e84d40...

OpenStack Infra (hudson-openstack)
Changed in tempest:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tempest (master)

Reviewed: https://review.opendev.org/c/openstack/tempest/+/802388
Committed: https://opendev.org/openstack/tempest/commit/3c7159d2e3974f74547969f45a317e2d2b47d678
Submitter: "Zuul (22348)"
Branch: master

commit 3c7159d2e3974f74547969f45a317e2d2b47d678
Author: Huy Tran <email address hidden>
Date: Mon Jul 26 13:49:53 2021 -0500

    Changes to ensure delete_containers is done at class level tear down

      Invoking delete containers at test level tearDown caused:
    1) unnecessary requests are being sent again in the tearDown for
    the subsequent methods,
    2) possible race condition can happen between methods during the
    tearDown.

    Closes-Bug: #1938593

    Change-Id: I24087607a9f0129b7b3d21473196fab1b5f08cd4

Changed in tempest:
status: In Progress → Fix Released
Huy Tran (ht095u)
Changed in tempest:
assignee: nobody → Huy Tran (ht095u)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tempest 28.1.0

This issue was fixed in the openstack/tempest 28.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.