tempest.api.compute.servers.test_multiple_create.MultipleCreateTestJSON teardown bug

[James]

This teardown fails on all freshly deployed openstack versions from ussuri, victoria, wallaby, and xena tempest.api.compute.servers.test_multiple_create.MultipleCreateTestJSON due to ports that exist. Below is the specific output from the tempest attached is the full refstack log.

{0} tearDownClass (tempest.api.compute.servers.test_multiple_create.MultipleCreateTestJSON) [0.000000s] ... FAILED
Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/home/ubuntu/refstack-client/.tempest/tempest/test.py", line 236, in tearDownClass
   six.reraise(etype, value, trace)
      File "/home/ubuntu/refstack-client/.tempest/.venv/lib/python3.6/site-packages/six.py", line 719, in reraise
    raise value
      File "/home/ubuntu/refstack-client/.tempest/tempest/test.py", line 208, in tearDownClass
    teardown()
      File "/home/ubuntu/refstack-client/.tempest/tempest/test.py", line 719, in clear_credentials
    cls._creds_provider.clear_creds()
      File "/home/ubuntu/refstack-client/.tempest/tempest/lib/common/dynamic_creds.py", line 558, in clear_creds
    self._clear_isolated_net_resources()
      File "/home/ubuntu/refstack-client/.tempest/tempest/lib/common/dynamic_creds.py", line 549, in _clear_isolated_net_resources
    creds.subnet['name'])
      File "/home/ubuntu/refstack-client/.tempest/tempest/lib/common/dynamic_creds.py", line 499, in _clear_isolated_subnet
    client.delete_subnet(subnet_id)
      File "/home/ubuntu/refstack-client/.tempest/tempest/lib/services/network/subnets_client.py", line 52, in delete_subnet
    return self.delete_resource(uri)
      File "/home/ubuntu/refstack-client/.tempest/tempest/lib/services/network/base.py", line 42, in delete_resource
    resp, body = self.delete(req_uri)
      File "/home/ubuntu/refstack-client/.tempest/tempest/lib/common/rest_client.py", line 331, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
      File "/home/ubuntu/refstack-client/.tempest/tempest/lib/common/rest_client.py", line 704, in request
    self._error_checker(resp, resp_body)
      File "/home/ubuntu/refstack-client/.tempest/tempest/lib/common/rest_client.py", line 825, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)
    tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'type': 'SubnetInUse', 'message': 'Unable to complete operation on subnet 5be11612-2a84-4723-b683-3cf06e3567d4: One or more ports have an IP allocation from this subnet.', 'detail': ''}

[Martin Kopec (mkopec)]

hmm, that's weird, I haven't seen anything like that.

We run refstack-client on all stable releases since Train currently and the test is passing fine, you can see the jobs and their logs f.e. in this review - https://review.opendev.org/c/openinfra/ansible-role-refstack-client/+/826888

I wonder what could have created the ports on that particular network.

[James (james-o-benson)]

@mkopec, I'm not sure if this helps or not, but I'm deploying a pretty vanilla OpenStack-Kolla, minimal global changes. We are using some older r710's hardware. Is it potentially a timing issue? If we run the job by itself, it does tear down properly. Only when we run the full suite does it fail.

[Lukas Piwowarski (lukas-piwowarski)]

Hi James, does the test fail every time when you run the full suite? Or does it fail from time to time? I checked again the test and it seems like it passes in the refstack jobs as already pointed out by mkopec.

[James (james-o-benson)]

Hi Lukas, Yes, it fails every time. However, since we originally posted this, we have begun to test Yoga and it does NOT fail there.

Below are the versions we are testing on everything including relevant micro-versions. Please note, that the deployment method is exactly the same on all of these with only minor, non-relevant changes.

REFSTACK_TEST_VERSION = {
    "ussuri": "2020.11",
    "victoria": "2021.11",
    "wallaby": "2021.11",
    "xena": "2021.11",
    "yoga": "2021.11",
}
TEMPEST_VERSION = {
    "ussuri": "29.2.0",
    "victoria": "29.2.0",
    "wallaby": "29.1.0",
    "xena": "31.1.0",
    "yoga": "31.1.0",
}
NOVA_MIN_MICROVERSION = {
    "ussuri": "2.1",
    "victoria": "2.1",
    "wallaby": "2.1",
    "xena": "2.1",
    "yoga": "2.1",
}
NOVA_MAX_MICROVERSION = {
    "ussuri": "2.87",
    "victoria": "2.87",
    "wallaby": "2.88",
    "xena": "2.90",
    "yoga": "2.92",
}

[Lukas Piwowarski (lukas-piwowarski)]

Hi James,

I'm sorry for taking me this long to respond. I can confirm that this bug is valid as I was able to reproduce the issue running the tempest.api.compute.servers.test_multiple_create.MultipleCreateTestJSON.test_multiple_create [1] test. I think that the issue is caused by this line [2] (the servers are deleted using the servers_client.delete_server which does not wait for the deletion of the servers). I will propose a patch to tempest that should fix this and I will probably propose a patch that flags the test in refstack for now. We will see what others think about this during the review.

[1] https://opendev.org/openstack/tempest/src/commit/cd354d8ce3745a96116d534c1428742c605c61e0/tempest/api/compute/servers/test_multiple_create.py#L26
[2] https://opendev.org/openstack/tempest/src/commit/cd354d8ce3745a96116d534c1428742c605c61e0/tempest/api/compute/servers/test_multiple_create.py#L38

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tempest/+/875751

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tempest/+/875751
Committed: https://opendev.org/openstack/tempest/commit/d78e405364ca3725820fc7bd4b2f2af198ccc624
Submitter: "Zuul (22348)"
Branch: master

commit d78e405364ca3725820fc7bd4b2f2af198ccc624
Author: Lukas Piwowarski <email address hidden>
Date: Tue Feb 28 15:11:43 2023 +0000

    Fix server cleanup in test_multiple_create test

    The above-mentioned test uses ServersClient.delete_server function to
    delete servers. However, this function does not wait for the successful
    deletion of the servers which sometimes leads to an error (see linked
    bug).

    This patch changes the test so that it uses
    BaseV2ComputeTest.delete_server function which waits for the successful
    deletion of the server.

    Closes-Bug: #1954535
    Change-Id: Ic1ed48fda03ce3c13198d2dd299b57d68b06e63d

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tempest 34.1.0 release.

[Martin Kopec (mkopec)]

we've just bumped the tempest version refstack-client uses - https://review.opendev.org/c/openinfra/refstack-client/+/877374 - that tempest version contains the needed fix for this issues. Therefore I'm gonna close this as solved.