Random failure of test test_aggregate_basic_ops with host already has 1 instance(s). Changing the AZ of an existing instance is not supported by this action

[yatin]

Failing as:-
2025-02-05 09:06:26.660305 | controller | Traceback (most recent call last):
2025-02-05 09:06:26.660311 | controller |
2025-02-05 09:06:26.660315 | controller | File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 65, in wrapper
2025-02-05 09:06:26.660326 | controller | return f(*func_args, **func_kwargs)
2025-02-05 09:06:26.660330 | controller |
2025-02-05 09:06:26.660334 | controller | File "/opt/stack/tempest/tempest/serial_tests/scenario/test_aggregates_basic_ops.py", line 134, in test_aggregate_basic_ops
2025-02-05 09:06:26.660338 | controller | self._add_host(aggregate['id'], host)
2025-02-05 09:06:26.660342 | controller |
2025-02-05 09:06:26.660346 | controller | File "/opt/stack/tempest/tempest/serial_tests/scenario/test_aggregates_basic_ops.py", line 81, in _add_host
2025-02-05 09:06:26.660350 | controller | aggregate = (self.aggregates_client.add_host(aggregate_id, host=host)
2025-02-05 09:06:26.660354 | controller |
2025-02-05 09:06:26.660358 | controller | File "/opt/stack/tempest/tempest/lib/services/compute/aggregates_client.py", line 106, in add_host
2025-02-05 09:06:26.660362 | controller | resp, body = self.post('os-aggregates/%s/action' % aggregate_id,
2025-02-05 09:06:26.660366 | controller |
2025-02-05 09:06:26.660370 | controller | File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 314, in post
2025-02-05 09:06:26.660373 | controller | resp_header, resp_body = self.request(
2025-02-05 09:06:26.660377 | controller |
2025-02-05 09:06:26.660381 | controller | File "/opt/stack/tempest/tempest/lib/services/compute/base_compute_client.py", line 47, in request
2025-02-05 09:06:26.660386 | controller | resp, resp_body = super(BaseComputeClient, self).request(
2025-02-05 09:06:26.660390 | controller |
2025-02-05 09:06:26.660393 | controller | File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 762, in request
2025-02-05 09:06:26.660397 | controller | self._error_checker(resp, resp_body)
2025-02-05 09:06:26.660401 | controller |
2025-02-05 09:06:26.660405 | controller | File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 892, in _error_checker
2025-02-05 09:06:26.660409 | controller | raise exceptions.Conflict(resp_body, resp=resp)
2025-02-05 09:06:26.660412 | controller |
2025-02-05 09:06:26.660416 | controller | tempest.lib.exceptions.Conflict: Conflict with state of target resource
2025-02-05 09:06:26.660433 | controller | Details: {'code': 409, 'message': "Cannot add host to aggregate 19. Reason: The host cannot be added to the aggregate as the availability zone of the host would change from 'None' to 'foo_zone' but the host already has 1 instance(s). Changing the AZ of an existing instance is not supported by this action. Move the instances away from this host then try again. If you need to move the instances between AZs then you can use shelve_offload and unshelve to achieve this.."}

Example failures:
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8e5/940342/6/check/neutron-ovs-tempest-dvr-ha-multinode-full/8e5d67e/testr_results.html
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_224/934241/6/check/neutron-ovs-tempest-dvr-ha-multinode-full/224ee14/testr_results.html
https://23ab8c1ad94c4f995cfb-353899952d3f58d374107c0e68cc7036.ssl.cf1.rackcdn.com/940474/1/check/neutron-ovs-tempest-dvr-ha-multinode-full/3751a44/testr_results.html
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8e1/939321/21/check/neutron-ovs-tempest-dvr-ha-multinode-full/8e1f075/testr_results.html
https://83bc65af26be6709ddc0-e84faa2b407329d2b0da24e899449f42.ssl.cf1.rackcdn.com/939117/3/check/neutron-ovs-tempest-multinode-full/a20575d/testr_results.html

The tests are running serially but the previous tests not waiting for instance Delete[1] so this test fails as expect no instance to be running. I see there is a method[2] in base class which also wait for server delete, using that should resolve it.

[1] https://opendev.org/openstack/tempest/src/branch/master/tempest/serial_tests/api/compute/admin/test_server_affinity.py#L74
[2] https://opendev.org/openstack/tempest/src/branch/master/tempest/api/compute/base.py#L431-L438

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tempest/+/941077

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tempest/+/941077
Committed: https://opendev.org/openstack/tempest/commit/3d7b364ce4fd230b3e5b4793d51d30e40151f7cd
Submitter: "Zuul (22348)"
Branch: master

commit 3d7b364ce4fd230b3e5b4793d51d30e40151f7cd
Author: yatinkarel <email address hidden>
Date: Mon Feb 10 12:17:28 2025 +0530

    Wait for server termination in server affinity tests

    These tests were not waiting for server termination and
    leading to failures in another serial tests which expects
    no server running on the compute nodes.
    Using method from parent class which already waits for
    server deletion.

    Closes-Bug: #2097594
    Change-Id: Iea6ea61c5b827be705777cca5b972a5441a69540

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tempest 43.0.0 Epoxy release.