[tempest] VM ports have status=DOWN when calling ``TestNetworkBasicOps._setup_network_and_servers``

[Rodolfo Alonso]

The method ``TestNetworkBasicOps._setup_network_and_servers`` is used in several tempest tests. It creates a set of resources (network, servers, FIPs, etc). This method has a race condition when the config option "project_networks_reachable" if False (by default).

The server is created [1] but there is no connectivity test [2] (due to project_networks_reachable=False). The next step is to create a FIP [3]. Because we are not passing the port_id, we first retrieve all the VM ports [4]. The issue happens at [5]: the ports are created but are still down.

An example of this can be seen in [5][6]:
1) The tempest test list the VM ports (only one in this case) but the port is down: https://paste.opendev.org/show/bSLi4joS6blqipbwa7Pq/

2) The Neutron API finishes processing the port activation at the same time the port list call was made: https://paste.opendev.org/show/brRqntkQYdDoVeEqCeXF/

It is needed to add an active wait in the method ``get_server_port_id_and_ip4`` in order to wait all ports to be active.

[1]https://github.com/openstack/tempest/blob/0a0e1070e573674332cb5126064b95f17099307e/tempest/scenario/test_network_basic_ops.py#L120
[2]https://github.com/openstack/tempest/blob/0a0e1070e573674332cb5126064b95f17099307e/tempest/scenario/test_network_basic_ops.py#L124
[3]https://github.com/openstack/tempest/blob/0a0e1070e573674332cb5126064b95f17099307e/tempest/scenario/test_network_basic_ops.py#L128
[4]https://github.com/openstack/tempest/blob/0a0e1070e573674332cb5126064b95f17099307e/tempest/scenario/manager.py#L1143
[5]https://3fdd3adccbbbca8893fe-55e7a9d33a731efe4f7611907a31a4a1.ssl.cf1.rackcdn.com/924317/10/experimental/neutron-ovn-tempest-ovs-master/038956b/controller/logs/screen-neutron-api.txt
[6]https://3fdd3adccbbbca8893fe-55e7a9d33a731efe4f7611907a31a4a1.ssl.cf1.rackcdn.com/924317/10/experimental/neutron-ovn-tempest-ovs-master/038956b/controller/logs/tempest_log.txt

[Brian Haley (brian-haley)]

I'll push a patch for this based on the comments above

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tempest/+/928471

[Brian Haley (brian-haley)]

Will mark the neutron bug as Invalid since it is a bug in tempest.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tempest/+/928471
Committed: https://opendev.org/openstack/tempest/commit/d6437c9dd175371cd13d0a5d3222205a8863bda5
Submitter: "Zuul (22348)"
Branch: master

commit d6437c9dd175371cd13d0a5d3222205a8863bda5
Author: Brian Haley <email address hidden>
Date: Fri Sep 6 16:09:26 2024 -0400

    Wait for all instance ports to become ACTIVE

    get_server_port_id_and_ip4() gets a list of neutron ports
    for an instance, but it could be one or more of those have
    not completed provisioning at the time of the call, so are
    still marked DOWN.

    Wait for all ports to become active since it could just be
    neutron has not completed its work yet.

    Added new waiter function and tests to verify it worked.

    Closes-bug: #2079831
    Change-Id: I758e5eeb8ab05e79d6bdb2b560aa0f9f38c5992c

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tempest 41.0.0 release.