Fullstack connectivity issues in tests

[Slawek Kaplonski]

From time to time we can see in the fullstack job that various tests are failing with errors like:

Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 183, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/fullstack/test_connectivity.py", line 166, in test_connectivity
    self._test_connectivity()
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/fullstack/test_connectivity.py", line 84, in _test_connectivity
    vms.ping_all()
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/fullstack/resources/machine.py", line 46, in ping_all
    vm_1.block_until_ping(vm_2.ip)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/common/machine_fixtures.py", line 67, in block_until_ping
    utils.wait_until_true(
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 722, in wait_until_true
    raise exception
neutron.tests.common.machine_fixtures.FakeMachineException: No ICMP reply obtained from IP address 20.0.0.63

Example: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_366/830623/1/check/neutron-fullstack-with-uwsgi/366febd/testr_results.html

[yatin (yatinkarel)]

In the failing test i see dhcp config is used, have seen similar issue in other test and there had pushed https://review.opendev.org/c/openstack/neutron/+/830732 to ensure dhcp config is done before moving forward. I think same can be done here, atleast that will fail if issue is with dhcp config and give more idea on what can be wrong.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/834070

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/834070
Committed: https://opendev.org/openstack/neutron/commit/282ccbb8f0b471d9bc96261ab1866ffca4ec2b97
Submitter: "Zuul (22348)"
Branch: master

commit 282ccbb8f0b471d9bc96261ab1866ffca4ec2b97
Author: Slawek Kaplonski <email address hidden>
Date: Wed Mar 16 21:19:37 2022 +0100

    [Fullstack] Remove dhcp agent in tests where it isn't necessary

    In fullstack tests like test_connectivity or test_securitygroup using
    dhcp agent isn't really needed. We are testing dhcp agents and
    configuration of the IP addresses using DHCP in the tests from the
    test_dhcp_agent module.
    So this patch disables usage of the dhcp agent where it's not really
    needed to save some resources (less agents running during tests) and to
    minimize potential failures in the tests.

    Additionally this patch also adds method
    block_until_all_dhcp_config_done() to the FakeFullstackMachinesList
    class and uses it in tests when fake VMs are using DHCP. That will
    hopefully help better understand where the issue is in case of failures
    like described in the related bug.

    Related-Bug: #1962854
    Change-Id: Ib6ca18b5a0ae101ad6424637abff3d992737f6f4