Fullstack test neutron.tests.fullstack.test_l3_agent.TestHAL3Agent.test_router_fip_qos_after_admin_state_down_up failing intermittently

[Slawek Kaplonski]

Failure example: https://8d5ef598bba78b1573a4-7dfe055f87ad090ed1b50745545f409a.ssl.cf1.rackcdn.com/805391/10/check/neutron-fullstack-with-uwsgi/6e03086/testr_results.html

Stacktrace:

ft1.6: neutron.tests.fullstack.test_l3_agent.TestHAL3Agent.test_router_fip_qos_after_admin_state_down_uptesttools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 703, in wait_until_true
    eventlet.sleep(sleep)
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.8/site-packages/eventlet/greenthread.py", line 36, in sleep
    hub.switch()
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 313, in switch
    return self.greenlet.switch()
eventlet.timeout.Timeout: 60 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 183, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/fullstack/test_l3_agent.py", line 565, in test_router_fip_qos_after_admin_state_down_up
    self._router_fip_qos_after_admin_state_down_up(ha=True)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/fullstack/test_l3_agent.py", line 208, in _router_fip_qos_after_admin_state_down_up
    vm.block_until_ping(external_vm.ip)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/common/machine_fixtures.py", line 67, in block_until_ping
    utils.wait_until_true(
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 707, in wait_until_true
    raise exception
neutron.tests.common.machine_fixtures.FakeMachineException: No ICMP reply obtained from IP address 240.0.83.1

In the test's logs there are errors that "Network is unreachable" so maybe it is some issue with test itself.

[Slawek Kaplonski (slaweq)]

It seems that this bug is hitting us a lot recently. We need to check it ASAP.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/813078

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/813128

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/813129

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/813078
Committed: https://opendev.org/openstack/neutron/commit/ec89cd23d18a6ec2324e36c62991f1f749d5637f
Submitter: "Zuul (22348)"
Branch: master

commit ec89cd23d18a6ec2324e36c62991f1f749d5637f
Author: Slawek Kaplonski <email address hidden>
Date: Thu Oct 7 21:47:49 2021 +0200

    [Fullstack] Mark TestHAL3Agent fip_qos test as unstable

    Mark TestHAL3Agent.test_router_fip_qos_after_admin_state_down_up
    from the fullstack tests as unstable. It is hitting us a lot
    in the gate recently so lets unblock the gate and buy some time
    to investigate what is the root cause of that issue.

    Related-Bug: #1946186
    Change-Id: I8f250fdb8c0c25378fc3c164b74d78a6f420f5f5

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/813128
Committed: https://opendev.org/openstack/neutron/commit/416b7fe2f77e8d0025327f53c6743d6eba0af439
Submitter: "Zuul (22348)"
Branch: master

commit 416b7fe2f77e8d0025327f53c6743d6eba0af439
Author: Slawek Kaplonski <email address hidden>
Date: Fri Oct 8 08:47:16 2021 +0200

    [Fullstack] Don't use dhcp in L3 agent tests

    It's not really needed and it is causing failures due to bug [1].
    We can revert that patch and use dhcp again when [1] will be fixed
    on the privsep's side.

    [1] https://review.opendev.org/c/openstack/neutron/+/794994

    Closes-Bug: #1946186
    Change-Id: I7f888fa0737a1e52cee758afee6b790ca6a80a32

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/813129
Committed: https://opendev.org/openstack/neutron/commit/b57fdf7038727a488bd73bfb05858347a9ebcc09
Submitter: "Zuul (22348)"
Branch: master

commit b57fdf7038727a488bd73bfb05858347a9ebcc09
Author: Slawek Kaplonski <email address hidden>
Date: Fri Oct 8 08:50:07 2021 +0200

    Revert "Use 2 dhcp agents in TestLegacyL3Agent"

    This reverts commit 07337f9e99cdcfb1af3546a537d5595330e8bded.

    Now we don't use dhcp in the L3 agent tests at all so this isn't
    needed anymore.

    Related-Bug: #1930401
    Related-Bug: #1946186
    Change-Id: If3a48251770c3e669ac5a9d6a44085d295809240

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/815235

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/815235
Committed: https://opendev.org/openstack/neutron/commit/bbdba95f408aeaf6603327883ff6795cc8d7a5ce
Submitter: "Zuul (22348)"
Branch: master

commit bbdba95f408aeaf6603327883ff6795cc8d7a5ce
Author: Slawek Kaplonski <email address hidden>
Date: Mon Oct 25 12:10:38 2021 +0000

    Revert "[Fullstack] Mark TestHAL3Agent fip_qos test as unstable"

    This reverts commit ec89cd23d18a6ec2324e36c62991f1f749d5637f.

    Reason for revert: as https://review.opendev.org/c/openstack/neutron/+/813128
    is merged we don't use DHCP agent in those L3 agent tests at all so
    those tests should be stable now.

    Change-Id: I6ded877b2b40929a4f4151a68a0de82ddfca0b71
    Related-Bug: #1946186

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 20.0.0.0rc1 release candidate.