scenario test test_floating_ip_update is failing often on Ubuntu 20.04

[Slawek Kaplonski]

Similar to https://bugs.launchpad.net/neutron/+bug/1896735 also neutron_tempest_plugin.scenario.test_floatingip.TestFloatingIPUpdate.test_floating_ip_update is failing often now after we switched our jobs to Ubuntu 20.04.
I think that both bugs may have same root cause but as it's not sure yet lets track them in separate LPs.

Example of failure: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_476/748367/15/check/neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid/47699ad/testr_results.html

Traceback:

Traceback (most recent call last):
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/common/utils.py", line 80, in wait_until_true
    eventlet.sleep(sleep)
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/eventlet/greenthread.py", line 36, in sleep
    hub.switch()
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 298, in switch
    return self.greenlet.switch()
eventlet.timeout.Timeout: 15 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/test_floatingip.py", line 437, in test_floating_ip_update
    common_utils.wait_until_true(_wait_for_fip_associated,
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/common/utils.py", line 85, in wait_until_true
    raise WaitTimeout("Timed out after %d seconds" % timeout)
neutron_tempest_plugin.common.utils.WaitTimeout: Timed out after 15 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/test_floatingip.py", line 441, in test_floating_ip_update
    self.fail(
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/unittest2/case.py", line 690, in fail
    raise self.failureException(msg)
AssertionError: Server 27f3e5ec-4117-443f-b730-2c852346bfa6 is not accessible via its floating ip 886230f3-e1ec-4a5d-90f8-350358e649fa

[Slawek Kaplonski (slaweq)]

Based on logs from https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_476/748367/15/check/neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid/47699ad/testr_results.html it seems that this test is just not waiting enough for FIP to be moved to the new server.

FIP was updated at:
2020-09-25 16:24:39.273 97643 INFO tempest.lib.common.rest_client [req-61752930-7315-4d54-8f3d-3f5a83c38580 ] Request (TestFloatingIPUpdate:test_floating_ip_update): 200 PUT http://10.209.97.145:9696/v2.0/floatingips/886230f3-e1ec-4a5d-90f8-350358e649fa 3.935s

And test is waiting 15 seconds for it to be updated. But L3 agent got info about this update at

Sep 25 16:24:59.261691 ubuntu-focal-rax-iad-0020040651 neutron-l3-agent[81604]: DEBUG neutron.agent.l3.router_info [-] Floating IP was moved from fixed IP 10.1.0.11 to 10.1.0.6 {{(pid=81604) process_floating_ip_addresses /opt/stack/neutron/neutron/agent/l3/router_info.py:414}}

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/755314

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/755314
Committed: https://git.openstack.org/cgit/openstack/neutron-tempest-plugin/commit/?id=0e9edc58654db495aaa76b78d2b647762f957da3
Submitter: Zuul
Branch: master

commit 0e9edc58654db495aaa76b78d2b647762f957da3
Author: Slawek Kaplonski <email address hidden>
Date: Wed Sep 30 16:54:13 2020 +0200

    Wait longer time for FIP to be updated by L3 agent

    In test test_floating_ip_update we should wait a bit more than
    15 seconds for FIP to be actually updated and moved to new server
    by the L3 agent.
    So lets wait 60 seconds which is default value in wait_unti_true()
    function.

    Change-Id: I25e85af8e626664c0a562ea0a5b8faf8b29210a5
    Closes-Bug: #1897326

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron-tempest-plugin stein-last release.