[CI][periodic] neutron-functional-with-uwsgi-fips job failing

[Mamatisa Nurmatov]

Sometimes periodic job neutron-functional-with-uwsgi-fips fails on following two tests:

neutron.tests.functional.agent.l3.test_dvr_router.TestDvrRouter
    test_dvr_router_with_centralized_fip_calls_keepalived_cidr [1]
    test_dvr_router_snat_namespace_with_interface_remove [2]

Latest builds: https://zuul.openstack.org/builds?job_name=neutron-functional-with-uwsgi-fips&project=openstack%2Fneutron&branch=master&pipeline=periodic&skip=0

1) https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_108/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-uwsgi-fips/10804fa/testr_results.html

2) https://662fbc83c91c32a8789e-45518917cf8baf33fe991d0324b9a061.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-uwsgi-fips/4cad1c3/testr_results.html

[Lajos Katona (lajos-katona)]

As I see different tests are failing but from the same test class: neutron.tests.functional.agent.l3.test_dvr_router.TestDvrRouter

[Lajos Katona (lajos-katona)]

I checked a few logs, but far from understanding the issue, and I can't reproduce it locally.

#1. what I can't process why we execute DVR tests with OVN behind? I am thinking to add a line like this: l3_plugin = 'neutron.services.l3_router.l3_router_plugin.L3RouterPlugin' to DvrRouterTestFramework. Strange is that when I execute these tests locally I can't see ovn lines in the logs....

#2. the failure is that the fg-xxxxxxxx-yy interface is not in the namespace. like plug is slow or fails (https://opendev.org/openstack/neutron/src/branch/master/neutron/agent/l3/dvr_fip_ns.py#L172)

#3. Perhaps unrelated but found these lines from slaweq: https://opendev.org/openstack/neutron/src/branch/master/neutron/agent/l3/agent.py#L651
If I understand well this is more related to cleanup, but have to understand better what happens here (by the way this was the patch for this comment: https://review.opendev.org/c/openstack/neutron/+/814892 )

[Slawek Kaplonski (slaweq)]

Another occurences of probably same (or similar) issue:

https://ad72b96aaf0d79e9068d-08f6dc72eccd53018dd797f31cdfe820.ssl.cf5.rackcdn.com/864051/4/gate/neutron-functional-with-uwsgi/7ab1211/testr_results.html

https://b256bf67e521eed0f172-1f0cd77f0a7abfeba3b9e78874b79e6b.ssl.cf1.rackcdn.com/865220/1/check/neutron-functional-with-uwsgi/dd22d7e/testr_results.html

https://675daf3418638bf15806-f7e1f8eddcfdd9404f4b72ab9bb1f324.ssl.cf1.rackcdn.com/865575/1/check/neutron-functional-with-uwsgi/bd983b3/testr_results.html

https://498c49910e2e429f6f6d-5ec4103c89b5d2feb0a0ca5aa5752932.ssl.cf1.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-uwsgi-fips/320abfd/testr_results.html

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_cf5/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-uwsgi-fips/cf5a77a/testr_results.html

[Lajos Katona (lajos-katona)]

recently I can't find new occurances of this issue (perhaps my search method is bad)
I was not able to reproduce it by even adding random timeout to interface setup (the same group of tests were executed in a loop)

[Slawek Kaplonski (slaweq)]

It is likely fixed by https://review.opendev.org/c/openstack/neutron/+/856261 so I'm closing it now.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/869666
Committed: https://opendev.org/openstack/neutron/commit/de0918cea3fa3c107d4cc08137c04573274716f8
Submitter: "Zuul (22348)"
Branch: master

commit de0918cea3fa3c107d4cc08137c04573274716f8
Author: elajkat <email address hidden>
Date: Tue Jan 10 10:51:17 2023 +0100

    Add debug logs to OVSInterfaceDriver.plug_new

    DVR lifecycle functional tests fail frequently with exceptions
    like this: "Network interface * not found in namespace *"
    The reproduction locally is hard if possible, so by adding
    extra log messages I hope we can have more clue where to find
    the possible root cause of this issue.

    Change-Id: I2c6acf3c29f3d1e7574240376f7bcb46c98579c6
    Related-Bug: #1995031

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/873111

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Lajos Katona <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/873111