Network interface not found in namespace

[Slawek Kaplonski]

This is intermittent error in the functional tests job. Failure example: https://89b1b88fa362b409cfb1-2a70ac574f4ba34d12afc72df211f1b3.ssl.cf5.rackcdn.com/828687/1/gate/neutron-functional-with-uwsgi/a5b844b/testr_results.html

Stacktrace:
ft1.47: neutron.tests.functional.agent.l3.extensions.test_port_forwarding_extension.TestL3AgentFipPortForwardingExtensionDVR.test_dvr_non_ha_router_updatetesttools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 183, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 183, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_dvr_router.py", line 1708, in test_dvr_non_ha_router_update
    router2 = self._create_dvr_ha_router(
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_dvr_router.py", line 1445, in _create_dvr_ha_router
    router = self.manage_router(agent, r_info)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_dvr_router.py", line 195, in manage_router
    return super(TestDvrRouter, self).manage_router(agent, router)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/framework.py", line 420, in manage_router
    agent._process_added_router(router)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l3/agent.py", line 663, in _process_added_router
    self._cleanup_failed_router(router['id'],
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l3/agent.py", line 653, in _process_added_router
    ri.process()
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l3/dvr_local_router.py", line 924, in process
    super(DvrLocalRouter, self).process()
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l3/dvr_router_base.py", line 29, in process
    super(DvrRouterBase, self).process()
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 177, in call
    self.logger(e)
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 174, in call
    return func(*args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l3/router_info.py", line 1312, in process
    self.process_external()
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l3/dvr_local_router.py", line 740, in process_external
    super(DvrLocalRouter, self).process_external()
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/coordination.py", line 78, in _synchronized
    return f(*a, **k)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l3/router_info.py", line 1046, in process_external
    self._process_external_gateway(ex_gw_port)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l3/router_info.py", line 927, in _process_external_gateway
    self.external_gateway_added(ex_gw_port, interface_name)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l3/dvr_edge_router.py", line 43, in external_gateway_added
    self._create_dvr_gateway(ex_gw_port, interface_name)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l3/dvr_edge_router.py", line 194, in _create_dvr_gateway
    self._external_gateway_added(ex_gw_port, gw_interface_name,
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l3/router_info.py", line 821, in _external_gateway_added
    self._plug_external_gateway(ex_gw_port, interface_name, ns_name)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l3/router_info.py", line 755, in _plug_external_gateway
    self.driver.plug(ex_gw_port['network_id'],
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/interface.py", line 263, in plug
    self.plug_new(network_id, port_id, device_name, mac_address,
  File "<string>", line 3, in plug_new
  File "/usr/lib/python3.8/unittest/mock.py", line 1081, in __call__
    return self._mock_call(*args, **kwargs)
  File "/usr/lib/python3.8/unittest/mock.py", line 1085, in _mock_call
    return self._execute_mock_call(*args, **kwargs)
  File "/usr/lib/python3.8/unittest/mock.py", line 1146, in _execute_mock_call
    result = effect(*args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/framework.py", line 408, in new_ovs_plug
    original_plug_new(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/interface.py", line 388, in plug_new
    namespace_obj.add_device_to_namespace(ns_dev)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 271, in add_device_to_namespace
    device.link.set_netns(self.namespace)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 462, in set_netns
    privileged.set_link_attribute(
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap
    return self.channel.remote_call(name, args, kwargs,
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_privsep/daemon.py", line 215, in remote_call
    raise exc_type(*result[2])
neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface qg-d9d41d8e-8b not found in namespace None.

[Slawek Kaplonski (slaweq)]

I did some analysis of that issue and it seems that it may be another "variant" of the old bug https://bugs.launchpad.net/neutron/+bug/1618987 as, according to the ovs logs port is added, then deleted and added again quickly:

2022-02-21T12:25:56.484Z|04524|bridge|INFO|bridge test-br0e5eb98d: added interface qg-d9d41d8e-8b on port 7
...
2022-02-21T12:25:56.707Z|04530|bridge|INFO|bridge test-br0e5eb98d: deleted interface qg-d9d41d8e-8b on port 7
...
2022-02-21T12:25:56.713Z|04532|bridge|INFO|bridge test-br0e5eb98d: added interface qg-d9d41d8e-8b on port 7

All of that within about 250 ms. And during that time test failed actually.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/830622

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/830622
Committed: https://opendev.org/openstack/neutron/commit/f7fac03ae1a4b1b1471f6054ba058382a1b9e1a1
Submitter: "Zuul (22348)"
Branch: master

commit f7fac03ae1a4b1b1471f6054ba058382a1b9e1a1
Author: Slawek Kaplonski <email address hidden>
Date: Wed Feb 23 15:12:37 2022 +0100

    Repeat few times put new interface in the namespace

    It seems that from time to time we may experience another variant of the
    shy ovs ports issue (see [1] for details) while trying to put interface,
    e.g. router's port into the namespace by the interface driver.
    To avoid that, this patch adds some repeats of the putting interface in
    the namespace, in the same way like it was done originally with
    set_address of the device.

    Additionally, this patch also refactors a bit part which is responsible
    to set mac address of the device to be able to clean ports in case of
    the permanent error there.

    [1] https://bugs.launchpad.net/neutron/+bug/1618987

    Closes-Bug: #1961740
    Change-Id: I3e0786fd8d0320036b9046746ae579c8ed2ecf27

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/835808

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/835809

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/835850

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/835851

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/835852

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/835809
Committed: https://opendev.org/openstack/neutron/commit/7c67460b1612c686edf941603e9250a5b3d5778f
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 7c67460b1612c686edf941603e9250a5b3d5778f
Author: Slawek Kaplonski <email address hidden>
Date: Wed Feb 23 15:12:37 2022 +0100

    Repeat few times put new interface in the namespace

    It seems that from time to time we may experience another variant of the
    shy ovs ports issue (see [1] for details) while trying to put interface,
    e.g. router's port into the namespace by the interface driver.
    To avoid that, this patch adds some repeats of the putting interface in
    the namespace, in the same way like it was done originally with
    set_address of the device.

    Additionally, this patch also refactors a bit part which is responsible
    to set mac address of the device to be able to clean ports in case of
    the permanent error there.

    [1] https://bugs.launchpad.net/neutron/+bug/1618987

    Closes-Bug: #1961740
    Change-Id: I3e0786fd8d0320036b9046746ae579c8ed2ecf27
    (cherry picked from commit f7fac03ae1a4b1b1471f6054ba058382a1b9e1a1)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/835808
Committed: https://opendev.org/openstack/neutron/commit/db3965b1563e12a6ef272f7ea75475aee8c7da75
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit db3965b1563e12a6ef272f7ea75475aee8c7da75
Author: Slawek Kaplonski <email address hidden>
Date: Wed Feb 23 15:12:37 2022 +0100

    Repeat few times put new interface in the namespace

    It seems that from time to time we may experience another variant of the
    shy ovs ports issue (see [1] for details) while trying to put interface,
    e.g. router's port into the namespace by the interface driver.
    To avoid that, this patch adds some repeats of the putting interface in
    the namespace, in the same way like it was done originally with
    set_address of the device.

    Additionally, this patch also refactors a bit part which is responsible
    to set mac address of the device to be able to clean ports in case of
    the permanent error there.

    [1] https://bugs.launchpad.net/neutron/+bug/1618987

    Closes-Bug: #1961740
    Change-Id: I3e0786fd8d0320036b9046746ae579c8ed2ecf27
    (cherry picked from commit f7fac03ae1a4b1b1471f6054ba058382a1b9e1a1)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/835850
Committed: https://opendev.org/openstack/neutron/commit/b889048202922966a3a315f60e3a731df372d17e
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit b889048202922966a3a315f60e3a731df372d17e
Author: Slawek Kaplonski <email address hidden>
Date: Wed Feb 23 15:12:37 2022 +0100

    Repeat few times put new interface in the namespace

    It seems that from time to time we may experience another variant of the
    shy ovs ports issue (see [1] for details) while trying to put interface,
    e.g. router's port into the namespace by the interface driver.
    To avoid that, this patch adds some repeats of the putting interface in
    the namespace, in the same way like it was done originally with
    set_address of the device.

    Additionally, this patch also refactors a bit part which is responsible
    to set mac address of the device to be able to clean ports in case of
    the permanent error there.

    [1] https://bugs.launchpad.net/neutron/+bug/1618987

    Closes-Bug: #1961740
    Change-Id: I3e0786fd8d0320036b9046746ae579c8ed2ecf27
    (cherry picked from commit f7fac03ae1a4b1b1471f6054ba058382a1b9e1a1)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/835851
Committed: https://opendev.org/openstack/neutron/commit/258dc360508f6daab0f260c9b06b46b10a2b7624
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 258dc360508f6daab0f260c9b06b46b10a2b7624
Author: Slawek Kaplonski <email address hidden>
Date: Wed Feb 23 15:12:37 2022 +0100

    Repeat few times put new interface in the namespace

    It seems that from time to time we may experience another variant of the
    shy ovs ports issue (see [1] for details) while trying to put interface,
    e.g. router's port into the namespace by the interface driver.
    To avoid that, this patch adds some repeats of the putting interface in
    the namespace, in the same way like it was done originally with
    set_address of the device.

    Additionally, this patch also refactors a bit part which is responsible
    to set mac address of the device to be able to clean ports in case of
    the permanent error there.

    [1] https://bugs.launchpad.net/neutron/+bug/1618987

    Closes-Bug: #1961740
    Change-Id: I3e0786fd8d0320036b9046746ae579c8ed2ecf27
    (cherry picked from commit f7fac03ae1a4b1b1471f6054ba058382a1b9e1a1)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/835852
Committed: https://opendev.org/openstack/neutron/commit/db3f5b3ff26ff7d748a3b04a6c38a47e37448e17
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit db3f5b3ff26ff7d748a3b04a6c38a47e37448e17
Author: Slawek Kaplonski <email address hidden>
Date: Wed Feb 23 15:12:37 2022 +0100

    Repeat few times put new interface in the namespace

    It seems that from time to time we may experience another variant of the
    shy ovs ports issue (see [1] for details) while trying to put interface,
    e.g. router's port into the namespace by the interface driver.
    To avoid that, this patch adds some repeats of the putting interface in
    the namespace, in the same way like it was done originally with
    set_address of the device.

    Additionally, this patch also refactors a bit part which is responsible
    to set mac address of the device to be able to clean ports in case of
    the permanent error there.

    [1] https://bugs.launchpad.net/neutron/+bug/1618987

    Closes-Bug: #1961740
    Change-Id: I3e0786fd8d0320036b9046746ae579c8ed2ecf27
    (cherry picked from commit f7fac03ae1a4b1b1471f6054ba058382a1b9e1a1)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 17.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.2.0 release.

[Slawek Kaplonski (slaweq)]

This issue still happens, like e.g. https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e14/841094/1/check/neutron-functional-with-uwsgi/e145a4a/testr_results.html

I checked it and there is something strange going on there. Interface which caused failure was added/removed/added again in short time:

2022-05-09T11:02:13.807Z|03542|bridge|INFO|bridge test-bre924f925: added interface qr-a3915f78-e1 on port 1
2022-05-09T11:02:13.965Z|03551|bridge|INFO|bridge test-bre924f925: deleted interface qr-a3915f78-e1 on port 1
2022-05-09T11:02:14.058Z|03555|bridge|INFO|bridge test-bre924f925: added interface qr-a3915f78-e1 on port 1
2022-05-09T11:02:28.371Z|03687|bridge|INFO|bridge test-bre924f925: deleted interface qr-a3915f78-e1 on port 1

The last delete is when test already failed and there was cleanup phase so that's not a problem for sure.
I checked also other qr- port in logs of the same job (random one) and it was just created and deleted:

2022-05-09T11:02:28.426Z|03688|bridge|INFO|bridge test-br7dd57a6d: added interface qr-c9468ae1-58 on port 2
2022-05-09T11:02:42.990Z|03797|bridge|INFO|bridge test-br7dd57a6d: deleted interface qr-c9468ae1-58 on port 2

Getting back to the failed interface qr-a3915f78-e1, even if it was added again after few miliseconds, neutron couldn't set it to namespace at all:

2022-05-09 11:02:18.877 68464 WARNING neutron.agent.linux.interface [req-552fe784-bb96-48e2-9b29-edd623021e43 - - - - -] Failed to set interface qr-a3915f78-e1 into namespace qrouter-02f072b2-b45f-427c-b708-33fad2beb401. Interface not found in namespace, attempt: 1, retrying.: neutron.common.utils.WaitTimeout: Timed out after 5 seconds
2022-05-09 11:02:19.942 68464 WARNING neutron.agent.linux.interface [req-552fe784-bb96-48e2-9b29-edd623021e43 - - - - -] Failed to set interface qr-a3915f78-e1 into namespace qrouter-02f072b2-b45f-427c-b708-33fad2beb401. Interface not found, attempt: 2, retrying.: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface qr-a3915f78-e1 not found in namespace qrouter-02f072b2-b45f-427c-b708-33fad2beb401.
2022-05-09 11:02:21.049 68464 WARNING neutron.agent.linux.interface [req-552fe784-bb96-48e2-9b29-edd623021e43 - - - - -] Failed to set interface qr-a3915f78-e1 into namespace qrouter-02f072b2-b45f-427c-b708-33fad2beb401. Interface not found, attempt: 3, retrying.: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface qr-a3915f78-e1 not found in namespace qrouter-02f072b2-b45f-427c-b708-33fad2beb401.
2022-05-09 11:02:22.081 68464 WARNING neutron.agent.linux.interface [req-552fe784-bb96-48e2-9b29-edd623021e43 - - - - -] Failed to set interface qr-a3915f78-e1 into namespace qrouter-02f072b2-b45f-427c-b708-33fad2beb401. Interface not found, attempt: 4, retrying.: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface qr-a3915f78-e1 not found in namespace qrouter-02f072b2-b45f-427c-b708-33fad2beb401.
2022-05-09 11:02:23.115 68464 WARNING neutron.agent.linux.interface [req-552fe784-bb96-48e2-9b29-edd623021e43 - - - - -] Failed to set interface qr-a3915f78-e1 into namespace qrouter-02f072b2-b45f-427c-b708-33...

[Lajos Katona (lajos-katona)]

I think this the filter for it:
https://opensearch.logs.openstack.org/_dashboards/app/discover/?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15d,to:now))&_a=(columns:!(build_status,build_name),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'94869730-aea8-11ec-9e6a-83741af3fdcd',key:build_status,negate:!f,params:(query:FAILURE),type:phrase),query:(match_phrase:(build_status:FAILURE))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'94869730-aea8-11ec-9e6a-83741af3fdcd',key:build_name,negate:!f,params:(query:neutron-functional-with-uwsgi),type:phrase),query:(match_phrase:(build_name:neutron-functional-with-uwsgi)))),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:kuery,query:'message:%22neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound:%20Network%20interface%22'),sort:!())

Perhaps other jobs also failed with similar issue

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 20.1.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/856261

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/856261
Committed: https://opendev.org/openstack/neutron/commit/76578393ab401a33084921c4355af8e82c5d02ee
Submitter: "Zuul (22348)"
Branch: master

commit 76578393ab401a33084921c4355af8e82c5d02ee
Author: Slawek Kaplonski <email address hidden>
Date: Wed Sep 7 12:07:38 2022 +0200

    Reset device namespace when adding to the namespace fails

    In case when during adding device to the namespace, device will be "shy"
    and will disappear for a moment and NetworkInterfaceNotFound exception
    will be raised, we need to reset device.namespace to be None.
    Otherwise, in the next attempt of adding interface to namespace, when it
    will be added back to ovs (and will be in global scope), Neutron will
    already look for it in the "namespace" and that will always be failing.

    Closes-bug: #1961740
    Change-Id: Ie9331c72c44084b0a382598c3359214cce2f2ebd

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 21.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/875767

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/875767
Committed: https://opendev.org/openstack/neutron/commit/2af5fd889b3286dcec21e2bc89f287a0e4129d0f
Submitter: "Zuul (22348)"
Branch: master

commit 2af5fd889b3286dcec21e2bc89f287a0e4129d0f
Author: Slawek Kaplonski <email address hidden>
Date: Tue Feb 28 18:27:29 2023 +0100

    Add sleep before checking if ovs port is in the namespace

    When network device which is ovs internal port is moved to the namespace
    it may happend sometimes that it will have "shy port syndrome" [1].
    Even though there is wait for device to be in namespace in the set_netns
    method it may happend that device is in namespace during this check but
    it dissapears for short time later and that causes failures e.g. in
    functional tests like described in [2].
    To avoid that, this patch proposed simple (and ugly) sleep for 1 second
    before checking if port really exists in the namespace. If it will be
    "shy" port it should already flap during that 1 second.

    [1] https://bugs.launchpad.net/neutron/+bug/1618987
    [2] https://bugs.launchpad.net/neutron/+bug/1961740

    Related-Bug: #1961740
    Related-Bug: #1998337
    Change-Id: I442587e7ef55917f4ea873e190bf8afbc0e911e1

[yatin (yatinkarel)]

Still seeing similar issue as below, reopening the bug

- https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ed8/901717/1/gate/neutron-functional-with-uwsgi/ed8e900/testr_results.html

Fails as:- neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface sg-aeb1f7f1-24 not found in namespace snat-89cb4e33-917d-4ba3-b89f-48963ea3d024.

Device added/deleted/added quickly:-
2023-11-29T10:17:44.507Z|06104|bridge|INFO|bridge test-br8645578e: added interface sg-aeb1f7f1-24 on port 4
2023-11-29T10:17:44.865Z|06115|bridge|INFO|bridge test-br8645578e: deleted interface sg-aeb1f7f1-24 on port 4
2023-11-29T10:17:44.877Z|06117|bridge|INFO|bridge test-br8645578e: added interface sg-aeb1f7f1-24 on port 4
2023-11-29T10:17:45.336Z|06131|bridge|INFO|bridge test-br8645578e: deleted interface sg-aeb1f7f1-24 on port 4

- https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_03d/901718/1/gate/neutron-functional-with-uwsgi/03d59c7/testr_results.html

Fails as:- neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface sg-44024bfc-76 not found in namespace snat-0afef646-ef9d-406e-b479-6cc7861f8804.

Device added/deleted/added quickly:-
2023-11-29T10:53:03.124Z|03237|bridge|INFO|bridge test-brf59c6dc7: added interface sg-44024bfc-76 on port 4
2023-11-29T10:53:03.804Z|03253|bridge|INFO|bridge test-brf59c6dc7: deleted interface sg-44024bfc-76 on port 4
2023-11-29T10:53:03.817Z|03256|bridge|INFO|bridge test-brf59c6dc7: added interface sg-44024bfc-76 on port 4
2023-11-29T10:53:05.215Z|03282|bridge|INFO|bridge test-brf59c6dc7: deleted interface sg-44024bfc-76 on port 4

- https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_109/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-sqlalchemy-master/10928da/testr_results.html

Fails as:- AssertionError: Device name: fg-1f3bd0ce-36, expected MAC: fa:16:3e:80:8d:89, expected CIDRs: ['19.4.5.8/24'], device MAC: 0e:d2:9d:26:57:e6

Device added/deleted/added quickly
2023-12-11T03:51:16.045Z|05000|bridge|INFO|bridge test-br4d6080f6: added interface fg-1f3bd0ce-36 on port 6
2023-12-11T03:51:16.437Z|05008|bridge|INFO|bridge test-br4d6080f6: deleted interface fg-1f3bd0ce-36 on port 6
2023-12-11T03:51:16.446Z|05010|bridge|WARN|could not add network device fg-1f3bd0ce-36 to ofproto (No such device)
2023-12-11T03:51:16.556Z|05013|bridge|INFO|bridge test-br4d6080f6: added interface fg-1f3bd0ce-36 on port 8
2023-12-11T03:51:24.190Z|05053|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on fg-1f3bd0ce-36 device failed: No such device
2023-12-11T03:51:29.670Z|05098|bridge|INFO|bridge test-br4d6080f6: deleted interface fg-1f3bd0ce-36 on port 8

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/904825

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/904826

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/904827

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/904829

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/904830

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/904831

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/904832

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/904833

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/904825
Committed: https://opendev.org/openstack/neutron/commit/f89a585dcc6e9f8aed33ef0e244c953f8275ad0a
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit f89a585dcc6e9f8aed33ef0e244c953f8275ad0a
Author: Slawek Kaplonski <email address hidden>
Date: Wed Sep 7 12:07:38 2022 +0200

    Reset device namespace when adding to the namespace fails

    In case when during adding device to the namespace, device will be "shy"
    and will disappear for a moment and NetworkInterfaceNotFound exception
    will be raised, we need to reset device.namespace to be None.
    Otherwise, in the next attempt of adding interface to namespace, when it
    will be added back to ovs (and will be in global scope), Neutron will
    already look for it in the "namespace" and that will always be failing.

    Closes-bug: #1961740
    Change-Id: Ie9331c72c44084b0a382598c3359214cce2f2ebd
    (cherry picked from commit 76578393ab401a33084921c4355af8e82c5d02ee)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/904826
Committed: https://opendev.org/openstack/neutron/commit/b27ea5fe2da47d134ac9b4a50341a0ecc0934eb3
Submitter: "Zuul (22348)"
Branch: stable/xena

commit b27ea5fe2da47d134ac9b4a50341a0ecc0934eb3
Author: Slawek Kaplonski <email address hidden>
Date: Wed Sep 7 12:07:38 2022 +0200

    Reset device namespace when adding to the namespace fails

    In case when during adding device to the namespace, device will be "shy"
    and will disappear for a moment and NetworkInterfaceNotFound exception
    will be raised, we need to reset device.namespace to be None.
    Otherwise, in the next attempt of adding interface to namespace, when it
    will be added back to ovs (and will be in global scope), Neutron will
    already look for it in the "namespace" and that will always be failing.

    Closes-bug: #1961740
    Change-Id: Ie9331c72c44084b0a382598c3359214cce2f2ebd
    (cherry picked from commit 76578393ab401a33084921c4355af8e82c5d02ee)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/904827
Committed: https://opendev.org/openstack/neutron/commit/758970a3d7dce1e1de10741df38a5b69e12ca9fc
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 758970a3d7dce1e1de10741df38a5b69e12ca9fc
Author: Slawek Kaplonski <email address hidden>
Date: Wed Sep 7 12:07:38 2022 +0200

    Reset device namespace when adding to the namespace fails

    In case when during adding device to the namespace, device will be "shy"
    and will disappear for a moment and NetworkInterfaceNotFound exception
    will be raised, we need to reset device.namespace to be None.
    Otherwise, in the next attempt of adding interface to namespace, when it
    will be added back to ovs (and will be in global scope), Neutron will
    already look for it in the "namespace" and that will always be failing.

    Closes-bug: #1961740
    Change-Id: Ie9331c72c44084b0a382598c3359214cce2f2ebd
    (cherry picked from commit 76578393ab401a33084921c4355af8e82c5d02ee)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/904830
Committed: https://opendev.org/openstack/neutron/commit/222c997022392561c2de2cb493f0f5214eb20dfc
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 222c997022392561c2de2cb493f0f5214eb20dfc
Author: Slawek Kaplonski <email address hidden>
Date: Tue Feb 28 18:27:29 2023 +0100

    Add sleep before checking if ovs port is in the namespace

    When network device which is ovs internal port is moved to the namespace
    it may happend sometimes that it will have "shy port syndrome" [1].
    Even though there is wait for device to be in namespace in the set_netns
    method it may happend that device is in namespace during this check but
    it dissapears for short time later and that causes failures e.g. in
    functional tests like described in [2].
    To avoid that, this patch proposed simple (and ugly) sleep for 1 second
    before checking if port really exists in the namespace. If it will be
    "shy" port it should already flap during that 1 second.

    [1] https://bugs.launchpad.net/neutron/+bug/1618987
    [2] https://bugs.launchpad.net/neutron/+bug/1961740

    Related-Bug: #1961740
    Related-Bug: #1998337
    Change-Id: I442587e7ef55917f4ea873e190bf8afbc0e911e1
    (cherry picked from commit 2af5fd889b3286dcec21e2bc89f287a0e4129d0f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/904831
Committed: https://opendev.org/openstack/neutron/commit/f4e0b023621a922260b35b37f9940826327efc6e
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit f4e0b023621a922260b35b37f9940826327efc6e
Author: Slawek Kaplonski <email address hidden>
Date: Tue Feb 28 18:27:29 2023 +0100

    Add sleep before checking if ovs port is in the namespace

    When network device which is ovs internal port is moved to the namespace
    it may happend sometimes that it will have "shy port syndrome" [1].
    Even though there is wait for device to be in namespace in the set_netns
    method it may happend that device is in namespace during this check but
    it dissapears for short time later and that causes failures e.g. in
    functional tests like described in [2].
    To avoid that, this patch proposed simple (and ugly) sleep for 1 second
    before checking if port really exists in the namespace. If it will be
    "shy" port it should already flap during that 1 second.

    [1] https://bugs.launchpad.net/neutron/+bug/1618987
    [2] https://bugs.launchpad.net/neutron/+bug/1961740

    Related-Bug: #1961740
    Related-Bug: #1998337
    Change-Id: I442587e7ef55917f4ea873e190bf8afbc0e911e1
    (cherry picked from commit 2af5fd889b3286dcec21e2bc89f287a0e4129d0f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/904829
Committed: https://opendev.org/openstack/neutron/commit/c5e70ad716604a7a379fa51ff16500cd0ad0094c
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit c5e70ad716604a7a379fa51ff16500cd0ad0094c
Author: Slawek Kaplonski <email address hidden>
Date: Tue Feb 28 18:27:29 2023 +0100

    Add sleep before checking if ovs port is in the namespace

    When network device which is ovs internal port is moved to the namespace
    it may happend sometimes that it will have "shy port syndrome" [1].
    Even though there is wait for device to be in namespace in the set_netns
    method it may happend that device is in namespace during this check but
    it dissapears for short time later and that causes failures e.g. in
    functional tests like described in [2].
    To avoid that, this patch proposed simple (and ugly) sleep for 1 second
    before checking if port really exists in the namespace. If it will be
    "shy" port it should already flap during that 1 second.

    [1] https://bugs.launchpad.net/neutron/+bug/1618987
    [2] https://bugs.launchpad.net/neutron/+bug/1961740

    Related-Bug: #1961740
    Related-Bug: #1998337
    Change-Id: I442587e7ef55917f4ea873e190bf8afbc0e911e1
    (cherry picked from commit 2af5fd889b3286dcec21e2bc89f287a0e4129d0f)

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Rodolfo Alonso <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/904833

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Rodolfo Alonso <email address hidden>" on branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/904832

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron ussuri-eol release.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/906483

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/906483
Committed: https://opendev.org/openstack/neutron/commit/0396b33bc4db43798490fefaf97a3bfcbdba29d0
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 0396b33bc4db43798490fefaf97a3bfcbdba29d0
Author: Rodolfo Alonso <email address hidden>
Date: Wed Jan 24 07:39:09 2024 +0000

    Revert "Add sleep before checking if ovs port is in the namespace"

    This reverts commit 222c997022392561c2de2cb493f0f5214eb20dfc.

    Reason for revert:
    This patch seems to be breaking the test
    "test_multiple_agents_for_network(Open vSwitch agent)" in the
    fullstack job.

    Change-Id: Ib93ebdcceb177c297b3b287fda01e3d57a275cb4
    Related-Bug: #1961740

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/906295
Committed: https://opendev.org/openstack/neutron/commit/34e97943a8190a33d72458ce39aaa5f82e0855b9
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 34e97943a8190a33d72458ce39aaa5f82e0855b9
Author: Rodolfo Alonso <email address hidden>
Date: Tue Jan 23 15:22:13 2024 +0000

    Revert "Add sleep before checking if ovs port is in the namespace"

    This reverts commit f4e0b023621a922260b35b37f9940826327efc6e.

    Reason for revert:
    This patch seems to be breaking the test
    "test_multiple_agents_for_network(Open vSwitch agent)" in the
    fullstack job.

    Related-Bug: #1961740

    Change-Id: Ia8764965b54e68f04864ef2ddd04f3d966af55ec

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron yoga-eom release.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/909385

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/909385
Committed: https://opendev.org/openstack/neutron/commit/016f5187a5503b835cd6803c5d8fe6f05091df12
Submitter: "Zuul (22348)"
Branch: master

commit 016f5187a5503b835cd6803c5d8fe6f05091df12
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Feb 19 09:38:07 2024 +0000

    Retry ``set|get_link_attribute(s)`` if the interface is not present

    After some interface operations (in particular the
    ``IpLinkCommand.set_ns`` operation), the network interface is
    temporarily not present in the destination namespace. This patch
    retries the interface "ip link set|show" command in that case.

    Related-Bug: #1961740
    Change-Id: I5a57cfc71ad59f1fe9ea65e19b1a32314d798729

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/neutron/+/910355

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/910356

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/910357

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/910355
Committed: https://opendev.org/openstack/neutron/commit/a512e5c624bee974ad68bf39b5792343f9e54349
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit a512e5c624bee974ad68bf39b5792343f9e54349
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Feb 19 09:38:07 2024 +0000

    Retry ``set|get_link_attribute(s)`` if the interface is not present

    After some interface operations (in particular the
    ``IpLinkCommand.set_ns`` operation), the network interface is
    temporarily not present in the destination namespace. This patch
    retries the interface "ip link set|show" command in that case.

    Related-Bug: #1961740
    Change-Id: I5a57cfc71ad59f1fe9ea65e19b1a32314d798729
    (cherry picked from commit 016f5187a5503b835cd6803c5d8fe6f05091df12)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/910356
Committed: https://opendev.org/openstack/neutron/commit/1d2a1c68c0a869576e359f5245495c743a11c17c
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 1d2a1c68c0a869576e359f5245495c743a11c17c
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Feb 19 09:38:07 2024 +0000

    Retry ``set|get_link_attribute(s)`` if the interface is not present

    After some interface operations (in particular the
    ``IpLinkCommand.set_ns`` operation), the network interface is
    temporarily not present in the destination namespace. This patch
    retries the interface "ip link set|show" command in that case.

    Related-Bug: #1961740
    Change-Id: I5a57cfc71ad59f1fe9ea65e19b1a32314d798729
    (cherry picked from commit 016f5187a5503b835cd6803c5d8fe6f05091df12)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/910357
Committed: https://opendev.org/openstack/neutron/commit/32d7efbfb03e1d5ce2c73e4341e1b5949254c0ed
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 32d7efbfb03e1d5ce2c73e4341e1b5949254c0ed
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Feb 19 09:38:07 2024 +0000

    Retry ``set|get_link_attribute(s)`` if the interface is not present

    After some interface operations (in particular the
    ``IpLinkCommand.set_ns`` operation), the network interface is
    temporarily not present in the destination namespace. This patch
    retries the interface "ip link set|show" command in that case.

    Related-Bug: #1961740
    Change-Id: I5a57cfc71ad59f1fe9ea65e19b1a32314d798729
    (cherry picked from commit 016f5187a5503b835cd6803c5d8fe6f05091df12)

[yatin (yatinkarel)]

Just to update, we still seeing this issue:-
Opensearch(creds openstack/openstack):- https://opensearch.logs.openstack.org/_dashboards/app/discover/?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_a=(columns:!(_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'94869730-aea8-11ec-9e6a-83741af3fdcd',key:build_status,negate:!f,params:(query:FAILURE),type:phrase),query:(match_phrase:(build_status:FAILURE))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'94869730-aea8-11ec-9e6a-83741af3fdcd',key:build_name,negate:!f,params:(query:neutron-functional-with-uwsgi),type:phrase),query:(match_phrase:(build_name:neutron-functional-with-uwsgi)))),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:kuery,query:'message:%22neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound:%20Network%20interface%22'),sort:!())

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron wallaby-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron xena-eom release.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/911647

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/915990

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/915990
Committed: https://opendev.org/openstack/neutron/commit/4d405ca6758eeb27ea6b3dbf08737a81e788c69e
Submitter: "Zuul (22348)"
Branch: master

commit 4d405ca6758eeb27ea6b3dbf08737a81e788c69e
Author: yatinkarel <email address hidden>
Date: Tue Apr 16 18:57:45 2024 +0530

    Mark L3HA dvr tests unstable

    These test randomly fails with NetworkInterfaceNotFound
    from quite some time. Until the issue is fixed added unstable_test
    decorator so the job do not fail for these.

    Related-Bug: #1961740
    Change-Id: I1a352526aac72b417f93f51ba50c3359359c240d

[Ihar Hrachyshka (ihar-hrachyshka)]

@yatin I am not sure https://review.opendev.org/c/openstack/neutron/+/915990 was effective, see this failure that happens a month after the 'unstable' marking patch merged: https://zuul.opendev.org/t/openstack/build/b6925d6c979c4831ba9cb5c600f51a65

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/911647
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.