Test neutron.tests.functional.agent.ovn.metadata.test_metadata_agent.TestMetadataAgent.test_agent_resync_on_non_existing_bridge is failing intermitelly.
I just reintroduced this test [1] into the code. The failure is does not happen all the time but I can be reproduce it locally with --until-failure with multiple concurrency(big hint) and running the whole TestMetadaAgent class of tests (another hint). Like this
When the failure happens following exception is found in the logs
2023-03-10 17:49:11.861 40848 INFO neutron.agent.ovn.metadata.agent [-] Port ovn-port-feb89eb1-fcf8-4b38-8aee-d8dd4b6b497e in datapath ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b bound to our chassis
2023-03-10 17:49:11.863 40848 INFO neutron.agent.ovn.metadata.agent [-] Provisioning metadata for network ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b
2023-03-10 17:49:11.917 40848 DEBUG neutron.agent.ovn.metadata.agent [-] Creating VETH tapovn-f376a61 in ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b namespace provision_datapath /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py:603
2023-03-10 17:49:11.923 41596 DEBUG neutron.privileged.agent.linux.ip_lib [-] Interface tapovn-f376a60 not found in namespace None get_link_id /home/zuul/src/opendev.org/openstack/neutron/neutron/privileged/agent/linux/ip_lib.py:204
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event [-] Unexpected exception in notify_loop: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface tapovn-f376a61 not found in namespace ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b.
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event Traceback (most recent call last):
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/event.py", line 177, in notify_loop
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event match.run(event, row, updates)
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py", line 110, in run
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event self.agent.provision_datapath(row.datapath)
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py", line 640, in provision_datapath
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event ip2.addr.add_multiple(ipv4_cidrs_to_add)
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 544, in add_multiple
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event add_ip_addresses(cidrs, self.name, self._parent.namespace, scope,
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 848, in add_ip_addresses
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event privileged.add_ip_addresses(
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event return self.channel.remote_call(name, args, kwargs,
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 215, in remote_call
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event raise exc_type(*result[2])
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface tapovn-f376a61 not found in namespace ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b.
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event
The thing is this line of code should actually be creating the new namespace[2] so not sure why its complaining that namespace was not found. I am suspecting there is some race condition or more likely some test interferance due to test runner concurrency.
Test neutron. tests.functiona l.agent. ovn.metadata. test_metadata_ agent.TestMetad ataAgent. test_agent_ resync_ on_non_ existing_ bridge is failing intermitelly.
I just reintroduced this test [1] into the code. The failure is does not happen all the time but I can be reproduce it locally with --until-failure with multiple concurrency(big hint) and running the whole TestMetadaAgent class of tests (another hint). Like this
`tox -e dsvm-functional -- neutron. tests.functiona l.agent. ovn.metadata. test_metadata_ agent.TestMetad ataAgent --until-failure --concurrency 0`
When the failure happens following exception is found in the logs
2023-03-10 17:49:11.861 40848 INFO neutron. agent.ovn. metadata. agent [-] Port ovn-port- feb89eb1- fcf8-4b38- 8aee-d8dd4b6b49 7e in datapath ovn-f376a6ca- 6f5b-4fa9- 9fa9-d5f450bb80 1b bound to our chassis agent.ovn. metadata. agent [-] Provisioning metadata for network ovn-f376a6ca- 6f5b-4fa9- 9fa9-d5f450bb80 1b agent.ovn. metadata. agent [-] Creating VETH tapovn-f376a61 in ovnmeta- ovn-f376a6ca- 6f5b-4fa9- 9fa9-d5f450bb80 1b namespace provision_datapath /home/zuul/ src/opendev. org/openstack/ neutron/ neutron/ agent/ovn/ metadata/ agent.py: 603 privileged. agent.linux. ip_lib [-] Interface tapovn-f376a60 not found in namespace None get_link_id /home/zuul/ src/opendev. org/openstack/ neutron/ neutron/ privileged/ agent/linux/ ip_lib. py:204 privileged. agent.linux. ip_lib. NetworkInterfac eNotFound: Network interface tapovn-f376a61 not found in namespace ovnmeta- ovn-f376a6ca- 6f5b-4fa9- 9fa9-d5f450bb80 1b. zuul/src/ opendev. org/openstack/ neutron/ .tox/dsvm- functional- gate/lib/ python3. 10/site- packages/ ovsdbapp/ event.py" , line 177, in notify_loop zuul/src/ opendev. org/openstack/ neutron/ neutron/ agent/ovn/ metadata/ agent.py" , line 110, in run provision_ datapath( row.datapath) zuul/src/ opendev. org/openstack/ neutron/ neutron/ agent/ovn/ metadata/ agent.py" , line 640, in provision_datapath add_multiple( ipv4_cidrs_ to_add) zuul/src/ opendev. org/openstack/ neutron/ neutron/ agent/linux/ ip_lib. py", line 544, in add_multiple addresses( cidrs, self.name, self._parent. namespace, scope, zuul/src/ opendev. org/openstack/ neutron/ neutron/ agent/linux/ ip_lib. py", line 848, in add_ip_addresses add_ip_ addresses( zuul/src/ opendev. org/openstack/ neutron/ .tox/dsvm- functional- gate/lib/ python3. 10/site- packages/ oslo_privsep/ priv_context. py", line 271, in _wrap remote_ call(name, args, kwargs, zuul/src/ opendev. org/openstack/ neutron/ .tox/dsvm- functional- gate/lib/ python3. 10/site- packages/ oslo_privsep/ daemon. py", line 215, in remote_call *result[ 2]) privileged. agent.linux. ip_lib. NetworkInterfac eNotFound: Network interface tapovn-f376a61 not found in namespace ovnmeta- ovn-f376a6ca- 6f5b-4fa9- 9fa9-d5f450bb80 1b.
2023-03-10 17:49:11.863 40848 INFO neutron.
2023-03-10 17:49:11.917 40848 DEBUG neutron.
2023-03-10 17:49:11.923 41596 DEBUG neutron.
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event [-] Unexpected exception in notify_loop: neutron.
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event Traceback (most recent call last):
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event match.run(event, row, updates)
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event self.agent.
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event ip2.addr.
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event add_ip_
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event privileged.
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event return self.channel.
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event raise exc_type(
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event neutron.
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event
The thing is this line of code should actually be creating the new namespace[2] so not sure why its complaining that namespace was not found. I am suspecting there is some race condition or more likely some test interferance due to test runner concurrency.
[1] https:/ /review. opendev. org/c/openstack /neutron/ +/875586 /github. com/openstack/ neutron/ blob/master/ neutron/ agent/ovn/ metadata/ agent.py# LL610-L611
[2] https:/