| 2023-03-13 03:25:42 |
Miro Tomaska |
description |
Test neutron.tests.functional.agent.ovn.metadata.test_metadata_agent.TestMetadataAgent.test_agent_resync_on_non_existing_bridge is failing intermitelly.
I just reintroduced this test [1] into the code. The failure is does not happen all the time but I can be reproduce it locally with --until-failure with multiple concurrency(big hint) and running the whole TestMetadaAgent class of tests (another hint). Like this
`tox -e dsvm-functional -- neutron.tests.functional.agent.ovn.metadata.test_metadata_agent.TestMetadataAgent --until-failure --concurrency 0`
When the failure happens following exception is found in the logs
2023-03-10 17:49:11.861 40848 INFO neutron.agent.ovn.metadata.agent [-] Port ovn-port-feb89eb1-fcf8-4b38-8aee-d8dd4b6b497e in datapath ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b bound to our chassis
2023-03-10 17:49:11.863 40848 INFO neutron.agent.ovn.metadata.agent [-] Provisioning metadata for network ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b
2023-03-10 17:49:11.917 40848 DEBUG neutron.agent.ovn.metadata.agent [-] Creating VETH tapovn-f376a61 in ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b namespace provision_datapath /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py:603
2023-03-10 17:49:11.923 41596 DEBUG neutron.privileged.agent.linux.ip_lib [-] Interface tapovn-f376a60 not found in namespace None get_link_id /home/zuul/src/opendev.org/openstack/neutron/neutron/privileged/agent/linux/ip_lib.py:204
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event [-] Unexpected exception in notify_loop: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface tapovn-f376a61 not found in namespace ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b.
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event Traceback (most recent call last):
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/event.py", line 177, in notify_loop
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event match.run(event, row, updates)
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py", line 110, in run
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event self.agent.provision_datapath(row.datapath)
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py", line 640, in provision_datapath
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event ip2.addr.add_multiple(ipv4_cidrs_to_add)
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 544, in add_multiple
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event add_ip_addresses(cidrs, self.name, self._parent.namespace, scope,
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 848, in add_ip_addresses
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event privileged.add_ip_addresses(
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event return self.channel.remote_call(name, args, kwargs,
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 215, in remote_call
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event raise exc_type(*result[2])
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface tapovn-f376a61 not found in namespace ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b.
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event
The thing is this line of code should actually be creating the new namespace[2] so not sure why its complaining that namespace was not found. I am suspecting there is some race condition or more likely some test interferance due to test runner concurrency.
[1] https://review.opendev.org/c/openstack/neutron/+/875586
[2] https://github.com/openstack/neutron/blob/master/neutron/agent/ovn/metadata/agent.py#LL610-L611 |
Test neutron.tests.functional.agent.ovn.metadata.test_metadata_agent.TestMetadataAgent.test_agent_resync_on_non_existing_bridge is failing intermitelly. Ex [0]
I just reintroduced this test [1] into the code. The failure is does not happen all the time but I can be reproduce it locally with --until-failure with multiple concurrency(big hint) and running the whole TestMetadaAgent class of tests (another hint). Like this
`tox -e dsvm-functional -- neutron.tests.functional.agent.ovn.metadata.test_metadata_agent.TestMetadataAgent --until-failure --concurrency 0`
When the failure happens following exception is found in the logs
2023-03-10 17:49:11.861 40848 INFO neutron.agent.ovn.metadata.agent [-] Port ovn-port-feb89eb1-fcf8-4b38-8aee-d8dd4b6b497e in datapath ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b bound to our chassis
2023-03-10 17:49:11.863 40848 INFO neutron.agent.ovn.metadata.agent [-] Provisioning metadata for network ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b
2023-03-10 17:49:11.917 40848 DEBUG neutron.agent.ovn.metadata.agent [-] Creating VETH tapovn-f376a61 in ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b namespace provision_datapath /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py:603
2023-03-10 17:49:11.923 41596 DEBUG neutron.privileged.agent.linux.ip_lib [-] Interface tapovn-f376a60 not found in namespace None get_link_id /home/zuul/src/opendev.org/openstack/neutron/neutron/privileged/agent/linux/ip_lib.py:204
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event [-] Unexpected exception in notify_loop: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface tapovn-f376a61 not found in namespace ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b.
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event Traceback (most recent call last):
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/event.py", line 177, in notify_loop
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event match.run(event, row, updates)
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py", line 110, in run
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event self.agent.provision_datapath(row.datapath)
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/ovn/metadata/agent.py", line 640, in provision_datapath
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event ip2.addr.add_multiple(ipv4_cidrs_to_add)
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 544, in add_multiple
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event add_ip_addresses(cidrs, self.name, self._parent.namespace, scope,
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 848, in add_ip_addresses
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event privileged.add_ip_addresses(
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event return self.channel.remote_call(name, args, kwargs,
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 215, in remote_call
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event raise exc_type(*result[2])
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface tapovn-f376a61 not found in namespace ovnmeta-ovn-f376a6ca-6f5b-4fa9-9fa9-d5f450bb801b.
2023-03-10 17:49:12.368 40848 ERROR ovsdbapp.event
The thing is this line of code should actually be creating the new namespace[2] so not sure why its complaining that namespace was not found. I am suspecting there is some race condition or more likely some test interferance due to test runner concurrency.
[0] https://zuul.opendev.org/t/openstack/build/5d1910037db844e88cf9ef694068cf17
[1] https://review.opendev.org/c/openstack/neutron/+/875586
[2] https://github.com/openstack/neutron/blob/master/neutron/agent/ovn/metadata/agent.py#LL610-L611 |
|
