Comment 17 for bug 1961740

This issue still happens, like e.g. https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e14/841094/1/check/neutron-functional-with-uwsgi/e145a4a/testr_results.html

I checked it and there is something strange going on there. Interface which caused failure was added/removed/added again in short time:

2022-05-09T11:02:13.807Z|03542|bridge|INFO|bridge test-bre924f925: added interface qr-a3915f78-e1 on port 1
2022-05-09T11:02:13.965Z|03551|bridge|INFO|bridge test-bre924f925: deleted interface qr-a3915f78-e1 on port 1
2022-05-09T11:02:14.058Z|03555|bridge|INFO|bridge test-bre924f925: added interface qr-a3915f78-e1 on port 1
2022-05-09T11:02:28.371Z|03687|bridge|INFO|bridge test-bre924f925: deleted interface qr-a3915f78-e1 on port 1

The last delete is when test already failed and there was cleanup phase so that's not a problem for sure.
I checked also other qr- port in logs of the same job (random one) and it was just created and deleted:

2022-05-09T11:02:28.426Z|03688|bridge|INFO|bridge test-br7dd57a6d: added interface qr-c9468ae1-58 on port 2
2022-05-09T11:02:42.990Z|03797|bridge|INFO|bridge test-br7dd57a6d: deleted interface qr-c9468ae1-58 on port 2

Getting back to the failed interface qr-a3915f78-e1, even if it was added again after few miliseconds, neutron couldn't set it to namespace at all:

2022-05-09 11:02:18.877 68464 WARNING neutron.agent.linux.interface [req-552fe784-bb96-48e2-9b29-edd623021e43 - - - - -] Failed to set interface qr-a3915f78-e1 into namespace qrouter-02f072b2-b45f-427c-b708-33fad2beb401. Interface not found in namespace, attempt: 1, retrying.: neutron.common.utils.WaitTimeout: Timed out after 5 seconds
2022-05-09 11:02:19.942 68464 WARNING neutron.agent.linux.interface [req-552fe784-bb96-48e2-9b29-edd623021e43 - - - - -] Failed to set interface qr-a3915f78-e1 into namespace qrouter-02f072b2-b45f-427c-b708-33fad2beb401. Interface not found, attempt: 2, retrying.: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface qr-a3915f78-e1 not found in namespace qrouter-02f072b2-b45f-427c-b708-33fad2beb401.
2022-05-09 11:02:21.049 68464 WARNING neutron.agent.linux.interface [req-552fe784-bb96-48e2-9b29-edd623021e43 - - - - -] Failed to set interface qr-a3915f78-e1 into namespace qrouter-02f072b2-b45f-427c-b708-33fad2beb401. Interface not found, attempt: 3, retrying.: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface qr-a3915f78-e1 not found in namespace qrouter-02f072b2-b45f-427c-b708-33fad2beb401.
2022-05-09 11:02:22.081 68464 WARNING neutron.agent.linux.interface [req-552fe784-bb96-48e2-9b29-edd623021e43 - - - - -] Failed to set interface qr-a3915f78-e1 into namespace qrouter-02f072b2-b45f-427c-b708-33fad2beb401. Interface not found, attempt: 4, retrying.: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface qr-a3915f78-e1 not found in namespace qrouter-02f072b2-b45f-427c-b708-33fad2beb401.
2022-05-09 11:02:23.115 68464 WARNING neutron.agent.linux.interface [req-552fe784-bb96-48e2-9b29-edd623021e43 - - - - -] Failed to set interface qr-a3915f78-e1 into namespace qrouter-02f072b2-b45f-427c-b708-33fad2beb401. Interface not found, attempt: 5, retrying.: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface qr-a3915f78-e1 not found in namespace qrouter-02f072b2-b45f-427c-b708-33fad2beb401.
2022-05-09 11:02:24.157 68464 WARNING neutron.agent.linux.interface [req-552fe784-bb96-48e2-9b29-edd623021e43 - - - - -] Failed to set interface qr-a3915f78-e1 into namespace qrouter-02f072b2-b45f-427c-b708-33fad2beb401. Interface not found, attempt: 6, retrying.: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface qr-a3915f78-e1 not found in namespace qrouter-02f072b2-b45f-427c-b708-33fad2beb401.
2022-05-09 11:02:25.188 68464 WARNING neutron.agent.linux.interface [req-552fe784-bb96-48e2-9b29-edd623021e43 - - - - -] Failed to set interface qr-a3915f78-e1 into namespace qrouter-02f072b2-b45f-427c-b708-33fad2beb401. Interface not found, attempt: 7, retrying.: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface qr-a3915f78-e1 not found in namespace qrouter-02f072b2-b45f-427c-b708-33fad2beb401.
2022-05-09 11:02:26.221 68464 WARNING neutron.agent.linux.interface [req-552fe784-bb96-48e2-9b29-edd623021e43 - - - - -] Failed to set interface qr-a3915f78-e1 into namespace qrouter-02f072b2-b45f-427c-b708-33fad2beb401. Interface not found, attempt: 8, retrying.: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface qr-a3915f78-e1 not found in namespace qrouter-02f072b2-b45f-427c-b708-33fad2beb401.
2022-05-09 11:02:27.294 68464 WARNING neutron.agent.linux.interface [req-552fe784-bb96-48e2-9b29-edd623021e43 - - - - -] Failed to set interface qr-a3915f78-e1 into namespace qrouter-02f072b2-b45f-427c-b708-33fad2beb401. Interface not found, attempt: 9, retrying.: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface qr-a3915f78-e1 not found in namespace qrouter-02f072b2-b45f-427c-b708-33fad2beb401.
2022-05-09 11:02:28.367 68464 WARNING neutron.agent.linux.interface [req-552fe784-bb96-48e2-9b29-edd623021e43 - - - - -] Failed to plug interface qr-a3915f78-e1 into bridge test-bre924f925, cleaning up: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface qr-a3915f78-e1 not found in namespace qrouter-02f072b2-b45f-427c-b708-33fad2beb401.
2022-05-09 11:02:28.418 68464 ERROR neutron.agent.l3.router_info [req-552fe784-bb96-48e2-9b29-edd623021e43 - - - - -] Network interface qr-a3915f78-e1 not found in namespace qrouter-02f072b2-b45f-427c-b708-33fad2beb401.: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface qr-a3915f78-e1 not found in namespace qrouter-02f072b2-b45f-427c-b708-33fad2beb401.

What is even more strange, is the fact that error which was seen in the logs is coming from the method _add_device_to_namespace(): https://github.com/openstack/neutron/blob/182044d628314290a38440bab203d27fb9db755d/neutron/agent/linux/interface.py#L361 which is called only in https://github.com/openstack/neutron/blob/182044d628314290a38440bab203d27fb9db755d/neutron/agent/linux/interface.py#L422 and it failed in that place BUT earlier there was self._set_device_address method called and this was fine, there was no errors in it at all.