I don't think this issue is related to keepalived as keepalived is not used in scenario where those tests are failing.
I deployed similar env as in dvr-multinode scenario job used in gate and I run those tests.
It's very often that one of FIPs configured during test is not pinging. What I found there is that there is no routing to such FIP configured in fip-xxx namespace on host where it should be. Even after restart L3 agent it's not there so I think it's something wrong on server's side instead of some race on L3 agent.
Also when I detached and attached broken FIP to instance again, I saw in L3 agent's logs something like:
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info [-] L3 agent failure to setup floating IPs: ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "snat-d623a0c3-feed-4d29-8644-5b992d301eca": No such file or directory
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info Traceback (most recent call last):
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 406, in configure_fip_addresses
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info return self.process_floating_ip_addresses(interface_name)
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 349, in process_floating_ip_addresses
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info centralized_fip_cidrs = self.get_centralized_fip_cidr_set()
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/l3/dvr_edge_router.py", line 279, in get_centralized_fip_cidr_set
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info return set([addr['cidr'] for addr in device.addr.list()])
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 672, in list
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info self.name, scope, to, filters, ip_version)
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 637, in get_devices_with_ip
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info for line in self._run(options, tuple(args)).split('\n'):
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 372, in _run
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info return self._parent._run(options, self.COMMAND, args)
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 90, in _run
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info return self._as_root(options, command, args)
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 102, in _as_root
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info namespace=namespace)
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 110, in _execute
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info log_fail_as_error=self.log_fail_as_error)
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 147, in execute
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info returncode=returncode)
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "snat-d623a0c3-feed-4d29-8644-5b992d301eca": No such file or directory
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info [-] Failed to process floating IPs.: FloatingIpSetupException: L3 agent failure to setup floating IPs
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info Traceback (most recent call last):
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 940, in process_external
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info fip_statuses = self.configure_fip_addresses(interface_name)
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 411, in configure_fip_addresses
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info raise n_exc.FloatingIpSetupException(msg)
Nov 16 22:19:20 devstack-ubuntu-allinone neutron-l3-agent[25687]: ERROR neutron.agent.l3.router_info FloatingIpSetupException: L3 agent failure to setup floating IPs
As my agent was configured as "dvr_snat" I tried to change it's mode to "dvr". After restart agent FIP was configured properly and worked fine.
I don't think this issue is related to keepalived as keepalived is not used in scenario where those tests are failing.
I deployed similar env as in dvr-multinode scenario job used in gate and I run those tests.
It's very often that one of FIPs configured during test is not pinging. What I found there is that there is no routing to such FIP configured in fip-xxx namespace on host where it should be. Even after restart L3 agent it's not there so I think it's something wrong on server's side instead of some race on L3 agent.
Also when I detached and attached broken FIP to instance again, I saw in L3 agent's logs something like:
Nov 16 22:19:20 devstack- ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info [-] L3 agent failure to setup floating IPs: ProcessExecutio nError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "snat-d623a0c3- feed-4d29- 8644-5b992d301e ca": No such file or directory ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info Traceback (most recent call last): ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info File "/opt/stack/ neutron/ neutron/ agent/l3/ router_ info.py" , line 406, in configure_ fip_addresses ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info return self.process_ floating_ ip_addresses( interface_ name) ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info File "/opt/stack/ neutron/ neutron/ agent/l3/ router_ info.py" , line 349, in process_ floating_ ip_addresses ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info centralized_ fip_cidrs = self.get_ centralized_ fip_cidr_ set() ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info File "/opt/stack/ neutron/ neutron/ agent/l3/ dvr_edge_ router. py", line 279, in get_centralized _fip_cidr_ set ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info return set([addr['cidr'] for addr in device. addr.list( )]) ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info File "/opt/stack/ neutron/ neutron/ agent/linux/ ip_lib. py", line 672, in list ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info self.name, scope, to, filters, ip_version) ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info File "/opt/stack/ neutron/ neutron/ agent/linux/ ip_lib. py", line 637, in get_devices_with_ip ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info for line in self._run(options, tuple(args) ).split( '\n'): ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info File "/opt/stack/ neutron/ neutron/ agent/linux/ ip_lib. py", line 372, in _run ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info return self._parent. _run(options, self.COMMAND, args) ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info File "/opt/stack/ neutron/ neutron/ agent/linux/ ip_lib. py", line 90, in _run ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info return self._as_ root(options, command, args) ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info File "/opt/stack/ neutron/ neutron/ agent/linux/ ip_lib. py", line 102, in _as_root ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info namespace= namespace) ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info File "/opt/stack/ neutron/ neutron/ agent/linux/ ip_lib. py", line 110, in _execute ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info log_fail_ as_error= self.log_ fail_as_ error) ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info File "/opt/stack/ neutron/ neutron/ agent/linux/ utils.py" , line 147, in execute ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info returncode= returncode) ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info ProcessExecutio nError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "snat-d623a0c3- feed-4d29- 8644-5b992d301e ca": No such file or directory ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info [-] Failed to process floating IPs.: FloatingIpSetup Exception: L3 agent failure to setup floating IPs ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info Traceback (most recent call last): ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info File "/opt/stack/ neutron/ neutron/ agent/l3/ router_ info.py" , line 940, in process_external ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info fip_statuses = self.configure_ fip_addresses( interface_ name) ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info File "/opt/stack/ neutron/ neutron/ agent/l3/ router_ info.py" , line 411, in configure_ fip_addresses ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info raise n_exc.FloatingI pSetupException (msg) ubuntu- allinone neutron- l3-agent[ 25687]: ERROR neutron. agent.l3. router_ info FloatingIpSetup Exception: L3 agent failure to setup floating IPs
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
Nov 16 22:19:20 devstack-
As my agent was configured as "dvr_snat" I tried to change it's mode to "dvr". After restart agent FIP was configured properly and worked fine.
I will continue digging into it.