Hello,
We have a Mitaka deploy with several projects running, with 3 controller nodes and 4 compute nodes. After restarting the L3 agent, for one specific tenant the floating IPs that are attached to an instance are not actually attached, L3 agent is throwing the following error:
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info [-] L3 agent failure to setup floating IPs
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 342, in configure_fip_addresses
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info return self.process_floating_ip_addresses(interface_name)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 314, in process_floating_ip_addresses
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info fip, interface_name, device)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_local_router.py", line 157, in add_floating_ip
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info self.floating_ip_added_dist(fip, ip_cidr)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_local_router.py", line 88, in floating_ip_added_dist
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info device.route.add_route(fip_cidr, str(rtr_2_fip.ip))
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 841, in add_route
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info self._run_as_root_detect_device_not_found([ip_version], tuple(args))
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 699, in _run_as_root_detect_device_not_found
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info raise exceptions.DeviceNotFoundError(device_name=self.name)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info self.force_reraise()
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info six.reraise(self.type_, self.value, self.tb)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 694, in _run_as_root_detect_device_not_found
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info return self._as_root(*args, **kwargs)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 361, in _as_root
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info use_root_namespace=use_root_namespace)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 94, in _as_root
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info log_fail_as_error=self.log_fail_as_error)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 103, in _execute
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info log_fail_as_error=log_fail_as_error)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 140, in execute
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info raise RuntimeError(msg)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info RuntimeError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Network is unreachable
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info [-] Failed to process floating IPs.
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 815, in process_external
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info fip_statuses = self.configure_fip_addresses(interface_name)
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 347, in configure_fip_addresses
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info raise n_exc.FloatingIpSetupException(msg)
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info FloatingIpSetupException: L3 agent failure to setup floating IPs
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info
As of now, every time we try to attach/detach the floating IP on an instance the L3 agent shows the same error. Neutron package versions are:
ii neutron-common 2:8.2.0-0ubuntu1~cloud0
ii neutron-dhcp-agent 2:8.2.0-0ubuntu1~cloud0
ii neutron-l3-agent 2:8.2.0-0ubuntu1~cloud0
ii neutron-metadata-agent 2:8.2.0-0ubuntu1~cloud0
ii neutron-openvswitch-agent 2:8.2.0-0ubuntu1~cloud0
ii neutron-plugin-ml2 2:8.2.0-0ubuntu1~cloud0
OS: Ubuntu 14.04, uname: Linux compute01 3.16.0-77-generic #99~14.04.1-Ubuntu SMP Tue Jun 28 19:17:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
To follow up with a little more detail, we have two external networks. There are working floating IPs on the project, the one thing that is not happening is on the FIP namespace, when I attach a floating IP the route that sends traffic for that floating IP to the corresponding fpr interface is not being created.
In the following paste: http:// paste.openstack .org/show/ 588391/ I'm showing the FIP namespace with its fpr interfaces and the existing routes for the currently working floating IPs, as well as the rfp interface on the router. All the existing stuff before restarting L3 agent works, but the new route is not being created (if we add the route by hand on the namespace, traffic DOES work).