neutron

linuxbridge-agent ERRORS with pyroute2.netlink.exceptions.NetlinkError if ipv6 link local address exist

Bug #1903835 reported by Jake Yip
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Invalid
Undecided
Unassigned

Bug Description

Hi all. We are running neutron train. After a minor version upgrade recently, linuxbridge-agent started throwing errors like this:

2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent [req-947d5add-cdd0-437d-a2eb-1c17d2556cd3 - - - - -] Error in agent loop. Devices info: {'current': {'tap25db63b7-9b', 'tapa6658716-0e', 'tapeae0251d-48', 'tapfeeae322-76', 'tap2dcf0e81-2d', 'tapd908490e-6f', 'tap4208ea46-9f', 'tape4a832ef
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent Traceback (most recent call last):
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/opt/neutron/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 465, in daemon_loop
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent sync = self.process_network_devices(device_info)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/opt/train/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent result = f(*args, **kwargs)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/opt/neutron/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 214, in process_network_devices
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent resync_a = self.treat_devices_added_updated(devices_added_updated)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/opt/train/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent result = f(*args, **kwargs)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/opt/neutron/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 231, in treat_devices_added_updated
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent self._process_device_if_exists(device_details)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/opt/neutron/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 258, in _process_device_if_exists
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent device, device_details['device_owner'])
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/opt/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 588, in plug_interface
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent network_segment.mtu)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/opt/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 524, in add_tap_interface
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent return False
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/opt/train/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent self.force_reraise()
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/opt/train/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent six.reraise(self.type_, self.value, self.tb)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/opt/train/lib/python3.6/site-packages/six.py", line 693, in reraise
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent raise value
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/opt/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 516, in add_tap_interface
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent tap_device_name, device_owner, mtu)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/opt/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 549, in _add_tap_interface
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent mtu):
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/opt/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 500, in ensure_physical_in_bridge
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent physical_interface)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/opt/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 288, in ensure_flat_bridge
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent if self.ensure_bridge(bridge_name, physical_interface):
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/opt/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 458, in ensure_bridge
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent self.update_interface_ip_details(bridge_name, interface)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/opt/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 420, in update_interface_ip_details
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent gateway)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/opt/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 396, in _update_interface_ip_details
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent dst_device.addr.add(cidr=ip['cidr'])
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/opt/neutron/neutron/agent/linux/ip_lib.py", line 528, in add
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent add_broadcast)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/opt/neutron/neutron/agent/linux/ip_lib.py", line 802, in add_ip_address
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent device, namespace, scope, broadcast)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/opt/neutron/neutron/privileged/agent/linux/ip_lib.py", line 73, in sync_inner
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent return input_func(*args, **kwargs)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/opt/train/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 245, in _wrap
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent return self.channel.remote_call(name, args, kwargs)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/opt/train/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 224, in remote_call
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent raise exc_type(*result[2])
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent pyroute2.netlink.exceptions.NetlinkError: (13, 'Permission denied')
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent

I think I've traced it down to the (backported) change https://review.opendev.org/#/c/747627/

It happens because the linuxbridge-agent interface has a link-local ipv6 address. Prior to this patch, linuxbridge-agent works fine. With this patch, it errors. Removing the link-local ipv6 address will solve the problem.

It may be the correct way is to not have ipv6 link-local addresses on linuxbridge interface; we are working towards removing them. Just thought that I might put this up as a bug, in case operators and developers run into the same problem.

Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Hello Jake:

This is a duplicate of 1899141.

Please check you code has https://review.opendev.org/#/q/I55a0f4341b328af52ea3bd758a72f633fbe3abcb

Regards.

Changed in neutron:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.