l3 agent Bad file descriptor

[Donny Davis]

The latest update for Stein has started throwing this error for some routers in the l3-agent. I have had this system in prod for about 8 months and have not had any issues until now with the l3-agent.

2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 25cd3192-d7cf-45a6-933d-4aa963900a74: OSError: [Errno 9] Bad file descriptor
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 723, in _process_routers_if_compatible
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 569, in _process_router_if_compatible
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent self._process_updated_router(router)
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 610, in _process_updated_router
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent ri.process()
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 161, in call
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent self.logger(e)
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent self.force_reraise()
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 158, in call
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent return func(*args, **kwargs)
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 1188, in process
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent self.process_external()
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 946, in process_external
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent self._process_external_gateway(ex_gw_port)
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 828, in _process_external_gateway
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent self.external_gateway_added(ex_gw_port, interface_name)
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 779, in external_gateway_added
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent ex_gw_port, interface_name, self.ns_name, preserve_ips)
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 728, in _external_gateway_added
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent self._plug_external_gateway(ex_gw_port, interface_name, ns_name)
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 670, in _plug_external_gateway
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent mtu=ex_gw_port.get('mtu'))
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 265, in plug
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent namespace=namespace):
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 818, in device_exists
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent return IPDevice(device_name, namespace=namespace).exists()
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 318, in exists
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent return privileged.interface_exists(self.name, self.namespace)
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/privileged/agent/linux/ip_lib.py", line 50, in sync_inner
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent return input_func(*args, **kwargs)
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 241, in _wrap
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent return self.channel.remote_call(name, args, kwargs)
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 203, in remote_call
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent raise exc_type(*result[2])
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent OSError: [Errno 9] Bad file descriptor
2019-12-16 12:13:36.355 20785 ERROR neutron.agent.l3.agent

[Slawek Kaplonski (slaweq)]

What version of pyroute You are using? I think You just hit bug: https://bugs.launchpad.net/neutron/+bug/1811515 and easiest solution for that would be to upgrade pyroute to 0.5.6 or 0.5.7

[Donny Davis (donny-g)]

I will give that a spin. Unfortunately 0.5.3 is the latest available in the RDO Stein repo.

I will get it upgraded and see if that fixes it. If that does in fact turn out to resolve the issue, I will open something up with RDO to bump the version in the stein repo.

[Donny Davis (donny-g)]

Yep, that fixed it. Thank you @slaweq

[Donny Davis (donny-g)]

Installing pyroute2 from the train repo has resolved this bug

http://mirror.centos.org/centos/7/cloud/x86_64/openstack-train/python2-pyroute2-0.5.6-1.el7.noarch.rpm

[chalansonnet (schalans)]

Hello,

Thank you for the resolution, we have the same bug on our RDO Stein (Kolla-Ansible) and that solved it !
Kolla communauty should package a new neutron_l3_agent with the rpm updated for Stein version

Regards,