It appears this started around Jan 9. Looking at changes to neutron and upper-constraints for dependent package versions we test with, these stand out given there are privsep errors in the neutron l3 agent logs:
In fact, the upper-constraints change depends on the neutron change, so I'm guessing those are the roots of the problem and probably require some more debugging. We do see privsep failures in the l3 agent log:
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[139657795965328]: 255 {{(pid=5905) _process_cmd /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:452}}
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: Traceback (most recent call last):
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: File "/usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py", line 447, in _process_cmd
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: ret = func(*f_args, **f_kwargs)
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: File "/usr/local/lib/python2.7/dist-packages/oslo_privsep/priv_context.py", line 216, in _wrap
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: return func(*args, **kwargs)
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: File "/opt/stack/neutron/neutron/privileged/agent/linux/ip_lib.py", line 331, in set_link_attribute
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: return _run_iproute_link("set", device, namespace, **attributes)
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: File "/opt/stack/neutron/neutron/privileged/agent/linux/ip_lib.py", line 194, in _run_iproute_link
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: return ip.link(command, index=idx, **kwargs)
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: File "/usr/local/lib/python2.7/dist-packages/pyroute2/iproute/linux.py", line 1115, in link
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: msg_flags=msg_flags)
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: File "/usr/local/lib/python2.7/dist-packages/pyroute2/netlink/nlsocket.py", line 352, in nlm_request
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: return tuple(self._genlm_request(*argv, **kwarg))
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: File "/usr/local/lib/python2.7/dist-packages/pyroute2/netlink/nlsocket.py", line 836, in nlm_request
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: callback=callback):
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: File "/usr/local/lib/python2.7/dist-packages/pyroute2/netlink/nlsocket.py", line 355, in get
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: return tuple(self._genlm_get(*argv, **kwarg))
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: File "/usr/local/lib/python2.7/dist-packages/pyroute2/netlink/nlsocket.py", line 752, in get
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: callback)
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: File "/usr/local/lib/python2.7/dist-packages/pyroute2/netlink/nlsocket.py", line 174, in parse
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: msg.decode()
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: File "/usr/local/lib/python2.7/dist-packages/pyroute2/netlink/__init__.py", line 981, in decode
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: self.unregister_clean_cb()
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: File "/usr/local/lib/python2.7/dist-packages/pyroute2/netlink/__init__.py", line 751, in unregister_clean_cb
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: del clean_cbs[seq]
Jan 13 17:57:16.887623 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: KeyError: 255
Jan 13 17:57:16.889489 ubuntu-bionic-vexxhost-sjc1-0001673055 neutron-l3-agent[4880]: DEBUG oslo.privsep.daemon [-] privsep: reply[139657795965328]: (5, 'exceptions.KeyError', (255,)) {{(pid=5905) _call_back /usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py:473}}
Looking at some failures in logstash:
http:// logstash. openstack. org/#dashboard/ file/logstash. json?query= message% 3A%5C%22OSError %3A%20% 5BErrno% 209%5D% 20Bad%20file% 20descriptor% 5C%22%20AND% 20build_ status% 3A%5C%22FAILURE %5C%22
It appears this started around Jan 9. Looking at changes to neutron and upper-constraints for dependent package versions we test with, these stand out given there are privsep errors in the neutron l3 agent logs:
https:/ /github. com/openstack/ neutron/ commit/ c6d358d4c692663 8fe9d5194e3da11 2c2750c6a4
https:/ /github. com/openstack/ requirements/ commit/ 6b45c47e53b8820 b68ff78eaec8062 c4fdf05a56# diff-0bdd949ed8 a7fdd4f95240bd9 51779c8
In fact, the upper-constraints change depends on the neutron change, so I'm guessing those are the roots of the problem and probably require some more debugging. We do see privsep failures in the l3 agent log:
http:// logs.openstack. org/43/ 628943/ 2/check/ tempest- full/07fe3fe/ controller/ logs/screen- q-l3.txt. gz#_Jan_ 13_17_57_ 16_887623
Jan 13 17:57:16.887623 ubuntu- bionic- vexxhost- sjc1-0001673055 neutron- l3-agent[ 4880]: DEBUG oslo.privsep.daemon [-] privsep: Exception during request[ 139657795965328 ]: 255 {{(pid=5905) _process_cmd /usr/local/ lib/python2. 7/dist- packages/ oslo_privsep/ daemon. py:452} } bionic- vexxhost- sjc1-0001673055 neutron- l3-agent[ 4880]: Traceback (most recent call last): bionic- vexxhost- sjc1-0001673055 neutron- l3-agent[ 4880]: File "/usr/local/ lib/python2. 7/dist- packages/ oslo_privsep/ daemon. py", line 447, in _process_cmd bionic- vexxhost- sjc1-0001673055 neutron- l3-agent[ 4880]: ret = func(*f_args, **f_kwargs) bionic- vexxhost- sjc1-0001673055 neutron- l3-agent[ 4880]: File "/usr/local/ lib/python2. 7/dist- packages/ oslo_privsep/ priv_context. py", line 216, in _wrap bionic- vexxhost- sjc1-0001673055 neutron- l3-agent[ 4880]: return func(*args, **kwargs) bionic- vexxhost- sjc1-0001673055 neutron- l3-agent[ 4880]: File "/opt/stack/ neutron/ neutron/ privileged/ agent/linux/ ip_lib. py", line 331, in set_link_attribute bionic- vexxhost- sjc1-0001673055 neutron- l3-agent[ 4880]: return _run_iproute_ link("set" , device, namespace, **attributes) bionic- vexxhost- sjc1-0001673055 neutron- l3-agent[ 4880]: File "/opt/stack/ neutron/ neutron/ privileged/ agent/linux/ ip_lib. py", line 194, in _run_iproute_link bionic- vexxhost- sjc1-0001673055 neutron- l3-agent[ 4880]: return ip.link(command, index=idx, **kwargs) bionic- vexxhost- sjc1-0001673055 neutron- l3-agent[ 4880]: File "/usr/local/ lib/python2. 7/dist- packages/ pyroute2/ iproute/ linux.py" , line 1115, in link bionic- vexxhost- sjc1-0001673055 neutron- l3-agent[ 4880]: msg_flags= msg_flags) bionic- vexxhost- sjc1-0001673055 neutron- l3-agent[ 4880]: File "/usr/local/ lib/python2. 7/dist- packages/ pyroute2/ netlink/ nlsocket. py", line 352, in nlm_request bionic- vexxhost- sjc1-0001673055 neutron- l3-agent[ 4880]: return tuple(self. _genlm_ request( *argv, **kwarg)) bionic- vexxhost- sjc1-0001673055 neutron- l3-agent[ 4880]: File "/usr/local/ lib/python2. 7/dist- packages/ pyroute2/ netlink/ nlsocket. py", line 836, in nlm_request bionic- vexxhost- sjc1-0001673055 neutron- l3-agent[ 4880]: callback=callback): bionic- vexxhost- sjc1-0001673055 neutron- l3-agent[ 4880]: File "/usr/local/ lib/python2. 7/dist- packages/ pyroute2/ netlink/ nlsocket. py", line 355, in get bionic- vexxhost- sjc1-0001673055 neutron- l3-agent[ 4880]: return tuple(self. _genlm_ get(*argv, **kwarg)) bionic- vexxhost- sjc1-0001673055 neutron- l3-agent[ 4880]: File "/usr/local/ lib/python2. 7/dist- packages/ pyroute2/ netlink/ nlsocket. py", line 752, in get bionic- vexxhost- sjc1-0001673055 neutron- l3-agent[ 4880]: callback) bionic- vexxhost- sjc1-0001673055 neutron- l3-agent[ 4880]: File "/usr/local/ lib/python2. 7/dist- packages/ pyroute2/ netlink/ nlsocket. py", line 174, in parse bionic- vexxhost- sjc1-0001673055 neutron- l3-agent[ 4880]: msg.decode() bionic- vexxhost- sjc1-0001673055 neutron- l3-agent[ 4880]: File "/usr/local/ lib/python2. 7/dist- packages/ pyroute2/ netlink/ __init_ _.py", line 981, in decode bionic- vexxhost- sjc1-0001673055 neutron- l3-agent[ 4880]: self.unregister _clean_ cb() bionic- vexxhost- sjc1-0001673055 neutron- l3-agent[ 4880]: File "/usr/local/ lib/python2. 7/dist- packages/ pyroute2/ netlink/ __init_ _.py", line 751, in unregister_clean_cb bionic- vexxhost- sjc1-0001673055 neutron- l3-agent[ 4880]: del clean_cbs[seq] bionic- vexxhost- sjc1-0001673055 neutron- l3-agent[ 4880]: KeyError: 255 bionic- vexxhost- sjc1-0001673055 neutron- l3-agent[ 4880]: DEBUG oslo.privsep.daemon [-] privsep: reply[139657795 965328] : (5, 'exceptions. KeyError' , (255,)) {{(pid=5905) _call_back /usr/local/ lib/python2. 7/dist- packages/ oslo_privsep/ daemon. py:473} }
Jan 13 17:57:16.887623 ubuntu-
Jan 13 17:57:16.887623 ubuntu-
Jan 13 17:57:16.887623 ubuntu-
Jan 13 17:57:16.887623 ubuntu-
Jan 13 17:57:16.887623 ubuntu-
Jan 13 17:57:16.887623 ubuntu-
Jan 13 17:57:16.887623 ubuntu-
Jan 13 17:57:16.887623 ubuntu-
Jan 13 17:57:16.887623 ubuntu-
Jan 13 17:57:16.887623 ubuntu-
Jan 13 17:57:16.887623 ubuntu-
Jan 13 17:57:16.887623 ubuntu-
Jan 13 17:57:16.887623 ubuntu-
Jan 13 17:57:16.887623 ubuntu-
Jan 13 17:57:16.887623 ubuntu-
Jan 13 17:57:16.887623 ubuntu-
Jan 13 17:57:16.887623 ubuntu-
Jan 13 17:57:16.887623 ubuntu-
Jan 13 17:57:16.887623 ubuntu-
Jan 13 17:57:16.887623 ubuntu-
Jan 13 17:57:16.887623 ubuntu-
Jan 13 17:57:16.887623 ubuntu-
Jan 13 17:57:16.887623 ubuntu-
Jan 13 17:57:16.887623 ubuntu-
Jan 13 17:57:16.887623 ubuntu-
Jan 13 17:57:16.887623 ubuntu-
Jan 13 17:57:16.889489 ubuntu-