devstack py3 get_link_devices() KeyError: 'index'

[iain MacDonnell]

devstack master with py3. openvswitch agent has suddenly stopped working, with no change in config or environment (other than rebuilding devstack). Stack trace below. For some reason (yet undetermined), privileged.get_link_devices() now seems to be returning byte arrays instead of strings as the dict keys:

>>> from neutron.privileged.agent.linux import ip_lib as privileged
>>> privileged.get_link_devices(None)[0].keys()
dict_keys([b'index', b'family', b'__align', b'header', b'flags', b'ifi_type', b'event', b'change', b'attrs'])
>>>

From agent startup:

neutron-openvswitch-agent[42936]: ERROR neutron Traceback (most recent call last):
neutron-openvswitch-agent[42936]: ERROR neutron File "/usr/local/bin/neutron-openvswitch-agent", line 10, in <module>
neutron-openvswitch-agent[42936]: ERROR neutron sys.exit(main())
neutron-openvswitch-agent[42936]: ERROR neutron File "/opt/stack/neutron/neutron/cmd/eventlet/plugins/ovs_neutron_agent.py", line 20, in main
neutron-openvswitch-agent[42936]: ERROR neutron agent_main.main()
neutron-openvswitch-agent[42936]: ERROR neutron File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/main.py", line 47, in main
neutron-openvswitch-agent[42936]: ERROR neutron mod.main()
neutron-openvswitch-agent[42936]: ERROR neutron File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/main.py", line 35, in main
neutron-openvswitch-agent[42936]: ERROR neutron 'neutron.plugins.ml2.drivers.openvswitch.agent.'
neutron-openvswitch-agent[42936]: ERROR neutron File "/usr/local/lib/python3.6/dist-packages/os_ken/base/app_manager.py", line 375, in run_apps
neutron-openvswitch-agent[42936]: ERROR neutron hub.joinall(services)
neutron-openvswitch-agent[42936]: ERROR neutron File "/usr/local/lib/python3.6/dist-packages/os_ken/lib/hub.py", line 102, in joinall
neutron-openvswitch-agent[42936]: ERROR neutron t.wait()
neutron-openvswitch-agent[42936]: ERROR neutron File "/usr/local/lib/python3.6/dist-packages/eventlet/greenthread.py", line 180, in wait
neutron-openvswitch-agent[42936]: ERROR neutron return self._exit_event.wait()
neutron-openvswitch-agent[42936]: ERROR neutron File "/usr/local/lib/python3.6/dist-packages/eventlet/event.py", line 132, in wait
neutron-openvswitch-agent[42936]: ERROR neutron current.throw(*self._exc)
neutron-openvswitch-agent[42936]: ERROR neutron File "/usr/local/lib/python3.6/dist-packages/eventlet/greenthread.py", line 219, in main
neutron-openvswitch-agent[42936]: ERROR neutron result = function(*args, **kwargs)
neutron-openvswitch-agent[42936]: ERROR neutron File "/usr/local/lib/python3.6/dist-packages/os_ken/lib/hub.py", line 64, in _launch
neutron-openvswitch-agent[42936]: ERROR neutron raise e
neutron-openvswitch-agent[42936]: ERROR neutron File "/usr/local/lib/python3.6/dist-packages/os_ken/lib/hub.py", line 59, in _launch
neutron-openvswitch-agent[42936]: ERROR neutron return func(*args, **kwargs)
neutron-openvswitch-agent[42936]: ERROR neutron File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_oskenapp.py", line 43, in agent_main_wrapper
neutron-openvswitch-agent[42936]: ERROR neutron LOG.exception("Agent main thread died of an exception")
neutron-openvswitch-agent[42936]: ERROR neutron File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
neutron-openvswitch-agent[42936]: ERROR neutron self.force_reraise()
neutron-openvswitch-agent[42936]: ERROR neutron File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
neutron-openvswitch-agent[42936]: ERROR neutron six.reraise(self.type_, self.value, self.tb)
neutron-openvswitch-agent[42936]: ERROR neutron File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise
neutron-openvswitch-agent[42936]: ERROR neutron raise value
neutron-openvswitch-agent[42936]: ERROR neutron File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_oskenapp.py", line 40, in agent_main_wrapper
neutron-openvswitch-agent[42936]: ERROR neutron ovs_agent.main(bridge_classes)
neutron-openvswitch-agent[42936]: ERROR neutron File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2393, in main
neutron-openvswitch-agent[42936]: ERROR neutron validate_tunnel_config(cfg.CONF.AGENT.tunnel_types, cfg.CONF.OVS.local_ip)
neutron-openvswitch-agent[42936]: ERROR neutron File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2362, in validate_tunnel_config
neutron-openvswitch-agent[42936]: ERROR neutron validate_local_ip(local_ip)
neutron-openvswitch-agent[42936]: ERROR neutron File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2350, in validate_local_ip
neutron-openvswitch-agent[42936]: ERROR neutron if not ip_lib.IPWrapper().get_device_by_ip(local_ip):
neutron-openvswitch-agent[42936]: ERROR neutron File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 187, in get_device_by_ip
neutron-openvswitch-agent[42936]: ERROR neutron devices = get_devices_with_ip(self.namespace, **kwargs)
neutron-openvswitch-agent[42936]: ERROR neutron File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 1417, in get_devices_with_ip
neutron-openvswitch-agent[42936]: ERROR neutron for device in devices):
neutron-openvswitch-agent[42936]: ERROR neutron File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 1417, in <genexpr>
neutron-openvswitch-agent[42936]: ERROR neutron for device in devices):
neutron-openvswitch-agent[42936]: ERROR neutron File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 1388, in _parse_link_device
neutron-openvswitch-agent[42936]: ERROR neutron index=device['index'],
neutron-openvswitch-agent[42936]: ERROR neutron KeyError: 'index'

[Brian Haley (brian-haley)]

What version of pyroute2 is installed?

And can you reproduce this outside of neutron?

$ python3 -i
>>> import pyroute2
>>> with pyroute2.IPRoute() as ip:
... ip.get_links()

[iain MacDonnell (imacdonn)]

pyroute2 is 0.5.3

I've been trying to reproduce it outside neutron, but have not been able to...

# python3
Python 3.6.8 (default, Jan 14 2019, 11:02:34)
[GCC 8.0.1 20180414 (experimental) [trunk revision 259383]] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyroute2
>>> with pyroute2.IPRoute() as ip:
... links = ip.get_links()
...
>>> links[0].keys()
dict_keys(['family', '__align', 'ifi_type', 'index', 'flags', 'change', 'attrs', 'header', 'event'])
>>>

[iain MacDonnell (imacdonn)]

privsep may be doing something funky - note the change in logging:

# cat test.py
#!/usr/bin/python3

import logging
from neutron.privileged.agent.linux import ip_lib as privileged

logging.basicConfig(level=logging.DEBUG)

print(privileged.get_link_devices(None)[0].keys());

# ./test.py
INFO:oslo.privsep.daemon:Running privsep helper: ['sudo', 'privsep-helper', '--privsep_context', 'neutron.privileged.default', '--privsep_sock_path', '/tmp/tmped_r6cdn/privsep.sock']
INFO:oslo.privsep.daemon:Spawned new privsep daemon via rootwrap
DEBUG:oslo.privsep.daemon:Accepted privsep connection to /tmp/tmped_r6cdn/privsep.sock
b'INFO':b'oslo.privsep.daemon':b'privsep daemon starting'
b'INFO':b'oslo.privsep.daemon':b'privsep process running with uid/gid: 0/0'
b'INFO':b'oslo.privsep.daemon':b'privsep process running with capabilities (eff/prm/inh): CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN/CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN/none'
b'INFO':b'oslo.privsep.daemon':b'privsep daemon running as pid 10419'
dict_keys([b'index', b'family', b'__align', b'header', b'flags', b'ifi_type', b'event', b'change', b'attrs'])

[iain MacDonnell (imacdonn)]

Yeah... downgrading oslo.privsep from 1.33.0 to 1.32.1 makes the problem go away.

[iain MacDonnell (imacdonn)]

It gets more strange .. I went back to 1.33.0 (planning to revert changes, one at a time, to figure out which makes the difference), but the problem did not come back!

I have run into this on two separate hosts .. one is Ubuntu 18.04, the other 16.04. The 16.04 still has the problem (have not reinstalled oslo.privsep yet - don't want to unbreak it in case I can repro now!).

[iain MacDonnell (imacdonn)]

Reinstalling oslo.privsep seems to have "fixed" it on the 16.04 box too. Still don't understand why - must have been some bad cache or something. Will put this down to "gremlins" unless it resurfaces :/