update-status failures on s390x

[Laurent Sesquès]

On an s390x compute node, neutron-openvswitch fails at the update-status hook. (this cloud is running ocata, with 17.11 charms)

The error is:

2018-02-02 08:19:32 DEBUG update-status Traceback (most recent call last):
2018-02-02 08:19:32 DEBUG update-status File "/var/lib/juju/agents/unit-neutron-openvswitch-4/charm/hooks/update-status", line 187, in <module>
2018-02-02 08:19:32 DEBUG update-status main()
2018-02-02 08:19:32 DEBUG update-status File "/var/lib/juju/agents/unit-neutron-openvswitch-4/charm/hooks/update-status", line 183, in main
2018-02-02 08:19:32 DEBUG update-status assess_status(CONFIGS)
2018-02-02 08:19:32 DEBUG update-status File "/var/lib/juju/agents/unit-neutron-openvswitch-4/charm/hooks/neutron_ovs_utils.py", line 770, in assess_status
2018-02-02 08:19:32 DEBUG update-status assess_status_func(configs)()
2018-02-02 08:19:32 DEBUG update-status File "/var/lib/juju/agents/unit-neutron-openvswitch-4/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1852, in _assess_status_func
2018-02-02 08:19:32 DEBUG update-status state, message = _determine_os_workload_status(*args, **kwargs)
2018-02-02 08:19:32 DEBUG update-status File "/var/lib/juju/agents/unit-neutron-openvswitch-4/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1252, in _determine_os_workload_status
2018-02-02 08:19:32 DEBUG update-status configs, required_interfaces)
2018-02-02 08:19:32 DEBUG update-status File "/var/lib/juju/agents/unit-neutron-openvswitch-4/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1306, in _ows_check_generic_interfaces
2018-02-02 08:19:32 DEBUG update-status required_interfaces)
2018-02-02 08:19:32 DEBUG update-status File "/var/lib/juju/agents/unit-neutron-openvswitch-4/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1587, in incomplete_relation_data
2018-02-02 08:19:32 DEBUG update-status complete_ctxts = configs.complete_contexts()
2018-02-02 08:19:32 DEBUG update-status File "/var/lib/juju/agents/unit-neutron-openvswitch-4/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 304, in complete_contexts
2018-02-02 08:19:32 DEBUG update-status for i in six.itervalues(self.templates)]
2018-02-02 08:19:32 DEBUG update-status File "/var/lib/juju/agents/unit-neutron-openvswitch-4/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 304, in <listcomp>
2018-02-02 08:19:32 DEBUG update-status for i in six.itervalues(self.templates)]
2018-02-02 08:19:32 DEBUG update-status File "/var/lib/juju/agents/unit-neutron-openvswitch-4/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 124, in complete_contexts
2018-02-02 08:19:32 DEBUG update-status self.context()
2018-02-02 08:19:32 DEBUG update-status File "/var/lib/juju/agents/unit-neutron-openvswitch-4/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 109, in context
2018-02-02 08:19:32 DEBUG update-status _ctxt = context()
2018-02-02 08:19:32 DEBUG update-status File "/var/lib/juju/agents/unit-neutron-openvswitch-4/charm/hooks/charmhelpers/contrib/openstack/context.py", line 981, in __call__
2018-02-02 08:19:32 DEBUG update-status ctxt.update(self.ovs_ctxt())
2018-02-02 08:19:32 DEBUG update-status File "/var/lib/juju/agents/unit-neutron-openvswitch-4/charm/hooks/neutron_ovs_context.py", line 108, in ovs_ctxt
2018-02-02 08:19:32 DEBUG update-status fallback)
2018-02-02 08:19:32 DEBUG update-status File "/var/lib/juju/agents/unit-neutron-openvswitch-4/charm/hooks/charmhelpers/contrib/network/ip.py", line 112, in get_address_in_network
2018-02-02 08:19:32 DEBUG update-status addresses = netifaces.ifaddresses(iface)
2018-02-02 08:19:32 DEBUG update-status ValueError: You must specify a valid interface name.
2018-02-02 08:19:32 ERROR juju.worker.uniter.operation runhook.go:107 hook "update-status" failed: exit status 1

As the stack trace says, it happens in ip.py", line 112:
        for iface in netifaces.interfaces():
            addresses = netifaces.ifaddresses(iface)

By running such a loop manually, I could see that it fails 50% of the time, either on a tap device or on a qbr* one:
tapc3c1d498-70
Traceback (most recent call last):
  File "./debug_netifaces.py", line 9, in <module>
    print(netifaces.ifaddresses(iface))
ValueError: You must specify a valid interface name.

My understanding is that this is a race condition, if an instance is present when netifaces.interfaces() but has been deleted when the loop reaches its netifaces.ifaddresses(iface).
At the moment, netifaces.interfaces() returns 198 interfaces.

This results in the neutron-openvswitch unit being in error 50% of the time.

[Laurent Sesquès (sajoupa)]

Fixed in https://github.com/juju/charm-helpers/pull/107

[James Page (james-page)]

Considering this was fixed in charmhelpers in Feb, this formed part of the 18.05 charm release.

Marking Fix Released.