On upgrading neutron on a compute node, instances on that node wound up losing some of their network plumbing via openvswitch.
This cloud: Mitaka, xenial, openvswitch with gre.
2017-07-05 16:17:52, the following auto-upgrade occurred: neutron-common:amd64 (2:8.4.0-0ubuntu2, 2:8.4.0-0ubuntu3), neutron-plugin-openvswitch-agent:amd64 (2:8.4.0-0ubuntu2, 2:8.4.0-0ubuntu3), neutron-openvswitch-agent:amd64 (2:8.4.0-0ubuntu2, 2:8.4.0-0ubuntu3), python-neutron:amd64 (2:8.4.0-0ubuntu2, 2:8.4.0-0ubuntu3)
The neutron logs:
2017-07-05 16:17:53.670 6156 CRITICAL neutron [req-1fcb315b-9f12-4657-aea8-1463f55f0106 - - - - -] ProcessExecutionError: Exit code: -15; Stdin: ; Stdout: ; Stderr: Signal 15 (TERM) caught by ps (procps-ng version 3.3.10).
ps:display.c:66: please report this bug
2017-07-05 16:17:53.670 6156 ERROR neutron Traceback (most recent call last):
2017-07-05 16:17:53.670 6156 ERROR neutron File "/usr/bin/neutron-openvswitch-agent", line 10, in <module>
2017-07-05 16:17:53.670 6156 ERROR neutron sys.exit(main())
2017-07-05 16:17:53.670 6156 ERROR neutron File "/usr/lib/python2.7/dist-packages/neutron/cmd/eventlet/plugins/ovs_neutron_agent.py", line 20, in main
2017-07-05 16:17:53.670 6156 ERROR neutron agent_main.main()
2017-07-05 16:17:53.670 6156 ERROR neutron File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/main.py", line 49, in main
2017-07-05 16:17:53.670 6156 ERROR neutron mod.main()
2017-07-05 16:17:53.670 6156 ERROR neutron File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/ovs_ofctl/main.py", line 36, in main
2017-07-05 16:17:53.670 6156 ERROR neutron ovs_neutron_agent.main(bridge_classes)
2017-07-05 16:17:53.670 6156 ERROR neutron File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2164, in main
2017-07-05 16:17:53.670 6156 ERROR neutron agent.daemon_loop()
2017-07-05 16:17:53.670 6156 ERROR neutron File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2087, in daemon_loop
2017-07-05 16:17:53.670 6156 ERROR neutron self.rpc_loop(polling_manager=pm)
2017-07-05 16:17:53.670 6156 ERROR neutron File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2000, in rpc_loop
2017-07-05 16:17:53.670 6156 ERROR neutron if (self._agent_has_updates(polling_manager) or sync
2017-07-05 16:17:53.670 6156 ERROR neutron File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1752, in _agent_has_updates
2017-07-05 16:17:53.670 6156 ERROR neutron return (polling_manager.is_polling_required or
2017-07-05 16:17:53.670 6156 ERROR neutron File "/usr/lib/python2.7/dist-packages/neutron/agent/common/base_polling.py", line 35, in is_polling_required
2017-07-05 16:17:53.670 6156 ERROR neutron polling_required = self._is_polling_required()
2017-07-05 16:17:53.670 6156 ERROR neutron File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/polling.py", line 69, in _is_polling_required
2017-07-05 16:17:53.670 6156 ERROR neutron return self._monitor.has_updates
2017-07-05 16:17:53.670 6156 ERROR neutron File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ovsdb_monitor.py", line 74, in has_updates
2017-07-05 16:17:53.670 6156 ERROR neutron if not self.is_active():
2017-07-05 16:17:53.670 6156 ERROR neutron File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/async_process.py", line 100, in is_active
2017-07-05 16:17:53.670 6156 ERROR neutron self.pid, self.cmd_without_namespace)
2017-07-05 16:17:53.670 6156 ERROR neutron File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/async_process.py", line 159, in pid
2017-07-05 16:17:53.670 6156 ERROR neutron run_as_root=self.run_as_root)
2017-07-05 16:17:53.670 6156 ERROR neutron File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 267, in get_root_helper_child_pid
2017-07-05 16:17:53.670 6156 ERROR neutron pid = find_child_pids(pid)[0]
2017-07-05 16:17:53.670 6156 ERROR neutron File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 202, in find_child_pids
2017-07-05 16:17:53.670 6156 ERROR neutron return []
2017-07-05 16:17:53.670 6156 ERROR neutron File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-05 16:17:53.670 6156 ERROR neutron self.force_reraise()
2017-07-05 16:17:53.670 6156 ERROR neutron File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-05 16:17:53.670 6156 ERROR neutron six.reraise(self.type_, self.value, self.tb)
2017-07-05 16:17:53.670 6156 ERROR neutron File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 194, in find_child_pids
2017-07-05 16:17:53.670 6156 ERROR neutron log_fail_as_error=False)
2017-07-05 16:17:53.670 6156 ERROR neutron File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 146, in execute
2017-07-05 16:17:53.670 6156 ERROR neutron raise ProcessExecutionError(msg, returncode=returncode)
2017-07-05 16:17:53.670 6156 ERROR neutron ProcessExecutionError: Exit code: -15; Stdin: ; Stdout: ; Stderr: Signal 15 (TERM) caught by ps (procps-ng version 3.3.10).
2017-07-05 16:17:53.670 6156 ERROR neutron ps:display.c:66: please report this bug
2017-07-05 16:17:53.670 6156 ERROR neutron
2017-07-05 16:17:53.670 6156 ERROR neutron
2017-07-05 16:17:58.483 84539 WARNING oslo_config.cfg [-] Option "prevent_arp_spoofing" from group "AGENT" is deprecated for removal. Its value may be silently ignored in the future.
<silence>
2017-07-05 17:24:16.795 84539 WARNING neutron.plugins.ml2.drivers.openvswitch.agent.openflow.ovs_ofctl.ofswitch [req-12b545ec-8a7d-4faf-b669-d000be9c6ee6 - - - - -] Deleting flow cookie=0xae1f44d4f69895b4, duration=521455.277s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=10,icmp6,in_port=5,icmp_type=136 actions=resubmit(,24)
2017-07-05 17:24:16.888 84539 WARNING neutron.plugins.ml2.drivers.openvswitch.agent.openflow.ovs_ofctl.ofswitch [req-12b545ec-8a7d-4faf-b669-d000be9c6ee6 - - - - -] Deleting flow cookie=0xae1f44d4f69895b4, duration=521452.099s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=10,icmp6,in_port=121,icmp_type=136 actions=resubmit(,24)
2017-07-05 17:24:16.992 84539 WARNING neutron.plugins.ml2.drivers.openvswitch.agent.openflow.ovs_ofctl.ofswitch [req-12b545ec-8a7d-4faf-b669-d000be9c6ee6 - - - - -] Deleting flow cookie=0xae1f44d4f69895b4, duration=521451.309s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=10,icmp6,in_port=4,icmp_type=136 actions=resubmit(,24)
2017-07-05 17:24:17.102 84539 WARNING neutron.plugins.ml2.drivers.openvswitch.agent.openflow.ovs_ofctl.ofswitch [req-12b545ec-8a7d-4faf-b669-d000be9c6ee6 - - - - -] Deleting flow cookie=0xae1f44d4f69895b4, duration=521448.587s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=10,icmp6,in_port=6,icmp_type=136 actions=resubmit(,24)
<etc>
nova-compute.log:
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [req-f3b2cae2-80e1-4a65-8b36-85b50e9e70f3 404192d1e0df4068ad985bd423db4cae 1f0043836a7d46c0a70d82056f1bc99c - - -] [instance: 4451405e-5f24-42e3-8984-d6cb2f41e65c] Failed while unplugging vif
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance: 4451405e-5f24-42e3-8984-d6cb2f41e65c] Traceback (most recent call last):
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance: 4451405e-5f24-42e3-8984-d6cb2f41e65c] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/vif.py", line 777, in unplug_ovs_hybrid
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance: 4451405e-5f24-42e3-8984-d6cb2f41e65c] run_as_root=True)
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance: 4451405e-5f24-42e3-8984-d6cb2f41e65c] File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 388, in execute
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance: 4451405e-5f24-42e3-8984-d6cb2f41e65c] return RootwrapProcessHelper().execute(*cmd, **kwargs)
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance: 4451405e-5f24-42e3-8984-d6cb2f41e65c] File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 271, in execute
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance: 4451405e-5f24-42e3-8984-d6cb2f41e65c] return processutils.execute(*cmd, **kwargs)
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance: 4451405e-5f24-42e3-8984-d6cb2f41e65c] File "/usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 389, in execute
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance: 4451405e-5f24-42e3-8984-d6cb2f41e65c] cmd=sanitized_cmd)
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance: 4451405e-5f24-42e3-8984-d6cb2f41e65c] ProcessExecutionError: Unexpected error while running command.
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance: 4451405e-5f24-42e3-8984-d6cb2f41e65c] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf brctl delif qbr0254be56-07 qvb0254be56-07
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance: 4451405e-5f24-42e3-8984-d6cb2f41e65c] Exit code: 1
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance: 4451405e-5f24-42e3-8984-d6cb2f41e65c] Stdout: u''
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance: 4451405e-5f24-42e3-8984-d6cb2f41e65c] Stderr: u'interface qvb0254be56-07 does not exist!\n'
2017-07-05 17:23:58.614 163228 ERROR nova.virt.libvirt.vif [instance: 4451405e-5f24-42e3-8984-d6cb2f41e65c]
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [req-734b5574-9973-4f5a-834d-158d66856248 404192d1e0df4068ad985bd423db4cae 1f0043836a7d46c0a70d82056f1bc99c - - -] [instance: 67d903ad-e751-46c5-9d50-d856bc119efa] Failed while unplugging vif
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance: 67d903ad-e751-46c5-9d50-d856bc119efa] Traceback (most recent call last):
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance: 67d903ad-e751-46c5-9d50-d856bc119efa] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/vif.py", line 777, in unplug_ovs_hybrid
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance: 67d903ad-e751-46c5-9d50-d856bc119efa] run_as_root=True)
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance: 67d903ad-e751-46c5-9d50-d856bc119efa] File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 388, in execute
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance: 67d903ad-e751-46c5-9d50-d856bc119efa] return RootwrapProcessHelper().execute(*cmd, **kwargs)
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance: 67d903ad-e751-46c5-9d50-d856bc119efa] File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 271, in execute
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance: 67d903ad-e751-46c5-9d50-d856bc119efa] return processutils.execute(*cmd, **kwargs)
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance: 67d903ad-e751-46c5-9d50-d856bc119efa] File "/usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 389, in execute
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance: 67d903ad-e751-46c5-9d50-d856bc119efa] cmd=sanitized_cmd)
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance: 67d903ad-e751-46c5-9d50-d856bc119efa] ProcessExecutionError: Unexpected error while running command.
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance: 67d903ad-e751-46c5-9d50-d856bc119efa] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf brctl delif qbr8e404216-8e qvb8e404216-8e
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance: 67d903ad-e751-46c5-9d50-d856bc119efa] Exit code: 1
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance: 67d903ad-e751-46c5-9d50-d856bc119efa] Stdout: u''
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance: 67d903ad-e751-46c5-9d50-d856bc119efa] Stderr: u'interface qvb8e404216-8e does not exist!\n'
2017-07-05 17:47:07.618 163228 ERROR nova.virt.libvirt.vif [instance: 67d903ad-e751-46c5-9d50-d856bc119efa]
One affected instance was noted to have lost it's 'qvb' interface (seen via ovs-vsctl show) resulting in loss of connectivity.
At the same time, on a host running the neutron api, we noted the following in the neutron openswitch log (package update was 2017-07-05 16:23:32):
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-8c6f6c00-d3b5-460c-b93a-ba9fb98ae899 - - - - -] Error while processing VIF ports
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2038, in rpc_loop
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info, ovs_restarted)
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1653, in process_network_ports
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent failed_devices['added'] |= self._bind_devices(need_binding_devices)
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 884, in _bind_devices
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.conf.host)
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/neutron/agent/rpc.py", line 180, in update_device_list
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent agent_id=agent_id, host=host)
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/neutron/common/rpc.py", line 155, in call
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent time.sleep(wait)
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.force_reraise()
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent six.reraise(self.type_, self.value, self.tb)
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/neutron/common/rpc.py", line 136, in call
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return self._original_context.call(ctxt, method, **kwargs)
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=self.retry)
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent timeout=timeout, retry=retry)
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry)
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in _send
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result = self._waiter.wait(msg_id, timeout)
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent message = self.waiters.get(msg_id, timeout=timeout)
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 'to message ID %s' % msg_id)
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID d2c512e487d3462982d99c4053472ea3
2017-07-05 16:25:20.736 56273 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
Restarting nova-compute on the compute node allowed the interface to recover and connectivity resumed immediately.
The effect on this cloud was losing connectivity to a large number of instances (although not all) until we identified the issue and restarted nova-compute.
apologies for the vile wrapping. For those with access, https:/