"error: [Errno 32] Broken pipe" when plugging OVS VIF into an instance

[Ihar Hrachyshka]

http://logs.openstack.org/85/486585/1/gate/gate-tempest-dsvm-neutron-dvr-ubuntu-xenial/acc8282/logs/screen-n-cpu.txt.gz#_2017-07-28_08_35_03_006

2017-07-28 08:35:03.006 1542 ERROR os_vif [req-398df22e-6d9b-4203-b2a9-1d7bc2011939 tempest-TestVolumeBootPattern-1767728258 tempest-TestVolumeBootPattern-1767728258] Failed to plug vif VIFBridge(active=False,address=fa:16:3e:fa:58:82,bridge_name='qbrd3b698dd-61',has_traffic_filtering=True,id=d3b698dd-61b1-4cf6-8967-5f57b204bcc1,network=Network(4cec1154-d5c3-4b79-9172-a20f55ea2c03),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tapd3b698dd-61')
2017-07-28 08:35:03.006 1542 ERROR os_vif Traceback (most recent call last):
2017-07-28 08:35:03.006 1542 ERROR os_vif File "/usr/local/lib/python2.7/dist-packages/os_vif/__init__.py", line 80, in plug
2017-07-28 08:35:03.006 1542 ERROR os_vif plugin.plug(vif, instance_info)
2017-07-28 08:35:03.006 1542 ERROR os_vif File "/usr/local/lib/python2.7/dist-packages/vif_plug_ovs/ovs.py", line 170, in plug
2017-07-28 08:35:03.006 1542 ERROR os_vif self._plug_bridge(vif, instance_info)
2017-07-28 08:35:03.006 1542 ERROR os_vif File "/usr/local/lib/python2.7/dist-packages/vif_plug_ovs/ovs.py", line 133, in _plug_bridge
2017-07-28 08:35:03.006 1542 ERROR os_vif linux_net.ensure_bridge(vif.bridge_name)
2017-07-28 08:35:03.006 1542 ERROR os_vif File "/usr/local/lib/python2.7/dist-packages/oslo_privsep/priv_context.py", line 205, in _wrap
2017-07-28 08:35:03.006 1542 ERROR os_vif return self.channel.remote_call(name, args, kwargs)
2017-07-28 08:35:03.006 1542 ERROR os_vif File "/usr/local/lib/python2.7/dist-packages/oslo_privsep/daemon.py", line 177, in remote_call
2017-07-28 08:35:03.006 1542 ERROR os_vif result = self.send_recv((Message.CALL.value, name, args, kwargs))
2017-07-28 08:35:03.006 1542 ERROR os_vif File "/usr/local/lib/python2.7/dist-packages/oslo_privsep/comm.py", line 160, in send_recv
2017-07-28 08:35:03.006 1542 ERROR os_vif self.writer.send((myid, msg))
2017-07-28 08:35:03.006 1542 ERROR os_vif File "/usr/local/lib/python2.7/dist-packages/oslo_privsep/comm.py", line 55, in send
2017-07-28 08:35:03.006 1542 ERROR os_vif self.writesock.sendall(buf)
2017-07-28 08:35:03.006 1542 ERROR os_vif File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 385, in sendall
2017-07-28 08:35:03.006 1542 ERROR os_vif tail = self.send(data, flags)
2017-07-28 08:35:03.006 1542 ERROR os_vif File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 379, in send
2017-07-28 08:35:03.006 1542 ERROR os_vif return self._send_loop(self.fd.send, data, flags)
2017-07-28 08:35:03.006 1542 ERROR os_vif File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 366, in _send_loop
2017-07-28 08:35:03.006 1542 ERROR os_vif return send_method(data, *args)
2017-07-28 08:35:03.006 1542 ERROR os_vif error: [Errno 32] Broken pipe
2017-07-28 08:35:03.006 1542 ERROR os_vif
2017-07-28 08:35:03.007 1542 ERROR nova.virt.libvirt.driver [req-398df22e-6d9b-4203-b2a9-1d7bc2011939 tempest-TestVolumeBootPattern-1767728258 tempest-TestVolumeBootPattern-1767728258] [instance: dffc7798-8ac8-4de6-9991-4930c5805208] Failed to start libvirt guest

Looking at libvirtd log, there is nothing that would immediately suggest what happened, but for the reference, that's what I see there:

2017-07-28 08:35:03.689+0000: 11804: debug : virThreadJobSet:96 : Thread 11804 (virNetServerHandleJob) is now running job remoteDispatchConnectGetLibVersion
2017-07-28 08:35:03.689+0000: 11804: debug : virThreadJobClear:121 : Thread 11804 (virNetServerHandleJob) finished job remoteDispatchConnectGetLibVersion with ret=0
2017-07-28 08:35:03.689+0000: 11807: debug : virThreadJobSet:96 : Thread 11807 (virNetServerHandleJob) is now running job remoteDispatchDomainLookupByName
2017-07-28 08:35:03.689+0000: 11807: debug : qemuDomainLookupByName:1595 : Domain not found: no domain with matching name 'instance-00000081'
2017-07-28 08:35:03.689+0000: 11807: debug : virThreadJobClear:121 : Thread 11807 (virNetServerHandleJob) finished job remoteDispatchDomainLookupByName with ret=-1
2017-07-28 08:35:03.697+0000: 11802: debug : virThreadJobSet:96 : Thread 11802 (virNetServerHandleJob) is now running job remoteDispatchConnectGetLibVersion
2017-07-28 08:35:03.697+0000: 11802: debug : virThreadJobClear:121 : Thread 11802 (virNetServerHandleJob) finished job remoteDispatchConnectGetLibVersion with ret=0
2017-07-28 08:35:03.698+0000: 11806: debug : virThreadJobSet:96 : Thread 11806 (virNetServerHandleJob) is now running job remoteDispatchDomainLookupByName
2017-07-28 08:35:03.698+0000: 11806: debug : qemuDomainLookupByName:1595 : Domain not found: no domain with matching name 'instance-00000081'
2017-07-28 08:35:03.698+0000: 11806: debug : virThreadJobClear:121 : Thread 11806 (virNetServerHandleJob) finished job remoteDispatchDomainLookupByName with ret=-1
2017-07-28 08:35:03.698+0000: 11810: debug : virThreadJobSet:96 : Thread 11810 (virNetServerHandleJob) is now running job remoteDispatchConnectGetLibVersion
2017-07-28 08:35:03.698+0000: 11810: debug : virThreadJobClear:121 : Thread 11810 (virNetServerHandleJob) finished job remoteDispatchConnectGetLibVersion with ret=0
2017-07-28 08:35:03.699+0000: 11804: debug : virThreadJobSet:96 : Thread 11804 (virNetServerHandleJob) is now running job remoteDispatchDomainLookupByName
2017-07-28 08:35:03.699+0000: 11804: debug : qemuDomainLookupByName:1595 : Domain not found: no domain with matching name 'instance-00000081'
2017-07-28 08:35:03.699+0000: 11804: debug : virThreadJobClear:121 : Thread 11804 (virNetServerHandleJob) finished job remoteDispatchDomainLookupByName with ret=-1
2017-07-28 08:35:03.857+0000: 11801: debug : virThreadJobSet:96 : Thread 11801 (virNetServerHandleJob) is now running job remoteDispatchConnectGetLibVersion
2017-07-28 08:35:03.857+0000: 11801: debug : virThreadJobClear:121 : Thread 11801 (virNetServerHandleJob) finished job remoteDispatchConnectGetLibVersion with ret=0
2017-07-28 08:35:03.858+0000: 11808: debug : virThreadJobSet:96 : Thread 11808 (virNetServerHandleJob) is now running job remoteDispatchDomainLookupByName
2017-07-28 08:35:03.858+0000: 11808: debug : qemuDomainLookupByName:1595 : Domain not found: no domain with matching name 'instance-00000081'
2017-07-28 08:35:03.858+0000: 11808: debug : virThreadJobClear:121 : Thread 11808 (virNetServerHandleJob) finished job remoteDispatchDomainLookupByName with ret=-1

This is Ocata.

[Sean Dague (sdague)]

That looks like a privsep issue? Any idea if the privsep daemon is still running in those cases?

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]