Oslo messaging is failed on vCenter's machine. [Errno 32] Broken pipe. Failed to publish message to topic 'None'

[Tatyana Dubyk]

I've had this bug on my vcenter's machine with configuration, which is described below:

api: '1.0'
astute_sha: bc60b7d027ab244039f48c505ac52ab8eb0a990c
auth_required: true
build_id: 2014-09-01_00-01-17
build_number: '491'
feature_groups:
- mirantis
fuellib_sha: 2cfa83119ae90b13a5bac6a844bdadfaf5aeb13f
fuelmain_sha: 109812be3425408dd7be192b5debf109cb1edd4c
nailgun_sha: d25ed02948a8be773e2bd87cfe583ef7be866bb2
ostf_sha: 4dcd99cc4bfa19f52d4b87ed321eb84ff03844da
production: docker
release: '5.1'

on vcenter machine 172.18.170.88:
1. Create new environment (CentOS, simple mode)
     Network: Nova Network Flat DHCP

     setting for vcenter creation:
                    VCENTER_IP='172.16.0.254'
                    <email address hidden>'
                    VCENTER_PASSWORD='Qwer!1234'
                    VCENTER_CLUSTERS='Cluster1'

2. Add 2 nodes with roles: 1 controller and 1 cinder (as storage type - VMDK)
3. Make deploy of environment
4. Verify network connectivity
5. Run ostf

Error and traceback:
<179>Sep 2 12:46:01 node-2 nova-api 2014-09-02 12:46:01.130 19317 ERROR oslo.messaging._drivers.impl_rabbit [req-684f1371-1d98-47a4-a437-4a860f7c60d2 ] Failed to publish message to topic 'notifications.info': [Errno 32] Broken pipe
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 648, in ensure
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit return method(*args, **kwargs)
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 753, in _publish
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit publisher = cls(self.conf, self.channel, topic, **kwargs)
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 420, in __init__
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit super(NotifyPublisher, self).__init__(conf, channel, topic, **kwargs)
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 396, in __init__
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit **options)
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 339, in __init__
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit self.reconnect(channel)
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 423, in reconnect
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit super(NotifyPublisher, self).reconnect(channel)
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 347, in reconnect
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit routing_key=self.routing_key)
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/kombu/messaging.py", line 84, in __init__
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit self.revive(self._channel)
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/kombu/messaging.py", line 218, in revive
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit self.declare()
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/kombu/messaging.py", line 104, in declare
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit self.exchange.declare()
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/kombu/entity.py", line 166, in declare
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit nowait=nowait, passive=passive,
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/amqp/channel.py", line 613, in exchange_declare
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit self._send_method((40, 10), args)
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/amqp/abstract_channel.py", line 56, in _send_method
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit self.channel_id, method_sig, args, content,
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/amqp/method_framing.py", line 221, in write_method
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit write_frame(1, channel, payload)
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/amqp/transport.py", line 177, in write_frame
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit frame_type, channel, size, payload, 0xce,
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 309, in sendall
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit tail = self.send(data, flags)
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 295, in send
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit total_sent += fd.send(data[total_sent:], flags)
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit error: [Errno 32] Broken pipe
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit

[Tatyana Dubyk (tdubyk)]

This bug is reproduced and for Fuel team on theirs env. Here bug: https://bugs.launchpad.net/fuel/+bug/1331969

[Viktor Serhieiev (vsergeyev)]

Should we mark this bug as duplicate?

[Tatyana Dubyk (tdubyk)]

I don't know, because there is another configuration and vcenter's instead of compute.

[Dennis Dmitriev (ddmitriev)]

Updating python-oslo.messaging package to the version 1.4.0.0~a3-ubuntu2 resolve the issue with broken pipe.

[Igor Marnat (imarnat)]

We might want to close the bug then, aren't we?

[Dmitry Mescheryakov (dmitrymex)]

Igor, the bug could be closed only if we upgrade python-oslo.messaging in MOS. We didn't consider that yet.

[Serg Melikyan (smelikyan)]

Indeed, "Broken pipe" is caused by our oslo.messaging, but so far we didn't find any correlation between this particular error and any issues found during testing of deployed cloud. We need more info about which exactly OSTF test failed. For now I see that whole log of nova-compute is filled with:

2014-09-02T12:23:22.647176+01:00 crit: 2014-09-02 11:23:22.632 19831 CRITICAL nova.virt.vmwareapi.driver [-] Unable to connect to server at 172.16.0.254, sleeping for 1 seconds
2014-09-02 11:23:22.632 19831 TRACE nova.virt.vmwareapi.driver Traceback (most recent call last):
2014-09-02 11:23:22.632 19831 TRACE nova.virt.vmwareapi.driver File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 809, in _create_session
2014-09-02 11:23:22.632 19831 TRACE nova.virt.vmwareapi.driver self.vim = self._get_vim_object()
2014-09-02 11:23:22.632 19831 TRACE nova.virt.vmwareapi.driver File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 798, in _get_vim_object
2014-09-02 11:23:22.632 19831 TRACE nova.virt.vmwareapi.driver return vim.Vim(protocol=self._scheme, host=self._host_ip)
2014-09-02 11:23:22.632 19831 TRACE nova.virt.vmwareapi.driver File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vim.py", line 116, in __init__
2014-09-02 11:23:22.632 19831 TRACE nova.virt.vmwareapi.driver plugins=[VIMMessagePlugin()])
2014-09-02 11:23:22.632 19831 TRACE nova.virt.vmwareapi.driver File "/usr/lib/python2.6/site-packages/suds/client.py", line 112, in __init__
2014-09-02 11:23:22.632 19831 TRACE nova.virt.vmwareapi.driver self.wsdl = reader.open(url)
2014-09-02 11:23:22.632 19831 TRACE nova.virt.vmwareapi.driver File "/usr/lib/python2.6/site-packages/suds/reader.py", line 152, in open
2014-09-02 11:23:22.632 19831 TRACE nova.virt.vmwareapi.driver d = self.fn(url, self.options)
2014-09-02 11:23:22.632 19831 TRACE nova.virt.vmwareapi.driver File "/usr/lib/python2.6/site-packages/suds/wsdl.py", line 136, in __init__
2014-09-02 11:23:22.632 19831 TRACE nova.virt.vmwareapi.driver d = reader.open(url)
2014-09-02 11:23:22.632 19831 TRACE nova.virt.vmwareapi.driver File "/usr/lib/python2.6/site-packages/suds/reader.py", line 79, in open
2014-09-02 11:23:22.632 19831 TRACE nova.virt.vmwareapi.driver d = self.download(url)
2014-09-02 11:23:22.632 19831 TRACE nova.virt.vmwareapi.driver File "/usr/lib/python2.6/site-packages/suds/reader.py", line 95, in download
2014-09-02 11:23:22.632 19831 TRACE nova.virt.vmwareapi.driver fp = self.options.transport.open(Request(url))
2014-09-02 11:23:22.632 19831 TRACE nova.virt.vmwareapi.driver File "/usr/lib/python2.6/site-packages/suds/transport/https.py", line 60, in open
2014-09-02 11:23:22.632 19831 TRACE nova.virt.vmwareapi.driver return HttpTransport.open(self, request)
2014-09-02 11:23:22.632 19831 TRACE nova.virt.vmwareapi.driver File "/usr/lib/python2.6/site-packages/suds/transport/http.py", line 62, in open
2014-09-02 11:23:22.632 19831 TRACE nova.virt.vmwareapi.driver return self.u2open(u2request)
2014-09-02 11:23:22.632 19831 TRACE nova.virt.vmwareapi.driver File "/usr/lib/python2.6/site-packages/suds/transport/http.py",...

[Tatyana Dubyk (tdubyk)]

When I've created this bug I haven't got info that environment around our 88 machine not set up properly. That's why we reproduced this bug. Now environment is set up correctly, so now we can close this bug.