I've reproduced this bug on my vcenter's machine with configuration, which is described below:
api: '1.0' astute_sha: bc60b7d027ab244039f48c505ac52ab8eb0a990c auth_required: true build_id: 2014-09-01_00-01-17 build_number: '491' feature_groups: - mirantis fuellib_sha: 2cfa83119ae90b13a5bac6a844bdadfaf5aeb13f fuelmain_sha: 109812be3425408dd7be192b5debf109cb1edd4c nailgun_sha: d25ed02948a8be773e2bd87cfe583ef7be866bb2 ostf_sha: 4dcd99cc4bfa19f52d4b87ed321eb84ff03844da production: docker release: '5.1'
on vcenter machine 172.18.170.88: 1. Create new environment (CentOS, simple mode) Network: Nova Network Flat DHCP
setting for vcenter creation: VCENTER_IP='172.16.0.254' <email address hidden>' VCENTER_PASSWORD='Qwer!1234' VCENTER_CLUSTERS='Cluster1'
2. Add 2 nodes with roles: 1 controller and 1 cinder (as storage type - VMDK) 3. Make deploy of environment 4. Verify network connectivity 5. Run ostf
Error and traceback: <179>Sep 2 12:46:01 node-2 nova-api 2014-09-02 12:46:01.130 19317 ERROR oslo.messaging._drivers.impl_rabbit [req-684f1371-1d98-47a4-a437-4a860f7c60d2 ] Failed to publish message to topic 'notifications.info': [Errno 32] Broken pipe 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 648, in ensure 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit return method(*args, **kwargs) 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 753, in _publish 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit publisher = cls(self.conf, self.channel, topic, **kwargs) 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 420, in __init__ 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit super(NotifyPublisher, self).__init__(conf, channel, topic, **kwargs) 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 396, in __init__ 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit **options) 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 339, in __init__ 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit self.reconnect(channel) 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 423, in reconnect 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit super(NotifyPublisher, self).reconnect(channel) 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 347, in reconnect 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit routing_key=self.routing_key) 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/kombu/messaging.py", line 84, in __init__ 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit self.revive(self._channel) 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/kombu/messaging.py", line 218, in revive 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit self.declare() 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/kombu/messaging.py", line 104, in declare 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit self.exchange.declare() 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/kombu/entity.py", line 166, in declare 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit nowait=nowait, passive=passive, 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/amqp/channel.py", line 613, in exchange_declare 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit self._send_method((40, 10), args) 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/amqp/abstract_channel.py", line 56, in _send_method 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit self.channel_id, method_sig, args, content, 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/amqp/method_framing.py", line 221, in write_method 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit write_frame(1, channel, payload) 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/amqp/transport.py", line 177, in write_frame 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit frame_type, channel, size, payload, 0xce, 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 309, in sendall 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit tail = self.send(data, flags) 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 295, in send 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit total_sent += fd.send(data[total_sent:], flags) 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit error: [Errno 32] Broken pipe 2014-09-02 12:46:01.130 19317 TRACE oslo.messaging._drivers.impl_rabbit
Diagnostic snapshot below:
I've reproduced this bug on my vcenter's machine with configuration, which is described below:
api: '1.0' 039f48c505ac52a b8eb0a990c 3a5bac6a844bdad faf5aeb13f dd7be192b5debf1 09cb1edd4c 73e2bd87cfe583e f7be866bb2 52d4b87ed321eb8 4ff03844da
astute_sha: bc60b7d027ab244
auth_required: true
build_id: 2014-09-01_00-01-17
build_number: '491'
feature_groups:
- mirantis
fuellib_sha: 2cfa83119ae90b1
fuelmain_sha: 109812be3425408
nailgun_sha: d25ed02948a8be7
ostf_sha: 4dcd99cc4bfa19f
production: docker
release: '5.1'
on vcenter machine 172.18.170.88:
1. Create new environment (CentOS, simple mode)
Network: Nova Network Flat DHCP
setting for vcenter creation:
VCENTER_ IP='172. 16.0.254'
< email address hidden>'
VCENTER_ PASSWORD= 'Qwer!1234'
VCENTER_ CLUSTERS= 'Cluster1'
2. Add 2 nodes with roles: 1 controller and 1 cinder (as storage type - VMDK)
3. Make deploy of environment
4. Verify network connectivity
5. Run ostf
Error and traceback: _drivers. impl_rabbit [req-684f1371- 1d98-47a4- a437-4a860f7c60 d2 ] Failed to publish message to topic 'notifications. info': [Errno 32] Broken pipe _drivers. impl_rabbit Traceback (most recent call last): _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ oslo/messaging/ _drivers/ impl_rabbit. py", line 648, in ensure _drivers. impl_rabbit return method(*args, **kwargs) _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ oslo/messaging/ _drivers/ impl_rabbit. py", line 753, in _publish _drivers. impl_rabbit publisher = cls(self.conf, self.channel, topic, **kwargs) _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ oslo/messaging/ _drivers/ impl_rabbit. py", line 420, in __init__ _drivers. impl_rabbit super(NotifyPub lisher, self)._ _init__ (conf, channel, topic, **kwargs) _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ oslo/messaging/ _drivers/ impl_rabbit. py", line 396, in __init__ _drivers. impl_rabbit **options) _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ oslo/messaging/ _drivers/ impl_rabbit. py", line 339, in __init__ _drivers. impl_rabbit self.reconnect( channel) _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ oslo/messaging/ _drivers/ impl_rabbit. py", line 423, in reconnect _drivers. impl_rabbit super(NotifyPub lisher, self).reconnect (channel) _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ oslo/messaging/ _drivers/ impl_rabbit. py", line 347, in reconnect _drivers. impl_rabbit routing_ key=self. routing_ key) _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ kombu/messaging .py", line 84, in __init__ _drivers. impl_rabbit self.revive( self._channel) _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ kombu/messaging .py", line 218, in revive _drivers. impl_rabbit self.declare() _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ kombu/messaging .py", line 104, in declare _drivers. impl_rabbit self.exchange. declare( ) _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ kombu/entity. py", line 166, in declare _drivers. impl_rabbit nowait=nowait, passive=passive, _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ amqp/channel. py", line 613, in exchange_declare _drivers. impl_rabbit self._send_ method( (40, 10), args) _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ amqp/abstract_ channel. py", line 56, in _send_method _drivers. impl_rabbit self.channel_id, method_sig, args, content, _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ amqp/method_ framing. py", line 221, in write_method _drivers. impl_rabbit write_frame(1, channel, payload) _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ amqp/transport. py", line 177, in write_frame _drivers. impl_rabbit frame_type, channel, size, payload, 0xce, _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ eventlet/ greenio. py", line 309, in sendall _drivers. impl_rabbit tail = self.send(data, flags) _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ eventlet/ greenio. py", line 295, in send _drivers. impl_rabbit total_sent += fd.send( data[total_ sent:], flags) _drivers. impl_rabbit error: [Errno 32] Broken pipe _drivers. impl_rabbit
<179>Sep 2 12:46:01 node-2 nova-api 2014-09-02 12:46:01.130 19317 ERROR oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
2014-09-02 12:46:01.130 19317 TRACE oslo.messaging.
Diagnostic snapshot below: