root@node-3:~# dpkg -l | grep oslo
ii python-oslo.config 1:1.4.0-fuel6.0~mira16 Common code for Openstack Projects (configuration API)
ii python-oslo.db 1.0.1-fuel6.0~mira19 oslo db library
ii python-oslo.i18n 1.0.0-fuel6.0~mira16 Oslo Internationalization Utilities - Python 2.x
ii python-oslo.messaging 1.4.1-fuel6.0~mira18 oslo messaging library
ii python-oslo.rootwrap 1.3.0.0a1-ubuntu2 Fine filtering of shell commands as 'root'
ii python-oslo.serialization 1.0.0-fuel6.0~mira15 utilities for serialization , especially JSON - Python 2.x
ii python-oslo.utils 1.0.0-fuel6.0~mira16 set of utility functions for OpenStack - Python 2.x
ii python-oslo.vmware 0.6.0-fuel6.0~mira16 VMware library for OpenStack projects - Python 2.x
I replaced the whole folder under oslo.messaging-1.4.1\oslo\messaging to /usr/share/pyshared/oslo/messaging, then restart all 3 controllers, the HA didn't work very well, nova service-list did not work with HTTP 503 error. Neutron components didn't work very well also.
Please check below error messages,
<166>Jul 16 14:22:12 node-3 neutron-server 2015-07-16 14:22:12.436 6138 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 127.0.0.1:5673
<166>Jul 16 14:22:12 node-3 neutron-server 2015-07-16 14:22:12.456 6138 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 127.0.0.1:5673
<163>Jul 16 14:22:12 node-3 neutron-server 2015-07-16 14:22:12.544 6061 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'reply_c2d631eb2efa4fc1b88d2d03f90d6bca': [Errno 32] Broken pipe
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 681, in ensure
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit return method()
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 784, in _publish
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 377, in __init__
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit type='direct', **options)
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 340, in __init__
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit self.reconnect(channel)
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 348, in reconnect
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit routing_key=self.routing_key)
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 84, in __init__
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit self.revive(self._channel)
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 218, in revive
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit self.declare()
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 104, in declare
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit self.exchange.declare()
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 166, in declare
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit nowait=nowait, passive=passive,
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 613, in exchange_declare
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit self._send_method((40, 10), args)
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 56, in _send_method
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit self.channel_id, method_sig, args, content,
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 221, in write_method
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit write_frame(1, channel, payload)
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/transport.py", line 177, in write_frame
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit frame_type, channel, size, payload, 0xce,
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 357, in sendall
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit tail = self.send(data, flags)
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 340, in send
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit total_sent += fd.send(data[total_sent:], flags)
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit error: [Errno 32] Broken pipe
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging._drivers.impl_rabbit
<166>Jul 16 14:22:12 node-3 neutron-server 2015-07-16 14:22:12.549 6061 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 5.0 seconds ...
<166>Jul 16 14:22:17 node-3 neutron-server 2015-07-16 14:22:17.549 6061 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 10.14.68.7:5673
<166>Jul 16 14:22:17 node-3 neutron-server 2015-07-16 14:22:17.566 6061 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.14.68.7:5673
Hello Chris J Arges, I tested your package https:/ /launchpad. net/ubuntu/ +source/ oslo.messaging/ 1.4.1-0ubuntu1. 1 on my fuel openstack environment, it didn't work very well.
Please check the below information for details,
[root@mvp-builder ~]# fuel --f 73030b8100cf8c5 ca6a970a91 f819d402c645c73 1af450aff0 8b4bee79ca45a54 b76c1361b8 f522ca9f6d34e7e 135f150a90 70460c29d6c3293 219d3624d4 73030b8100cf8c5 ca6a970a91 groups: f819d402c645c73 1af450aff0 8b4bee79ca45a54 b76c1361b8 f522ca9f6d34e7e 135f150a90 70460c29d6c3293 219d3624d4
api: '1.0'
astute_sha: 16b252d93be6aaa
auth_required: true
build_id: 2014-12-26_14-25-46
build_number: '58'
feature_groups:
- mirantis
- experimental
fuellib_sha: fde8ba5e11a1aca
fuelmain_sha: 81d38d6f2903b5a
nailgun_sha: 5f91157daa6798f
ostf_sha: a9afb68710d8095
production: docker
release: '6.0'
release_versions:
2014.2-6.0:
VERSION:
api: '1.0'
astute_sha: 16b252d93be6aaa
build_id: 2014-12-26_14-25-46
build_number: '58'
feature_
- mirantis
fuellib_sha: fde8ba5e11a1aca
fuelmain_sha: 81d38d6f2903b5a
nailgun_sha: 5f91157daa6798f
ostf_sha: a9afb68710d8095
production: docker
release: '6.0'
root@node-3:~# dpkg -l | grep oslo fuel6.0~ mira16 Common code for Openstack Projects (configuration API) 0~mira19 oslo db library 0~mira16 Oslo Internationaliz ation Utilities - Python 2.x oslo.messaging 1.4.1-fuel6. 0~mira18 oslo messaging library oslo.rootwrap 1.3.0.0a1-ubuntu2 Fine filtering of shell commands as 'root' oslo.serializat ion 1.0.0-fuel6. 0~mira15 utilities for serialization , especially JSON - Python 2.x 0~mira16 set of utility functions for OpenStack - Python 2.x 0~mira16 VMware library for OpenStack projects - Python 2.x
ii python-oslo.config 1:1.4.0-
ii python-oslo.db 1.0.1-fuel6.
ii python-oslo.i18n 1.0.0-fuel6.
ii python-
ii python-
ii python-
ii python-oslo.utils 1.0.0-fuel6.
ii python-oslo.vmware 0.6.0-fuel6.
I replaced the whole folder under oslo.messaging- 1.4.1\oslo\ messaging to /usr/share/ pyshared/ oslo/messaging, then restart all 3 controllers, the HA didn't work very well, nova service-list did not work with HTTP 503 error. Neutron components didn't work very well also.
Please check below error messages, _drivers. impl_rabbit [-] Connecting to AMQP server on 127.0.0.1:5673 _drivers. impl_rabbit [-] Connected to AMQP server on 127.0.0.1:5673 _drivers. impl_rabbit [-] Failed to publish message to topic 'reply_ c2d631eb2efa4fc 1b88d2d03f90d6b ca': [Errno 32] Broken pipe _drivers. impl_rabbit Traceback (most recent call last): _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ oslo/messaging/ _drivers/ impl_rabbit. py", line 681, in ensure _drivers. impl_rabbit return method() _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ oslo/messaging/ _drivers/ impl_rabbit. py", line 784, in _publish _drivers. impl_rabbit _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ oslo/messaging/ _drivers/ impl_rabbit. py", line 377, in __init__ _drivers. impl_rabbit type='direct', **options) _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ oslo/messaging/ _drivers/ impl_rabbit. py", line 340, in __init__ _drivers. impl_rabbit self.reconnect( channel) _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ oslo/messaging/ _drivers/ impl_rabbit. py", line 348, in reconnect _drivers. impl_rabbit routing_ key=self. routing_ key) _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ kombu/messaging .py", line 84, in __init__ _drivers. impl_rabbit self.revive( self._channel) _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ kombu/messaging .py", line 218, in revive _drivers. impl_rabbit self.declare() _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ kombu/messaging .py", line 104, in declare _drivers. impl_rabbit self.exchange. declare( ) _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ kombu/entity. py", line 166, in declare _drivers. impl_rabbit nowait=nowait, passive=passive, _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ amqp/channel. py", line 613, in exchange_declare _drivers. impl_rabbit self._send_ method( (40, 10), args) _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ amqp/abstract_ channel. py", line 56, in _send_method _drivers. impl_rabbit self.channel_id, method_sig, args, content, _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ amqp/method_ framing. py", line 221, in write_method _drivers. impl_rabbit write_frame(1, channel, payload) _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ amqp/transport. py", line 177, in write_frame _drivers. impl_rabbit frame_type, channel, size, payload, 0xce, _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ eventlet/ greenio. py", line 357, in sendall _drivers. impl_rabbit tail = self.send(data, flags) _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ eventlet/ greenio. py", line 340, in send _drivers. impl_rabbit total_sent += fd.send( data[total_ sent:], flags) _drivers. impl_rabbit error: [Errno 32] Broken pipe _drivers. impl_rabbit _drivers. impl_rabbit [-] Delaying reconnect for 5.0 seconds ... _drivers. impl_rabbit [-] Connecting to AMQP server on 10.14.68.7:5673 _drivers. impl_rabbit [-] Connected to AMQP server on 10.14.68.7:5673
<166>Jul 16 14:22:12 node-3 neutron-server 2015-07-16 14:22:12.436 6138 INFO oslo.messaging.
<166>Jul 16 14:22:12 node-3 neutron-server 2015-07-16 14:22:12.456 6138 INFO oslo.messaging.
<163>Jul 16 14:22:12 node-3 neutron-server 2015-07-16 14:22:12.544 6061 ERROR oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
2015-07-16 14:22:12.544 6061 TRACE oslo.messaging.
<166>Jul 16 14:22:12 node-3 neutron-server 2015-07-16 14:22:12.549 6061 INFO oslo.messaging.
<166>Jul 16 14:22:17 node-3 neutron-server 2015-07-16 14:22:17.549 6061 INFO oslo.messaging.
<166>Jul 16 14:22:17 node-3 neutron-server 2015-07-16 14:22:17.566 6061 INFO oslo.messaging.
Thanks.