Ceilometer complains about rabbit being unreachable, although it is configured

[Vladimir Kuklin]

fuel-library10.0-10.0.0-1.mos8393.noarch

MODULE LIST
stdlib: 05c6587d852800cc69e3a9381ff466ac3aa630d2
pacemaker: 8780696d6f1367f1a3b3c0ef737a89167e61732e
concat: 5997e65f18eb78cd7dbce818eb29354e49d5a038
inifile: 1de47a3ba2dc62d2a535131f8d3a8e8ad59c9860
xinetd: 5e31388979e8f3a667899fad5cefa95289e22281
ssh: 3216cd87ae97ee74f06edd0e4868cedbc90e86d9
ntp: 442690656f9a598509ee0c42f87ff26d66cf12f2
apache: d76699aa0be710f2063a9172cf32536a547888f9
apt: eee0a6b1e71a34321a67eb78608fc716c1ff739a
firewall: 8655623f0a71c0ea43fa9153936b85ad092ae976
memcached: 5a6c247a757671094d477d07c14b59c484d560d4
sysctl: 65ffe839a4ce785bc3901452488197a0ef158cd8
staging: bc434a71e19aae54223d57c274e2e1a7f9546d5e
vcsrepo: f98e8f0e552addd79974ae4523183639a06142e0
postgresql: 392f17ad8abb5b6b21c5f62525baab0365006ce7
rsyslog: c0c3991fe36d4e552c441e53d8441cf5d528d896
openssl: 9c3c613a1d1a34466dcd9916c413ac22257ba279
mongodb: d6141b5ed417ec62421d7481a571174a56a5bee0
rsync: fba3de1a9adbfeff27cae2999e3e1b6b6625f005
filemapper: 9b53310278e76827bbe12a36cc6470d77071abb2
tftp: e7e5b5f1a38833e769453a848a7b20741039b415
datacat: b0595d5ec3be472975403a136ca202e9dd4e432a
mcollective: f0734a6a0818495aebb2abcbcd92b73fe14df24f
corosync: c8525e402b49ee9a2f0b0e3ea9a5a4805e88f169
rabbitmq: dc0964da2399da6a4a2654419b8479257e59d76a
mysql: 10b873c717a497878b1fdf939a98c5a67ddf5fba
galera: e2cee1fce92025041f11c58e3a804529720747c6
aodh: 4d6b781b7c0ac7e8abf8e4738d10c5b58ccce2f3
ceilometer: ba21901512fdda95a3290933d7e0218eaa936828
cinder: 96e98e24f0b3186f0c757bc06a1ff9e2c08fe00d
glance: d70f4c54342b4bdb71a5cb6e4ea2637e0917ddf6
heat: 099774011c321fe0d148d4e36ac585c4a96393ef
horizon: ce8d41488f74defd9c55dbdedc58538ab8c4b4e7
ironic: 3b801043542f99cf295d8c5114950da28c35b006
keystone: f8ea9959587e5b7b57cf8debe938e3716c8e28fe
openstacklib: 45f8a57770f8b0e2c8a11bf3badfb1dde2721d78
oslo: b79dda880a5f71254d42872029500487f47037f8
neutron: 56d2a467384d61c63ff777f33bc47657c2fc68b8
nova: 189a662c869afbe918c8a80ace5f7478d766dc6a
murano: eaea162702f44ad183f44750896ec8f983cf5995
sahara: e5f369e5a7fed9a30b55d898fea084c18713d0c8
swift: f569ed0edaa46f986c8a844d6ed0ee41ed86ad8a
ceph: 0d68c38980521cb8826f9294f747b4c55e58fc44

ceilometer.conf contents:

[oslo_messaging_rabbit]
# EXPERIMENTAL: Possible values are: gzip, bz2. If not set compression will not be used. This option may notbe available in future versions.
# (string value)
#kombu_compression = <None>

# How long to wait a missing client beforce abandoning to send it its replies. This value should not be longer than rpc_response_timeout.
# (integer value)
# Deprecated group/name - [DEFAULT]/kombu_reconnect_timeout
#kombu_missing_consumer_retry_timeout = 60

# Determines how the next RabbitMQ node is chosen in case the one we are currently connected to becomes unavailable. Takes effect only if
# more than one RabbitMQ node is provided in config. (string value)
# Allowed values: round-robin, shuffle
#kombu_failover_strategy = round-robin

# The RabbitMQ broker address where a single node is used. (string value)
# Deprecated group/name - [DEFAULT]/rabbit_host

# The RabbitMQ broker port where a single node is used. (port value)
# Minimum value: 0
# Maximum value: 65535
# Deprecated group/name - [DEFAULT]/rabbit_port
#rabbit_port = 5672

# RabbitMQ HA cluster host:port pairs. (list value)
# Deprecated group/name - [DEFAULT]/rabbit_hosts
#rabbit_hosts = $rabbit_host:$rabbit_port
rabbit_hosts= 10.109.1.7:5673, 10.109.1.8:5673, 10.109.1.5:5673

meanwhile in logs:

2016-05-13 16:01:27.974 23206 ERROR oslo.messaging._drivers.impl_rabbit self._connection = self._establish_connection()
2016-05-13 16:01:27.974 23206 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 696, in _establish_connection
2016-05-13 16:01:27.974 23206 ERROR oslo.messaging._drivers.impl_rabbit conn = self.transport.establish_connection()
2016-05-13 16:01:27.974 23206 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
2016-05-13 16:01:27.974 23206 ERROR oslo.messaging._drivers.impl_rabbit conn = self.Connection(**opts)
2016-05-13 16:01:27.974 23206 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 165, in __init__
2016-05-13 16:01:27.974 23206 ERROR oslo.messaging._drivers.impl_rabbit self.transport = self.Transport(host, connect_timeout, ssl)
2016-05-13 16:01:
2016-05-13T16:01:27.978610+00:00 err: 2016-05-13 16:01:27.977 23206 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 127.0.0.1:5672 is unreachable: [Errno 111] Connection refused. Trying again in 32 seconds.
2016-05-13T16:01:27.978957+00:00 err: 2016-05-13 16:01:27.977 23206 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 127.0.0.1:5672 is unreachable: [Errno 111] Connection refused. Trying again in 32 seconds.

[Ivan Berezovskiy (iberezovskiy)]

Ceilometer is configured correctly. It looks like some of the ceilometer-collector threads still using default config file (they weren't restarted after config changes), and some other are working as expected.

[Igor Degtiarov (idegtiarov)]

I do not see any issues on current MOS 9.0 release. And it seems that this bug related to that one https://bugs.launchpad.net/mos/9.x/+bug/1566202 which was already fixed. So I change status to invalid.

If there are any concerns that bug is still actual feel free to reopen it.