ceilometer-agent does not address all units of a rabbitmq-server cluster

[James Hebden]

Juju 2.0.2
ceilometer & ceilometer-agent charms from 17.02 release
OpenStack Mitaka

Deployed ceilometer-agent is not publishing nova meters to RMQ.
ceilometer meters-list shows glance and neutron meters only.
ceilometer-agent has been related to nova-compute and ceilometer, and no juju errors are present.

With RabbitMQ deployed in a three-node cluster, I did noticed that there is a difference in ceilometer.conf for ceilometer and ceilometer-agent units. The ceilometer-agent units are not picking up all three RMQ servers, instead only one server is rendered into the configuration file. It is not the leader from Juju or RMQ's perspective, and I suspect this is related to the data not being published.

ceilometer.conf from ceilometer unit:
# mitaka
###############################################################################
# [ WARNING ]
# ceilometer configuration file maintained by Juju
# local changes may be overwritten.
###############################################################################
[DEFAULT]
debug = False
verbose = False
use_syslog = False
event_pipeline_cfg_file = /etc/ceilometer/event_pipeline_alarm.yaml

[api]
port = 8767
workers = 4

[service_credentials]
auth_url = http://redacted:5000
project_name = services
username = ceilometer
password = redacted
project_domain_name = service_domain
user_domain_name = service_domain
auth_type = password
[database]
connection = mongodb://redacted1:27017,redacted2:27017,redacted3:27017/ceilometer?readPreference=primaryPreferred&replicaSet=myset
mongodb_replica_set = myset

metering_time_to_live = 432000
event_time_to_live = 432000

[publisher]
telemetry_secret = redacted

[keystone_authtoken]
auth_uri = http://redacted:5000
auth_url = http://redacted:35357
auth_type = password
project_domain_name = service_domain
user_domain_name = service_domain
project_name = services
username = ceilometer
password = redacted
signing_dir = /var/cache/ceilometer

memcached_servers = inet6:[::1]:11211

[oslo_messaging_rabbit]
rabbit_userid = ceilometer
rabbit_virtual_host = openstack
rabbit_password = redacted
rabbit_hosts = redacted1,redacted2,redacted3

and from ceilometer-agent:
# mitaka
###############################################################################
# [ WARNING ]
# ceilometer configuration file maintained by Juju
# local changes may be overwritten.
###############################################################################
[DEFAULT]
debug = False
verbose = False
logdir = /var/log/ceilometer

[service_credentials]
auth_url = http://redacted:5000
project_name = services
username = ceilometer
password = redacted
project_domain_name = service_domain
user_domain_name = service_domain
auth_type = password
[publisher]
telemetry_secret = redacted
[keystone_authtoken]
auth_uri = http://redacted:5000
auth_url = http://redacted:35357
auth_type = password
project_domain_name = service_domain
user_domain_name = service_domain
project_name = services
username = ceilometer
password = redacted
signing_dir = /var/cache/ceilometer

memcached_servers = inet6:[::1]:11211

[oslo_messaging_rabbit]
rabbit_userid = ceilometer
rabbit_virtual_host = openstack
rabbit_password = redacted
rabbit_host = redacted1

[James Page (james-page)]

Not able to reproduce with a single rmq instance and the latest charm release (which does tidy some of the notification configuration across the charms) - checking with a cluster.

[James Page (james-page)]

ceilometer resource-list: http://paste.ubuntu.com/25638309/

ceilometer meter-list: http://paste.ubuntu.com/25638313/

[James Page (james-page)]

Confirmed that ceilometer-agent does not update rabbitmq configuration to address the full cluster in a multi-unit deployment; this is probably due to its lack of direct relation to rmq - it gains access via the ceilometer charm.

[James Page (james-page)]

Note that this should not impact on the collection of metrics - the queues are mirrored so messages published to one of the units should be propagated to consumers of the topic connected via other brokers.

[James Page (james-page)]

Even with the single rmq host entry, I see metrics being collected for all instances across my test cloud with a 3 unit rabbitmq deployment as well.

That said we do need to fix the issue with regards to addressing a cluster rabbitmq-server from ceilometer; I suggest we do this by adding a relation between ceilometer-agent and rabbitmq-server, rather than mutating the ceilometer-service relation to include clustering information.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/512997

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/512997
Committed: https://git.openstack.org/cgit/openstack/charm-ceilometer-agent/commit/?id=958dfc3e2fd44f51171b781ba5c969bfb45d2eed
Submitter: Zuul
Branch: master

commit 958dfc3e2fd44f51171b781ba5c969bfb45d2eed
Author: Liam Young <email address hidden>
Date: Wed Oct 18 10:28:07 2017 +0000

    Address all rabbit units in clustered deploy

    If rabbitmq is deployed in a cluster then the ceilometer charm will
    pass rabbitmq_hosts and rabbitmq_host to ceilometer-agent. This charm
    was ignoring the rabbitmq_hosts key and thus always talking to a
    single rabbit even if there were multiple.

    Change-Id: I42c2e3ee4ee2bd1feb9815a4d9895a7121801995
    Closes-Bug: #1670273