Comment 0 for bug 1522611

Revision history for this message
Edward Hope-Morley (hopem) wrote :

We have observed that upgrading an existing 3 unit rabbitmq-server cluster to the latest stable charms can cause the cluster to break. This is because will the charm will check that the rabbitmq-server (at least on the leader) is configured by ensuring that RABBITMQ_NODENAME=rabbit@<hostname> is set in /etc/rabbitmq/rabbitmq-env.conf. Trouble is, when the hook runs to perform this action, it will use whatever is returned by charmhelpers.contrib.openstack.utils.get_hostname(). Now, in some of the environments we recently upgraded, the MAAS dns entry for the rabbitmq-server host IP has multiple cname entries e.g.

;; ANSWER SECTION:
1.5.27.172.in-addr.arpa. 300 IN PTR ctrlnode1.maas.
1.5.27.172.in-addr.arpa. 300 IN PTR 172-27-5-1.maas.

(see http://paste.ubuntu.com/13653431/ for full answer)

The result is that a rabbitmq node whose nodename was originally configured with 172-27-5-1 now switches to ctrlnode1 which, when the charm attempts to restart rabbitmq, causes it to fail since it no longer recognises the nodename e.g. https://pastebin.canonical.com/145396/

The problem he appears to be simple, if the charm is presented with multiple cnames it needs be aware of this and chose the correct one based on what it may have previously configured in rabbitmq.