charm changes nodename on upgrade causing rabbitmq cluster to break

Bug #1522611 reported by Edward Hope-Morley on 2015-12-03
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
rabbitmq-server (Juju Charms Collection)
Critical
Unassigned

Bug Description

We have observed that upgrading an existing 3 unit rabbitmq-server cluster to the latest stable charms can cause the cluster to break. This is because the charm will check that the rabbitmq-server (at least on the leader) is configured by ensuring that RABBITMQ_NODENAME=rabbit@<hostname> is set in /etc/rabbitmq/rabbitmq-env.conf. Trouble is, when the hook runs to perform this action, it will use whatever is returned by charmhelpers.contrib.openstack.utils.get_hostname(). Now, in some of the environments we recently upgraded, the MAAS dns entry for the rabbitmq-server host IP has multiple ptr entries e.g.

;; ANSWER SECTION:
1.5.27.172.in-addr.arpa. 300 IN PTR ctrlnode1.maas.
1.5.27.172.in-addr.arpa. 300 IN PTR 172-27-5-1.maas.

(see http://paste.ubuntu.com/13653431/ for full answer)

The result is that a rabbitmq node whose nodename was originally configured with 172-27-5-1 now switches to ctrlnode1 which, when the charm attempts to restart rabbitmq, causes it to fail since it no longer recognises the nodename e.g. https://pastebin.canonical.com/145396/

Edward Hope-Morley (hopem) wrote :

For the record this was with MAAS 1.5.4.

description: updated
Changed in rabbitmq-server (Juju Charms Collection):
assignee: nobody → Jorge Niedbalski (niedbalski)
Edward Hope-Morley (hopem) wrote :

It is not actually legitimate to have > 1 PTR record for an IP address so this is actually a bug in the DNS (in this case MAAS 1.5.4 which is very old and newer versions do not exhibit this behaviour).

description: updated
Changed in rabbitmq-server (Juju Charms Collection):
status: New → Invalid
Changed in rabbitmq-server (Juju Charms Collection):
assignee: Jorge Niedbalski (niedbalski) → nobody
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers