| 2015-12-03 22:44:36 |
Edward Hope-Morley |
bug |
|
|
added bug |
| 2015-12-04 09:47:42 |
Edward Hope-Morley |
description |
We have observed that upgrading an existing 3 unit rabbitmq-server cluster to the latest stable charms can cause the cluster to break. This is because will the charm will check that the rabbitmq-server (at least on the leader) is configured by ensuring that RABBITMQ_NODENAME=rabbit@<hostname> is set in /etc/rabbitmq/rabbitmq-env.conf. Trouble is, when the hook runs to perform this action, it will use whatever is returned by charmhelpers.contrib.openstack.utils.get_hostname(). Now, in some of the environments we recently upgraded, the MAAS dns entry for the rabbitmq-server host IP has multiple cname entries e.g.
;; ANSWER SECTION:
1.5.27.172.in-addr.arpa. 300 IN PTR ctrlnode1.maas.
1.5.27.172.in-addr.arpa. 300 IN PTR 172-27-5-1.maas.
(see http://paste.ubuntu.com/13653431/ for full answer)
The result is that a rabbitmq node whose nodename was originally configured with 172-27-5-1 now switches to ctrlnode1 which, when the charm attempts to restart rabbitmq, causes it to fail since it no longer recognises the nodename e.g. https://pastebin.canonical.com/145396/
The problem he appears to be simple, if the charm is presented with multiple cnames it needs be aware of this and chose the correct one based on what it may have previously configured in rabbitmq. |
We have observed that upgrading an existing 3 unit rabbitmq-server cluster to the latest stable charms can cause the cluster to break. This is because the charm will check that the rabbitmq-server (at least on the leader) is configured by ensuring that RABBITMQ_NODENAME=rabbit@<hostname> is set in /etc/rabbitmq/rabbitmq-env.conf. Trouble is, when the hook runs to perform this action, it will use whatever is returned by charmhelpers.contrib.openstack.utils.get_hostname(). Now, in some of the environments we recently upgraded, the MAAS dns entry for the rabbitmq-server host IP has multiple cname entries e.g.
;; ANSWER SECTION:
1.5.27.172.in-addr.arpa. 300 IN PTR ctrlnode1.maas.
1.5.27.172.in-addr.arpa. 300 IN PTR 172-27-5-1.maas.
(see http://paste.ubuntu.com/13653431/ for full answer)
The result is that a rabbitmq node whose nodename was originally configured with 172-27-5-1 now switches to ctrlnode1 which, when the charm attempts to restart rabbitmq, causes it to fail since it no longer recognises the nodename e.g. https://pastebin.canonical.com/145396/
The problem he appears to be simple, if the charm is presented with multiple cnames it needs be aware of this and chose the correct one based on what it may have previously configured in rabbitmq. |
|
| 2015-12-04 14:27:24 |
Jorge Niedbalski |
rabbitmq-server (Juju Charms Collection): assignee |
|
Jorge Niedbalski (niedbalski) |
|
| 2015-12-08 10:32:50 |
Edward Hope-Morley |
description |
We have observed that upgrading an existing 3 unit rabbitmq-server cluster to the latest stable charms can cause the cluster to break. This is because the charm will check that the rabbitmq-server (at least on the leader) is configured by ensuring that RABBITMQ_NODENAME=rabbit@<hostname> is set in /etc/rabbitmq/rabbitmq-env.conf. Trouble is, when the hook runs to perform this action, it will use whatever is returned by charmhelpers.contrib.openstack.utils.get_hostname(). Now, in some of the environments we recently upgraded, the MAAS dns entry for the rabbitmq-server host IP has multiple cname entries e.g.
;; ANSWER SECTION:
1.5.27.172.in-addr.arpa. 300 IN PTR ctrlnode1.maas.
1.5.27.172.in-addr.arpa. 300 IN PTR 172-27-5-1.maas.
(see http://paste.ubuntu.com/13653431/ for full answer)
The result is that a rabbitmq node whose nodename was originally configured with 172-27-5-1 now switches to ctrlnode1 which, when the charm attempts to restart rabbitmq, causes it to fail since it no longer recognises the nodename e.g. https://pastebin.canonical.com/145396/
The problem he appears to be simple, if the charm is presented with multiple cnames it needs be aware of this and chose the correct one based on what it may have previously configured in rabbitmq. |
We have observed that upgrading an existing 3 unit rabbitmq-server cluster to the latest stable charms can cause the cluster to break. This is because the charm will check that the rabbitmq-server (at least on the leader) is configured by ensuring that RABBITMQ_NODENAME=rabbit@<hostname> is set in /etc/rabbitmq/rabbitmq-env.conf. Trouble is, when the hook runs to perform this action, it will use whatever is returned by charmhelpers.contrib.openstack.utils.get_hostname(). Now, in some of the environments we recently upgraded, the MAAS dns entry for the rabbitmq-server host IP has multiple ptr entries e.g.
;; ANSWER SECTION:
1.5.27.172.in-addr.arpa. 300 IN PTR ctrlnode1.maas.
1.5.27.172.in-addr.arpa. 300 IN PTR 172-27-5-1.maas.
(see http://paste.ubuntu.com/13653431/ for full answer)
The result is that a rabbitmq node whose nodename was originally configured with 172-27-5-1 now switches to ctrlnode1 which, when the charm attempts to restart rabbitmq, causes it to fail since it no longer recognises the nodename e.g. https://pastebin.canonical.com/145396/ |
|
| 2015-12-08 10:35:22 |
Edward Hope-Morley |
rabbitmq-server (Juju Charms Collection): status |
New |
Invalid |
|
| 2015-12-09 17:39:23 |
Jorge Niedbalski |
rabbitmq-server (Juju Charms Collection): assignee |
Jorge Niedbalski (niedbalski) |
|
|
