hook failed: "cluster-relation-changed" when adding third node to haclustered rabbitmq-server

[Nobuto Murata]

'hook failed: "cluster-relation-changed"' when adding third node to rabbitmq-server service which is ha-clustered with ceph.

How to reproduce:
 1. deploy 3 ceph nodes
 2. deploy 2 rabbitmq-server nodes
 3. deploy hacluster charm
 4. add-relation rabbitmq-server and ceph
 5. add-relation rabbitmq-server and hacluster
 6. confirm 2 rabbitmq-server nodes online by `crm status` on either side of rabbitmq-server node.
 7. add-unit rabbitmq-server

workaround:
with just one try of 'juju resolved --retry rabbitmq-server/[error_node]', the error will be cleared.

====

node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK Stopping node 'rabbit@node-1467d3-6' ...
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK ...done.
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK Clustering node 'rabbit@node-1467d3-6' with ['rabbit@node-a19002-5']
 ...
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK Error: {no_running_cluster_nodes,['rabbit@node-a19002-5'],
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK ['rabbit@node-a19002-5']}
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK Traceback (most recent call last):
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/clus
ter-relation-changed", line 342, in <module>
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK utils.do_hooks(hooks)
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/lib/
utils.py", line 28, in do_hooks
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK hook_func()
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/clus
ter-relation-changed", line 122, in cluster_changed
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK rabbit.cluster_with(remote_host)
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/rabb
it_utils.py", line 101, in cluster_with
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK subprocess.check_call(cmd)
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK raise CalledProcessError(retcode, cmd)
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK subprocess.CalledProcessError: Command '['/usr/sbin/rabbitmqctl', 'c
luster', 'rabbit@node-a19002-5']' returned non-zero exit status 2
node-1467d3-6:2013-11-29 05:04:57 ERROR juju.worker.uniter uniter.go:350 hook failed: exit status 1
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter modes.go:421 ModeHookError starting

[Nobuto Murata (nobuto)]

I used this deployer file to reproduce step 5, then I executed `juju add-unit rabbitmq-server`.

[Nobuto Murata (nobuto)]

This debug-log contains whole logs of deploy, add-relation, add-unit, resolved --retry.

[Matt Rae (mattrae)]

I've seen this issue as well deploying 3 rabbitmq-server nodes

[James Page (james-page)]

My preference is for uses to start using the native rabbitmq clustering rather than hacluster + ceph.

Marking low for this reason; we'll probably deprecate and remove the hacluster+ceph support.

[James Page (james-page)]

hacluster based rabbitmq is no longer supported and will be removed from the charm.