'hook failed: "cluster-relation-changed"' when adding third node to rabbitmq-server service which is ha-clustered with ceph.
How to reproduce:
1. deploy 3 ceph nodes
2. deploy 2 rabbitmq-server nodes
3. deploy hacluster charm
4. add-relation rabbitmq-server and ceph
5. add-relation rabbitmq-server and hacluster
6. confirm 2 rabbitmq-server nodes online by `crm status` on either side of rabbitmq-server node.
7. add-unit rabbitmq-server
workaround:
with just one try of 'juju resolved --retry rabbitmq-server/[error_node]', the error will be cleared.
====
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK Stopping node 'rabbit@node-1467d3-6' ...
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK ...done.
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK Clustering node 'rabbit@node-1467d3-6' with ['rabbit@node-a19002-5']
...
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK Error: {no_running_cluster_nodes,['rabbit@node-a19002-5'],
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK ['rabbit@node-a19002-5']}
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK Traceback (most recent call last):
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/clus
ter-relation-changed", line 342, in <module>
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK utils.do_hooks(hooks)
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/lib/
utils.py", line 28, in do_hooks
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK hook_func()
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/clus
ter-relation-changed", line 122, in cluster_changed
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK rabbit.cluster_with(remote_host)
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK File "/var/lib/juju/agents/unit-rabbitmq-server-2/charm/hooks/rabb
it_utils.py", line 101, in cluster_with
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK subprocess.check_call(cmd)
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK raise CalledProcessError(retcode, cmd)
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter context.go:255 HOOK subprocess.CalledProcessError: Command '['/usr/sbin/rabbitmqctl', 'c
luster', 'rabbit@node-a19002-5']' returned non-zero exit status 2
node-1467d3-6:2013-11-29 05:04:57 ERROR juju.worker.uniter uniter.go:350 hook failed: exit status 1
node-1467d3-6:2013-11-29 05:04:57 INFO juju.worker.uniter modes.go:421 ModeHookError starting
I used this deployer file to reproduce step 5, then I executed `juju add-unit rabbitmq-server`.