In the meantime, while testing the leadership bug it revealed this bug as a charm bug. The rabbitmq-server charm is not waiting until the cluster is completely formed before running amqp-relation-* hooks. This at least in theory could lead to the split brain problems described here.
Working on a change that
1) Will use the min-cluster-size to try and guarantee waiting for the full cluster to form
2) When min-cluster is not set try and determine the number of cluster relations and wait on them
3) Once the cluster is formed only the leader node will run amqp-relation-*
The extant charm does this to some degree already and thus the suspicion of Bug#1654116
The leader election issues in bug https:/ /bugs.launchpad .net/charms/ +source/ rabbitmq- server/ +bug/1654116/ is a prime suspect for this bug. We'll follow the juju-core response there.
In the meantime, while testing the leadership bug it revealed this bug as a charm bug. The rabbitmq-server charm is not waiting until the cluster is completely formed before running amqp-relation-* hooks. This at least in theory could lead to the split brain problems described here.
Working on a change that
1) Will use the min-cluster-size to try and guarantee waiting for the full cluster to form
2) When min-cluster is not set try and determine the number of cluster relations and wait on them
3) Once the cluster is formed only the leader node will run amqp-relation-*
The extant charm does this to some degree already and thus the suspicion of Bug#1654116