Set RabbitMQ cluster_partition_handling mode to pause_minority

[Alexey Khivin]

Set RabbitMQ cluster_partition_handling mode to pause_minority instead of autoheal

this mode should protect cluster out of issues about syncing after split brain when two parts of the cluster could be used independently

[Alexey Khivin (akhivin)]

But I should note that this feature could be more useful if HA would not break and rebuild RMQ cluster

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/179171

[Bogdan Dobrelya (bogdando)]

Please describe a test case for QA team - how this change would help to prevent permanent rabbit partitions comparing to existing auto_heal value

[Bogdan Dobrelya (bogdando)]

I'm marking this one as a duplicate as switching to the pause-minority is a part of suggested work around for the main bug, which is https://bugs.launchpad.net/bugs/1447619

[Bogdan Dobrelya (bogdando)]

As testing shows, there is no reasons to switch to the pause minority. Autoheal works as well as pause minority, when it comes partitioning recovery.

[Alexey Khivin (akhivin)]

¨As testing shows¨ is only about rabbitmq cluster recovery, but not in the sense of rabbitmq usage.

For example, one controller loose connection with other two.
Does this mean that clients that using this rabbit node loose connection too? Is it possible that in the autoheal mode one of the controllers works separately and it´s clients knows nothing about split brain? I think, yes. With the autoheal mode clients will not try to move to another node.

I think it is more desirable to be sure that separated node would not handle incoming connections. Thus clients should try to move to another node.

The main reason why I want to change cluster mode to pause_minority is to guarantee that all of the clients will always connected to the same part of the splitted cluster

[Bogdan Dobrelya (bogdando)]

@Alex, there is no difference at app level, which is Oslo and its rabbit_hosts parameter, either the underlying AMQP is using pause_minority or autoheal. When failover occurs, Oslo will have to reconnect anyway.

[Alexey Khivin (akhivin)]

yes, oslo should reconnect anyway

but I want a guarantee that if one node gone from the cluster then it should not handle incoming connection until rejoined to the cluster

[Bogdan Dobrelya (bogdando)]

I could be wrong, but split brain by definition assumes this situation *had* happened, and there were incoming connections for each partition. Pause minority and autoheal is about partition recovery, which is how to merge changes and which decisions for cluster logic to be made in order to exit partitioned state.

[Alexey Khivin (akhivin)]

pause_minority is not only about merging strategy

the single node should never handle incoming connections
it must reset all connections for moving all clients to the one partition of cluster

[Matthew Mosesohn (raytrac3r)]

Alexey Khivin's patch on review says we need a new RabbitMQ, but the version # isn't known. Moving to incomplete.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alexey Khivin (<email address hidden>) on branch: master
Review: https://review.openstack.org/179171
Reason: I think too many things changed since the patch