Comment 0 for bug 1458828

Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

Steps to reproduce:

0) deploy any HA cluster of 3 controllers.
Assume we have a node-1 as a primary controller with the rabbit multistate clone resource master running, and
node-2, node-3 as running multistate resource slaves.

1) Move rabbit master resource to node-2
2) wait for ostf ha passed
3) kill the node-2, which should be a rabbit master now
4) wait for ostf ha passed
5) power on node-2
6) wait for it joined the rabbit cluster (wait for ostf ha passed)
7) repeat 1-6

Expected:
A. The rabbitmq cluster assembles from two remaining nodes, having 1 master and 1 slave after the step #4, no longer than in 5 minutes.
B. The rabbitmq cluster assembles from three nodes, having 1 master and 2 slaves after the steps #2, #6, no longer than in 5 minutes.
C. At least one node is always available for AMQP connections and it's queues and messages synced with other nodes, if any available as well.

Actual: after the step #6, the node-2 will have its rabbit app stopped and will not be shown as running db node in rabbitmcrl report by other nodes and by itself.

Note, sometimes this case can be reproduced after few iterations, sometimes it may take up to 20 or 40 - it looks random.