Version: 9.1
Steps to reproduce:
1. Install environment consisting of 3 controllers.
2. On one of the controllers make 'rabbitmqctl cluster_status' command run for more than 1 minute. You can do that for instance by applying the following patch to /usr/sbin/rabbitmqctl - http://paste.openstack.org/show/565123/
Another alternative is to execute the following command on one of the controllers:
rabbitmqctl eval 'sys:suspend(whereis(rabbit_node_monitor)).'
That will make cluster_status hang on all controllers simultaneously until RabbitMQ is restarted.
Expected result:
After some time (up to 5 minutes) Pacemaker should restart the RabbitMQ on the controller where rabbitmqctl was corrupted.
Actual result:
RabbitMQ is not restarted, not even in days.
Reproducibility: 100%
Details:
Look into /var/log/remote/node-X/lrmd.log on master node, where node-X is hostname of controller on which you have corrupted rabbitmqctl.
After some time you will see the following entries appear in the log:
2016-08-29T22:16:36.741121+00:00 err: ERROR: p_rabbitmq-server[4041]: get_monitor():: is_cluster_status_ok: 'rabbitmqctl cluster_status' timed out 1 of max. 3 time(s) in a row and is not responding. The resource is failed.
At some point the counter will go over the limit (3), but nothing will change - Pacemaker will not restart the resource even though OCF script clearly reports error. In pacemaker.log it can be seen that Pacemaker sends pre-stop notifications about that node. If one adds logging of such notifications to the RabbitMQ OCF script, it can be seen that they do reach the nodes. Such pre-stop notifications are sent with some period 3-5 minutes). But nothing else happens, Pacemaker does not initiate the stop action itself.
Fix proposed to branch: master
Review: https:/