RabbitMQ is assembled into cluster by pacemaker in several separate stages ('start' to checking Mnesia database consistensy, 'pre-promote', 'promote' and 'post-promote'  to choose the Master and join other nodes to it).

Pacemaker runs each stage for 'rabbitmq' resource together with other resources ('heat' and 'mysql'), and goes to the next stage only when all resources are processed in the current stage, one-by-one.

We often facing broken galera cluster that takes a long time when restoring the cluster. 

The script /usr/lib/ocf/resource.d/mirantis/mysql-wss on the controller consumes for about 7 minutes for every try to start the galera, not allowing pacemeker to process other resources. This leads to about a seven-minute period between processing the 'rabbitmq' stages. 

Taking into account other resources, we have to wait for about 20 minutes before RabbitMQ will be functional ('start' ... 10 minutes wait for others ... 'promote to master' ... 10 minutes wait for others ... 'join to cluster and allow access to rabbitmq').

Unfortunatelly, logging from 'mysql-wss' is broken, so here is output of mysql-wss script started manually:


================================================================================
[root@node-2 mirantis]# date
Thu Oct 23 18:24:23 UTC 2014

[root@node-2 mirantis]# OCF_ROOT=/usr/lib/ocf/ /usr/lib/ocf/resource.d/mirantis/mysql-wss start
INFO:  mysql_status: ====================== i = 1 ; sleeptime = 5
INFO: PIDFile /var/run/mysql/mysqld.pid of MySQL server not found. Sleeping for 5 seconds. 0 retries left
INFO: MySQL is not running
Resource 'default' not found: No such device or address
Error performing operation: No such device or address
INFO: Checking if galera primary controller
INFO: GTID OK: 96cc782e-5aa0-11e4-b985-066f5a65a8fa:24134
INFO: GTID OK: 96cc782e-5aa0-11e4-b985-066f5a65a8fa:24114
INFO: GTID OK: 96cc782e-5aa0-11e4-b985-066f5a65a8fa:24209
INFO: Possible masters:  node-4.test.domain.local
INFO: Choosed master: node-4.test.domain.local
date
INFO: Waiting for master. 300 seconds to go
Resource 'default' not found: No such device or address
Error performing operation: No such device or address
INFO: Waiting for master. 270 seconds to go
Resource 'default' not found: No such device or address
Error performing operation: No such device or address
INFO: Waiting for master. 240 seconds to go
Resource 'default' not found: No such device or address
Error performing operation: No such device or address
INFO: Waiting for master. 210 seconds to go
Resource 'default' not found: No such device or address
Error performing operation: No such device or address
INFO: Waiting for master. 180 seconds to go
Resource 'default' not found: No such device or address
Error performing operation: No such device or address
INFO: Waiting for master. 150 seconds to go
Resource 'default' not found: No such device or address
Error performing operation: No such device or address
INFO: Waiting for master. 120 seconds to go
Resource 'default' not found: No such device or address
Error performing operation: No such device or address
INFO: Waiting for master. 90 seconds to go
Resource 'default' not found: No such device or address
Error performing operation: No such device or address
INFO: Waiting for master. 60 seconds to go
Resource 'default' not found: No such device or address
Error performing operation: No such device or address
INFO: Waiting for master. 30 seconds to go
Resource 'default' not found: No such device or address
Error performing operation: No such device or address
INFO: GTID OK: 96cc782e-5aa0-11e4-b985-066f5a65a8fa:24591
INFO: GTID OK: 96cc782e-5aa0-11e4-b985-066f5a65a8fa:24809
INFO: GTID OK: 96cc782e-5aa0-11e4-b985-066f5a65a8fa:24905
INFO: Possible masters:  node-4.test.domain.local
INFO: Choosed master: node-4.test.domain.local
INFO:  mysql_status: ====================== i = 3 ; sleeptime = 5
INFO: MySQL not running: removing old PID file
INFO:  mysql_status: ====================== i = 3 ; sleeptime = 5
INFO: PIDFile /var/run/mysql/mysqld.pid of MySQL server not found. Sleeping for 5 seconds. 2 retries left
INFO: PIDFile /var/run/mysql/mysqld.pid of MySQL server not found. Sleeping for 5 seconds. 1 retries left
INFO: PIDFile /var/run/mysql/mysqld.pid of MySQL server not found. Sleeping for 5 seconds. 0 retries left
INFO: MySQL is not running
INFO:  mysql_status: ====================== i = 3 ; sleeptime = 5
INFO: PIDFile /var/run/mysql/mysqld.pid of MySQL server not found. Sleeping for 5 seconds. 2 retries left
INFO: PIDFile /var/run/mysql/mysqld.pid of MySQL server not found. Sleeping for 5 seconds. 1 retries left
INFO: MySQL started

[root@node-2 mirantis]# date
Thu Oct 23 18:31:53 UTC 2014
================================================================================