The issue is caused by pacemaker, which for some reason can not select a new master. It can be seen in 'pcs resource' output that it continues to run with 2 slaves, though there is no master: http://paste.openstack.org/show/479567/
Here is 'crm_mon -fotAW -1' output, though I can not make anythin useful out of it: http://paste.openstack.org/show/479568/
Also, pacemaker logs contain these entries repeating every 10-20 seconds:
Nov 20 11:49:51 [16248] node-1.test.domain.local pengine: info: clone_print: Master/Slave Set: master_p_rabbitmq-server [p_rabbitmq-server] Nov 20 11:49:51 [16248] node-1.test.domain.local pengine: info: native_color: Resource p_rabbitmq-server:2 cannot run anywhere Nov 20 11:49:51 [16248] node-1.test.domain.local pengine: info: master_color: master_p_rabbitmq-server: Promoted 0 instances of a possible 1 to master Nov 20 11:49:51 [16248] node-1.test.domain.local pengine: info: LogActions: Leave p_rabbitmq-server:0 (Slave node-1.test.domain.local) Nov 20 11:49:51 [16248] node-1.test.domain.local pengine: info: LogActions: Leave p_rabbitmq-server:1 (Slave node-3.test.domain.local) Nov 20 11:49:51 [16248] node-1.test.domain.local pengine: info: LogActions: Leave p_rabbitmq-server:2 (Stopped)
The issue is caused by pacemaker, which for some reason can not select a new master. It can be seen in 'pcs resource' output that it continues to run with 2 slaves, though there is no master: http:// paste.openstack .org/show/ 479567/
Here is 'crm_mon -fotAW -1' output, though I can not make anythin useful out of it: http:// paste.openstack .org/show/ 479568/
Also, pacemaker logs contain these entries repeating every 10-20 seconds:
Nov 20 11:49:51 [16248] node-1. test.domain. local pengine: info: clone_print: Master/Slave Set: master_ p_rabbitmq- server [p_rabbitmq-server] test.domain. local pengine: info: native_color: Resource p_rabbitmq-server:2 cannot run anywhere test.domain. local pengine: info: master_color: master_ p_rabbitmq- server: Promoted 0 instances of a possible 1 to master test.domain. local pengine: info: LogActions: Leave p_rabbitmq-server:0 (Slave node-1. test.domain. local) test.domain. local pengine: info: LogActions: Leave p_rabbitmq-server:1 (Slave node-3. test.domain. local) test.domain. local pengine: info: LogActions: Leave p_rabbitmq-server:2 (Stopped)
Nov 20 11:49:51 [16248] node-1.
Nov 20 11:49:51 [16248] node-1.
Nov 20 11:49:51 [16248] node-1.
Nov 20 11:49:51 [16248] node-1.
Nov 20 11:49:51 [16248] node-1.