Steps to reproduce:
1. Deploy ha environment with 3 controller 1 compute and cinder nodes.
2. When cluster is ready, run health check, all tests should be passed
3. ssh on first controller and find controller where rabbit master is running using pcs status
4. Destroy it (virsh destroy <domain_name>)
6. wait for 10 minutes while cluster recovers
7. Run ostf
Expected result:
New rabbit master re-election happens, there is one rabbit cluster, pacs status on the last 2 controllers shows the same, rabbitmqctl cluster_status shows the same on each online controller
Actual:
Master rabbit was not elected, for each online controller rabbitmqctl shows that there is single one node cluster:
root@node-3:~# rabbitmqctl cluster_status
Cluster status of node 'rabbit@node-3' ...
[{nodes,[{disc,['rabbit@node-3']}]}]
root@node-3:~# OCF_ROOT=/usr/lib/ocf /usr/lib/ocf/resource.d/fuel/rabbitmq-server status ; echo $?
0
on other online controller
root@node-1:~# rabbitmqctl cluster_status
Cluster status of node 'rabbit@node-1' ...
[{nodes,[{disc,['rabbit@node-1']}]},
{running_nodes,['rabbit@node-1']},
{cluster_name,<<"<email address hidden>">>},
{partitions,[]}]
root@node-1:~# OCF_ROOT=/usr/lib/ocf /usr/lib/ocf/resource.d/fuel/rabbitmq-server status ; echo $?
0
root@node-1:~#
pcs status
Cluster name:
Last updated: Fri Nov 20 10:55:27 2015
Last change: Fri Nov 20 05:42:14 2015
Stack: corosync
Current DC: node-1.test.domain.local (1) - partition with quorum
Version: 1.1.12-561c4cf
3 Nodes configured
46 Resources configured
Online: [ node-1.test.domain.local node-3.test.domain.local ]
OFFLINE: [ node-2.test.domain.local ]
Full list of resources:
Clone Set: clone_p_vrouter [p_vrouter]
Started: [ node-1.test.domain.local node-3.test.domain.local ]
vip__management (ocf::fuel:ns_IPaddr2): Started node-1.test.domain.local
vip__vrouter_pub (ocf::fuel:ns_IPaddr2): Started node-1.test.domain.local
vip__vrouter (ocf::fuel:ns_IPaddr2): Started node-1.test.domain.local
vip__public (ocf::fuel:ns_IPaddr2): Started node-3.test.domain.local
Clone Set: clone_p_haproxy [p_haproxy]
Started: [ node-1.test.domain.local node-3.test.domain.local ]
Clone Set: clone_p_mysql [p_mysql]
Started: [ node-1.test.domain.local node-3.test.domain.local ]
Clone Set: clone_p_dns [p_dns]
Started: [ node-1.test.domain.local node-3.test.domain.local ]
Master/Slave Set: master_p_rabbitmq-server [p_rabbitmq-server]
Slaves: [ node-1.test.domain.local node-3.test.domain.local ]
Clone Set: clone_p_neutron-plugin-openvswitch-agent [p_neutron-plugin-openvswitch-agent]
Started: [ node-1.test.domain.local node-3.test.domain.local ]
Clone Set: clone_p_neutron-l3-agent [p_neutron-l3-agent]
Started: [ node-1.test.domain.local node-3.test.domain.local ]
Clone Set: clone_p_neutron-dhcp-agent [p_neutron-dhcp-agent]
Started: [ node-1.test.domain.local node-3.test.domain.local ]
Clone Set: clone_p_neutron-metadata-agent [p_neutron-metadata-agent]
Started: [ node-1.test.domain.local node-3.test.domain.local ]
Clone Set: clone_p_heat-engine [p_heat-engine]
Started: [ node-1.test.domain.local node-3.test.domain.local ]
Master/Slave Set: master_p_conntrackd [p_conntrackd]
Masters: [ node-1.test.domain.local ]
Slaves: [ node-3.test.domain.local ]
sysinfo_node-1.test.domain.local (ocf::pacemaker:SysInfo): Started node-1.test.domain.local
sysinfo_node-3.test.domain.local (ocf::pacemaker:SysInfo): Started node-3.test.domain.local
sysinfo_node-2.test.domain.local (ocf::pacemaker:SysInfo): Stopped
Clone Set: clone_ping_vip__public [ping_vip__public]
Started: [ node-1.test.domain.local node-3.test.domain.local ]
Clone Set: clone_p_ntp [p_ntp]
Started: [ node-1.test.domain.local node-3.test.domain.local ]
healthcheck tests are failed
[root@nailgun ~]# cat /etc/fuel/version.yaml
VERSION:
feature_groups:
- mirantis
production: "docker"
release: "8.0"
openstack_version: "2015.1.0-8.0"
api: "1.0"
build_number: "181"
build_id: "181"
fuel-nailgun_sha: "e5600e8a87745323c2a9c524b7c0647c9829ebb9"
python-fuelclient_sha: "e685d68c1c0d0fa0491a250f07d9c3a8d0f9608c"
fuel-agent_sha: "6faa1e0ba836ef114b3c1a6c4d12469fc66ae402"
fuel-nailgun-agent_sha: "db1738a65012d2a1d2b7e83cc2a44a196e4290b4"
astute_sha: "c8400f51b0b92254da206de55ef89d17fdf35393"
fuel-library_sha: "f9281f9c3b08ed16b2fb411e30f5d48809446030"
fuel-ostf_sha: "7e24fc802a95d2f0627512e585f8977f587aea18"
fuel-createmirror_sha: "22a7aacd95bbdca69f9e0f08b70facabdec8fb28"
fuelmenu_sha: "d12061b1aee82f81b3d074de74ea27a6e962a686"
shotgun_sha: "c377d163519f6d10b69a654019d6086ba5f14edc"
network-checker_sha: "a57e1d69acb5e765eb22cab0251c589cd76f51da"
fuel-upgrade_sha: "1e894e26d4e1423a9b0d66abd6a79505f4175ff6"
fuelmain_sha: "cd084cf5c4372a46184fb7c2f24568da4e030be2"
[root@nailgun ~]#
https:/