Ha ipv6 configuration take a lot of time to finish. After the deployment is complete a nova list on the overcloud succeeds. However, pcs status shows rabbitmq has not started on two of the tree overcloud nodes.
Cluster name: tripleo_cluster
Last updated: Mon Sep 26 12:26:00 2016 Last change: Mon Sep 26 09:11:16 2016 by root via cibadmin on overcloud-controller-0
Stack: corosync
Current DC: overcloud-controller-2 (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum
3 nodes and 19 resources configured
Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
Full list of resources:
ip-fd00.fd00.fd00.2000..14 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0
Clone Set: haproxy-clone [haproxy]
Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
ip-192.0.2.8 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1
Master/Slave Set: galera-master [galera]
Masters: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
ip-2001.db8.fd00.1000..19 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2
ip-fd00.fd00.fd00.3000..11 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0
Clone Set: rabbitmq-clone [rabbitmq]
Started: [ overcloud-controller-0 ]
Stopped: [ overcloud-controller-1 overcloud-controller-2 ]
Master/Slave Set: redis-master [redis]
Masters: [ overcloud-controller-0 ]
Slaves: [ overcloud-controller-1 overcloud-controller-2 ]
ip-fd00.fd00.fd00.2000..10 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1
ip-fd00.fd00.fd00.4000..19 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2
openstack-cinder-volume (systemd:openstack-cinder-volume): Started overcloud-controller-1
Failed Actions:
* rabbitmq_start_0 on overcloud-controller-2 'unknown error' (1): call=66, status=complete, exitreason='none',
last-rc-change='Mon Sep 26 08:47:31 2016', queued=0ms, exec=5945ms
* rabbitmq_start_0 on overcloud-controller-1 'unknown error' (1): call=61, status=complete, exitreason='none',
last-rc-change='Mon Sep 26 08:47:02 2016', queued=0ms, exec=18237ms
log in /var/log/rabbitmq show a lot of crashes with this report
=CRASH REPORT==== 26-Sep-2016::09:43:50 ===
crasher:
initial call: rabbit_reader:init/4
pid: <0.1570.0>
registered_name: []
exception exit: {aborted,
{no_exists,[rabbit_runtime_parameters,cluster_name]}}
in function mnesia:abort/1 (mnesia.erl, line 313)
in call from rabbit_runtime_parameters:lookup0/2 (src/rabbit_runtime_parameters.erl, line 272)
in call from rabbit_runtime_parameters:value0/2 (src/rabbit_runtime_parameters.erl, line 268)
in call from rabbit_reader:server_properties/1 (src/rabbit_reader.erl, line 282)
in call from rabbit_reader:start_connection/3 (src/rabbit_reader.erl, line 1091)
in call from rabbit_reader:handle_input/3 (src/rabbit_reader.erl, line 1041)
in call from rabbit_reader:recvloop/4 (src/rabbit_reader.erl, line 446)
in call from rabbit_reader:run/1 (src/rabbit_reader.erl, line 428)
ancestors: [<0.1568.0>,<0.847.0>,<0.846.0>,<0.845.0>,rabbit_sup,
<0.697.0>]
messages: [{'EXIT',#Port<0.9969>,normal}]
links: [<0.1568.0>]
dictionary: [{process_name,
{rabbit_reader,
<<"[FD00:FD00:FD00:2000::10]:57696 -> [FD00:FD00:FD00:2000::18]:5672">>}}]
trap_exit: true
status: running
heap_size: 1598
stack_size: 27
reductions: 1613
neighbours:
journalctl on one of the failing nodes show
Sep 26 12:56:19 overcloud-controller-2 pengine[3007]: warning: Forcing rabbitmq-clone away from overcloud-controller-2 after 1000000 failures (max=1000000)
look at http://logs.openstack.org/74/363674/27/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-ha-ipv6/74c9d65/ for more informations
Latest findings:
pacemaker reports rabbitmq has failed, but appears to be up and running, altought there are a lot of crash reports on the logs.
ocf::heartbeat: