Comment 0 for bug 1627729

Revision history for this message
Gabriele Cerami (gcerami) wrote :

Ha ipv6 configuration take a lot of time to finish. After the deployment is complete a nova list on the overcloud succeeds. However, pcs status shows rabbitmq has not started on two of the tree overcloud nodes.

Cluster name: tripleo_cluster
Last updated: Mon Sep 26 12:26:00 2016 Last change: Mon Sep 26 09:11:16 2016 by root via cibadmin on overcloud-controller-0
Stack: corosync
Current DC: overcloud-controller-2 (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum
3 nodes and 19 resources configured

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Full list of resources:

 ip-fd00.fd00.fd00.2000..14 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0
 Clone Set: haproxy-clone [haproxy]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 ip-192.0.2.8 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1
 Master/Slave Set: galera-master [galera]
     Masters: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 ip-2001.db8.fd00.1000..19 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2
 ip-fd00.fd00.fd00.3000..11 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0
 Clone Set: rabbitmq-clone [rabbitmq]
     Started: [ overcloud-controller-0 ]
     Stopped: [ overcloud-controller-1 overcloud-controller-2 ]
 Master/Slave Set: redis-master [redis]
     Masters: [ overcloud-controller-0 ]
     Slaves: [ overcloud-controller-1 overcloud-controller-2 ]
 ip-fd00.fd00.fd00.2000..10 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1
 ip-fd00.fd00.fd00.4000..19 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2
 openstack-cinder-volume (systemd:openstack-cinder-volume): Started overcloud-controller-1

Failed Actions:
* rabbitmq_start_0 on overcloud-controller-2 'unknown error' (1): call=66, status=complete, exitreason='none',
    last-rc-change='Mon Sep 26 08:47:31 2016', queued=0ms, exec=5945ms
* rabbitmq_start_0 on overcloud-controller-1 'unknown error' (1): call=61, status=complete, exitreason='none',
    last-rc-change='Mon Sep 26 08:47:02 2016', queued=0ms, exec=18237ms

log in /var/log/rabbitmq show a lot of crashes with this report
=CRASH REPORT==== 26-Sep-2016::09:43:50 ===
  crasher:
    initial call: rabbit_reader:init/4
    pid: <0.1570.0>
    registered_name: []
    exception exit: {aborted,
                        {no_exists,[rabbit_runtime_parameters,cluster_name]}}
      in function mnesia:abort/1 (mnesia.erl, line 313)
      in call from rabbit_runtime_parameters:lookup0/2 (src/rabbit_runtime_parameters.erl, line 272)
      in call from rabbit_runtime_parameters:value0/2 (src/rabbit_runtime_parameters.erl, line 268)
      in call from rabbit_reader:server_properties/1 (src/rabbit_reader.erl, line 282)
      in call from rabbit_reader:start_connection/3 (src/rabbit_reader.erl, line 1091)
      in call from rabbit_reader:handle_input/3 (src/rabbit_reader.erl, line 1041)
      in call from rabbit_reader:recvloop/4 (src/rabbit_reader.erl, line 446)
      in call from rabbit_reader:run/1 (src/rabbit_reader.erl, line 428)
    ancestors: [<0.1568.0>,<0.847.0>,<0.846.0>,<0.845.0>,rabbit_sup,
                  <0.697.0>]
    messages: [{'EXIT',#Port<0.9969>,normal}]
    links: [<0.1568.0>]
    dictionary: [{process_name,
                      {rabbit_reader,
                          <<"[FD00:FD00:FD00:2000::10]:57696 -> [FD00:FD00:FD00:2000::18]:5672">>}}]
    trap_exit: true
    status: running
    heap_size: 1598
    stack_size: 27
    reductions: 1613
  neighbours:

look at http://logs.openstack.org/74/363674/27/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-ha-ipv6/74c9d65/ for more informations