Comment 5 for bug 1498167

Revision history for this message
Leontii Istomin (listomin) wrote :

Ilya Shakhat has found the following in neutron.log:
2015-09-19 23:17:13.980 2733 ERROR oslo_messaging._drivers.impl_rabbit [req-6e630e90-66c3-46a7-9079-b29412d3033c ] AMQP server on 192.168.0.149:5673 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker
forced connection closure with reason 'shutdown'. Trying again in 5 seconds.
192.168.0.149 is node-174
I've checket rabbitmq on node-174 at the time:
=ERROR REPORT==== 19-Sep-2015::23:16:20 ===
Error on AMQP connection <0.8579.1> (192.168.0.149:44638 -> 192.168.0.149:5673, vhost: '/', user: 'nova', state: running), channel 0:
{amqp_error,connection_forced,
            "broker forced connection closure with reason 'shutdown'",none}

=INFO REPORT==== 19-Sep-2015::23:16:21 ===
Stopped RabbitMQ application

=INFO REPORT==== 19-Sep-2015::23:16:21 ===
Halting Erlang VM

=INFO REPORT==== 19-Sep-2015::23:17:33 ===
Error description:
   {error,{inconsistent_cluster,"Node 'rabbit@node-174' thinks it's clustered with node 'rabbit@node-170', but 'rabbit@node-170' disagrees"}}

Log files (may contain more information):
   /<email address hidden>
   /<email address hidden>

Stack trace:
   [{rabbit_mnesia,check_cluster_consistency,0,
                   [{file,"src/rabbit_mnesia.erl"},{line,588}]},
    {rabbit,'-start/0-fun-0-',0,[{file,"src/rabbit.erl"},{line,317}]},
    {rabbit,start_it,1,[{file,"src/rabbit.erl"},{line,353}]},
    {rpc,'-handle_call_call/6-fun-0-',5,[{file,"rpc.erl"},{line,205}]}]

=INFO REPORT==== 19-Sep-2015::23:17:35 ===
Resetting Rabbit

Rabbitmq was broken on node-174 at the time. Rabbitmq was alive on two other controller nodes at the time