Ilya Shakhat has found the following in neutron.log:
2015-09-19 23:17:13.980 2733 ERROR oslo_messaging._drivers.impl_rabbit [req-6e630e90-66c3-46a7-9079-b29412d3033c ] AMQP server on 192.168.0.149:5673 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker
forced connection closure with reason 'shutdown'. Trying again in 5 seconds.
192.168.0.149 is node-174
I've checket rabbitmq on node-174 at the time:
=ERROR REPORT==== 19-Sep-2015::23:16:20 ===
Error on AMQP connection <0.8579.1> (192.168.0.149:44638 -> 192.168.0.149:5673, vhost: '/', user: 'nova', state: running), channel 0:
{amqp_error,connection_forced,
"broker forced connection closure with reason 'shutdown'",none}
Ilya Shakhat has found the following in neutron.log: _drivers. impl_rabbit [req-6e630e90- 66c3-46a7- 9079-b29412d303 3c ] AMQP server on 192.168.0.149:5673 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker 2015::23: 16:20 === 0.149:44638 -> 192.168.0.149:5673, vhost: '/', user: 'nova', state: running), channel 0: connection_ forced,
2015-09-19 23:17:13.980 2733 ERROR oslo_messaging.
forced connection closure with reason 'shutdown'. Trying again in 5 seconds.
192.168.0.149 is node-174
I've checket rabbitmq on node-174 at the time:
=ERROR REPORT==== 19-Sep-
Error on AMQP connection <0.8579.1> (192.168.
{amqp_error,
"broker forced connection closure with reason 'shutdown'",none}
=INFO REPORT==== 19-Sep- 2015::23: 16:21 ===
Stopped RabbitMQ application
=INFO REPORT==== 19-Sep- 2015::23: 16:21 ===
Halting Erlang VM
=INFO REPORT==== 19-Sep- 2015::23: 17:33 === {inconsistent_ cluster, "Node 'rabbit@node-174' thinks it's clustered with node 'rabbit@node-170', but 'rabbit@node-170' disagrees"}}
Error description:
{error,
Log files (may contain more information):
/<email address hidden>
/<email address hidden>
Stack trace: mnesia, check_cluster_ consistency, 0,
[{ file,"src/ rabbit_ mnesia. erl"},{ line,588} ]}, '-start/ 0-fun-0- ',0,[{file, "src/rabbit. erl"},{ line,317} ]}, start_it, 1,[{file, "src/rabbit. erl"},{ line,353} ]}, '-handle_ call_call/ 6-fun-0- ',5,[{file, "rpc.erl" },{line, 205}]}]
[{rabbit_
{rabbit,
{rabbit,
{rpc,
=INFO REPORT==== 19-Sep- 2015::23: 17:35 ===
Resetting Rabbit
Rabbitmq was broken on node-174 at the time. Rabbitmq was alive on two other controller nodes at the time