Fuel for OpenStack

ceilometer-rabbitmq-failover

Bug #1503363 reported by Michael Petersen on 2015-10-06

This bug report is a duplicate of: Bug #1510916: Ceilometer collector cannot reconnect to rabbitmq after RabbitMQ failover. Edit Remove

This bug affects 2 people

	Status	Importance	Assigned to	Milestone
Fuel for OpenStack	New	High	MOS Ceilometer	Fuel for OpenStack 9.0
6.1.x	Confirmed	High	MOS Ceilometer	Fuel for OpenStack 6.1-updates
7.0.x	New	Undecided	MOS Ceilometer	Fuel for OpenStack 7.0-mu-3
8.0.x	New	Undecided	MOS Ceilometer	Fuel for OpenStack 8.0
Future	New	Undecided	MOS Ceilometer	Fuel for OpenStack next

Bug Description

Fuel - 6.1

The non-graceful failover of RabbitMQ connections in Ceilometer causes issues in active environments when Ceilometer cannot connect to the next RabbitMQ system in line. This was found in a customer environment. When trying to reproduce:

Kill RabbitMQ on the master node
Tail the Ceilometer logs to watch connections to the next RabbitMQ cluster. It does't look like it actually does this correctly and it waits for the original cluster to come back up.

The Cluster ends up coming back up with the original node and it starts working, however if the node did not come back up it looks like it would continue to have issues connecting to RabbitMQ servers on the other nodes.

Logs:

2015-10-06 16:28:45.706 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server 192.168.0.5:5673 closed the connection. Check login credentials: Socket closed
2015-10-06 16:28:46.412 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server 192.168.0.5:5673 closed the connection. Check login credentials: Socket closed
2015-10-06 16:28:46.459 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server 192.168.0.5:5673 closed the connection. Check login credentials: Socket closed
2015-10-06 16:28:48.089 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.5:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2015-10-06 16:28:48.335 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.5:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2015-10-06 16:28:48.337 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.5:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2015-10-06 16:28:49.538 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.5:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2015-10-06 16:28:49.704 5914 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 192.168.0.7:5673
2015-10-06 16:28:49.727 5914 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 192.168.0.7:5673
2015-10-06 16:28:50.555 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.5:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.
2015-10-06 16:28:52.593 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.5:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2015-10-06 16:28:53.698 5914 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 192.168.0.4:5673
2015-10-06 16:29:55.229 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.4:5673 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds.
2015-10-06 16:29:56.415 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.4:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2015-10-06 16:29:57.555 5914 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 192.168.0.7:5673
2015-10-06 16:32:07.102 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.7:5673 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds.
2015-10-06 16:32:07.314 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.7:5673 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds.
2015-10-06 16:32:08.238 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.7:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2015-10-06 16:32:08.298 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.7:5673 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds.
2015-10-06 16:32:08.337 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.7:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2015-10-06 16:32:09.256 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.4:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2015-10-06 16:32:09.313 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.7:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2015-10-06 16:32:09.420 5914 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 192.168.0.5:5673
2015-10-06 16:32:10.270 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.4:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.
2015-10-06 16:32:10.330 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.4:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2015-10-06 16:32:11.357 5914 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 192.168.0.5:5673
2015-10-06 16:32:12.291 5914 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 192.168.0.5:5673

See original description

Tags:

Andrey Grebennikov (agrebennikov) on 2015-10-06

tags:

added: customer-found