Fuel for OpenStack

  1. Bug #1503363
  2. Activity log

Activity log for bug #1503363

Date Who What changed Old value New value Message
2015-10-06 16:42:19 Michael Petersen bug added bug
2015-10-06 16:43:52 Javier Diaz Jr bug added subscriber Javier Diaz Jr
2015-10-06 16:44:21 Javier Diaz Jr bug added subscriber Rene Soto
2015-10-06 16:48:00 Andrey Grebennikov tags customer-found
2015-10-07 08:38:08 Vadim Rovachev fuel: assignee MOS Ceilometer (mos-ceilometer)
2015-10-07 08:38:37 Vadim Rovachev fuel: importance Undecided High
2015-10-07 08:39:14 Vadim Rovachev tags customer-found ceilometer customer-found
2015-10-07 09:10:27 Ivan Berezovskiy fuel: status New Incomplete
2015-10-07 17:20:13 Michael Petersen description The non-graceful failover of RabbitMQ connections in Ceilometer causes issues in active environments when Ceilometer cannot connect to the next RabbitMQ system in line. This was found in a customer environment. When trying to reproduce: Kill RabbitMQ on the master node Tail the Ceilometer logs to watch connections to the next RabbitMQ cluster. It does't look like it actually does this correctly and it waits for the original cluster to come back up. The Cluster ends up coming back up with the original node and it starts working, however if the node did not come back up it looks like it would continue to have issues connecting to RabbitMQ servers on the other nodes. Logs: 2015-10-06 16:28:45.706 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server 192.168.0.5:5673 closed the connection. Check login credentials: Socket closed 2015-10-06 16:28:46.412 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server 192.168.0.5:5673 closed the connection. Check login credentials: Socket closed 2015-10-06 16:28:46.459 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server 192.168.0.5:5673 closed the connection. Check login credentials: Socket closed 2015-10-06 16:28:48.089 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.5:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2015-10-06 16:28:48.335 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.5:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2015-10-06 16:28:48.337 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.5:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2015-10-06 16:28:49.538 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.5:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2015-10-06 16:28:49.704 5914 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 192.168.0.7:5673 2015-10-06 16:28:49.727 5914 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 192.168.0.7:5673 2015-10-06 16:28:50.555 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.5:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. 2015-10-06 16:28:52.593 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.5:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2015-10-06 16:28:53.698 5914 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 192.168.0.4:5673 2015-10-06 16:29:55.229 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.4:5673 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds. 2015-10-06 16:29:56.415 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.4:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2015-10-06 16:29:57.555 5914 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 192.168.0.7:5673 2015-10-06 16:32:07.102 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.7:5673 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds. 2015-10-06 16:32:07.314 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.7:5673 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds. 2015-10-06 16:32:08.238 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.7:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2015-10-06 16:32:08.298 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.7:5673 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds. 2015-10-06 16:32:08.337 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.7:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2015-10-06 16:32:09.256 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.4:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2015-10-06 16:32:09.313 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.7:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2015-10-06 16:32:09.420 5914 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 192.168.0.5:5673 2015-10-06 16:32:10.270 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.4:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. 2015-10-06 16:32:10.330 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.4:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2015-10-06 16:32:11.357 5914 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 192.168.0.5:5673 2015-10-06 16:32:12.291 5914 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 192.168.0.5:5673 Fuel - 6.1 The non-graceful failover of RabbitMQ connections in Ceilometer causes issues in active environments when Ceilometer cannot connect to the next RabbitMQ system in line. This was found in a customer environment. When trying to reproduce: Kill RabbitMQ on the master node Tail the Ceilometer logs to watch connections to the next RabbitMQ cluster. It does't look like it actually does this correctly and it waits for the original cluster to come back up. The Cluster ends up coming back up with the original node and it starts working, however if the node did not come back up it looks like it would continue to have issues connecting to RabbitMQ servers on the other nodes. Logs: 2015-10-06 16:28:45.706 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server 192.168.0.5:5673 closed the connection. Check login credentials: Socket closed 2015-10-06 16:28:46.412 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server 192.168.0.5:5673 closed the connection. Check login credentials: Socket closed 2015-10-06 16:28:46.459 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server 192.168.0.5:5673 closed the connection. Check login credentials: Socket closed 2015-10-06 16:28:48.089 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.5:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2015-10-06 16:28:48.335 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.5:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2015-10-06 16:28:48.337 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.5:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2015-10-06 16:28:49.538 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.5:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2015-10-06 16:28:49.704 5914 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 192.168.0.7:5673 2015-10-06 16:28:49.727 5914 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 192.168.0.7:5673 2015-10-06 16:28:50.555 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.5:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. 2015-10-06 16:28:52.593 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.5:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2015-10-06 16:28:53.698 5914 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 192.168.0.4:5673 2015-10-06 16:29:55.229 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.4:5673 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds. 2015-10-06 16:29:56.415 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.4:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2015-10-06 16:29:57.555 5914 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 192.168.0.7:5673 2015-10-06 16:32:07.102 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.7:5673 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds. 2015-10-06 16:32:07.314 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.7:5673 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds. 2015-10-06 16:32:08.238 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.7:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2015-10-06 16:32:08.298 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.7:5673 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds. 2015-10-06 16:32:08.337 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.7:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2015-10-06 16:32:09.256 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.4:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2015-10-06 16:32:09.313 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.7:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2015-10-06 16:32:09.420 5914 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 192.168.0.5:5673 2015-10-06 16:32:10.270 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.4:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. 2015-10-06 16:32:10.330 5914 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.4:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. 2015-10-06 16:32:11.357 5914 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 192.168.0.5:5673 2015-10-06 16:32:12.291 5914 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 192.168.0.5:5673
2015-11-10 13:04:47 Timur Nurlygayanov fuel: milestone 6.1-updates
2015-11-10 13:04:50 Timur Nurlygayanov fuel: status Incomplete Confirmed
2015-12-30 13:40:28 Fuel Devops McRobotson fuel/7.0.x: milestone 7.0-mu-3
2015-12-30 13:40:28 Fuel Devops McRobotson fuel/7.0.x: assignee MOS Ceilometer (mos-ceilometer)
2015-12-30 13:40:31 Fuel Devops McRobotson fuel/8.0.x: milestone 8.0
2015-12-30 13:40:31 Fuel Devops McRobotson fuel/8.0.x: assignee MOS Ceilometer (mos-ceilometer)
2015-12-30 13:40:33 Fuel Devops McRobotson fuel: status Confirmed New
2015-12-30 13:40:33 Fuel Devops McRobotson fuel: milestone 6.1-updates 9.0
2015-12-30 13:40:36 Fuel Devops McRobotson fuel/6.1.x: importance Undecided High
2015-12-30 13:40:36 Fuel Devops McRobotson fuel/6.1.x: status New Confirmed
2015-12-30 13:40:36 Fuel Devops McRobotson fuel/6.1.x: milestone 6.1-updates
2015-12-30 13:40:36 Fuel Devops McRobotson fuel/6.1.x: assignee MOS Ceilometer (mos-ceilometer)
2015-12-30 13:40:39 Fuel Devops McRobotson fuel/future: milestone next
2015-12-30 13:40:39 Fuel Devops McRobotson fuel/future: assignee MOS Ceilometer (mos-ceilometer)
2015-12-30 13:47:19 Ivan Berezovskiy marked as duplicate 1510916