Comment 9 for bug 1460762

Revision history for this message
Alexander Nevenchannyy (anevenchannyy) wrote : Re: RabbitMQ cluster got restarted under load of Openstack services

According to the logs, of lrmd from ticket https://bugs.launchpad.net/fuel/+bug/1461586

2015-06-03T04:54:03.191168+00:00 info: INFO: p_rabbitmq-server: su_rabbit_cmd(): the invoked command exited 137: /usr/sbin/rabbitmqctl list_channels 2>&1 > /dev/null
2015-06-03T04:54:03.197232+00:00 err: ERROR: p_rabbitmq-server: get_monitor(): rabbitmqctl is not responding. The resource is failed.
2015-06-03T04:54:03.873629+00:00 info: INFO: p_rabbitmq-server: demote: action begin.
2015-06-03T04:54:03.879774+00:00 info: INFO: p_rabbitmq-server: get_monitor(): CHECK LEVEL IS: 0
2015-06-03T04:54:04.121298+00:00 info: INFO: p_rabbitmq-server: get_monitor(): get_status() returns 0.
2015-06-03T04:54:04.126604+00:00 info: INFO: p_rabbitmq-server: get_monitor(): also checking if we are master.
2015-06-03T04:54:04.378931+00:00 info: INFO: p_rabbitmq-server: get_monitor(): master attribute is 0
2015-06-03T04:54:04.613379+00:00 info: INFO: p_rabbitmq-server: get_monitor(): checking if rabbit app is running
2015-06-03T04:54:04.618044+00:00 info: INFO: p_rabbitmq-server: get_monitor(): rabbit app is running. checking if we are the part of healthy cluster
2015-06-03T04:54:04.636400+00:00 info: INFO: p_rabbitmq-server: get_monitor(): rabbit app is running. looking for master on node-49.domain.tld
2015-06-03T04:54:04.655893+00:00 info: INFO: p_rabbitmq-server: get_monitor(): fetched master attribute for node-49.domain.tld. attr value is 1
2015-06-03T04:54:04.661757+00:00 info: INFO: p_rabbitmq-server: get_monitor(): rabbit app is running. looking for master on node-1.domain.tld
2015-06-03T04:54:04.681031+00:00 info: INFO: p_rabbitmq-server: get_monitor(): fetched master attribute for node-1.domain.tld. attr value is 0
2015-06-03T04:54:04.687028+00:00 info: INFO: p_rabbitmq-server: get_monitor(): rabbit app is running. master is node-1.domain.tld
2015-06-03T04:54:04.927305+00:00 info: INFO: p_rabbitmq-server: get_monitor(): rabbit app is running and is member of healthy cluster
2015-06-03T04:54:04.932689+00:00 info: INFO: p_rabbitmq-server: get_monitor(): preparing to update master score for node
2015-06-03T04:54:04.974043+00:00 info: INFO: p_rabbitmq-server: get_monitor(): comparing our uptime (14477) with node-49.domain.tld (12530)
2015-06-03T04:54:05.001216+00:00 info: INFO: p_rabbitmq-server: get_monitor(): comparing our uptime (14477) with node-44.domain.tld (12549)
2015-06-03T04:54:05.016135+00:00 info: INFO: p_rabbitmq-server: get_monitor(): we are the oldest node

And CPU usage at this oment at node: http://paste.openstack.org/show/259735/
I'm think that root cause of this ticket not a high CPU load.