RABBITMQ cluster crash on CentOS+GRE+525

Bug #1467298 reported by Viktoria Efimova
18
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Invalid
High
Fuel Library (Deprecated)
6.1.x
Invalid
High
Fuel Library (Deprecated)
7.0.x
Invalid
High
Fuel Library (Deprecated)

Bug Description

Rabbit logs reports about cluster crash under rally tests:
logs on node-8 (controller_0)
=INFO REPORT==== 21-Jun-2015::16:14:41 ===
rabbit on node 'rabbit@node-15' down
..
=INFO REPORT==== 21-Jun-2015::16:14:42 ===
node 'rabbit@node-15' down: connection_closed
..
=INFO REPORT==== 21-Jun-2015::16:15:52 ===
node 'rabbit@node-15' down: connection_closed
...
=INFO REPORT==== 21-Jun-2015::16:15:52 ===
node 'rabbit@node-15' down: connection_closed

from node-8
 atop -r /var/log/atop/atop_current -b "16:14"
http://paste.openstack.org/show/311046/

from node-13
 atop -r /var/log/atop/atop_current -b "16:14"
http://paste.openstack.org/show/311047/

from node-15
 atop -r /var/log/atop/atop_current -b "16:14"
http://paste.openstack.org/show/311048/

According to the time crash happened during nova/boot_and_delete_server_with_secgroups.yaml
(use for reference for rally test running sequence http://mos-scale.vm.mirantis.net:8080/job/13_env_run_rally/33/console)

Topology:
UBUNTU+VLAN+525(RC3)
Controllers: 3
Computes: 17

Network configuration
eth0 - public
eth2 - Admin, Storage, Mgmt
eth3 - private

Snapsot at http://mos-scale-share.mirantis.com/fuel-snapshot-2015-06-21_17-39-12.tar.xz

Tags: rabbitmq scale
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

Was the cluster recovered after the crash?

Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

According to the logs, the failure of rabbit node-15 was recovered at 2015-06-21T16:16:01.440535+00:00, less than in 2 minutes.
Other nodes , node-8, node-13 remained running available. Marking this bug as invalid as it has nothing to be addressed and contains no issue

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.