Sometimes RabbitMQ processes are not starting again after they were killed on all rabbit nodes.
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Confirmed
|
High
|
MOS Oslo |
Bug Description
Hello,
Please take a look at the following issue:
Sometimes RabbitMQ processes are not starting again after they were killed on all rabbit nodes.
Configuration:
MOS 9.0 official ISO: MirantisOpenSta
[root@nailgun ~]# fuel nodes
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
-
1 | ready | slave-01_controller | 1 | 10.109.2.3 | 64:9f:ca:b0:49:c0 | controller | | 1 | 1
2 | ready | slave-02_controller | 1 | 10.109.2.4 | 64:6d:92:eb:98:6f | controller | | 1 | 1
4 | ready | slave-03_controller | 1 | 10.109.2.5 | 64:bd:82:d3:a5:64 | controller | | 1 | 1
5 | ready | slave-04_compute | 1 | 10.109.2.6 | 64:ba:bf:de:02:4f | compute | | 1 | 1
6 | ready | slave-05_compute | 1 | 10.109.2.7 | 64:25:a3:5a:26:08 | compute | | 1 | 1
3 | ready | slave-06_cinder | 1 | 10.109.2.8 | 64:8a:c2:f8:de:e3 | cinder | | 1 | 1
Actions:
!! Perform all actions below from all RabbitMQ nodes (controllers) step-by-step.
In order to kill rabbit processes on all rabbit nodes almost at the same time.
1) OK - Check RabbitMQ processes on node:
# ps aux | grep -v grep | grep 'beam.smp'
2) OK - Kill all RabbitMQ processes on node:
# date ; for pid in $(pgrep 'beam.smp'); do kill -9 $pid; echo 'killed: ' $pid; done
3) OK - Wait 5-15 minutes till RabbitMQ proccesse will start and cluster will recover.
4) OK - Check RabbitMQ cluster status:
# for cmd in 'rabbitmqctl cluster_status' 'rabbitmqctl status' 'pcs resource show p_rabbitmq-server' 'pcs resource show master_
Expected that command above should return '0' exit code launched from _ALL_ Rabbit nodes.
If not - wait 5 more minutes.
Waiting more then 30 minutes probably will indicate that the RabbitMQ cluster will not recover on node.
5) NOK - Repeat steps 1-4 on all RabbitMQ nodes (controllers) 10 times.
Usually after 2-6 rounds of processes kills and automatic restarts on all nodes, RabbitMQ process is not starting on one node even after 30 minutes of waiting.
root@node-4:~# ps aux | grep -v grep | grep 'beam.smp'
rabbitmq 18238 7.5 1.9 553272 79504 ? Sl 09:45 0:08 /usr/lib/
root@node-4:~# date ; for pid in $(pgrep 'beam.smp'); do kill -9 $pid; echo 'killed: ' $pid; done
Thu Aug 18 09:47:08 UTC 2016
killed: 18238
killed: 31842
Wait 30 minutes.
On all other two nodes rabbit is ok. Problem only in node-4.
root@node-4:~# for cmd in 'rabbitmqctl cluster_status' 'rabbitmqctl status' 'pcs resource show p_rabbitmq-server' 'pcs resource show master_
rabbitmqctl cluster_status => 69
rabbitmqctl status => 69
pcs resource show p_rabbitmq-server => 0
pcs resource show master_
root@node-4:~# ps aux | grep -v grep | grep 'beam.smp'
{empty}
root@node-4:~# rabbitmqctl cluster_status
Cluster status of node 'rabbit@
Error: unable to connect to node 'rabbit@
DIAGNOSTICS
===========
attempted to contact: ['rabbit@
* connected to epmd (port 4369) on messaging-node-4
* epmd reports: node 'rabbit' not running at all
* suggestion: start the node
current node details:
- node name: 'rabbitmq-
- home dir: /var/lib/rabbitmq
- cookie hash: soeIWU2jk2YNseT
Changed in fuel: | |
status: | New → Confirmed |
Hi MOS Oslo team, could you please check if we kill the services correctly and what is the expected behaviour for the scenario.
Thank you!