Pacemaker migration of management vip causes RabbitMQ, MySQL lockups
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Fix Released
|
Critical
|
Dmitry Borodaenko |
Bug Description
mirantis: "yes"
release: "4.1"
build_number: "208"
build_id: "2014-02-
fuellib_sha: "0a2e5bdc01c1e3
nailgun_sha: "ea08cef3e06a72
astute_sha: "10cccc87f2ee35
ostf_sha: "8e6681b6d06c7c
fuelmain_sha: "7939e28a5b3ab6
1. Create new environment (Ubuntu, HA, Neutron/GRE, Ceph for everything)
2. Add 3 controller + ceph-osd nodes, 1 compute node
3. Deployment is successful, able to create cinder volumes in Ceph.
4. Force shutdown primary controller.
5. Try to create another cinder volume.
Result: new volume is stuck in "creating" state and never becomes "available".
Changed in fuel: | |
milestone: | 5.0 → 4.1 |
importance: | High → Critical |
Changed in fuel: | |
assignee: | Fuel Library Team (fuel-library) → Ryan Moe (rmoe) |
status: | New → Confirmed |
Changed in fuel: | |
assignee: | Dmitry Borodaenko (dborodaenko) → Sergey Vasilenko (xenolog) |
Changed in fuel: | |
assignee: | Sergey Vasilenko (xenolog) → Dmitry Borodaenko (dborodaenko) |
Changed in fuel: | |
assignee: | Dmitry Borodaenko (dborodaenko) → Sergey Vasilenko (xenolog) |
Changed in fuel: | |
assignee: | Sergey Vasilenko (xenolog) → Matthew Mosesohn (raytrac3r) |
Changed in fuel: | |
assignee: | Matthew Mosesohn (raytrac3r) → Sergey Vasilenko (xenolog) |
Changed in fuel: | |
assignee: | Matthew Mosesohn (raytrac3r) → Sergey Vasilenko (xenolog) |
Changed in fuel: | |
assignee: | Sergey Vasilenko (xenolog) → Matthew Mosesohn (raytrac3r) |
Changed in fuel: | |
assignee: | Matthew Mosesohn (raytrac3r) → Dmitry Borodaenko (dborodaenko) |
Changed in fuel: | |
status: | In Progress → Fix Committed |
tags: | added: in progress |
tags: | removed: in progress |
tags: | added: in progress |
tags: | removed: in progress |
Changed in fuel: | |
milestone: | 4.1 → 5.0 |
tags: | added: backports-4.1.1 |
tags: | removed: ceph |
tags: | added: backports-4.1.1 |
summary: |
- Moving management vip breaks rabbitmq sessions + Pacemaker migration of management vip causes RabbitMQ, MySQL lockups |
You don't even need to shut down a node to reproduce this problem, all you have to do is move the management vip to a different node with the following command (replace node-1 with hostname of the controller node that doesn't currently have the vip):
crm_resource -r vip__management_old --move --node node-1
After that, most OpenStack services become unable to either put messages on RabbitMQ queues, take messages off the queues, or acknowledge the messages.