Killing one RabbitMQ node causes complete Swift outage
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Fix Released
|
High
|
Kyrylo Galanov | ||
8.0.x |
Won't Fix
|
High
|
Kyrylo Galanov | ||
Mitaka |
Fix Released
|
High
|
Kyrylo Galanov |
Bug Description
Version: 8.0
1. Install environment with Ceilometer and Swift consisting of 1 controller node, 1 compute node and 3 rabbitmq nodes (using https:/
2. Log into controller and examine /etc/swift/
url = rabbit:
3. Find out, which RabbitMQ node has the first IP in the list and power it off.
4. Try running
swift --debug list --os-auth-url http://<VIP>:35357/v2.0
Expected result
The command succeeds
Actual result
The command hangs until HA proxy returns 504
The problem is that credentials need to be passed to each server separately, like that:
url = rabbit:
Otherwise, Swift goes to 2nd and 3rd RabbitMQ node with default guest/guest credentials.
9.0 seems to be affected as well, as it has the same parameter in proxy-server.conf. Most probably issue could be reproduced without detach-rabbitmq plugin if you install 3 controllers and shut down 2 of 3 proxy-server processes.
tags: | added: area-library |
tags: | added: swift team-bugfix |
tags: | added: low-hanging-fruit |
tags: |
added: area-mos removed: area-library |
tags: | added: on-verification |
Changed in fuel: | |
status: | Fix Committed → Fix Released |
This is not valid. Dmitry had invalid network settings on the added node.