swift-ring-builder rebalance - endless loop
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Object Storage (swift) |
Fix Released
|
Undecided
|
Tim Burke | ||
Ubuntu Cloud Archive |
Fix Released
|
High
|
Unassigned | ||
Mitaka |
Fix Released
|
High
|
Unassigned | ||
Newton |
Fix Released
|
High
|
Unassigned | ||
Ocata |
Fix Released
|
High
|
Unassigned | ||
swift (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Xenial |
Fix Released
|
High
|
Unassigned | ||
Yakkety |
Fix Released
|
High
|
Unassigned | ||
Zesty |
Fix Released
|
High
|
Unassigned |
Bug Description
There is an issue with swift-ring-builder and rebalancing under certain conditions beginning in Mitaka.
Steps to reproduce:
Create a ring with part power of 12. 3 servers with 11 devices each, BUT 2 servers in zone 1 and one server in zone 2. After adding all devices as described above, swift-ring-builder <ring>.builder rebalance does not return and seems to run into an endless loop.
The weirdness starts with the 11th device, 10 devices still work. It does not matter whether you add each 10 devices from the 3 servers and rebalance, and than add the 11th device from each server afterwards or do it all at once.
This behaviour does not exists in Liberty and starts with Mitaka.
For connivence I am adding the statements to reproduce:
swift-ring-builder account.builder create 12 3 1
swift-ring-builder account.builder add --region 1 --zone 1 --ip 10.46.15.60 --port 6002 --replication-ip 10.46.15.60 --replication-port 6002 --device swift-01 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 1 --ip 10.46.15.60 --port 6002 --replication-ip 10.46.15.60 --replication-port 6002 --device swift-02 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 1 --ip 10.46.15.60 --port 6002 --replication-ip 10.46.15.60 --replication-port 6002 --device swift-03 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 1 --ip 10.46.15.60 --port 6002 --replication-ip 10.46.15.60 --replication-port 6002 --device swift-04 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 1 --ip 10.46.15.60 --port 6002 --replication-ip 10.46.15.60 --replication-port 6002 --device swift-05 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 1 --ip 10.46.15.60 --port 6002 --replication-ip 10.46.15.60 --replication-port 6002 --device swift-06 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 1 --ip 10.46.15.60 --port 6002 --replication-ip 10.46.15.60 --replication-port 6002 --device swift-07 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 1 --ip 10.46.15.60 --port 6002 --replication-ip 10.46.15.60 --replication-port 6002 --device swift-08 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 1 --ip 10.46.15.60 --port 6002 --replication-ip 10.46.15.60 --replication-port 6002 --device swift-09 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 1 --ip 10.46.15.60 --port 6002 --replication-ip 10.46.15.60 --replication-port 6002 --device swift-10 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 1 --ip 10.46.15.60 --port 6002 --replication-ip 10.46.15.60 --replication-port 6002 --device swift-11 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 1 --ip 10.46.15.61 --port 6002 --replication-ip 10.46.15.61 --replication-port 6002 --device swift-01 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 1 --ip 10.46.15.61 --port 6002 --replication-ip 10.46.15.61 --replication-port 6002 --device swift-02 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 1 --ip 10.46.15.61 --port 6002 --replication-ip 10.46.15.61 --replication-port 6002 --device swift-03 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 1 --ip 10.46.15.61 --port 6002 --replication-ip 10.46.15.61 --replication-port 6002 --device swift-04 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 1 --ip 10.46.15.61 --port 6002 --replication-ip 10.46.15.61 --replication-port 6002 --device swift-05 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 1 --ip 10.46.15.61 --port 6002 --replication-ip 10.46.15.61 --replication-port 6002 --device swift-06 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 1 --ip 10.46.15.61 --port 6002 --replication-ip 10.46.15.61 --replication-port 6002 --device swift-07 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 1 --ip 10.46.15.61 --port 6002 --replication-ip 10.46.15.61 --replication-port 6002 --device swift-08 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 1 --ip 10.46.15.61 --port 6002 --replication-ip 10.46.15.61 --replication-port 6002 --device swift-09 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 1 --ip 10.46.15.61 --port 6002 --replication-ip 10.46.15.61 --replication-port 6002 --device swift-10 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 1 --ip 10.46.15.61 --port 6002 --replication-ip 10.46.15.61 --replication-port 6002 --device swift-11 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 2 --ip 10.46.15.62 --port 6002 --replication-ip 10.46.15.62 --replication-port 6002 --device swift-01 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 2 --ip 10.46.15.62 --port 6002 --replication-ip 10.46.15.62 --replication-port 6002 --device swift-02 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 2 --ip 10.46.15.62 --port 6002 --replication-ip 10.46.15.62 --replication-port 6002 --device swift-03 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 2 --ip 10.46.15.62 --port 6002 --replication-ip 10.46.15.62 --replication-port 6002 --device swift-04 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 2 --ip 10.46.15.62 --port 6002 --replication-ip 10.46.15.62 --replication-port 6002 --device swift-05 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 2 --ip 10.46.15.62 --port 6002 --replication-ip 10.46.15.62 --replication-port 6002 --device swift-06 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 2 --ip 10.46.15.62 --port 6002 --replication-ip 10.46.15.62 --replication-port 6002 --device swift-07 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 2 --ip 10.46.15.62 --port 6002 --replication-ip 10.46.15.62 --replication-port 6002 --device swift-08 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 2 --ip 10.46.15.62 --port 6002 --replication-ip 10.46.15.62 --replication-port 6002 --device swift-09 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 2 --ip 10.46.15.62 --port 6002 --replication-ip 10.46.15.62 --replication-port 6002 --device swift-10 --weight 100.00
swift-ring-builder account.builder add --region 1 --zone 2 --ip 10.46.15.62 --port 6002 --replication-ip 10.46.15.62 --replication-port 6002 --device swift-11 --weight 100.00
swift-ring-builder account.builder rebalance
Changed in swift (Ubuntu Xenial): | |
status: | New → Triaged |
Changed in swift (Ubuntu Yakkety): | |
status: | New → Triaged |
Changed in swift (Ubuntu Zesty): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in swift (Ubuntu Yakkety): | |
importance: | Undecided → High |
Changed in swift (Ubuntu Xenial): | |
importance: | Undecided → High |
Changed in cloud-archive: | |
status: | Triaged → Fix Committed |
tags: |
added: verification-done removed: verification-needed |
tags: |
added: verification-mitaka-done removed: verification-mitaka-needed |
FWIW, running with --debug, I see something like https:/ /gist.github. com/tipabu/ 9687531671aa9a1 93ddd51151c5112 67
I wonder if maybe the "requires -3.05311331772e-16 overload" is indicating some numerical analysis troubles?