Bad balance and excess part movement when reducing fractional replica count

[Cheng Li]

in swift/common/ring/builder.py
the rebalance function calls self._set_parts_wanted(replica_plan) at 443 line.
dev['parts_wanted'] = parts_by_tier[tier] - dev['parts']

so dev_losing_part['parts'] -= 1

should be followed with

dev_losing_part['parts_wanted'] += 1

but in _adjust_replica2part2dev_size(...),
dev_losing_part['parts_wanted'] += 1 does exist. This gets wrong parts_wanted number.

You will meet this case if you reduce replicas and then rebalance

current case:
swift-ring-builder test.builder create 8 3.5 1
swift-ring-builder test.builder add r1z1-127.0.0.1:6000/d1 1.0
swift-ring-builder test.builder add r1z1-127.0.0.1:6000/d2 1.0
swift-ring-builder test.builder add r1z1-127.0.0.1:6000/d3 1.0
swift-ring-builder test.builder add r1z1-127.0.0.1:6000/d4 1.0
swift-ring-builder test.builder rebalance -s 1
Reassigned 512 (200.00%) partitions. Balance is now 0.00. Dispersion is now 0.00
swift-ring-builder test.builder set_replicas 3
swift-ring-builder test.builder pretend_min_part_hours_passed
swift-ring-builder test.builder rebalance -s 1
Reassigned *128 (50.00%)* partitions. Balance is now 3.65. Dispersion is now 0.00

after this issue fixed:
swift-ring-builder test.builder create 8 3.5 1
swift-ring-builder test.builder add r1z1-127.0.0.1:6000/d1 1.0
swift-ring-builder test.builder add r1z1-127.0.0.1:6000/d2 1.0
swift-ring-builder test.builder add r1z1-127.0.0.1:6000/d3 1.0
swift-ring-builder test.builder add r1z1-127.0.0.1:6000/d4 1.0
swift-ring-builder test.builder rebalance -s 1
Reassigned 512 (200.00%) partitions. Balance is now 0.00. Dispersion is now 0.00
swift-ring-builder test.builder set_replicas 3
swift-ring-builder test.builder pretend_min_part_hours_passed
swift-ring-builder test.builder rebalance -s 1
Reassigned *9 (3.52%)* partitions. Balance is now 0.00. Dispersion is now 0.00

[Cheng Li (shcli)]

gerrit link: https://review.openstack.org/#/c/344898/

[clayg (clay-gerrard)]

Cheng - reducing fraction replicas (3.5 => 3.0) is a *very* rare process compared to *increasing* the replica count (like when gradually adding a region) - I'm not even sure I understand the use-case?

Do you have any additional testing for increasing fractional replica counts - or does this issue only come up when reducing replica count?

As you know, there's always always more than one solution that would deliver the desired result, but some solutions might better align with existing assumptions and invariants in the ring builder code - so we need to make sure we first clearly state the problem.

Is the problem mainly an efficiency issue? How quickly (multiple rebalance; move too many parts?) the rebalance makes the reduced replica count change? Or is there a correctness issue as well (some invalid internal state that prevents the ring for ever fixing the balance issues)?

[Cheng Li (shcli)]

@Clay, this case happens with dev_losing_part['parts'] -= 1 in _adjust_replica2part2dev_size(...).
So this case always happens with reducing replicas, not only reduce fraction replicas.

but not happens with increasing replicas.