An easy way to understand this behavior is to create a ring with min_part_hours of 300 then add some devices and rebalance, then add another device, and pretend_min_part_hours and rebalance.
... nothing moves
With min_part_hours > 255 nothing will ever get gathered if it requires a last_part_moves check, no matter how far you are from the epoch apparently.
This is probably bad, it's a confusing semantic. Unfortunately I've leveraged this in the past to minimize unneeded partition movement during a gradual capacity increase that gets interrupted by a failed device:
#!/bin/bash
rm forever.builder || true
swift-ring-builder forever.builder create 8 3 1
swift-ring-builder forever.builder add r1z1-127.0.0.1:6001/d0 10
swift-ring-builder forever.builder add r1z1-127.0.0.1:6001/d1 10
swift-ring-builder forever.builder add r1z1-127.0.0.1:6001/d2 10
swift-ring-builder forever.builder add r1z1-127.0.0.1:6001/d3 10
swift-ring-builder forever.builder rebalance
swift-ring-builder forever.builder
swift-ring-builder forever.builder add r1z1-127.0.0.1:6001/d4 1
swift-ring-builder forever.builder add r1z1-127.0.0.1:6001/d5 1
swift-ring-builder forever.builder add r1z1-127.0.0.1:6001/d6 1
echo "*** no parts can get reassigned!"
swift-ring-builder forever.builder rebalance
swift-ring-builder forever.builder
echo "*** ... time passes"
swift-ring-builder forever.builder pretend_min_part_hours_passed
echo "*** gradually parts move"
swift-ring-builder forever.builder rebalance
swift-ring-builder forever.builder
swift-ring-builder forever.builder set_weight d4 10
swift-ring-builder forever.builder set_weight d5 10
swift-ring-builder forever.builder set_weight d6 10
echo "*** now it's un-balanced"
swift-ring-builder forever.builder rebalance
swift-ring-builder forever.builder
echo "*** ... time passes"
swift-ring-builder forever.builder pretend_min_part_hours_passed
echo "*** and a device *fails!*"
swift-ring-builder forever.builder remove d2
cp forever.builder fast.builder
echo "*** here can we do a little trick to prevent unneeded movement"
swift-ring-builder forever.builder set_min_part_hours 300
echo "*** ... and failed deivce parts reassign no matter what!"
swift-ring-builder forever.builder rebalance | grep -i Reassigned
echo "*** compared to not doing the trick"
swift-ring-builder forever.builder set_min_part_hours 254
swift-ring-builder fast.builder rebalance | grep -i Reassigned
In this scenario it's not unreasonable I think to want to ask rebalance to reassign *only* the parts it *has* to for the failed device. With the plan of waiting for that replication cycle to finish before continuing the normal gradual capacity adjustment and rebalance cycle.
But I think there's a subtle mistake in the bug report. It says "if min_part_hours is larger that 0xff, rebalance will never work" - but that's *only* true for some types of gather parts - the common ones people deal with when "playing" with rings. But in the real world there's some type of gather parts that must avoid min_part_hours checks, and when they do your goals as an operator can come into conflict with other priorities in the ring, so the only escape hatch we have gets used.
I'd really like to get some additional opinions to help align my thinking based on input from other operators and developers familiar with the ring and thinking about how its going to continue to evolve.
An easy way to understand this behavior is to create a ring with min_part_hours of 300 then add some devices and rebalance, then add another device, and pretend_ min_part_ hours and rebalance.
... nothing moves
With min_part_hours > 255 nothing will ever get gathered if it requires a last_part_moves check, no matter how far you are from the epoch apparently.
This is probably bad, it's a confusing semantic. Unfortunately I've leveraged this in the past to minimize unneeded partition movement during a gradual capacity increase that gets interrupted by a failed device:
#!/bin/bash ring-builder forever.builder create 8 3 1 ring-builder forever.builder add r1z1-127. 0.0.1:6001/ d0 10 ring-builder forever.builder add r1z1-127. 0.0.1:6001/ d1 10 ring-builder forever.builder add r1z1-127. 0.0.1:6001/ d2 10 ring-builder forever.builder add r1z1-127. 0.0.1:6001/ d3 10 ring-builder forever.builder rebalance ring-builder forever.builder ring-builder forever.builder add r1z1-127. 0.0.1:6001/ d4 1 ring-builder forever.builder add r1z1-127. 0.0.1:6001/ d5 1 ring-builder forever.builder add r1z1-127. 0.0.1:6001/ d6 1 ring-builder forever.builder rebalance ring-builder forever.builder ring-builder forever.builder pretend_ min_part_ hours_passed ring-builder forever.builder rebalance ring-builder forever.builder ring-builder forever.builder set_weight d4 10 ring-builder forever.builder set_weight d5 10 ring-builder forever.builder set_weight d6 10 ring-builder forever.builder rebalance ring-builder forever.builder ring-builder forever.builder pretend_ min_part_ hours_passed ring-builder forever.builder remove d2 ring-builder forever.builder set_min_part_hours 300 ring-builder forever.builder rebalance | grep -i Reassigned ring-builder forever.builder set_min_part_hours 254 ring-builder fast.builder rebalance | grep -i Reassigned
rm forever.builder || true
swift-
swift-
swift-
swift-
swift-
swift-
swift-
swift-
swift-
swift-
echo "*** no parts can get reassigned!"
swift-
swift-
echo "*** ... time passes"
swift-
echo "*** gradually parts move"
swift-
swift-
swift-
swift-
swift-
echo "*** now it's un-balanced"
swift-
swift-
echo "*** ... time passes"
swift-
echo "*** and a device *fails!*"
swift-
cp forever.builder fast.builder
echo "*** here can we do a little trick to prevent unneeded movement"
swift-
echo "*** ... and failed deivce parts reassign no matter what!"
swift-
echo "*** compared to not doing the trick"
swift-
swift-
In this scenario it's not unreasonable I think to want to ask rebalance to reassign *only* the parts it *has* to for the failed device. With the plan of waiting for that replication cycle to finish before continuing the normal gradual capacity adjustment and rebalance cycle.
It's possible this use-case is just an example of https:/ /xkcd.com/ 1172/ ;)
But I think there's a subtle mistake in the bug report. It says "if min_part_hours is larger that 0xff, rebalance will never work" - but that's *only* true for some types of gather parts - the common ones people deal with when "playing" with rings. But in the real world there's some type of gather parts that must avoid min_part_hours checks, and when they do your goals as an operator can come into conflict with other priorities in the ring, so the only escape hatch we have gets used.
I'd really like to get some additional opinions to help align my thinking based on input from other operators and developers familiar with the ring and thinking about how its going to continue to evolve.