Allow increasing of part_power

[Samuel Merritt]

When you first set up a Swift cluster, you have to choose a part_power for your rings. If it's too big, your Swift processes will use too much memory, but if it's too small, your data doesn't get distributed well across your disks. It would be really nice to be able to start with a smallish part_power and increase it as your cluster grows.

It's possible to do this without causing all your cluster's data to get moved. The attached patch has the code; the commit comment on Github (https://github.com/smerritt/swift/tree/increment-part-power) has a pretty detailed explanation of how it works.

[gholt (gholt)]

One thing to consider with such a re-ringing is that the servers store data in paths like /<device-path>/<device>/objects/<partition>/<hash>/<data-files>

If the partition that was 0 is now 0 and 1, you now have half the objects in that partition in the wrong on disk location. To continue, if partition 1 is now 2 and 3, all the on disk locations for that partition are wrong, etc. You can alleviate this some by, with this two->four partition change, by making 0=0,2 and 1=1,3, but you still have half your data in the wrong location.

Some discussion about this included altering the servers to look in the primary location and fall back to the old partition if needed, but then you have problems with tombstones in one location and not in the other, etc. It's worse with containers and accounts since they're SQLite databases. You really need to lock a changed partition and move half the data around, obviously severely impacting availability.

[Samuel Merritt (torgomatic)]

Ah, crud. That certainly throws a large monkey wrench into things.

Thanks for the clear explanation. I've got a lot of reconsidering to do.

I'm marking this bug as invalid to keep the bug tracker clean.