The current ring building code in tripleo-image-elements/elements/swift/os-refresh-config/configure.d/73-swift should only be used to create the initial swift rings i.e. in its current form it cannot be used to include additional devices into the ring after initial system standup.
To add/replace devices to existing swift rings you need to use the 'swift-ring-builder add' command to add devices. Then run 'swift-ring-builder rebalance' after all the devices have been added. Note the rebalance command is simply rebalancing partitions in the swift rings. The movement of swift data happens after the rings are changed. Also, time is needed to allow swift data to move to new locations. Typically subsequent ring rebalances are required to ensure the ring is properly balanced.
Note care is needed to ensure the ring is not changed too much in one step. Large ring changes result in increased replication traffic and longer replication times.
When removing devices from the ring the safest approach is to drain the data from these devices rather than simply removing the devices from the ring. Devices can be drained by reducing the weight associated with the device, do this with the swift-ring-builder command. It is best to drop the weight on a device in stages, wait for the data to drain then drop the weight further. The final step after the weight is set to 0 and all the data is drained from the device is to remove the device from the ring.
Thanks for the excellent write up Gerry.
I think we have a severe lack of documentation on evolutionary components of a TripleO system. We are good at defining how we'll deploy new things, but not how we'll change things as the system evolves.
I think this is a valid bug, but it is also in some ways a feature request too since we've never really talked about evolving swift rings in place and none of the elements or larger documents about TripleO mention this. We need to accurately document how to add and remove disks to and from boxes already in the swift ring. So a partial fix for this would be to simply allow turning off ring maintenance in Heat and documenting the steps necessary to use a workflow-oriented tool to do these operations on standing resources.
A proposed plan:
* Flesh out documentation in tripleo-
* Add a disable flag to allow runtime disabling of ring managmeent.