Support replicating objects in handoff partitions first

[Drew Freiberger]

In extreme cases of object rebalance or node failure, swift environments with millions of objects may have a hard time with objects landing in handoff or misplaced partitions and the operator of the cloud may need to prioritize migrating data in handoff partitions back to primary partitions over the 3 primary partitions replicating amongst each other.

To allow for this, the object-server.conf configuration can have handoffs_first set to True which will force the object-replicators to focus on handoff partitions before syncing primary partitions for any given object partition.

This will priorize draining the 4th, 5th, 6th, ... nth copy of objects from incorrect locations back to the 3 primary disks for that object's partition over copying that object from one primary copy to the other 2 primary disks.

This is an extreme setting as noted by the product documentation, but is needed for environments where multiple swift-ring rebalances have been performed before the prior replications have had a chance to complete.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/18.08
Review: https://review.opendev.org/726939

[Drew Freiberger (afreiberger)]

It appears that this feature is congruent to the standalone_replicator for > trusty and should potentially only be allowed on trusty clouds.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Drew Freiberger (<email address hidden>) on branch: stable/18.08
Review: https://review.opendev.org/726939
Reason: merging to master instead. abandoning stable backport

[Andrea Ieri (aieri)]

Subscribing field-medium since we need this urgently for a customer.
We have a merge proposal ready, we only need a review and a merge.

[Paul Goins (vultaire)]

For clarity, the updated review appears to be here: https://review.opendev.org/#/c/726942/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/726942
Committed: https://git.openstack.org/cgit/openstack/charm-swift-storage/commit/?id=9809a3b347ce68aacaef6a7b4284fc58fe28d790
Submitter: Zuul
Branch: master

commit 9809a3b347ce68aacaef6a7b4284fc58fe28d790
Author: Drew Freiberger <email address hidden>
Date: Mon May 11 15:00:03 2020 -0500

    Add support for object replication handoffs_first

    In extreme cases of object rebalance or node failure, swift environments
    with millions of objects may have a hard time with objects landing in
    handoff or misplaced partitions and the operator of the cloud may need
    to prioritize migrating data in handoff partitions back to primary
    partitions over the 3 primary partitions replicating amongst each other.

    To allow for this, the object-server.conf [object-replicator]
    configuration can have handoffs_first set to True which will force
    the object-replicators to focus on handoff partitions before syncing
    primary partitions for any given object partition.

    Change-Id: I8b44c287567a0e6d634def0b13baf0fe4ad4aa7b
    Closes-Bug: 1878087