reconciler should skip objects in policies involved in a partition power increase

Bug #1934314 reported by Alistair Coles
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
Fix Released
High
Unassigned

Bug Description

The object relinker step does not cope well with object files with same name but different inodes existing in the current and next part power. These file pairs can cause relinking to log errors and confuse whether relinking has succeeded.

These circumstances can be created by a process writing the same object concurrently and specifying the same x-timestamp. The reconciler does precisely that. The scenario is explained in more detail in https://bugs.launchpad.net/swift/+bug/1934142. Bug 1934142 reports a case where, by good fortune, only tombstone files were being written by the reconciler. In other circumstances .data files could be written with different inodes in the old and new partition power location. Determining that these .data have identical content is more onerous than with tombstones and currently not supported by the relinker: this would be a bad state to end up in.

Possible fixes:

* The reconciler should NOT migrate objects that are either in or destined for a policy that is currently being relinked, much like how the replicator skips any such policy. Note that if either the new or old policy are being relinked then the reconciler should not attempt to move an object.

and/or

* The reconciler should be enhanced to make less (no) concurrent requests with same x-timestamp to the same object.

A workaround is to stop all reconciler processes on all nodes before deploying a ring with part_power and next_part_power and starting the relink step of a part power increase. The reconcilers can be started again after the relink step is complete and a new ring has been deployed with part_power == next_part_power.

clayg (clay-gerrard)
Changed in swift:
status: New → Confirmed
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to swift (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/swift/+/799561

Changed in swift:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to swift (master)

Reviewed: https://review.opendev.org/c/openstack/swift/+/799561
Committed: https://opendev.org/openstack/swift/commit/e491693e36e5b6b6b8eab6933fb247ae756ea28b
Submitter: "Zuul (22348)"
Branch: master

commit e491693e36e5b6b6b8eab6933fb247ae756ea28b
Author: Matthew Oliver <email address hidden>
Date: Tue Jul 6 16:32:08 2021 +1000

    reconciler: PPI aware reconciler

    This patch makes the reconciler PPI aware. It does this by adding a
    helper method `can_reconcile_policy` that is used to check that the
    policies used for the source and destination aren't in the middle of a
    PPI (their ring doesn't have next_part_power set).

    In order to accomplish this the reconciler has had to include the
    POLICIES singleton and grown swift_dir and ring_check_interval config options.

    Closes-Bug: #1934314
    Change-Id: I78a94dd1be90913a7a75d90850ec5ef4a85be4db

Changed in swift:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/swift 2.28.0

This issue was fixed in the openstack/swift 2.28.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.