EC reconstructor revert tombstone SSYNCs with too many primaries

Bug #1668857 reported by clayg
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
Medium
Mahati Chamarthy

Bug Description

During a busy EC rebalance it's far too common for SSYNC requests to get rejected for concurrency, to allow a revert tombstone only parts try to talk to ALL primary nodes [1]. Because such a large sync_to list will often encounter at least ONE failure, revert of tombstone only parts creates a lot of IO doing suffix calculation [2] and doing SSYNC requests - only to ultimately not be able to remove the handoff part.

As the comment in the code alludes - it's overly pessimistic; rather (ec_npartiy + 1) would be far superior and entirely sufficient.

1. https://github.com/openstack/swift/blob/ee2f94226849c577eaf25b4b554dc4ac46576e7c/swift/obj/reconstructor.py#L853
2. both locally and remotely, although the remote suffix hashing is remediated by lp bug #1665141

Revision history for this message
clayg (clay-gerrard) wrote :
Revision history for this message
clayg (clay-gerrard) wrote :
Changed in swift:
assignee: nobody → Mahati Chamarthy (mahati-chamarthy)
Changed in swift:
status: New → In Progress
clayg (clay-gerrard)
Changed in swift:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to swift (master)

Reviewed: https://review.openstack.org/439572
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=188c07e12ab49ce71a76f2426a943a8f4265d374
Submitter: Jenkins
Branch: master

commit 188c07e12ab49ce71a76f2426a943a8f4265d374
Author: Mahati Chamarthy <email address hidden>
Date: Wed Mar 1 23:18:09 2017 +0530

    Limit number of revert tombstone SSYNC requests

    Revert tombstone only parts try to talk to all primary nodes - this
    fixes it to randomize selection within part_nodes. Corresponding probe
    test is modified to reflect this change.

    The primary improvement of this patch is the reconstuctor at a handoff
    node is being able to delete local tombstones when it succeeds to sync
    to less than all primary nodes. (Before this patch, it requires all
    nodes are responsible for the REVERT requests)

    The number of primary nodes to communicate with the reconstructor can be
    in dicsussion more but, right now with this patch, it's (replicas - k + 1)
    that is able to prevent stale read.

    *BONUS*

    - Fix mis-testsetting (was setting less replicas than ec_k + ec_m)
      for reconstructor ring in the unit test

    Co-Authored-By: Kota Tsuyuzaki <email address hidden>
    Co-Authored-By: Clay Gerrard <email address hidden>

    Change-Id: I05ce8fe75f1c4a7971cc8995b003df818b69b3c1
    Closes-Bug: #1668857

Changed in swift:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/swift 2.15.0

This issue was fixed in the openstack/swift 2.15.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers