multiple runs of swift-dispersion-populate create duplicate partitions

Bug #1233045 reported by Grahame Bowland
18
This bug affects 4 people
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
Fix Released
Undecided
Unassigned

Bug Description

it seems that multiple runs of swift-dispersion-populate create a situation where swift-dispersion-report will never run successfully; the output will note overlapping partitions and will never report 100% of object copies found.

root@swiftn4:~# swift-dispersion-report
Queried 982 containers for dispersion reporting, 1s, 0 retries
There were 4 overlapping partitions
99.59% of container copies found (2934 of 2946)
Sample represents 2.98% of the container partition space
Queried 981 objects for dispersion reporting, 1s, 0 retries
There were 9 overlapping partitions
99.08% of object copies found (2916 of 2943)
Sample represents 2.97% of the object partition space

should swift-dispersion-populate clean up containers and objects from the previous run?

Revision history for this message
Lorcan Browne (lorcan-browne) wrote :

Once swift-dispersion-populate is run once, do we even want it to be able to run a second time? Running it twice, should in theory double the partition coverage required in the dispersion.conf file (is this desired behaviour) but as pointed out in this bug, there can be overlaps with partitions.

Could a potential solution here be to add an "--override" option to the swift-dispersion-populate tool which would remove all the containers/objects from the previous populate run and keep the coverage percentage consistent with the dispersion.conf file.

As an adition or alternative to this, a check could be added to the code where if "swift-dispersion-populate" has been run before, it will not populate again.

Revision history for this message
Lorcan Browne (lorcan-browne) wrote :

Have put a change up for review that may solve this issue by adding a "--no-overlap" option to the swift-dispersion-populate tool.
https://review.openstack.org/118411

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to swift (master)

Reviewed: https://review.openstack.org/118411
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=4faf1702706b289521329243e5802cf86e54c7f7
Submitter: Jenkins
Branch: master

commit 4faf1702706b289521329243e5802cf86e54c7f7
Author: Lorcan <email address hidden>
Date: Tue Sep 2 18:12:18 2014 +0100

    Add "--no-overlap" option to swift-dispersion populate

    This change allows the user to use a "--no-overlap" parameter when
    running the tool multiple times. It will increase the coverage by
    whatever is specified in the dispersion_coverage field of the conf
    file in a manner where existing container/objects are left in place
    and no partition is populated more than once.

    Related-Bug: #1233045

    Change-Id: I139fed2f4c967ba18d073b7ecd1e946ed4da1271

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to swift (feature/ec)

Related fix proposed to branch: feature/ec
Review: https://review.openstack.org/124503

Revision history for this message
paul luse (paul-e-luse) wrote :

Ignore the previous auto-added comment. Accidentally had topic for a totally unrelated review associated with this....

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to swift (feature/ec)
Download full text (9.1 KiB)

Reviewed: https://review.openstack.org/124503
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=bcaa00f25f3e8bd4123645256c535d0efe8a6733
Submitter: Jenkins
Branch: feature/ec

commit 15fbf9fe7cf33ed4b56569078400a2ba070b6bce
Author: paul luse <email address hidden>
Date: Thu Sep 11 06:55:45 2014 -0700

    Add container_count to policy_stat table

    Start tracking the container count per policy including reporting
    it in account HEAD and supporting installations where the DB
    existed before the updated schema.

    Migration is triggered by the account audtior; if the database is
    un-migrated it will continue to report policy_stats without the per
    policy container_count keys.

    Closes-Bug: #1367514
    Change-Id: I07331cea177e19b3df303609a4ac510765a19162

commit d10462e8704e7f5efe03c4024b541a3780348615
Author: Darrell Bishop <email address hidden>
Date: Tue Sep 23 09:11:44 2014 -0700

    Fix profile tests to clean up its tempdirs.

    Change-Id: I363651046cf414e14f15affd834043aabd5427c0

commit b68258a322cb004151b84584d00b3c76ee01bc29
Author: Mahati Chamarthy <email address hidden>
Date: Fri Sep 5 01:42:17 2014 +0530

    Added instructions to create a label or UUID to the XFS volume and mount using it.

    Change-Id: Idcaf16a278d6c34770af9b1f17d69bdd94fb86b7

commit 4d23a0fcf5faa6339a1a58fcbdab8687a6c88feb
Author: Samuel Merritt <email address hidden>
Date: Thu Aug 28 09:39:38 2014 -0800

    Reject overly-taxing ranged-GET requests

    RFC 7233 says that servers MAY reject egregious range-GET requests
    such as requests with hundreds of ranges, requests with non-ascending
    ranges, and so on.

    Such requests are fairly hard for Swift to process. Consider a Range
    header that asks for the first byte of every 10th MiB in a 4 GiB
    object, but in some random order. That'll cause a lot of seeks on the
    object server, but the corresponding response body is quite small in
    comparison to the workload.

    This commit makes Swift reject, with a 416 response, any ranged GET
    request with more than fifty ranges, more than three overlapping
    ranges, or more than eight non-increasing ranges.

    This is a necessary prerequisite for supporting multi-range GETs on
    large objects. Otherwise, a malicious user could construct a Range
    header with hundreds of byte ranges where each individual byterange
    requires the proxy to contact a different object server. If seeking
    all over a disk is bad, connecting all over the cluster is way worse.

    DocImpact

    Change-Id: I4dcedcaae6c3deada06a0223479e611094d57234

commit 5efdab6055bc99638e4e1388bef685b19682025d
Author: OpenStack Proposal Bot <email address hidden>
Date: Mon Sep 22 06:07:56 2014 +0000

    Imported Translations from Transifex

    Change-Id: Ibd8882766a87c6d77e786f7635b1290391e43f10

commit 4faf1702706b289521329243e5802cf86e54c7f7
Author: Lorcan <email address hidden>
Date: Tue Sep 2 18:12:18 2014 +0100

    Add "--no-overlap" option to swift-dispersion populate

    This change allows the user to use a "--no-overlap" parameter w...

Read more...

Changed in swift:
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.