OpenStack Object Storage (swift)

multiple runs of swift-dispersion-populate create duplicate partitions

Bug #1233045 reported by Grahame Bowland on 2013-09-30

This bug affects 4 people

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Object Storage (swift)	Fix Released	Undecided	Unassigned

Bug Description

it seems that multiple runs of swift-dispersion-populate create a situation where swift-dispersion-report will never run successfully; the output will note overlapping partitions and will never report 100% of object copies found.

root@swiftn4:~# swift-dispersion-report
Queried 982 containers for dispersion reporting, 1s, 0 retries
There were 4 overlapping partitions
99.59% of container copies found (2934 of 2946)
Sample represents 2.98% of the container partition space
Queried 981 objects for dispersion reporting, 1s, 0 retries
There were 9 overlapping partitions
99.08% of object copies found (2916 of 2943)
Sample represents 2.97% of the object partition space

should swift-dispersion-populate clean up containers and objects from the previous run?

Revision history for this message

Lorcan Browne (lorcan-browne) wrote on 2014-08-28:

Once swift-dispersion-populate is run once, do we even want it to be able to run a second time? Running it twice, should in theory double the partition coverage required in the dispersion.conf file (is this desired behaviour) but as pointed out in this bug, there can be overlaps with partitions.

Could a potential solution here be to add an "--override" option to the swift-dispersion-populate tool which would remove all the containers/objects from the previous populate run and keep the coverage percentage consistent with the dispersion.conf file.

As an adition or alternative to this, a check could be added to the code where if "swift-dispersion-populate" has been run before, it will not populate again.

Revision history for this message

Lorcan Browne (lorcan-browne) wrote on 2014-09-02:

Have put a change up for review that may solve this issue by adding a "--no-overlap" option to the swift-dispersion-populate tool.
https://review.openstack.org/118411

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-09-22: Related fix merged to swift (master)

Reviewed: https://review.openstack.org/118411
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=4faf1702706b289521329243e5802cf86e54c7f7
Submitter: Jenkins
Branch: master

commit 4faf1702706b289521329243e5802cf86e54c7f7
Author: Lorcan <email address hidden>
Date: Tue Sep 2 18:12:18 2014 +0100

Add "--no-overlap" option to swift-dispersion populate

    This change allows the user to use a "--no-overlap" parameter when
    running the tool multiple times. It will increase the coverage by
    whatever is specified in the dispersion_coverage field of the conf
    file in a manner where existing container/objects are left in place
    and no partition is populated more than once.

Related-Bug: #1233045

Change-Id: I139fed2f4c967ba18d073b7ecd1e946ed4da1271

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-09-26: Related fix proposed to swift (feature/ec)

Related fix proposed to branch: feature/ec
Review: https://review.openstack.org/124503

Revision history for this message

paul luse (paul-e-luse) wrote on 2014-09-26:

Ignore the previous auto-added comment. Accidentally had topic for a totally unrelated review associated with this....

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-09-27: Related fix merged to swift (feature/ec)

Download full text (9.1 KiB)

Reviewed: https://review.openstack.org/124503
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=bcaa00f25f3e8bd4123645256c535d0efe8a6733
Submitter: Jenkins
Branch: feature/ec

commit 15fbf9fe7cf33ed4b56569078400a2ba070b6bce
Author: paul luse <email address hidden>
Date: Thu Sep 11 06:55:45 2014 -0700

Add container_count to policy_stat table

    Start tracking the container count per policy including reporting
    it in account HEAD and supporting installations where the DB
    existed before the updated schema.

    Migration is triggered by the account audtior; if the database is
    un-migrated it will continue to report policy_stats without the per
    policy container_count keys.

Closes-Bug: #1367514
Change-Id: I07331cea177e19b3df303609a4ac510765a19162

commit d10462e8704e7f5efe03c4024b541a3780348615
Author: Darrell Bishop <email address hidden>
Date: Tue Sep 23 09:11:44 2014 -0700

Fix profile tests to clean up its tempdirs.

Change-Id: I363651046cf414e14f15affd834043aabd5427c0

commit b68258a322cb004151b84584d00b3c76ee01bc29
Author: Mahati Chamarthy <email address hidden>
Date: Fri Sep 5 01:42:17 2014 +0530

Added instructions to create a label or UUID to the XFS volume and mount using it.

Change-Id: Idcaf16a278d6c34770af9b1f17d69bdd94fb86b7

commit 4d23a0fcf5faa6339a1a58fcbdab8687a6c88feb
Author: Samuel Merritt <email address hidden>
Date: Thu Aug 28 09:39:38 2014 -0800

Reject overly-taxing ranged-GET requests

    RFC 7233 says that servers MAY reject egregious range-GET requests
    such as requests with hundreds of ranges, requests with non-ascending
    ranges, and so on.

    Such requests are fairly hard for Swift to process. Consider a Range
    header that asks for the first byte of every 10th MiB in a 4 GiB
    object, but in some random order. That'll cause a lot of seeks on the
    object server, but the corresponding response body is quite small in
    comparison to the workload.

    This commit makes Swift reject, with a 416 response, any ranged GET
    request with more than fifty ranges, more than three overlapping
    ranges, or more than eight non-increasing ranges.

    This is a necessary prerequisite for supporting multi-range GETs on
    large objects. Otherwise, a malicious user could construct a Range
    header with hundreds of byte ranges where each individual byterange
    requires the proxy to contact a different object server. If seeking
    all over a disk is bad, connecting all over the cluster is way worse.

DocImpact

Change-Id: I4dcedcaae6c3deada06a0223479e611094d57234

commit 5efdab6055bc99638e4e1388bef685b19682025d
Author: OpenStack Proposal Bot <email address hidden>
Date: Mon Sep 22 06:07:56 2014 +0000

Imported Translations from Transifex

Change-Id: Ibd8882766a87c6d77e786f7635b1290391e43f10

commit 4faf1702706b289521329243e5802cf86e54c7f7
Author: Lorcan <email address hidden>
Date: Tue Sep 2 18:12:18 2014 +0100

Add "--no-overlap" option to swift-dispersion populate

This change allows the user to use a "--no-overlap" parameter w...

Reviewed:  https://review.openstack.org/124503
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=bcaa00f25f3e8bd4123645256c535d0efe8a6733
Submitter: Jenkins
Branch:    feature/ec

commit 15fbf9fe7cf33ed4b56569078400a2ba070b6bce
Author: paul luse <paul.e.luse@intel.com>
Date:   Thu Sep 11 06:55:45 2014 -0700

Add container_count to policy_stat table
    
    Start tracking the container count per policy including reporting
    it in account HEAD and supporting installations where the DB
    existed before the updated schema.
    
    Migration is triggered by the account audtior; if the database is
    un-migrated it will continue to report policy_stats without the per
    policy container_count keys.
    
    Closes-Bug: #1367514
    Change-Id: I07331cea177e19b3df303609a4ac510765a19162

commit d10462e8704e7f5efe03c4024b541a3780348615
Author: Darrell Bishop <darrell@swiftstack.com>
Date:   Tue Sep 23 09:11:44 2014 -0700

Fix profile tests to clean up its tempdirs.
    
    Change-Id: I363651046cf414e14f15affd834043aabd5427c0

commit b68258a322cb004151b84584d00b3c76ee01bc29
Author: Mahati Chamarthy <mahati.chamarthy@gmail.com>
Date:   Fri Sep 5 01:42:17 2014 +0530

Added instructions to create a label or UUID to the XFS volume and mount using it.
    
    Change-Id: Idcaf16a278d6c34770af9b1f17d69bdd94fb86b7

commit 4d23a0fcf5faa6339a1a58fcbdab8687a6c88feb
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Thu Aug 28 09:39:38 2014 -0800

Reject overly-taxing ranged-GET requests
    
    RFC 7233 says that servers MAY reject egregious range-GET requests
    such as requests with hundreds of ranges, requests with non-ascending
    ranges, and so on.
    
    Such requests are fairly hard for Swift to process. Consider a Range
    header that asks for the first byte of every 10th MiB in a 4 GiB
    object, but in some random order. That'll cause a lot of seeks on the
    object server, but the corresponding response body is quite small in
    comparison to the workload.
    
    This commit makes Swift reject, with a 416 response, any ranged GET
    request with more than fifty ranges, more than three overlapping
    ranges, or more than eight non-increasing ranges.
    
    This is a necessary prerequisite for supporting multi-range GETs on
    large objects. Otherwise, a malicious user could construct a Range
    header with hundreds of byte ranges where each individual byterange
    requires the proxy to contact a different object server. If seeking
    all over a disk is bad, connecting all over the cluster is way worse.
    
    DocImpact
    
    Change-Id: I4dcedcaae6c3deada06a0223479e611094d57234

commit 5efdab6055bc99638e4e1388bef685b19682025d
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Mon Sep 22 06:07:56 2014 +0000

Imported Translations from Transifex
    
    Change-Id: Ibd8882766a87c6d77e786f7635b1290391e43f10

commit 4faf1702706b289521329243e5802cf86e54c7f7
Author: Lorcan <lorcan.browne@hp.com>
Date:   Tue Sep 2 18:12:18 2014 +0100

Add "--no-overlap" option to swift-dispersion populate
    
    This change allows the user to use a "--no-overlap" parameter when
    running the tool multiple times. It will increase the coverage by
    whatever is specified in the dispersion_coverage field of the conf
    file in a manner where existing container/objects are left in place
    and no partition is populated more than once.
    
    Related-Bug: #1233045
    
    Change-Id: I139fed2f4c967ba18d073b7ecd1e946ed4da1271

commit c1f6569c00951e3b57d58a0cd32dc28a638f5a81
Author: Rafael Rivero <rafael@cloudscaling.com>
Date:   Thu Sep 18 21:16:35 2014 -0700

Fixes several typos (Swift)
    
    Corrects spelling errors found in comments.
    
    Change-Id: I228a888e3f256569ea32ef1613092dbd63e13c62

commit 1678083d0e48f24e6a74eada874f01ac2755cf78
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Thu Sep 18 17:04:30 2014 -0700

Test for unicode names in AccountBroker.merge_items()
    
    ContainerBroker.merge_items() had a bug in it where non-ASCII Unicode
    names would possibly result in duplicate entries in container
    databases. AccountBroker.merge_items() doesn't do the same
    bulk-operations tricks that ContainerBroker does, so it doesn't
    currently have the bug. This commit just adds a test to ensure the bug
    doesn't creep in should someone decide to make AccountBroker look more
    like ContainerBroker someday.
    
    Change-Id: Id2ac129828dbdf55b609d839ce4d9d42437ee0a3

commit 7d0e5ebe690bf3cf41ccd970281d532a23284e58
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Tue Jun 10 14:15:27 2014 -0700

Zero-copy object-server GET responses with splice()
    
    This commit lets the object server use splice() and tee() to move data
    from disk to the network without ever copying it into user space.
    
    Requires Linux. Sorry, FreeBSD folks. You still have the old
    mechanism, as does anyone who doesn't want to use splice. This
    requires a relatively recent kernel (2.6.38+) to work, which includes
    the two most recent Ubuntu LTS releases (Precise and Trusty) as well
    as RHEL 7. However, it excludes Lucid and RHEL 6. On those systems,
    setting "splice = on" will result in warnings in the logs but no
    actual use of splice.
    
    Note that this only applies to GET responses without Range headers. It
    can easily be extended to single-range GET requests, but this commit
    leaves that for future work. Same goes for PUT requests, or at least
    non-chunked ones.
    
    On some real hardware I had laying around (not a VM), this produced a
    37% reduction in CPU usage for GETs made directly to the object
    server. Measurements were done by looking at /proc/<pid>/stat,
    specifically the utime and stime fields (user and kernel CPU jiffies,
    respectively).
    
    Note: There is a Python module called "splicetee" available on PyPi,
    but it's licensed under the GPL, so it cannot easily be added to
    OpenStack's requirements. That's why this patch uses ctypes instead.
    
    Also fixed a long-standing annoyance in FakeLogger:
    
        >>> fake_logger.warn('stuff')
        >>> fake_logger.get_lines_for_level('warn')
        []
        >>>
    
    This, of course, is because the correct log level is 'warning'. Now
    you get a KeyError if you call get_lines_for_level with a bogus log
    level.
    
    Change-Id: Ic6d6b833a5b04ca2019be94b1b90d941929d21c8

commit eaab4d3fd6b7a330c1b904dc69d7ea4f0fbe8781
Author: Michael Barton <mike@weirdlooking.com>
Date:   Thu Sep 18 19:20:51 2014 +0000

container.merge_items bug
    
    When replicated container entries get round-tripped through json, they wind up
    with unicode objects for names.  This causes equality checks to fail against
    container entries, and you can wind up with duplicate records.  My bad.
    
    Change-Id: I3aee2ad8dbd3a617efe37e887cfb902a3e4a1646

commit 09bdc87cbc1e7bc1918f9b5094bec266b6761d75
Author: Christian Schwede <christian.schwede@enovance.com>
Date:   Thu Sep 11 08:01:51 2014 +0000

Return correct number of changed partitions
    
    When a ring is rebalanced the number of changed partitions is counted.
    Before this patch partitions might be rebalanced, but actually no data
    is moved - for example, when a partition is assigned to the same device
    as before. This results in a wrong number of reassigned partitions that
    is shown to the user.
    
    This patch remembers the partition allocation before the rebalance, and
    compares it to the new allocation after a rebalance. Only partitions
    that are stored on a different device than before are counted.
    
    Partial-Bug: 1367826
    Also-By: Florent Flament <florent.flament-ext@cloudwatt.com>
    Change-Id: Iacfd514df3af351791f9191cef78cff1b3e2645f

commit a7604da065f025931f42bbcf9307bbabda3a37a9
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Wed Sep 10 17:29:05 2014 -0700

Move multipart MIME parser into utils
    
    Over on the EC branch, we need to be able to parse multipart MIME
    documents in the object server. The formpost middleware has a
    perfectly good MIME parser, but it seems sort of awful to import
    things from formpost in swift/obj/server.py, so I pulled it out into
    common.utils.
    
    Change-Id: Ieb4c05d02d8e4ef51a3a11d26c503786b1897f60

commit 49fa5b8fb467bb5900dda36da47d46d4c5882bb0
Author: Alistair Coles <alistair.coles@hp.com>
Date:   Wed Sep 10 16:09:13 2014 +0100

Update documentation for using keystone auth
    
    Cleanup and add clarification to the documentation
    for using Keystone auth.
    
    Update to refer to auth_token middleware being
    distributed as part of the keystomemiddelware project
    rather than keystone.
    
    Include capabilities (/info) in the list of reasons
    why delay_auth_decision might need to be set in
    auth_token middleware config.
    
    Add description of the project_id:user_id format
    for container ACLs and emphasize that ids rather than
    names should be used since this patch has now merged:
    https://review.openstack.org/#/c/86430
    
    DocImpact
    blueprint keystone-v3-support
    Change-Id: Idda4a3dcf8240474f1d2d163016ca2d40ec2d589

John Dickinson (notmyname) on 2015-07-14

Changed in swift:
status:	New → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.