Bug #1076202 “Expirer could be better optimized” : Bugs : OpenStack Object Storage (swift)

Chuck Thier (cthier) on 2012-12-04

Changed in swift:
status:	New → Triaged

Li Ma (nick-ma-z) on 2013-06-14

Changed in swift:
assignee:	nobody → Li Ma (nick-ma-b)
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2013-08-20: Fix proposed to swift (master)

#1

Fix proposed to branch: master
Review: https://review.openstack.org/42897

Li Ma (nick-ma-z) on 2013-08-29

Changed in swift:
assignee:	Li Ma (nick-ma-b) → nobody

Tom Fifield (fifieldt) on 2013-12-06

Changed in swift:
status:	In Progress → Confirmed

Revision history for this message

Cristian A Sanchez (cristian-a-sanchez) wrote on 2013-12-18:

#2

After analyzing the code, I arrived to the following conclusion described in code comments.
Is it correct?

http://paste.openstack.org/show/55330/

Revision history for this message

clayg (clay-gerrard) wrote on 2013-12-19:

#3

@Cristian - pretty sure that's deleting the container record for the x-delete-at timestamp in the .expiring_objects account.

It *is* needed to prevent future attempts to delete the same object - the weird part is that there is no ".data file" behind that "object".

Objects in the .expiring_objects account are just a record in the container db's, but greg is levering a bunch of code that already works to get the container updates for the delete all done correctly. But routing the request through the object servers is... wasteful? genius? Not sure the best way to describe it :P

Thiago da Silva (thiagodasilva) on 2014-03-03

Changed in swift:
assignee:	nobody → Thiago da Silva (thiagodasilva)

Revision history for this message

John Dickinson (notmyname) wrote on 2015-06-03:

#4

Thiago, what's going on here?

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-01-18: Fix merged to swift (master)

#5

Reviewed: https://review.openstack.org/532383
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=745581ff2f5b2c6f6340eb60918d8ba842297f89
Submitter: Zuul
Branch: master

commit 745581ff2f5b2c6f6340eb60918d8ba842297f89
Author: Samuel Merritt <email address hidden>
Date: Tue Jan 9 19:19:36 2018 -0800

Don't make async_pendings during object expiration

    After deleting an object, the object expirer deletes the corresponding
    row from the expirer queue by making DELETE requests directly to the
    container servers. The same thing happens after attempting to delete
    an object, but failing because the object has already been deleted. If
    the DELETE requests fail, then the expirer will encounter that row
    again on its next pass and retry the DELETE at that time. Therefore,
    it is not necessary for the object server to write an async_pending
    for that queue row's deletion.

    Currently, however, two of the object servers do write such
    async_pendings. Given Rc container replicas, that's 2 * Rc updates
    from async_pendings and another Rc from the object expirer
    directly. Given a typical Rc of 3, that's 9 container updates per
    expiring object.

    This commit makes the object server write no async_pendings for DELETE
    requests coming from the object expirer. This reduces the number of
    container server requests to Rc (typically 3), all issued directly
    from the object expirer.

Closes-Bug: 1076202
Change-Id: Icd63c80c73f864d2561e745c3154fbfda02bd0cc

Changed in swift:
status:	Confirmed → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-01-19: Fix proposed to swift (feature/deep)

#6

Fix proposed to branch: feature/deep
Review: https://review.openstack.org/535990

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-01-22: Fix merged to swift (feature/deep)

#7

Download full text (32.4 KiB)

Reviewed: https://review.openstack.org/535990
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=3122895118111b2b11f5ef9d0b3410b337624b1b
Submitter: Zuul
Branch: feature/deep

commit ddb13aa5eab03b6993887eb02260b4bc0b256922
Author: vxlinux <email address hidden>
Date: Sat Jan 20 17:23:35 2018 +0800

Remove redundant blank space in README.rst

Change-Id: If347476e3b9185921ff174d3f8170a1c4d0622e8

commit 12f874534925b52f9d1c91580794eb9e5e9a4589
Author: vxlinux <email address hidden>
Date: Fri Jan 19 16:54:26 2018 +0800

Add Docstrings to validate_replicas_by_tier

New common functions should have Docstrings

Change-Id: Icbb3cdf38509fd6d034cbb2271786559780a7b68

commit d2034cd7b6946829a7d95c4d2c71d4322f80e855
Author: Clay Gerrard <email address hidden>
Date: Tue Jan 16 17:03:38 2018 -0800

Keep object-updater stats logging consistent

    If we're going to encapsulate the stats tracking it seems reasonable if
    we ever add any more metrics we can reduce the number of places we need
    to update log messages.

Change-Id: I187cf6cfec1e0a9138b709fa298e1991aa809ec4

commit cd2c73fd955317a3f40758cef45ee48bef8fbc79
Author: Tim Burke <email address hidden>
Date: Tue Jan 16 01:07:35 2018 +0000

internal_client: Don't retry when we expect the same reponse

This boils down to 404, 412, or 416; or 409 when we provided an
X-Timestamp.

This means, among other things, that the expirer won't issue 3 DELETEs
every cycle for every stale work item.

Related-Change: Icd63c80c73f864d2561e745c3154fbfda02bd0cc
Change-Id: Ie5f2d3824e040bbc76d511a54d1316c4c2503732

commit 222df9185782f59ffdc96c3534afaa2fb1361235
Author: chengebj5238 <email address hidden>
Date: Thu Jan 18 17:03:11 2018 +0800

Modify redirection URL and broken URL

Change-Id: I9a04cb2fbe61e1fbd8185ab2fac9abbcea4d55cc

commit d1656e334959e09d13eea98c2696e58c77e4ab91
Author: Tim Burke <email address hidden>
Date: Fri Jan 12 13:17:45 2018 -0800

slo: Send ETag header in 206 responses

    Why weren't we doing that before?? The etag should be the same as for
    GET/HEAD, and by sending it, we can assure resuming clients that they're
    downlading the same object even if they didn't include an If-Match
    header.

Change-Id: I4ccbd1ae3a909ecb4606ef18211d1b868f5cad86
Related-Change: Ic11662eb5c7176fbf422a6fc87a569928d6f85a1

commit 88eea33ccd1875af811b59d15df55e2bffa27f77
Author: Clay Gerrard <email address hidden>
Date: Thu Jan 11 13:36:09 2018 -0800

Recenter builder test expectation around random variance

... in order to make the test pass with more seeds and fail less
frequently in the gate.

Change-Id: I059e80af87fd33a3b6c0731fbad62e035215eca5

commit f64c00b00aa8df31a937448917421891904abdc8
Author: Samuel Merritt <email address hidden>
Date: Fri Jan 12 07:17:18 2018 -0800

Improve object-updater's stats logging

    The object updater has five different stats, but its logging only told
    you two of them (successes and failures), and it only told you after
    finishing all the async_pendings for a device. If y...

Reviewed:  https://review.openstack.org/535990
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=3122895118111b2b11f5ef9d0b3410b337624b1b
Submitter: Zuul
Branch:    feature/deep

commit ddb13aa5eab03b6993887eb02260b4bc0b256922
Author: vxlinux <yan.wei7@zte.com.cn>
Date:   Sat Jan 20 17:23:35 2018 +0800

Remove redundant blank space in README.rst
    
    Change-Id: If347476e3b9185921ff174d3f8170a1c4d0622e8

commit 12f874534925b52f9d1c91580794eb9e5e9a4589
Author: vxlinux <yan.wei7@zte.com.cn>
Date:   Fri Jan 19 16:54:26 2018 +0800

Add Docstrings to validate_replicas_by_tier
    
    New common functions should have Docstrings
    
    Change-Id: Icbb3cdf38509fd6d034cbb2271786559780a7b68

commit d2034cd7b6946829a7d95c4d2c71d4322f80e855
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Tue Jan 16 17:03:38 2018 -0800

Keep object-updater stats logging consistent
    
    If we're going to encapsulate the stats tracking it seems reasonable if
    we ever add any more metrics we can reduce the number of places we need
    to update log messages.
    
    Change-Id: I187cf6cfec1e0a9138b709fa298e1991aa809ec4

commit cd2c73fd955317a3f40758cef45ee48bef8fbc79
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue Jan 16 01:07:35 2018 +0000

internal_client: Don't retry when we expect the same reponse
    
    This boils down to 404, 412, or 416; or 409 when we provided an
    X-Timestamp.
    
    This means, among other things, that the expirer won't issue 3 DELETEs
    every cycle for every stale work item.
    
    Related-Change: Icd63c80c73f864d2561e745c3154fbfda02bd0cc
    Change-Id: Ie5f2d3824e040bbc76d511a54d1316c4c2503732

commit 222df9185782f59ffdc96c3534afaa2fb1361235
Author: chengebj5238 <chengebj@inspur.com>
Date:   Thu Jan 18 17:03:11 2018 +0800

Modify redirection URL and broken URL
    
    Change-Id: I9a04cb2fbe61e1fbd8185ab2fac9abbcea4d55cc

commit d1656e334959e09d13eea98c2696e58c77e4ab91
Author: Tim Burke <tim.burke@gmail.com>
Date:   Fri Jan 12 13:17:45 2018 -0800

slo: Send ETag header in 206 responses
    
    Why weren't we doing that before?? The etag should be the same as for
    GET/HEAD, and by sending it, we can assure resuming clients that they're
    downlading the same object even if they didn't include an If-Match
    header.
    
    Change-Id: I4ccbd1ae3a909ecb4606ef18211d1b868f5cad86
    Related-Change: Ic11662eb5c7176fbf422a6fc87a569928d6f85a1

commit 88eea33ccd1875af811b59d15df55e2bffa27f77
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Thu Jan 11 13:36:09 2018 -0800

Recenter builder test expectation around random variance
    
    ... in order to make the test pass with more seeds and fail less
    frequently in the gate.
    
    Change-Id: I059e80af87fd33a3b6c0731fbad62e035215eca5

commit f64c00b00aa8df31a937448917421891904abdc8
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Fri Jan 12 07:17:18 2018 -0800

Improve object-updater's stats logging
    
    The object updater has five different stats, but its logging only told
    you two of them (successes and failures), and it only told you after
    finishing all the async_pendings for a device. If you have a cluster
    that's been sick and has millions upon millions of async_pendings
    laying around, then your object-updaters are frustratingly
    silent. I've seen one cluster with around 8 million async_pendings per
    disk where the object-updaters only emitted stats every 12 hours.
    
    Yes, if you have StatsD logging set up properly, you can go look at
    your graphs and get real-time feedback on what it's doing. If you
    don't have that, all you get is a frustrating silence.
    
    Now, the object updater tells you all of its stats (successes,
    failures, quarantines due to bad pickles, unlinks, and errors), and it
    tells you incremental progress every five minutes. The logging at the
    end of a pass remains and has been expanded to also include all stats.
    
    Also included is a small change to what counts as an error: unmounted
    drives no longer do. The goal is that only abnormal things count as
    errors, like permission problems, malformed filenames, and so
    on. These are things that should never happen, but if they do, may
    require operator intervention. Drives fail, so logging an error upon
    encountering an unmounted drive is not useful.
    
    Change-Id: Idbddd507f0b633d14dffb7a9834fce93a10359ab

commit d924fa759967b7cdca0d91f21112725f6099a254
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Tue Jan 16 22:19:09 2018 -0800

Remove old post-as-copy leftovers from tests.
    
    Since commit 1e79f828, we don't need to test with post_as_copy=True
    any more since we haven't got post_as_copy at all.
    
    Change-Id: I9c96ce0b812d877bbe11bdb50eb160d6ffa5933d

commit dfa0c4e604fb931d232395599bd0e7b0f11441ee
Author: Alistair Coles <alistairncoles@gmail.com>
Date:   Wed Jan 17 12:04:45 2018 +0000

Preserve expiring object behaviour with old proxy-server
    
    The related change [1] causes expiring object records to no longer be
    created if the X-Delete-At-Container header is not sent to the object
    server, but old proxies prior to [2] (i.e. releases prior to 1.9.0)
    did not send this header.
    
    The goal of [1] can be alternatively achieved by making expiring
    object record creation be conditional on the X-Delete-At-Host header.
    
    [1] Related-Change: I20fc2f42f590fda995814a2fa7ba86019f9fddc1
    [2] Related-Change: Id0873a3f2198ce285fe0b0c777738eff38bc2438
    
    Change-Id: Ia0081693f01631d3f2a59612308683e939ced76a

commit 745581ff2f5b2c6f6340eb60918d8ba842297f89
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Tue Jan 9 19:19:36 2018 -0800

Don't make async_pendings during object expiration
    
    After deleting an object, the object expirer deletes the corresponding
    row from the expirer queue by making DELETE requests directly to the
    container servers. The same thing happens after attempting to delete
    an object, but failing because the object has already been deleted. If
    the DELETE requests fail, then the expirer will encounter that row
    again on its next pass and retry the DELETE at that time. Therefore,
    it is not necessary for the object server to write an async_pending
    for that queue row's deletion.
    
    Currently, however, two of the object servers do write such
    async_pendings. Given Rc container replicas, that's 2 * Rc updates
    from async_pendings and another Rc from the object expirer
    directly. Given a typical Rc of 3, that's 9 container updates per
    expiring object.
    
    This commit makes the object server write no async_pendings for DELETE
    requests coming from the object expirer. This reduces the number of
    container server requests to Rc (typically 3), all issued directly
    from the object expirer.
    
    Closes-Bug: 1076202
    Change-Id: Icd63c80c73f864d2561e745c3154fbfda02bd0cc

commit d707fc7b6d0ceb4556dddfc258c5de8c4baff05c
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Tue Jan 16 16:30:13 2018 -0800

DRY out tests until the stone bleeds
    
    Can we go deeper!?
    
    Change-Id: Ibd3b06542aa1bfcbcb71cc98e6bb21a6a67c12f4

commit ba8f1b1c3786df4e79fc3f9e4747d7cfb9072b6f
Author: Alistair Coles <alistairncoles@gmail.com>
Date:   Wed Jan 17 15:25:33 2018 +0000

Fix intermittent unit test failure
    
    test_check_delete_headers_removes_delete_after was
    failing intermittently due to rounding of float time
    values.
    
    Change-Id: Ia126ad6988f387bbd2d1f5ddff0a56d457a1fc9b
    Closes-Bug: #1743804

commit e747f94313f315fdf8d8fc01fb0c5aac60c33897
Author: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Date:   Wed Dec 27 14:37:29 2017 +0900

Fix InternalClient to drain response body if the request fails
    
    If we don't drain the body, the proxy logging in the internal client
    pipeline will log 499 client disconnect instead of actual error response
    code.
    
    For error responses, we try to do the most helpful thing using swob's
    closing and caching response body attribute.  For non-error responses
    which are returned to the client, we endeavour to keep the app_iter
    intact and unconsumed, trusting expecting the caller to do the right
    thing is the only reasonable interface.  We must cleanly close any WSGI
    app_iter which we do not return to the client regardless of status code
    and allow the logging of the 499 if needed.
    
    Closes-Bug: #1675650
    Change-Id: I455b5c38074ad0e72aa5e0b05771e193208905eb

commit d8f9045518035cbd1a40d0a94227952a384143ec
Author: Christopher Bartz <bartz@dkrz.de>
Date:   Fri Dec 1 11:13:10 2017 +0100

Send correct number of X-Delete-At-* headers
    
    Send just as many requests with X-Delete-At-* as we do X-Container-* to
    the object server.  Furthermore, stop the object server on making an
    update to the expirer queue when it wasn't told to do so and remove the
    log warning which would have been produced.
    
    Reason:
    
    It can be the case that the number of object replicas (OR) is larger
    than the number of container replicas (CR) for a given storage policy
    (most likely in case of EC).  Before this commit, only CR object servers
    received the x-delete-at-* headers, which means that OR - CR object
    servers did not receive the headers.  The servers missing the header
    would produce a log warning and create the x-delete-at-container header
    and async update on their own, which could lead to a bug, if the
    expiring_objects_container_divisor option was misconfigured.
    
    Change-Id: I20fc2f42f590fda995814a2fa7ba86019f9fddc1
    Closes-Bug: #1733588

commit cf1a1e89bbca50e285e99d31209c6eac6c697083
Author: Tim Burke <tim.burke@gmail.com>
Date:   Wed Aug 23 07:25:09 2017 +0000

expirer: unexpected responses don't warrant tracebacks
    
    If you want more information, you need to go check out the *other* node.
    
    Maybe this should be further refined to only log at debug for specific
    statuses like 404 and 412?
    
    Partial-Bug: 1688558
    Related-Bug: 1455221
    Change-Id: Ieefd8841154faba40dcf2a03abc5f056bdccd54f

commit 56b84c9295f3860a1fa0033774a7498105a1597f
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Tue Jan 16 12:02:13 2018 -0800

Minor cleanup in monitoring doc.
    
    Change-Id: Ia21f8743bfd745f2579db8658624f888461c2cc2

commit bc6fb8995123d08e672bdcf55bd35165509d60d9
Author: Monty Taylor <mordred@inaugust.com>
Date:   Tue Jan 16 11:32:12 2018 -0600

Add a note about the cost of COPY for setting metadata
    
    The pointer to using COPY to the same object as a mechanism to set only
    a subset of the metadata, it does not mention that doing so results in
    a full copy of the object in question on the backend.
    
    Add a note so it's clear that there is a tradeoff involved.
    
    Change-Id: I0c20a4909a6c3ff672f753d26cb9fb2f5f33d1f4

commit 0d324c16deacbd025314bd2211c824aa971a65b8
Author: guotao <guotao.bj@inspur.com>
Date:   Tue Jan 16 14:28:41 2018 +0800

Update http with https
    
    Use https instead of http for some links in readme.rst
    
    Change-Id: Idd382f58108e96129c69c6dc149c694fd7833fb3

commit 6e394bba0a783cf6bf06c6f60d4ccda150a87e67
Author: Alistair Coles <alistairncoles@gmail.com>
Date:   Mon Jan 15 15:16:08 2018 +0000

Add request_tries option to object-expirer.conf-sample
    
    ...and update the object-expirer man page.
    
    Change-Id: Idca1b8e3b7d5b40481af0d60477510e2557b88c0

commit 68906dac43ac4e0a2a7e9ee62d9893138369b27a
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Tue Jan 9 11:16:15 2018 -0800

Further extract a builder method to a util function
    
    ... to make testing more targeted and obvious
    
    Related-Change-Id: I89439286b211f2c5ef19deffa77c202f48f07cf8
    Change-Id: I93b99128a4fb35395e8e9caf11649e216f824fdf

commit a1ae142d5bcaf83ddde568ba4b957e23cc2b5e1c
Author: vxlinux <yan.wei7@zte.com.cn>
Date:   Thu Jan 4 16:18:37 2018 +0800

Merge repeat code for rebalance
    
    There are three similar code segments in rebalance process as follows：
    
        tiers = ['cluster', 'regions', 'zones', 'servers', 'devices']
        for i, tier_name in enumerate(tiers):
            replicas_at_tier = sum(weighted_replicas_by_tier[t] for t in
                                   weighted_replicas_by_tier if len(t) == i)
            if abs(self.replicas - replicas_at_tier) > 1e-10:
                raise exceptions.RingValidationError(
                    '%s != %s at tier %s' % (
                        replicas_at_tier, self.replicas, tier_name))
    
    I think we can encapsulate this code segment to a private function and
    replace those code segments with a function call
    
    Change-Id: I89439286b211f2c5ef19deffa77c202f48f07cf8

commit 5cbd5cf303ca085af2a61dbb5b98d1b98099bd7e
Author: cheng <li.chenga@h3c.com>
Date:   Mon Jan 15 12:05:39 2018 +0800

Return HTTPServerError instead of HTTPNotFound
    
    Swift allows autocreate account. It should be treat as server error
    instead of 404 when it fails to create account
    
    Change-Id: I726271bc06e3c1b07a4af504c3fd7ddb789bd512
    Closes-bug: 1718810

commit a9964a7fc3df5e16d066914498e83ac065282e2e
Author: Thiago da Silva <thiago@redhat.com>
Date:   Thu Jan 11 07:15:48 2018 -0500

fix barbican integration
    
    Added auth_url to the context we pass to castellan
    library. In a change [1] intended to deprecate the use
    of auth_endpoint passed as the oslo config, it actually
    completely removed the use of it[2], so this change became
    necessary or the integration is broken.
    
    [1] - https://review.openstack.org/#/c/483457
    [2] - https://review.openstack.org/#/c/483457/6/castellan/key_manager/barbican_key_manager.py@143
    
    Change-Id: I933367fa46aa0a3dc9aedf078b1be715bfa8c054

commit 35ad4e874522dd749582233ead8a023e042493bb
Author: Alistair Coles <alistairncoles@gmail.com>
Date:   Fri Jan 12 17:26:26 2018 +0000

Add tests for X-Backend-Clean-Expiring-Object-Queue true
    
    Check that when X-Backend-Clean-Expiring-Object-Queue is true
    the object server does indeed call async_update.
    
    Change-Id: I0a87979147591f15349b868a12ac6dd15ac4e37f
    Related-Change: I4d64f4d1d107c437fd3c23e19160157fdafbcd42

commit 7afc6a06eed1e9e3fdbea756074111b8a209d266
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Thu Jan 11 14:21:39 2018 -0800

Remove un-needed hack in probetest
    
    If you ran this probe test with ssync before the related change it would
    demonstrate the related bug.  The hack isn't harmful, but it isn't
    needed anymore.
    
    Related-Change-Id: I7f90b732c3268cb852b64f17555c631d668044a8
    Related-Bug: 1652323
    
    Change-Id: I09e3984a0500a0f4eceec392e7970b84070a5b39

commit 55a1b63db501f18ba62e86a29db47465dce8eb26
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Wed Jan 10 15:53:06 2018 -0800

Let recon-cron work with conf.d
    
    Change-Id: I862b74e0d9b20ba149581c1add6473dc1e5b2859

commit 48da3c1ed783a2b69cc74b02e8fd45e9d36cf80a
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Tue Jan 9 13:27:48 2018 -0800

Limit object-expirer queue updates on object DELETE, PUT, POST
    
    Currently, on deletion of an expiring object, each object server
    writes an async_pending to update the expirer queue and remove the row
    for that object. Each async_pending is processed by the object updater
    and results in all container replicas being updated. This is also true
    for PUT and POST requests for existing expiring objects.
    
    If you have Rc container replicas and Ro object replicas (or EC
    pieces), then the number of expirer-queue requests made is Rc * Ro [1].
    
    For a 3-replica cluster, that number is 9, which is not terrible. For
    a cluster with 3 container replicas and a 15+4 EC scheme, that number
    is 57, which is terrible.
    
    This commit makes it so at most two object servers will write out the
    async_pending files needed to update the queue, dropping the request
    count to 2 * Rc [2]. The object server now looks for a header
    "X-Backend-Clean-Expiring-Object-Queue: <true|false>" and writes or
    does not write expirer-queue async_pendings as appropriate. The proxy
    sends that header to 2 object servers.
    
    The queue update is not necessary for the proper functioning of the
    object expirer; if the queue update fails, then the object expirer
    will try to delete the object, receive 404s or 412s, and remove the
    queue entry. Removal on object PUT/POST/DELETE is helpful but not
    required.
    
    [1] assuming no retries needed by the object updater
    
    [2] or Rc, if a cluster has only one object replica
    
    Change-Id: I4d64f4d1d107c437fd3c23e19160157fdafbcd42

commit 9754a2ebe3f0ef66efd6fe0ff6c32fbb2e992617
Author: Christian Schwede <cschwede@redhat.com>
Date:   Wed Jan 10 11:35:00 2018 +0100

Change exit code when displaying empty rings
    
    Displaying an empty ring should not be an error, thus
    changing the exit code back to the former value of 0.
    
    Closes-Bug: 1742417
    Change-Id: I779c30cff1b4d24483f993221a8c6d944b7ae98d

commit a41c458c90b12c52688dd8b2b8a818b79b4e9693
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Fri Jan 5 16:54:44 2018 -0800

proxy: make the right number of container updates
    
    When the proxy is putting X-Container headers into object PUT
    requests, it should put out just enough to make the container update
    durable in the worst case. It shouldn't do more, since that results in
    extra work for the container servers; and it shouldn't do less, since
    that results in objects not showing up in listings.
    
    The current code gets the number right as long as you have 3 container
    replicas and an odd number of object replicas, but it comes up with
    some bogus numbers in other cases. The number it computes is
    (object-quorum + 1).
    
    This patch changes the number to (container-quorum +
    max_put_failures).
    
    Example: given an EC 12+5 policy and 3 container replicas, you can
    lose up to 4 connections and still succeed. Since you need to have 2
    container updates happen for durability, you need 6 connections to
    have X-Container headers. That way, you can lose 4 and still have 2
    left. The current code would put X-Container headers on 14 of the
    connections, resulting in more than double the workload on the
    container servers; this patch changes the number to 6.
    
    Example 2: given a (crazy) EC 3+6 policy and 3 container replicas, you
    can lose up to 5 connections, so you need X-Container headers on
    7. The current code only sends 5, giving a worst-case result of a PUT
    succeeds but never reaches the containers. This patch changes the
    number to 7.
    
    Other examples:
                              |  current  |  this change  |
                            --+-----------+---------------+
    EC 10+4, 3x container     |    12     |      5        |
    EC 10+4, 5x container     |    12     |      6        |
    EC 15+4, 3x container     |    17     |      5        |
    EC 15+4, 5x container     |    17     |      6        |
    EC 4+8, 3x container      |    6      |      9        |
    7x object, 3x container   |    5      |      5        |
    6x object, 3x container   |    4      |      5        |
    
    Change-Id: I34efd48655b890340912810ab111bb63445e5c8b

commit e7ffda5d0b75bd85c3b886c3aad0c938c7d476d6
Author: Alistair Coles <alistairncoles@gmail.com>
Date:   Fri Jan 5 17:41:21 2018 +0000

Use _update_x_timestamp method in object controller DELETE method
    
    The DELETE method repeats inline the same behaviour as provided by
    _update_x_timestamp, so just call the method.
    
    Also add unit tests for the behaviour of _update_x_timestamp.
    
    Change-Id: I8b6cfdbfb54b6d43ac507f23d84309ab543374aa

commit bf13d64cd07c0957a88687a24a8ca861189fb5ac
Author: Matthew Oliver <matt@oliver.net.au>
Date:   Wed Jan 3 04:48:44 2018 +0000

Show devices marked as deleted on empty rings
    
    This is a follow up patch to 530258 which will show
    extra infromation on empty rings.
    
    This patch goes one step further. On a completely empty ring:
    
      $ swift-ring-builder my.builder create  8 3 1
      $ swift-ring-builder my.builder
      my.builder, build version 0, id 33b4e117056340feae7d40430180c6bb
      256 partitions, 3.000000 replicas, 0 regions, 0 zones, 0 devices, 0.00 balance, 0.00 dispersion
      The minimum number of hours before a partition can be reassigned is 1 (0:00:00 remaining)
      The overload factor is 0.00% (0.000000)
      Ring file my.ring.gz not found, probably it hasn't been written yet
      Devices:   id region zone ip address:port replication ip:port  name weight partitions balance flags meta
      There are no devices in this ring, or all devices have been deleted
    
    It will still start the device list and then say no devices.. Why. let's
    see what happens now on an empty ring with devices still marked as
    deleted:
    
      $ swift-ring-builder my.builder add r1z1-127.0.0.1:6010/sdb1 1
      Device d0r1z1-127.0.0.1:6010R127.0.0.1:6010/sdb1_"" with 1.0 weight got id 0
      $ swift-ring-builder my.builder add r1z1-127.0.0.1:6010/sdb2 1
      Device d1r1z1-127.0.0.1:6010R127.0.0.1:6010/sdb2_"" with 1.0 weight got id 1
      $ swift-ring-builder my.builder remove r1z1-127.0.0.1
      Matched more than one device:
          d0r1z1-127.0.0.1:6010R127.0.0.1:6010/sdb1_""
          d1r1z1-127.0.0.1:6010R127.0.0.1:6010/sdb2_""
      Are you sure you want to remove these 2 devices? (y/N) y
      d0r1z1-127.0.0.1:6010R127.0.0.1:6010/sdb1_"" marked for removal and will be removed next rebalance.
      d1r1z1-127.0.0.1:6010R127.0.0.1:6010/sdb2_"" marked for removal and will be removed next rebalance.
    
      $ swift-ring-builder my.builder
      my.builder, build version 4, id 33b4e117056340feae7d40430180c6bb
      256 partitions, 3.000000 replicas, 1 regions, 1 zones, 2 devices, 0.00 balance, 0.00 dispersion
      The minimum number of hours before a partition can be reassigned is 1 (0:00:00 remaining)
      The overload factor is 0.00% (0.000000)
      Ring file my.ring.gz not found, probably it hasn't been written yet
      Devices:   id region zone ip address:port replication ip:port  name weight partitions balance flags meta
                  0      1    1  127.0.0.1:6010      127.0.0.1:6010  sdb1 0.00          0    0.00   DEL
                  1      1    1  127.0.0.1:6010      127.0.0.1:6010  sdb2 0.00          0    0.00   DEL
      There are no devices in this ring, or all devices have been deleted
    
    Now even when all devices are removed we can still see them as they are still there, only marked as deleted.
    
    Change-Id: Ib39f734deb67ad50bcdad5333cba716161a47e95

commit e343452394780b1f555777bc7083912ac68633d3
Author: Tim Burke <tim.burke@gmail.com>
Date:   Mon Jan 8 20:02:50 2018 +0000

Support existing builders with None _last_part_moves
    
    These were likely written before the first related change, or created
    from an existing ring file.
    
    Also, tolerate missing dispersion when rebalancing -- that may not exist
    in the builder file.
    
    Change-Id: I26e3b4429c747c23206e4671f7c86543bb182a15
    Related-Change: Ib165cf974c865d47c2d9e8f7b3641971d2e9f404
    Related-Change: Ie239b958fc7e0547ffda2bebf61546bd4ef3d829
    Related-Change: I551fcaf274876861feb12848749590f220842d68

commit 94565d9137275f4c6c775835cf2c0b81693137be
Author: Alistair Coles <alistairncoles@gmail.com>
Date:   Mon Jan 8 10:14:06 2018 +0000

Disallow x-delete-at equal to x-timestamp
    
    Previously an x-delete-at value equal to the x-timestamp value was
    allowed.  This could only occur when x-timestamp happened to take an
    integer value and would result in an object that was immediately
    unreadable.
    
    Similarly an x-delete-after value of zero may previously have been
    accepted if x-timestamp happened to be an integer value.
    
    With this change an x-delete-at value equal to x-timestamp or an
    x-delete-after value of zero always results in a 400 BadRequest.
    
    Also cleans up check_delete_headers docstring.
    
    Related-Change: Ia8d00fcef8893e3b3dd5720da2c8a5ae1e6e4cb8
    Related-Change: Ib2483444d3999e13ba83ca2edd3a8ef8e5c48548
    Change-Id: I27fdd800d8e149302ff4d6531101e9726a14d471

commit 79ac3a3c311b393d7b64ce4c41d68b52801b52cb
Author: Alistair Coles <alistairncoles@gmail.com>
Date:   Mon Jan 8 14:36:53 2018 +0000

Fix intermittent check_delete_headers failure
    
    Use a utils.Timestamp object to set a more realistic x-timestamp
    header to avoid intermittent failure when str(time.time()) results
    in a rounded up value.
    
    Closes-Bug: 1741912
    Change-Id: I0c54d07e30ecb391f9429e7bcfb782f965ece1ea

commit 6151554a89216934c0be242b93b28d87adc421e0
Author: Alistair Coles <alistairncoles@gmail.com>
Date:   Fri Jan 5 14:44:52 2018 +0000

Correct 400 response message when x-delete-after is zero
    
    Before an x-delete-after header with value '0' would almost
    certainly result in a 400 response, but the response body would
    report a problem with x-delete-at. Now the response correctly
    blames the x-delete-after header.
    
    Related-Change: I9a1b6826c4c553f0442cfe2bb78cdf49508fa4a5
    Change-Id: Ia8d00fcef8893e3b3dd5720da2c8a5ae1e6e4cb8

commit b22d3c1115609a62b3fce5177be213ed3fa587c5
Author: xhancar <pavel.hancar@gmail.com>
Date:   Sat Jan 6 20:48:10 2018 +0000

fix of type error
    
    There was incorrect path starting /home/swift, but /home/<your-user-name> is correct for common users.
    
    Change-Id: Ia81b2119c87dd88417428e55c82dac1ab7c028b3
    Closes-Bug: 1741378

commit 582460ecf9d503c10fd73c055301634b0b009dbc
Author: Alistair Coles <alistairncoles@gmail.com>
Date:   Fri Jan 5 14:43:12 2018 +0000

Document that x-delete-after takes precedence over x-delete-at
    
    Change-Id: Ib2483444d3999e13ba83ca2edd3a8ef8e5c48548

commit 31c294de797be30f499750ccbed3ec18a717f9b1
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Thu Jan 4 20:28:28 2018 -0800

Fix time skew when using X-Delete-After
    
    When a client sent "X-Delete-After: <n>", the proxy and all object
    servers would each compute X-Delete-At as "int(time.time() +
    n)". However, since they don't all compute it at exactly the same
    time, the objects stored on disk can end up with differing values for
    X-Delete-At, and in that case, the object-expirer queue has multiple
    entries for the same object (one for each distinct X-Delete-At value).
    
    This commit makes two changes, either one of which is sufficient to
    fix the bug.
    
    First, after computing X-Delete-At from X-Delete-After, X-Delete-After
    is removed from the request's headers. Thus, the proxy computes
    X-Delete-At, and the object servers don't, so there's only a single
    value.
    
    Second, computation of X-Delete-At now uses the request's X-Timestamp
    instead of time.time(). In the proxy, these values are essentially the
    same; the proxy is responsible for setting X-Timestamp. In the object
    server, this ensures that all computed X-Delete-At values are
    identical, even if the object servers' clocks are not, or if one
    object server takes an extra second to respond to a PUT request.
    
    Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
    Change-Id: I9a1b6826c4c553f0442cfe2bb78cdf49508fa4a5
    Closes-Bug: 1741371

commit a13e0ee76b1b5e56096878930fd24c638148155f
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Thu Jan 4 20:40:11 2018 -0800

Ignore directory .stestr
    
    After running the functional tests, this directory shows up. I don't
    know what's in it, but I'm fairly certain I don't want to commit it.
    
    Change-Id: If9179330c337daf2ae0a01e6c8aa8d349969e737

commit 49de7db532ffaba9fbd3c7e912e007b9d8a36d7c
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Fri Dec 29 08:53:21 2017 -0800

add swift-ring-builder option to recalculate dispersion
    
    Since dispersion info is cached, this can easily happen if we make
    changes to how dispersion info is calculated or stored (e.g. we extend
    the dispersion calculation to consider dispersion of all part-replicas
    in the related change)
    
    Related-Change-Id: Ifefff0260deac0c3e8b369a1e158686c89936686
    
    Change-Id: I714deb9e349cd114a21ec591216a9496aaf9e0d1

commit 9189f51d7601254181b2458f8c3e64c74d6dfad0
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Wed Dec 27 12:10:57 2017 -0800

Display more info on empty rings
    
    Related-Bug: #1737068
    Related-Change-Id: Ibadaf64748728a47a8f3f861ec1af601dbfeb9e0
    Change-Id: I683677f33764fa56dadfb7f6208f7f6ee25c8557

commit 56126b28392986636e99f47c3003ed6058b62fd9
Author: vxlinux <yan.wei7@zte.com.cn>
Date:   Fri Dec 22 11:40:40 2017 +0800

Handle EmptyRingError in swift-ring-builder's default command
    
    When the default display command for swift-ring-error encounters a
    EmptyRingError trying to calculate balance it should not raise exception
    and display the traceback in a command line environment.
    
    Instead handle the exceptional condition and provide the user with
    useful feedback.
    
    Closes-Bug: #1737068
    Change-Id: Ibadaf64748728a47a8f3f861ec1af601dbfeb9e0

commit f709eed41b9579cf5a8ca9180143301a2b452d47
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Thu Dec 28 14:56:08 2017 -0800

Fix socket leak on 416 EC GET responses.
    
    Sometimes, when handling an EC GET request with a Range header, the
    object servers reply 206 to the proxy, but the proxy (correctly)
    replies 416 to the client[1]. In that case, the connections to the object
    servers were not being closed. This was due to improper error handling
    in ECAppIter.
    
    Since ECAppIter is intended to be a WSGI iterable, it expects to have
    its close() method called when the caller is done with it. In this
    particular case, the caller (ECAppIter.kickoff()) was not calling
    close() when an exception was raised. Now it is.
    
    [1] consider a 4+2 EC policy with segment size 1024, an 20 byte
    object, and a request with "Range: bytes=21-50". The proxy needs whole
    fragments to decode, so it asks the object server for "Range:
    bytes=0-255" [2], the object server says 206, and then the proxy
    realizes that the client's request is unsatisfiable and tells the
    client 416.
    
    [2] segment size 1024 and 4 data fragments means the fragments have
    size 1024 / 4 = 256, hence "bytes=0-255" asks for the first whole
    fragment
    
    Change-Id: Ide2edf8c449c97d45f48c2dbbbff7aebefa4b158
    Closes-Bug: 1738804

commit 7013e70ca67891e94664e9eca70925b61ee8f689
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Thu Dec 14 20:03:24 2017 -0800

Represent dispersion worse than one replicanth
    
    With a sufficiently undispersed ring it's possible to move an entire
    replicas worth of parts and yet the value of dispersion may not get any
    better (even though in reality dispersion has dramatically improved).
    The problem is dispersion will currently only represent up to one whole
    replica worth of parts being undispersed.
    
    However with EC rings it's possible for more than one whole replicas
    worth of partitions to be undispersed, in these cases the builder will
    require multiple rebalance operations to fully disperse replicas - but
    the dispersion value should improve with every rebalance.
    
    N.B. with this change it's possible for rings with a bad dispersion
    value to measure as having a significantly smaller dispersion value
    after a rebalance (even though they may not have had their dispersion
    change) because the total amount of bad dispersion we can measure has
    been increased but we're normalizing within a similar range.
    
    Closes-Bug: #1697543
    
    Change-Id: Ifefff0260deac0c3e8b369a1e158686c89936686

commit 1449532fb82b4fe3a5484b547d425dcda82df259
Author: Kazuhiro MIYAHARA <miyahara.kazuhiro@lab.ntt.co.jp>
Date:   Mon Dec 25 07:09:49 2017 +0000

Allow InternalClient to container/object listing with prefix
    
    This patch adds 'prefix' argument to iter_containers/iter_objects
    method of InternalClient.
    This change will be used in general task queue feature [1].
    
    [1]: https://review.openstack.org/#/c/517389/
    
    Change-Id: I8c2067c07fe35681fdc9403da771f451c21136d3

commit 2f2899eb1df57c3ef0498c4a0d3295863a1bf798
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Thu Dec 14 14:57:48 2017 -0800

Show missing branches in coverage report.
    
    This used to be the default in coverage 3.x, but 4.0+ requires you to
    explicitly configure it.
    
    Change-Id: I3b06154c7862c300b5a2b3afb14cced1e8411468

tags:

added: in-feature-deep

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-02-05: Fix included in openstack/swift 2.17.0

#8

This issue was fixed in the openstack/swift 2.17.0 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-02-26: Fix proposed to swift (feature/s3api)

#9

Fix proposed to branch: feature/s3api
Review: https://review.openstack.org/548052

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-02-27:

#10

Fix proposed to branch: feature/s3api
Review: https://review.openstack.org/548193

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-02-27: Change abandoned on swift (feature/s3api)

#11

Change abandoned by Kota Tsuyuzaki (<email address hidden>) on branch: feature/s3api
Review: https://review.openstack.org/548052
Reason: This is affected by the new s3api functests added in recent. Use https://review.openstack.org/#/c/548193/ instead.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-02-27: Fix merged to swift (feature/s3api)

#12

Download full text (30.7 KiB)

Reviewed: https://review.openstack.org/548193
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=9d43f4e190802cb0cad507a65245e4dd70fda7db
Submitter: Zuul
Branch: feature/s3api

commit b3f1558acd2f5a4df3f039070ca5bbc33935e822
Author: Kazuhiro MIYAHARA <email address hidden>
Date: Tue Feb 13 05:16:27 2018 +0000

Fix expirer's invalid task object names in unit tests

    Object-expirer's task name should be in format of
    "<timestamp>-<account>/<container>/<obj>". In object-expirer
    implementation, ValueError is catched and handled when expirer's task
    objects have invalid name. But in actual swift cluster, invalid task
    object name is not created because task object is created by
    object-server.
    However, without the ValueError catching, some unit tests fail,
    because the unit tests create invalid task object names.

    This patch fixes invalid task object names in unit tests. The
    ValueError catch is remained for unexpected errors, but in the case
    the task will be skipped.

This patch will help to refactor expirer's task object parsing.

Change-Id: I8fab8fd180481ce9e97c945904c5c89eec037110

commit 4b19ac772364778a4b96d7e18834db9a7645f482
Author: Tim Burke <email address hidden>
Date: Thu Feb 1 14:30:19 2018 -0800

py3: port common/storage_policy.py

Change-Id: I7030280a8495628df9ed8edcc8abc31f901da72e

commit 25540a415e7e36bb08a01a14ca41e2d7591e62cb
Author: Tim Burke <email address hidden>
Date: Thu Feb 22 11:08:49 2018 -0800

Tighten up assertions around expirer's concurrency

In particular, test that each work item is only done *once*.

Change-Id: I9cc610bffb2aa9a2f2b05f4c49e574ab56d05201
Related-Change: Ic0075a3718face8c509ed0524b63d9171f5b7d7a

commit 5017864133b5af289f205afaf76ffe4518688b3f
Author: melissaml <ma.lei@99cloud.net>
Date: Mon Feb 26 15:48:31 2018 +0800

Fix the incorrect reference links

    TrivialFix
    [1] is the installation guide for OpenStack components, obviously,
    we need [1] in the docs.

[1] https://docs.openstack.org/latest/install/

Change-Id: I3c6fe7327f5552cc2b8f0f0e42b41f8e989a0a7e

commit 58f5d89066adbd54403ad315ffe1f9b2f05aa583
Author: Kazuhiro MIYAHARA <email address hidden>
Date: Tue Feb 13 03:36:04 2018 +0000

Remove confusing assertion from expirer's unit test

    In test_expirer.TestObjectExpirer.test_process_based_concurrency,
    an assertion checks that expirer execute tasks in round-robin order
    for target containers. But the assertion depends on task object path,
    because task assignation for each process depends on md5 of task
    object path. The dependency makes the assetion confusing.

Now, we have test_expirer.TestObjectExpirer.test_round_robin_order which
is added in [1]. So this patch remove the confusing assertion.

This patch will help to refactor expirer's task object parsing.
I will push patch for the refactoring after this patch.

[1]: https://review.openstack.org/#/c/538171

Change-Id: Ic0075a3718face8c509ed0524b63d9171f5b7d7a

commit 6060af8db96e23d32f3ecc3d02f7f...

Reviewed:  https://review.openstack.org/548193
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=9d43f4e190802cb0cad507a65245e4dd70fda7db
Submitter: Zuul
Branch:    feature/s3api

commit b3f1558acd2f5a4df3f039070ca5bbc33935e822
Author: Kazuhiro MIYAHARA <miyahara.kazuhiro@lab.ntt.co.jp>
Date:   Tue Feb 13 05:16:27 2018 +0000

Fix expirer's invalid task object names in unit tests
    
    Object-expirer's task name should be in format of
    "<timestamp>-<account>/<container>/<obj>". In object-expirer
    implementation, ValueError is catched and handled when expirer's task
    objects have invalid name. But in actual swift cluster, invalid task
    object name is not created because task object is created by
    object-server.
    However, without the ValueError catching, some unit tests fail,
    because the unit tests create invalid task object names.
    
    This patch fixes invalid task object names in unit tests. The
    ValueError catch is remained for unexpected errors, but in the case
    the task will be skipped.
    
    This patch will help to refactor expirer's task object parsing.
    
    Change-Id: I8fab8fd180481ce9e97c945904c5c89eec037110

commit 4b19ac772364778a4b96d7e18834db9a7645f482
Author: Tim Burke <tim.burke@gmail.com>
Date:   Thu Feb 1 14:30:19 2018 -0800

py3: port common/storage_policy.py
    
    Change-Id: I7030280a8495628df9ed8edcc8abc31f901da72e

commit 25540a415e7e36bb08a01a14ca41e2d7591e62cb
Author: Tim Burke <tim.burke@gmail.com>
Date:   Thu Feb 22 11:08:49 2018 -0800

Tighten up assertions around expirer's concurrency
    
    In particular, test that each work item is only done *once*.
    
    Change-Id: I9cc610bffb2aa9a2f2b05f4c49e574ab56d05201
    Related-Change: Ic0075a3718face8c509ed0524b63d9171f5b7d7a

commit 5017864133b5af289f205afaf76ffe4518688b3f
Author: melissaml <ma.lei@99cloud.net>
Date:   Mon Feb 26 15:48:31 2018 +0800

Fix the incorrect reference links
    
    TrivialFix
    [1] is the installation guide for OpenStack components, obviously,
    we need [1] in the docs.
    
    [1] https://docs.openstack.org/latest/install/
    
    Change-Id: I3c6fe7327f5552cc2b8f0f0e42b41f8e989a0a7e

commit 58f5d89066adbd54403ad315ffe1f9b2f05aa583
Author: Kazuhiro MIYAHARA <miyahara.kazuhiro@lab.ntt.co.jp>
Date:   Tue Feb 13 03:36:04 2018 +0000

Remove confusing assertion from expirer's unit test
    
    In test_expirer.TestObjectExpirer.test_process_based_concurrency,
    an assertion checks that expirer execute tasks in round-robin order
    for target containers. But the assertion depends on task object path,
    because task assignation for each process depends on md5 of task
    object path. The dependency makes the assetion confusing.
    
    Now, we have test_expirer.TestObjectExpirer.test_round_robin_order which
    is added in [1]. So this patch remove the confusing assertion.
    
    This patch will help to refactor expirer's task object parsing.
    I will push patch for the refactoring after this patch.
    
    [1]: https://review.openstack.org/#/c/538171
    
    Change-Id: Ic0075a3718face8c509ed0524b63d9171f5b7d7a

commit 6060af8db96e23d32f3ecc3d02f7f0dd27a57eba
Author: Tim Burke <tim.burke@gmail.com>
Date:   Thu Feb 22 10:26:00 2018 -0800

Add more tests around ObjectExpirer.round_robin_order
    
    Change-Id: I43b5e8d9513fd0566a61ff585dfdc1dde5b28343
    Related-Change: Ibf61eb1f767a48cb457dd494e1f7c12acfd205de

commit 303635348b6cd30b13522d5c483934882178982b
Author: Kazuhiro MIYAHARA <miyahara.kazuhiro@lab.ntt.co.jp>
Date:   Fri Jan 26 07:54:31 2018 +0000

Refactor expirer's task round robin implementation
    
    Object-expirer changes order of expiration tasks to avoid deleting
    objects in a certain container continuously.
    
    To make review for expirer's task queue update patch [1] easy,
    this patch refactors the implementation of the order change. In this
    patch, the order change is divided as a function.
    
    In [1], there will be two implementations for legacy task queue
    and for general task queue. The two implementations have similar
    codes. This patch helps to avoid copying codes in the two implementations.
    
    Other than dividing function, this patch tries to resolve:
    - Separate container iteration and object iteration to avoid the generator
      termination with (container, None) tuple.
    - Using Timestamp class for delete_timestamp to be consist with other modules
    - Change yielded delete task object info from tuple to dict because that
      includes several complex info (e.g. task_container, task_object,
      and target_path)
    - Fix minor docs and tests depends on the changes above
    
    [1]: https://review.openstack.org/#/c/517389
    
    Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
    
    Change-Id: Ibf61eb1f767a48cb457dd494e1f7c12acfd205de

commit c6169cd7146cd80a3a5562eb729cde97f801cb84
Author: Kazuhiro MIYAHARA <miyahara.kazuhiro@lab.ntt.co.jp>
Date:   Wed Feb 21 04:58:59 2018 +0000

Fix a typo in swift/common/swob.py
    
    This patch just fixing a typo.
    
    Change-Id: I8a3c5e86b39e442157874a426d664fce33c9d36a

commit 8539b9388793fa5588c9f184d52f7834f1e77128
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Thu Feb 15 16:03:06 2018 -0800

Small cleanup in DB replicator's error handling
    
    Generally, we shouldn't compare integers with "is". It happens to work
    because (a) CPython has only one instance of each integer betwen -5
    and 256, and (b) errno.ENOTEMPTY == 66 on Linux, but it's better not
    to rely on those details.
    
    Change-Id: I494a4adbb2bb35adcf83b2d23e790c1b220e75c7

commit 57306181f1ffde3da1849c6d27e0b001d3bf5ea3
Author: Gerard Gine <ggine@swiftstack.com>
Date:   Tue Feb 13 17:00:22 2018 -0800

Improved usage of args in .functests
    
    If we're calling the script with any arguments, --pretty will not
    be passed to ostestr.
    Also redirected cd commands' output to /dev/null in .functests.
    
    Change-Id: I6e7e391c7e1659b86ab12eae4362b565218917b2

commit 8dc40125ce8da616c713a778029b1b29903a1537
Author: John Dickinson <me@not.mn>
Date:   Tue Feb 13 14:38:22 2018 -0800

add keystonemiddleware to extras area
    
    Change-Id: I805cf220aee4af69bf4984cc148d69520e958073

commit 642f79965a5e77b20660d2f0144d1624b0fa451c
Author: Tim Burke <tim.burke@gmail.com>
Date:   Wed Nov 23 10:14:21 2016 -0800

py3: port common/ring/ and common/utils.py
    
    I can't imagine us *not* having a py3 proxy server at some point, and
    that proxy server is going to need a ring.
    
    While we're at it (and since they were so close anyway), port
    
    * cli/ringbuilder.py and
    * common/linkat.py
    * common/daemon.py
    
    Change-Id: Iec8d97e0ce925614a86b516c4c6ed82809d0ba9b

commit 1dddcb533e16602cca5a057dbda12ca4049ea64e
Author: Tim Burke <tim.burke@gmail.com>
Date:   Thu Feb 8 13:45:21 2018 -0800

probetests: Close our handle to subprocess' stdouts
    
    Otherwise, you could start running out of file handles when running all
    tests. Previously, I'd regularly exceed 1k fds and then all subsequent
    tests would fail; now, it maxes out around 90.
    
    Change-Id: Ib53f15b5d2be95c70cce8903ea0ef5c334479837

commit f0f3de34626269052b1d7022d2e3cd3f9f0a377e
Author: malei <malei@maleideMacBook-Pro.local>
Date:   Fri Feb 9 22:43:21 2018 +0800

fix typos in swift
    
    Change-Id: I14dd433ed7c7fe789d5d04defbf9eec5bffc301a

commit 54918ad1db574bbc2c0b429836609a7ddb1efb13
Author: baiwenteng <baiwenteng@inspur.com>
Date:   Fri Feb 9 16:39:32 2018 +0800

Fix grammar error
    
    Change-Id: Ic375b6c6ebf3f66860b065785e75b8d47552be28

commit 753715499c96c1602bfec5ed241b81b82dd1f1db
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Fri Feb 9 07:09:42 2018 +0000

Imported Translations from Zanata
    
    For more information about this automatic import see:
    https://docs.openstack.org/i18n/latest/reviewing-translation-import.html
    
    Change-Id: I1409c8eaca4fbe1170d2e998b35e7fdacf4b86b4

commit a3d2aaba6470645345b19d52f0dbbfd51a4d5cf4
Author: baiwenteng <baiwenteng@inspur.com>
Date:   Fri Feb 9 12:20:59 2018 +0800

Fix typos in swift
    
    Change-Id: I0982b0046a16fda0a39d9b31402b2e4b3160a5c4

commit 1f4ebbc9900c113fa4447dd51079cc2cb9aa3ceb
Author: Alistair Coles <alistairncoles@gmail.com>
Date:   Thu Feb 8 12:59:36 2018 +0000

kill orphans during probe test setup
    
    orphans processes sometimes cause probe test failures so
    get rid of them before each test.
    
    Change-Id: I4ba6748d30fbb28371f13aa95387c49bc8223402

commit c9410c7dd482cc1faefdfd9d9c83d225e7d28e8f
Author: Thiago da Silva <thiago@redhat.com>
Date:   Thu Feb 8 18:18:15 2018 -0500

Move eventlet patch before call to loadapp
    
    Ran into an eventlet bug[0] while integration Swift/Barbican
    in TripleO. It is very similar to a previous bug related
    to keystonemiddleware[1]. Suggestion from urllib3[2] is to
    patch eventlet "as early as possible". Traceback[3] shows that
    urllib3 is being imported before the eventlet patch, so moved
    the patch to before the loadapp call.
    
    [0] - http://paste.openstack.org/show/658046/
    [1] - https://bugs.launchpad.net/swift/+bug/1662473
    [2] - https://github.com/shazow/urllib3/issues/1104
    [3] - https://gist.github.com/thiagodasilva/12dad7dc4f940b046dd0863b6f82a78b
    
    Change-Id: I74e580f31349bdefd187cc5d6770a7041a936bef

commit 0a993437d1dd837be80a036b1525a3195c86a10d
Author: Tim Burke <tim.burke@gmail.com>
Date:   Thu Feb 8 10:56:28 2018 -0800

Update install-guide URLs to point to stable/queens
    
    Change-Id: I7b2dd19f18cbfac3bc3ea763cd3333e3fb68f1cd

commit 6b9098eead2e0de36dd5b9a5d6a136eedf58ed47
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Thu Feb 8 07:06:31 2018 +0000

Imported Translations from Zanata
    
    For more information about this automatic import see:
    https://docs.openstack.org/i18n/latest/reviewing-translation-import.html
    
    Change-Id: Ib339f89f634a22ea874dcf858c43a14d1145ed77

commit bfe52a2e3562243b1b488e339a965642d846a30e
Author: Ondřej Nový <ondrej.novy@firma.seznam.cz>
Date:   Tue Feb 6 13:08:42 2018 +0100

Quarantine DB without *_stat row
    
    Closes-Bug: #1747689
    Change-Id: Ief6bd0ba6cf675edd8ba939a36fb9d90d3f4447f

commit 5b30c1f8115f2a85407b836be47d8376832b4f7b
Author: Tim Burke <tim.burke@gmail.com>
Date:   Wed Feb 7 05:50:12 2018 +0000

Fix flakey test_check_delete_headers_sets_delete_at
    
    It was rare (saw it once in 10k runs running locally), but it's
    ocassionally blown up in the gate [1]. With this, no fails locally even
    after 100k runs.
    
    [1] http://logs.openstack.org/11/538011/3/gate/swift-tox-py27/06c06f0/job-output.txt.gz#_2018-02-07_03_29_09_578389
    
    Change-Id: I7701d2db2ec82b48559c5b74a2e08c4403fd5dec
    Related-Change: Ia126ad6988f387bbd2d1f5ddff0a56d457a1fc9b

commit d9eec63a588e1e1e574b71d67b5d7456c3ad468d
Author: James E. Blair <jeblair@redhat.com>
Date:   Wed Jan 24 17:05:13 2018 -0800

Zuul: Remove project name
    
    Zuul no longer requires the project-name for in-repo configuration.
    Omitting it makes forking or renaming projects easier.
    
    Change-Id: I2226881d18e69bf25ad93ee8b8db67248e14f697

commit f4cfe81e593b52e11e67916073a050cd2dde2e00
Author: John Dickinson <me@not.mn>
Date:   Tue Jan 16 14:51:29 2018 -0800

authors/changelog updates for 2.17.0 release
    
    Change-Id: I577d169022916676a20a9ac24c7cc7b63ae46778

commit 8140b7e7ad8eaef0ca110c0f5e185a046b69fd6f
Author: Kazuhiro MIYAHARA <miyahara.kazuhiro@lab.ntt.co.jp>
Date:   Thu Feb 1 09:43:46 2018 +0000

Fix inconsistency of account info in expirer's unit tests
    
    In expirer's unit tests, FakeInternalClient instances simulates
    expirer's task queue behavior. But get_account_info method of
    the FakeInternalClient returns container count = 1 and object
    count = 2, even if it simulate different count of containers or
    objects.
    
    This patch fixes the behavior. The return values of get_account_info
    will be equal to simulated container and object counts.
    
    This patch will make review for expirer's task queue upgrade patch [1]
    more easy.
    
    [1]: https://review.openstack.org/#/c/517389
    
    Change-Id: Id5339ea7e10e4577ff22daeb91ec90f08704c98d

commit 924a043c70ba38c5d81758b5e0c29fa73674404a
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Wed Jan 31 16:40:21 2018 -0800

Remove some cruft from ratelimit tests
    
    The tests were carefully setting up a mock for the 'http_connect'
    function in the ratelimit module, but there is no such function
    imported by the ratelimit module.
    
    As far as I can tell, this has been the case since the ratelimit
    middleware first appeared in 72d40bd (Mon Oct 4 14:11:48 2010 -0700).
    
    Change-Id: If047184c6435aa1647050f50b499dc9feff4318d

commit 5cac37284e1d3f3019309ca86f73e3dbb176df59
Author: Kazuhiro MIYAHARA <miyahara.kazuhiro@lab.ntt.co.jp>
Date:   Thu Jan 25 10:49:00 2018 +0000

Refactor expirer unit tests
    
    In expirer's unit tests, fake InternalClient classes are defined
    and its instance simulates expirer's task queue behaviors.
    
    To make review for expirer's task queue update patch [1] easy,
    this patch refactors the implementation of the fake InternalClient
    classes. In this patch, unit tests are refactored by the following
    two approaches:
        #1: Summarizing duplicated fake InternalClient implementation
        #2: Make task account name variable
    The #2 approach is for multiple task accounts in [1].
    
    The patch [1] will be rebased after this patch merged.
    
    [1]: https://review.openstack.org/#/c/517389
    
    Change-Id: I10a7151cfdd43460ad38c47f672d3c31b77e7990

commit 1c306660a53447bb0a696873bd3f88e66e3781eb
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Wed Jan 31 08:18:59 2018 +0000

Imported Translations from Zanata
    
    For more information about this automatic import see:
    https://docs.openstack.org/i18n/latest/reviewing-translation-import.html
    
    Change-Id: I6e420ea5241838192571dd9d857cca4056699314

commit 5a4d3bdfc49d61ee6e333c8eb1dad4383ae9308b
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue Dec 5 20:19:37 2017 +0000

tempurl: Make the digest algorithm configurable
    
    ... and add support for SHA-256 and SHA-512 by default. This allows us
    to start moving toward replacing SHA-1-based signatures. We've known
    this would eventually be necessary for a while [1], and earlier this
    year we've seen SHA-1 collisions [2].
    
    Additionally, allow signatures to be base64-encoded, provided they start
    with a digest name followed by a colon. Trailing padding is optional for
    base64-encoded signatures, and both normal and "url-safe" modes are
    supported. For example, all of the following SHA-1 signatures are
    equivalent:
    
       da39a3ee5e6b4b0d3255bfef95601890afd80709
       sha1:2jmj7l5rSw0yVb/vlWAYkK/YBwk=
       sha1:2jmj7l5rSw0yVb/vlWAYkK/YBwk
       sha1:2jmj7l5rSw0yVb_vlWAYkK_YBwk=
       sha1:2jmj7l5rSw0yVb_vlWAYkK_YBwk
    
    (Note that "normal" base64 encodings will require that you url encode
    all "+" characters as "%2B" so they aren't misinterpretted as spaces.)
    
    This was done for two reasons:
    
       1. A hex-encoded SHA-512 is rather lengthy at 128 characters -- 88
          isn't *that* much better, but it's something.
       2. This will allow us to more-easily add support for different
          digests with the same bit length in the future.
    
    Base64-encoding is required for SHA-512 signatures; hex-encoding is
    supported for SHA-256 signatures so we aren't needlessly breaking from
    what Rackspace is doing.
    
    [1] https://www.schneier.com/blog/archives/2012/10/when_will_we_se.html
    [2] https://security.googleblog.com/2017/02/announcing-first-sha1-collision.html
    
    Change-Id: Ia9dd1a91cc3c9c946f5f029cdefc9e66bcf01046
    Related-Bug: #1733634

commit 11bf9e45884aa5122e76a97cf62062d7b0bdfe54
Author: Joel Wright <joel.wright@sohonet.com>
Date:   Sun Sep 4 23:51:53 2016 +0100

Add support for data segments to SLO and SegmentedIterable
    
    This patch updates the SLO middleware and SegmentedIterable to add
    support for user-specified inlined-data segments. Such segments will
    contain base64-encoded data to be added before/after an object-backed
    segment within an SLO. To accommodate the potential extra data we
    increase the default SLO maximum manifest size from 2MiB to 8MiB.
    The default maximum number of segments remains 1000, but this will
    only be enforced for object-backed segments.
    
    This patch is a prerequisite for a future patch enabling the
    download of large objects as tarballs. The TLO patch will be added
    as a dependent patch later.
    
    UpgradeImpact
    =============
    During a rolling upgrade, an updated proxy may write a manifest that
    out-of-date proxies will not be able to read. This will resolve itself
    once the upgrade completes on all nodes.
    
    Change-Id: Ib8dc216a84d370e6da7d6b819af79582b671d699

commit 89a5c9d56fda8282292253d2d056121694ef7654
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue Sep 12 22:46:14 2017 +0000

Disallow fractional replicas in EC policies
    
    Change-Id: I873d7bf7de54e4b1dccdafc8a61f03c09a65dfbc
    Closes-Bug: 1554391
    Closes-Bug: 1677547

commit 10542d00ea131250a184803a03ce2928fadf274e
Author: gaofei <gao.fei@inspur.com>
Date:   Tue Jan 23 16:57:55 2018 +0800

Replace Chinese punctuation with English punctuation
    
    Curly quotes(Chinese punctuation) usually input from Chinese input method.
    When read from english context, it makes some confusion.
    
    Change-Id: Ibd50299ee287c56ec4759ea8ff53d47d006144f8

commit fdf817b3188459bed60bf8668fac7ee147de2a9e
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue Jan 23 18:42:40 2018 +0000

Include policy in ran-out-of-handoffs log message
    
    Related-Change: I2f5bb12f3704880df1750229425f64f419ff9aef
    Change-Id: I14b70e949a57d2c31b71d3efa6ee1200d0717db6

commit 05b2a14fe04d097bfc2155a325fa240983311e97
Author: gaofei <gao.fei@inspur.com>
Date:   Tue Jan 23 14:57:18 2018 +0800

Update broken link
    
    Change-Id: I8d13696970593d47d0f07cf97aa1284eed5f7a15

commit ddb13aa5eab03b6993887eb02260b4bc0b256922
Author: vxlinux <yan.wei7@zte.com.cn>
Date:   Sat Jan 20 17:23:35 2018 +0800

Remove redundant blank space in README.rst
    
    Change-Id: If347476e3b9185921ff174d3f8170a1c4d0622e8

commit cbcbc774681ad1a90f8fd260787c29943bedb675
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Thu Jan 18 18:33:08 2018 -0800

Internal Client should only retry server errors
    
    Change-Id: I0a3f4a01d50b6a5d5528f527faa34573c7dde619

commit 12f874534925b52f9d1c91580794eb9e5e9a4589
Author: vxlinux <yan.wei7@zte.com.cn>
Date:   Fri Jan 19 16:54:26 2018 +0800

Add Docstrings to validate_replicas_by_tier
    
    New common functions should have Docstrings
    
    Change-Id: Icbb3cdf38509fd6d034cbb2271786559780a7b68

commit d2034cd7b6946829a7d95c4d2c71d4322f80e855
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Tue Jan 16 17:03:38 2018 -0800

Keep object-updater stats logging consistent
    
    If we're going to encapsulate the stats tracking it seems reasonable if
    we ever add any more metrics we can reduce the number of places we need
    to update log messages.
    
    Change-Id: I187cf6cfec1e0a9138b709fa298e1991aa809ec4

commit f4cff9fbba76932a4b67f1130d630098497bb4bc
Author: Tim Burke <tim.burke@gmail.com>
Date:   Wed Jan 17 22:07:26 2018 +0000

object-server can 409 in response to x-if-delete-at
    
    Previously, if the expirer had a stale work item (because the object
    was overwritten or deleted, or some other process handled the delete),
    then it would keep retrying for reclaim_age, but every time it'd get
    back a 412.
    
    Now, have the object-server be smart enough to say, "I have more recent
    information than you" and let the expirer accept that as success.
    
    Change-Id: I0a94482ed16cb30ce79074e053e6177fe97bcaa9