Bug #1624088 “EC missing durable can prevent reconstruction” : Bugs : OpenStack Object Storage (swift)

Revision history for this message

clayg (clay-gerrard) wrote on 2016-09-15:

#1

failing probetest Edit (3.8 KiB, text/x-diff)

Revision history for this message

clayg (clay-gerrard) wrote on 2016-09-16:

#2

Making the Object Server prefer to serve non-durable frags rather than 404 seems like the most straight forward way to address this terrible terrible failure:

https://gist.github.com/clayg/2988c8804ab4b704905d203d534a5342

I'm not aware of any situation where the proxy or reconstructor would ask for a fragment and not be robust to a potentially uncommited minority response. If there was an uncommited *majority* response - it would be served whiled it's available - but the eventual durability would still be subject to the existence and propagation of the durable file.

As such, this solution should be considered at *least* a partial fix for lp bug #1469094 - additional work to improve availability of a minority overwrite while additional primaries off line (i.e. servicing a durable majority from "under" a non-durable minority using secondary frags from the same nodes) should fall under lp bug #1484598

Revision history for this message

Alistair Coles (alistair-coles) wrote on 2016-09-16:

#3

If we just make the object server always return most recent fragment even if it is not durable, (which is how I understood your attached patch), then don't we just swap one "bug" for another?

On master, if a partial PUT results in >= ec_ndata frags at t1 but < ec_ndata durables, then we cannot GET the object at t1

With your change, if a partial PUT at t1 results in < ec_ndata frags with no durables then we cannot GET the object at t1 *nor* an underlying object at t0, which remains on disk (no durable -> old durable data remains on disk) because optimism failed and there is no fallback. In this case no reconstruction can happen either.

With patch https://review.openstack.org/#/c/215276/ both scenarios are "fixed" because the proxy is smart enough to see that optimism failed and fallback to whatever else can be fetched from disks. Although that patch does not cover the reconstructor, I think the strategy (smart proxy plus adaptive object server) will allow to fix more cases.

Revision history for this message

clayg (clay-gerrard) wrote on 2016-09-16:

#5

> If we just make the object server always return most recent fragment
> even if it is not durable, (which is how I understood your attached
> patch), then don't we just swap one "bug" for another?

That would definately not be acceptable, we'd need unitests for the
suggested change to assuage any concerns that there is no bad behaviors.

> On master, if a partial PUT results in >= ec_ndata frags at t1 but <
> ec_ndata durables, then we cannot GET the object at t1

If we have >= ec_ndata frags at t1 we should serve the it. If the worry
is some of those frags would be unavailable on a subsequent read but an
older stale read might be *possible* - I think we'd have to conceeed
then any improvments would have to include a second request back to
those nodes.

So, I think the question is about what the object server should do in
the default case (which is most likely to achive our goals of highly
available read your writes?) I have learend that missing durables can
happen under load, and failing to serve undurable fragments is causing
us a lot of pain.

I'm not really sure about servicing stale durable fragments from "under"
more recent writes - now we're getting into lp bug #1484598 - which adds
a bunch of complexity.

> With your change, if a partial PUT at t1 results in < ec_ndata frags
> with no durables then we cannot GET the object at t1 *nor* an
> underlying object at t0, which remains on disk (no durable -> old
> durable data remains on disk) because optimism failed and there is no
> fallback. In this case no reconstruction can happen either.

It think this is correct. It's not good. Worth considering...

https://gist.github.com/clayg/1f2590ea4608a13823865e27a6666c2a

Yes, I think we would have to address this case. Still feels like lp
bug #1484598

If a GET at t2 were able to get *something* from the t0 write (which
might not be available until that TOR switch is fixed) - the ideal
behavior would be for the proxy to make a second round of requests to
the nodes - I'd probably advocate for a "X-Require-Durable" so Node 1-6
could return the "stale" write.

> With patch https://review.openstack.org/#/c/215276/ both scenarios are
> "fixed" because the proxy is smart enough to see that optimism failed
> and fallback to whatever else can be fetched from disks. Although that
> patch does not cover the reconstructor, I think the strategy (smart
> proxy plus adaptive object server) will allow to fix more cases.

Yes, there does keep seeming to be some failure scenarios that would
require a proxy smart enough to reach back into an object server for
"something else" that it might be holding. The API and adaptiveness
that the object server might need to be expressable is more contentious
- I'm definately triaging failure modes based on likeleness and trying
to deal with those in a manor as simple as possible. I'm not scared of
complexity, but I think complex APIs are complicated to get right. So I
think it's reasonable to only let in as much complexity as needed into
the API.

> If we just make the object server always return most recent fragment
> even if it is not durable, (which is how I understood your attached
> patch), then don't we just swap one "bug" for another?

That would definately not be acceptable, we'd need unitests for the
suggested change to assuage any concerns that there is no bad behaviors.

> On master, if a partial PUT results in >= ec_ndata frags at t1 but <
> ec_ndata durables, then we cannot GET the object at t1

If we have >= ec_ndata frags at t1 we should serve the it.  If the worry
is some of those frags would be unavailable on a subsequent read but an
older stale read might be *possible*  - I think we'd have to conceeed
then any improvments would have to include a second request back to
those nodes.

So, I think the question is about what the object server should do in
the default case (which is most likely to achive our goals of highly
available read your writes?) I have learend that missing durables can
happen under load, and failing to serve undurable fragments is causing
us a lot of pain.

I'm not really sure about servicing stale durable fragments from "under"
more recent writes - now we're getting into lp bug #1484598 - which adds
a bunch of complexity.

> With your change, if a partial PUT at t1 results in < ec_ndata frags
> with no durables then we cannot GET the object at t1 *nor* an
> underlying object at t0, which remains on disk (no durable -> old
> durable data remains on disk) because optimism failed and there is no
> fallback. In this case no reconstruction can happen either.

It think this is correct.  It's not good.  Worth considering...

https://gist.github.com/clayg/1f2590ea4608a13823865e27a6666c2a

Yes, I think we would have to address this case.  Still feels like lp
bug #1484598

If a GET at t2 were able to get *something* from the t0 write (which
might not be available until that TOR switch is fixed) - the ideal
behavior would be for the proxy to make a second round of requests to
the nodes - I'd probably advocate for a "X-Require-Durable" so Node 1-6
could return the "stale" write.

> With patch https://review.openstack.org/#/c/215276/ both scenarios are
> "fixed" because the proxy is smart enough to see that optimism failed
> and fallback to whatever else can be fetched from disks. Although that
> patch does not cover the reconstructor, I think the strategy (smart
> proxy plus adaptive object server) will allow to fix more cases.

Yes, there does keep seeming to be some failure scenarios that would
require a proxy smart enough to reach back into an object server for
"something else" that it might be holding.  The API and adaptiveness
that the object server might need to be expressable is more contentious
- I'm definately triaging failure modes based on likeleness and trying
to deal with those in a manor as simple as possible.  I'm not scared of
complexity, but I think complex APIs are complicated to get right.  So I
think it's reasonable to only let in as much complexity as needed into
the API.

Revision history for this message

clayg (clay-gerrard) wrote on 2016-09-23:

#6

No one has ever reported this failure from the wild. The probetest probes it's possible - but it's going out of it's way to mess with ondisk state so it's kind of unfair.

Most everyone seems convinced it can happen with timeouts writing durables; and a couple of unlucky disk failures - so it should be fixed; but the solution is not straight forward (it took us a year to fix essentially the same problem in the proxy [1])

It being "hard" isn't a good reason to reduce the severity of the bug - but not known to have ever occurred is probably worth a little breathing room.

1. lp bug #1484598

Changed in swift:
importance:	Critical → High

Alistair Coles (alistair-coles) on 2016-09-26

tags:

added: ec

Revision history for this message

Alistair Coles (alistair-coles) wrote on 2016-09-28:

#7

fix proposed here https://review.openstack.org/376630

John Dickinson (notmyname) on 2016-10-19

Changed in swift:
importance:	High → Critical

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-11-07: Fix merged to swift (master)

#8

Reviewed: https://review.openstack.org/376630
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=6574ce31eeda1bdffa2534f21eed7ab0d62bce97
Submitter: Jenkins
Branch: master

commit 6574ce31eeda1bdffa2534f21eed7ab0d62bce97
Author: Alistair Coles <email address hidden>
Date: Mon Sep 26 17:25:09 2016 +0100

EC: reconstruct using non-durable fragments

    Previously the reconstructor would only reconstruct a missing fragment
    when a set of ec_ndata other fragments was available, *all* of which
    were durable. Since change [1] it has been possible to retrieve
    non-durable fragments from object servers. This patch changes the
    reconstructor to take advantage of [1] and use non-durable fragments.

    A new probe test is added to test scenarios with a mix of failed and
    non-durable nodes. The existing probe tests in
    test_reconstructor_rebuild.py and test_reconstructor_durable.py were
    broken. These were intended to simulate cases where combinations of
    nodes were either failed or had non-durable fragments, but the test
    scenarios defined were not actually created - every test scenario
    broke only one node instead of the intent of breaking multiple
    nodes. The existing tests have been refactored to re-use most of their
    setup and assertion code, and merged with the new test into a single
    class in test_reconstructor_rebuild.py.

test_reconstructor_durable.py is removed.

[1] Related-Change: I2310981fd1c4622ff5d1a739cbcc59637ffe3fc3

    Change-Id: Ic0cdbc7cee657cea0330c2eb1edabe8eb52c0567
    Co-Authored-By: Clay Gerrard <email address hidden>
    Closes-Bug: #1624088

Changed in swift:
status:	Confirmed → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-11-18: Fix included in openstack/swift 2.11.0

#9

This issue was fixed in the openstack/swift 2.11.0 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-11-22: Fix proposed to swift (feature/hummingbird)

#10

Fix proposed to branch: feature/hummingbird
Review: https://review.openstack.org/400985

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-11-23: Fix merged to swift (feature/hummingbird)

#11

Download full text (78.0 KiB)

Reviewed: https://review.openstack.org/400985
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=0c3f8f87104af8717115c5badffd243dbaa1c430
Submitter: Jenkins
Branch: feature/hummingbird

commit 2d25fe6ad3573b2a06b6b3e5e66493d7b0c55693
Author: Tim Burke <email address hidden>
Date: Mon Jul 25 15:06:23 2016 -0700

Reduce backend requests for SLO If-Match / HEAD requests

... by storing SLO Etag and size in sysmeta.

    Previously, we had to GET the manifest for every HEAD or conditional
    request to an SLO. Worse, since SLO PUTs require that we HEAD every
    segment, we'd GET all included sub-SLO manifests. This was necessary so
    we could recompute the large object's Etag and content-length.

Since we already know both of those during PUT, we'll now store it in
object sysmeta. This allows us to:

     * satisfy HEAD requests based purely off of the manifest's HEAD
       response, and
     * perform the If-(None-)Match comparison on the object server, without
       any additional subrequests.

    Note that the large object content-length can't just be parsed from
    content-type -- with fast-POST enabled, the content-type coming out of
    the object-server won't necessarily include swift_bytes.

    Also note that we must still fall back to GETting the manifest if the
    sysmeta headers were not found. Otherwise, we'd break existing large
    objects.

Change-Id: Ia6ad32354105515560b005cea750aa64a88c96f9

commit ae7dddd801e28217d7dc46bd45cd6b621f29340c
Author: Ondřej Nový <email address hidden>
Date: Mon Nov 21 22:13:11 2016 +0100

Added comment for "user" option in drive-audit config

Change-Id: I24362826bee85ac3304e9b63504c9465da673014

commit c3e1d847f4b9d6cc6212aae4dc1b1e6dff45fb40
Author: Thiago da Silva <email address hidden>
Date: Thu Nov 17 17:17:00 2016 -0500

breaking down tests.py into smaller pieces

tests.py is currently at ~5500 lines of code, it's
time to break it down into smaller files.

    I started with an easy middleware set of tests
    (i.e., versioned writes, ~600 lines of code ) so I can get
    some feedback. There are more complicated tests that cover
    multiple middlewares for example, it is not so clear where
    those should go.

Change-Id: I2aa6c18ee5b68d0aae73cc6add8cac6fbf7f33da
Signed-off-by: Thiago da Silva <email address hidden>

commit 5d7a3a4172f0f11ab870252eec784cf24b247dea
Author: Ondřej Nový <email address hidden>
Date: Sat Nov 19 23:24:30 2016 +0100

Removed "in-process-" from func env tox name

This shorten shebang in infra, because we are hitting 128 bytes limit.

Change-Id: I02477d81b836df71780942189d37d616944c4dce

commit 9ea340256996a03c8c744201297b47a0e91fe65b
Author: Kota Tsuyuzaki <email address hidden>
Date: Fri Nov 18 01:50:11 2016 -0800

Don't overwrite built-in 'id'

This is a follow up for https://review.openstack.org/#/c/399237

'id' is assigned as a builtin function so that we should not use 'id'
for the local variable name.

Change-Id: Ic27460d49e68f6cd50bda1d5b3810e01ccb07a37

commit bf...

Reviewed:  https://review.openstack.org/400985
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=0c3f8f87104af8717115c5badffd243dbaa1c430
Submitter: Jenkins
Branch:    feature/hummingbird

commit 2d25fe6ad3573b2a06b6b3e5e66493d7b0c55693
Author: Tim Burke <tim.burke@gmail.com>
Date:   Mon Jul 25 15:06:23 2016 -0700

Reduce backend requests for SLO If-Match / HEAD requests
    
    ... by storing SLO Etag and size in sysmeta.
    
    Previously, we had to GET the manifest for every HEAD or conditional
    request to an SLO. Worse, since SLO PUTs require that we HEAD every
    segment, we'd GET all included sub-SLO manifests. This was necessary so
    we could recompute the large object's Etag and content-length.
    
    Since we already know both of those during PUT, we'll now store it in
    object sysmeta. This allows us to:
    
     * satisfy HEAD requests based purely off of the manifest's HEAD
       response, and
     * perform the If-(None-)Match comparison on the object server, without
       any additional subrequests.
    
    Note that the large object content-length can't just be parsed from
    content-type -- with fast-POST enabled, the content-type coming out of
    the object-server won't necessarily include swift_bytes.
    
    Also note that we must still fall back to GETting the manifest if the
    sysmeta headers were not found. Otherwise, we'd break existing large
    objects.
    
    Change-Id: Ia6ad32354105515560b005cea750aa64a88c96f9

commit ae7dddd801e28217d7dc46bd45cd6b621f29340c
Author: Ondřej Nový <ondrej.novy@firma.seznam.cz>
Date:   Mon Nov 21 22:13:11 2016 +0100

Added comment for "user" option in drive-audit config
    
    Change-Id: I24362826bee85ac3304e9b63504c9465da673014

commit c3e1d847f4b9d6cc6212aae4dc1b1e6dff45fb40
Author: Thiago da Silva <thiago@redhat.com>
Date:   Thu Nov 17 17:17:00 2016 -0500

breaking down tests.py into smaller pieces
    
    tests.py is currently at ~5500 lines of code, it's
    time to break it down into smaller files.
    
    I started with an easy middleware set of tests
    (i.e., versioned writes, ~600 lines of code ) so I can get
    some feedback. There are more complicated tests that cover
    multiple middlewares for example, it is not so clear where
    those should go.
    
    Change-Id: I2aa6c18ee5b68d0aae73cc6add8cac6fbf7f33da
    Signed-off-by: Thiago da Silva <thiago@redhat.com>

commit 5d7a3a4172f0f11ab870252eec784cf24b247dea
Author: Ondřej Nový <ondrej.novy@firma.seznam.cz>
Date:   Sat Nov 19 23:24:30 2016 +0100

Removed "in-process-" from func env tox name
    
    This shorten shebang in infra, because we are hitting 128 bytes limit.
    
    Change-Id: I02477d81b836df71780942189d37d616944c4dce

commit 9ea340256996a03c8c744201297b47a0e91fe65b
Author: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Date:   Fri Nov 18 01:50:11 2016 -0800

Don't overwrite built-in 'id'
    
    This is a follow up for https://review.openstack.org/#/c/399237
    
    'id' is assigned as a builtin function so that we should not use 'id'
    for the local variable name.
    
    Change-Id: Ic27460d49e68f6cd50bda1d5b3810e01ccb07a37

commit bf473ddc9d3fcb3a3fe516b5221a588c5b196a1d
Author: John Dickinson <me@not.mn>
Date:   Wed Nov 9 12:37:43 2016 -0800

2.11.0 release notes
    
    I have reordered the Contributors section of the AUTHORS
    file to simply be ordered by the first character in the
    author name.
    
    Change-Id: I824022e582cadc361484b20c411840f46c4c5b50

commit 2e7a7347fc58676fbaabce3d87a15866796d32e4
Author: Tim Burke <tim.burke@gmail.com>
Date:   Thu Nov 17 13:02:06 2016 -0800

Avoid infinite loop while placing parts
    
    Previously, we could over-assign how many parts should be in a tier.
    This would cause the local `parts` variable to go negative, which meant
    that our `while parts` loop would never terminate.
    
    Change-Id: Id7e7889742ca37cf1a9c0d55fba78d967e90e8d0
    Closes-Bug: 1642538

commit b94d0db9aaf117080bdcebda4b44bd02eb25749e
Author: Bryan Keller <kellerbr@us.ibm.com>
Date:   Wed Nov 16 22:16:53 2016 +0000

Correctly send 412 Precondition Failed in copy middleware
    
    Previously in copy middleware, if a user entered an invalid destination
    path with an invalid `container/object` path the server would return
    a 500 Internal Server Error. However, the correct response should be
    a 412 Precondition Failed. This patch updates copy so that it catches
    the 412 Precondition Failed exception and returns it to the client.
    
    Closes-Bug: #1641980
    
    Change-Id: Ic4677ae033d05b8730c6ad1041bd9c07268e11a9

commit ba1f285fe6fc8729d7961a34023f2b29c00bac39
Author: zhangyanxian <zhangyanxianmail@163.com>
Date:   Thu Nov 17 02:06:25 2016 +0000

Fix typos in test_container.py
    
    TrivialFix
    
    Change-Id: I6d3422a88b6fd6d2a495d17ff75d292ac96cc251

commit e8a5448b0767026925b5ad396f6de1be79ac2325
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue Nov 15 14:35:42 2016 -0800

Add X-Openstack-Request-Id to Access-Control-Expose-Headers
    
    Change-Id: Ib95a693042f0b3cf204033eb5957660cb3573dcf
    Related-Change: I56cd4738808b99c0a08463f83c100be51a62db05

commit f850ff065e70fb56f9b9e224194553770787f516
Author: Tim Burke <tim.burke@gmail.com>
Date:   Sat Oct 29 22:00:01 2016 +0200

SLO: Concurrently HEAD segments
    
    Before creating a static large object, we must verify that all of the
    referenced segments exist. Previously, this was done sequentially; due
    to latency between proxy and object nodes, clients must be careful to
    either keep their segment count low or use very long (minute+) timeouts.
    We mitigate this somewhat by enforcing a hard limit on segment count,
    but even then, HEADing a thousand segments (the default limit) with an
    average latency of (say) 100ms will require more than a minute and a
    half.
    
    Further, the nested-SLO approach requires multiple requests from the
    client -- as a result, Swift3 is in the position of enforcing a lower
    limit than S3's 10,000 (which will break some clients) or requiring that
    clients have timeouts on the order of 15-20 minutes (!).
    
    Now, we'll perform the segment HEADs in parallel, with a concurrency
    factor set by the operator. This is very similar to (and builds upon)
    the parallel-bulk-delete work. By default, two HEAD requests will be
    allowed at a time.
    
    As a side-effect, we'll also only ever HEAD a path once per manifest.
    Previously, if a manifest alternated between two paths repeatedly (for
    instance, because the user wanted to splice together various ranges from
    two sub-SLOs), then each entry in the manifest would trigger a fresh
    HEAD request.
    
    Upgrade Consideration
    =====================
    If operators would like to preserve the prior (single-threaded) SLO
    creation behavior, they must add the following line to their
    [filter:slo] proxy config section:
    
       concurrency = 1
    
    This may be done prior to upgrading Swift.
    
    UpgradeImpact
    Closes-Bug: #1637133
    Related-Change: I128374d74a4cef7a479b221fd15eec785cc4694a
    Change-Id: I567949567ecdbd94fa06d1dd5d3cdab0d97207b6

commit 40152f4242dfe46a4c63ae69b0a0e1c83172f556
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue Nov 15 14:12:02 2016 -0800

Fix up X-Trans-Id-Extra docs
    
    Previously, they would render as "X -Trans-Id-Extra"
    
    Change-Id: Ibaf9b365467a2c91a0f03baed4d7bc36b0fa9967

commit 8e122c6fe8aa0f3e3b96d124023728b98820147e
Author: Janie Richling <jrichli@us.ibm.com>
Date:   Tue Nov 15 13:15:59 2016 -0600

Document X-Openstack-Request-Id in all responses
    
    Swift returns `X-Openstack-Request-Id` for container and
    object requests as well as account.  A couple api-ref docs are
    missing this value in the response examples.
    
    Change-Id: Ifcd67a620e04be5e92b43c7749ee4acb50bb171d

commit d25216bddabc77fb54c514269ee55800c87f558c
Author: Thiago da Silva <thiago@redhat.com>
Date:   Mon May 16 15:57:47 2016 -0400

remove double checks on account/container info
    
    Continuing the clean up in account and container
    info, removed duplicated validation from account_info
    and container_info methods, since the same validations
    were recently added to get_account_info and get_container_info.
    
    Change-Id: I1ad745fe809367d22649c83f38c4de7a74cac44e
    Signed-off-by: Thiago da Silva <thiago@redhat.com>

commit 721cd1634a82b735d0d65062c15c30fffb204a0e
Author: Alistair Coles <alistair.coles@hpe.com>
Date:   Fri Nov 11 16:34:15 2016 +0000

Show file parameters as mandatory in swift-*-info man pages
    
    Changes the file parameters on command line to use the convention of
    <file> for a mandatory option rather than [file] which suggests an
    optional flag. Also drop to lower case as is convention in other man
    pages.
    
    Change-Id: Icfb4b4786343eb493a73b092cad1fdadc935d066

commit 243dd97c5848c5ef634d630fcc945bd12aff102f
Author: Mohit Motiani <mohit.motiani@intel.com>
Date:   Tue Oct 25 21:13:55 2016 +0000

Add OPTIONS in manpages
    
    swift-account-info, swift-container-info, swift-object-info
    and swift-get-node do not contain any information about the available
    options. This patch adds OPTIONS in these manpages.
    
    Change-Id: I832c621460ee6bf19adac8e0aadeab32be4b8da0

commit b69caeae7d717cd0b5c1d643d7f7fc12e09530a5
Author: Christian Hugo <hugo.christian@web.de>
Date:   Sun Oct 30 12:07:29 2016 +0100

Include debug message for rsync tempfiles
    
    The warning message for rsync tempfiles was removed in the related
    change.  However, because our regex match might result in a false
    positive maybe it's still useful to log a debug message.  Instead of
    silently ignoring rsync tempfiles, when running in debug we note the
    file and how we classified it - but still no warning will occur.
    
    I also consolidate our use of the regex for rsync tempfiles into the
    diskfile module and move the negative test for the warning logger
    message a little next to the positive test.
    
    Change-Id: Idb2a1a76aa275c9c2e9bad8aceea913b8f5b1c71
    Related-Change: I5a5d6e24710e4880776b32edcbc07021acf77676

commit e6dea4d69995272724c772f33385530f5a35bd89
Author: John Dickinson <me@not.mn>
Date:   Mon Oct 3 17:18:22 2016 -0700

adding reno sphinx tree
    
    This should give the necessary tooling to set up the right
    links in the right places so Swift has release notes listed
    alongside other projects.
    
    Change-Id: I4e5f1ce1fcfbb2943036c821a24a0b4a3a2d9fc8

commit 5de9747e1629565653a3f15d28bc4b4cb03c4abd
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue Nov 8 12:32:18 2016 -0800

Stop shadowing built-in iter
    
    Change-Id: I0e3fcddfef6968353e4c0aa6871a2e7a22598459

commit 2f31c0469dd48c3a7052faeb8b5d786d5a32c21a
Author: Lokesh S <lokesh.s@hp.com>
Date:   Thu Jul 28 13:31:47 2016 +0000

Py3: Fixes header key dict
    
    Decode header to latin1 on python3
    encode header to utf-8 on python2.
    
    Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>
    
    Change-Id: I10f205a05bb3a566e52a597d9315b3a8b8c14664

commit f4adb2f28f5517772174aac18e3fe9ab1d08c913
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Mon Nov 7 18:06:44 2016 -0800

Fix ZeroDivisionError in reconstructor.stats_line
    
    Despite a check to prevent zero values in the denominator python
    integer division could result in ZeroDivisionError in the compute_eta
    helper function.  Make sure we always have a non-zero value even if it
    is small.
    
    NotImplemented:
    
     * stats calculation is still not great, see lp bug #1488608
    
    Closes-Bug: #1549110
    Change-Id: I54f2081c92c2a0b8f02c31e82f44f4250043d837

commit c2ce92acd6fa25706e271dc2b1504a8db54bd2ae
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Tue Oct 11 13:23:11 2016 -0700

Fix signal handling for daemons with InternalClient
    
    The intentional use of "bare except" handling in catch_errors and some
    daemons to prevent propagation on unexpected errors that do not
    inherit from Exception (like eventlet.Timeout) or even BaseException
    (like old-style classes) has the side effect of spuriously "handling"
    *expected* errors like when a signal handler raises SystemExit.
    
    The signal handler installed in our Daemon is intended to ensure first
    that the entire process group and any forked processes (like rsync's)
    receive the SIGTERM signal and also that the process itself
    terminates.
    
    The use of sys.exit was not a concious grandiose plans for graceful
    shutdown (like the running[0] = False trick that wsgi server parent
    process do) - the desired behavior for SIGTERM is to stop - hard.
    
    This change ensures the original goals and intentions of our signal
    handler are fulfilled without the undesirable side effect that can
    cause our daemons to confusingly log an expected message to stop as an
    unexpected error, and start ignoring additional SIGTERM messages;
    forcing our kind operators to resort to brutal process murder.
    
    Closes-Bug: #1489209
    Change-Id: I9d2886611f6db2498cd6a8f81a58f2a611f40905

commit 116462d8a5a45ec876403234cf83b96dfad58ddd
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Fri Sep 16 11:58:46 2016 -0700

Add probetest for response with duplicate frags
    
    When an SSYNC connection fails after shipping a fragment, but before
    cleaning itself up - it can lead to multiple replicas of the same
    fragment index serviceable in the node_iter at the same time.  Or
    perhaps more likely if a partner nodes win a race to rebuild waiting
    on a handoff.  In this case, the proxy need not fail to service a GET
    just because of extra information.  A similar guard was added to the
    reconstructor in a related change [1].
    
    This underlying bug is acctually fixed by the related change [2], this
    probetest is just encoding the failure to help with future maintenance.
    
    1. Related-Change: I91f3d4af52cbc66c9f7ce00726f247b5462e66f9
    2. Related-Change: I2310981fd1c4622ff5d1a739cbcc59637ffe3fc3
    
    Closes-Bug: 1624176
    
    Change-Id: I06fc381a25613585dd18916d50e7ca2c68d406b6

commit 6574ce31eeda1bdffa2534f21eed7ab0d62bce97
Author: Alistair Coles <alistair.coles@hpe.com>
Date:   Mon Sep 26 17:25:09 2016 +0100

EC: reconstruct using non-durable fragments
    
    Previously the reconstructor would only reconstruct a missing fragment
    when a set of ec_ndata other fragments was available, *all* of which
    were durable. Since change [1] it has been possible to retrieve
    non-durable fragments from object servers. This patch changes the
    reconstructor to take advantage of [1] and use non-durable fragments.
    
    A new probe test is added to test scenarios with a mix of failed and
    non-durable nodes. The existing probe tests in
    test_reconstructor_rebuild.py and test_reconstructor_durable.py were
    broken. These were intended to simulate cases where combinations of
    nodes were either failed or had non-durable fragments, but the test
    scenarios defined were not actually created - every test scenario
    broke only one node instead of the intent of breaking multiple
    nodes. The existing tests have been refactored to re-use most of their
    setup and assertion code, and merged with the new test into a single
    class in test_reconstructor_rebuild.py.
    
    test_reconstructor_durable.py is removed.
    
    [1] Related-Change: I2310981fd1c4622ff5d1a739cbcc59637ffe3fc3
    
    Change-Id: Ic0cdbc7cee657cea0330c2eb1edabe8eb52c0567
    Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
    Closes-Bug: #1624088

commit a4d77d918d99dfa8a1e7f1b0561769a17ba64932
Author: Alistair Coles <alistair.coles@hpe.com>
Date:   Thu Nov 3 15:19:25 2016 +0000

Fix KeyError when auditor finds an expired object
    
    Since the related change [1] the auditor checks deleted objects in
    case their tombstone is ready to be reclaimed. However, the auditor
    mistakes an expired object for a deleted object because
    DiskFileExpired is a subclass of DiskFileDeleted. This causes a
    KeyError to be raised and logged because the expired object has no
    tombstone info in the ondisk_info data structure.
    
    This patch makes the auditor catch and ignore DiskFileExpired
    exceptions before handling DiskFileDeleted exceptions.
    
    [1] Related-Change: I3e99dc702d55a7424c6482969e03cb4afac854a4
    
    Change-Id: I9872b6997d09dcfd8a868c4b91b9379407283b8e
    Closes-Bug: #1638016

commit 3b889331b30f9a02efc38bc451b2ab625caffdd1
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Tue Sep 20 23:44:32 2016 -0700

Make NullLogger more logger like
    
    eventlet.wsgi.server is adaptive to dealing with the named log
    argument being either a file like or a log like [1], but message
    handling is a more direct pass through if it's a log like.
    
    Since all we want to do is make eventlet's logging it a noop - it's
    easy to provide either interface!
    
    1. https://github.com/eventlet/eventlet/blob/4d2cdc/eventlet/wsgi.py#L246
    
    Change-Id: I2d792176c96931eafb3f140e6653ba8b31eda429

commit 300d388825c228ae1f51cdb8abcdf4c76ec24637
Author: Gábor Antal <antal@inf.u-szeged.hu>
Date:   Fri Jul 15 14:11:09 2016 +0200

Use more specific asserts in test/probe tests
    
    I changed asserts with more specific assert methods.
    
    e.g.: from assertTrue(sth == None) to assertIsNone(*) or
    assertTrue(isinstance(inst, type)) to assertIsInstace(inst, type) or
    assertTrue(not sth) to assertFalse(sth).
    
    The code gets more readable, and a better description will be shown on fail.
    
    Change-Id: I3768faa568e3964e726ecc48ac8cb133cb088284

commit 2a75091c58948fb664016c0e91e72acd313e4610
Author: Alistair Coles <alistair.coles@hpe.com>
Date:   Mon Oct 17 20:38:52 2016 +0100

Make ECDiskFileReader check fragment metadata
    
    This patch makes the ECDiskFileReader check the validity of EC
    fragment metadata as it reads chunks from disk and quarantine a
    diskfile with bad metadata. This in turn means that both the object
    auditor and a proxy GET request will cause bad EC fragments to be
    quarantined.
    
    This change is motivated by bug 1631144 which may result in corrupt EC
    fragments being written to disk but appear valid to the object auditor
    md5 hash and content-length checks.
    
    NotImplemented:
    
     * perform metadata check when a read starts on any frag_size
       boundary, not just at zero
    
    Related-Bug: #1631144
    Closes-Bug: #1633647
    
    Change-Id: Ifa6a7f8aaca94c7d39f4aeb9d4fa3f59c4f6ee13
    Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
    Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>

commit e166e19b235e1fba8e5eba5720d1c07fe32c40c1
Author: Kazuhiro MIYAHARA <miyahara.kazuhiro@lab.ntt.co.jp>
Date:   Thu Oct 20 14:01:14 2016 +0900

Remove duplicated 'User-Agent' header from object-updater's requests
    
    When Object-Server saves async-update in pickle files,
    'User-Agent: object-server %(pid)' (the key is title style) is included
    in the pickles. However, Object-Updater will add
    'user-agent: object-updater %(pid)' (the key is lower case style) to the
    pickled headers and use it to make http connection. According to RFC
    7230, each header field consists of a case-insensitive field name,
    therefore either of 'User-Agent' and 'user-agent' should not be included
    in the headers.
    
    This patch removes old 'User-Agent' header from object-updater's
    requests.
    
    Change-Id: Ia624558395584457c718b311fe80e1a8406e22ad
    Closes-Bug:#1635114

commit 33c18c579e12474015bdd4995120e9f47e351e72
Author: Ondřej Nový <ondrej.novy@firma.seznam.cz>
Date:   Sat Oct 29 15:00:13 2016 +0200

Remove executable flag from some test modules
    
    Change-Id: I36560c2b54c43d1674b007b8105200869b5f7987

commit 2bd8d050fb5dec140ad23d5db1d281049954255b
Author: Christian Hugo <hugo.christian@web.de>
Date:   Sun Oct 30 12:07:29 2016 +0100

Suppress unexpected file warnings for rsync temp files
    
    Do not log unexpected file warning for rsync temp files when the parsing
    of the timespamp fails.  If the file passes a regex test suppress the
    logger waring, but still return it as an unexpected file from
    get_ondisk_files.
    
    Closes-Bug: 1616504
    
    Change-Id: I5a5d6e24710e4880776b32edcbc07021acf77676

commit e3e457da662e92885620981a445530fbd8370604
Author: Ondřej Nový <ondrej.novy@firma.seznam.cz>
Date:   Sun Nov 1 19:32:19 2015 +0100

Respect server type for --md5 check in swift-recon
    
    MD5 sum compare of server ring ignores server_type parameter
    now. This could be confusing and this should only test
    specified server_type, same as --replication, --validate-servers,
    etc.
    
    Change-Id: I9095309c24f42aec1330757c88e7b242b2a6a865

commit d0fe922904c8016267e89c97bef16052a2c3bb7e
Author: karen chan <karen@karen-chan.com>
Date:   Mon Oct 17 03:37:35 2016 -0700

Mirror X-Trans-Id to X-Openstack-Request-Id
    
    Many other OpenStack services use a `[X-]OpenStack-Request-Id` header
    to return a unique identifier for the request.  Swift will now return
    `X-Trans-Id` as well as `X-Openstack-Request-Id`.
    
    Change-Id: I56cd4738808b99c0a08463f83c100be51a62db05
    Closes-Bug: #1572786

commit c91ca5335fb86b3b260d88285fe0248d502276f3
Author: Tim Burke <tim.burke@gmail.com>
Date:   Sat Oct 29 18:29:48 2016 +0200

Ignore Range values like "bytes=--0"
    
    This is consistent with what we already do for other
    semantically-invalid values like "bytes=--1". Previously, we would
    return a 416 Requested Range Not Satisfiable response like we do for the
    semantically-valid-but-not-really-meaningful "bytes=-0"
    
    Change-Id: I932b42406c9a5ee7eaa6978a655e61022a957415

commit 30f672e7201e214fa625a43dcad867e5936e1cef
Author: Tim Burke <tim.burke@gmail.com>
Date:   Fri Oct 28 13:07:27 2016 +0200

Refactor SLO If-Match / HEAD tests
    
    Previously, we didn't make any assertions about the backend requests
    but rather just verified the user-visible behavior.
    
    Change-Id: Iddd4b705ee9b724a4a8a88c6fbaff36cca9612cf

commit c3bc9f31ca6c99c4d54478ca7846bbfad2f26427
Author: Hanxi Liu <hanxi.liu@easystack.cn>
Date:   Tue Oct 4 14:55:27 2016 +0800

Add links for more detailed overview in overview_architecture
    
    Change-Id: Ie1f171d0945e3f1f4daa61bc66786c01e68e9a0a

commit 99a13d93861240bfdc48730913bd6e445b645932
Author: Ondřej Nový <ondrej.novy@firma.seznam.cz>
Date:   Wed Oct 19 20:17:00 2016 +0200

Fixed rysnc -> rsync typo
    
    Change-Id: I671b4206072c6e22f4ae38033502336ec32e86ad

commit 9847796f01571dc7be332707dc935efac66ee5d4
Author: Ondřej Nový <ondrej.novy@firma.seznam.cz>
Date:   Mon Oct 17 19:46:57 2016 +0200

Set owner of drive-audit recon cache to swift user
    
    Fixies this problem:
    * swift-drive-audit needs to be run by root, because only root have
      "umount" permission
    * swift-object servers typically runs as user swift
    * if swift-drive-audit is run by root, /var/cache/swift/drive.recon is
      owned by root, with 0o600
    * recon middleware (inside swift-object-server) can't read this cache
      file: swift-object: Error reading recon cache file
    
    This patch adds "user" option to drive-audit config file. Recon cache
    is chowned to this user.
    
    Change-Id: Ibf20543ee690b7c5a37fabd1540fd5c0c7b638c9

commit a8b80f0727bcf760026bfa76539baec6fe706ce8
Author: Thiago da Silva <thiago@redhat.com>
Date:   Tue Oct 18 15:33:02 2016 -0400

update urls to newton
    
    Change-Id: I9eb8c004e100b16de906d340b54e154cf21ddaab
    Signed-off-by: Thiago da Silva <thiago@redhat.com>

commit 5064ebb7441fdf02e2ca895a2ae81102e751d978
Author: Charles Hsu <charles0126@gmail.com>
Date:   Wed Oct 12 18:02:16 2016 +0800

Make Content-Disposition support `inline; filename=...` format.
    
    A client can process a object contents immediately when it has knowledge,
    if it doesn't, it stores the content to local with the filename that
    assigned in `Content-Disposition`.
    
    According to https://tools.ietf.org/html/rfc2183, `filename` can be use
    whatever `inline` or `attachment` (disposition-type) in the disposition
    string.
    
    Change-Id: Iba94aedfad06c1dc8bd7be5eb3c73b33fb8d5198

commit c3bc83013f94671c48df18858c566d6cf58a89dc
Author: Nandini Tata <nandini.tata@intel.com>
Date:   Thu Oct 13 18:05:05 2016 +0000

SAIO install bash scripts for a novice install
    
    Added SAIO bash scripts as an option under developer tools. They are
    well commented and simple to understand for a novice developer.
    
    Change-Id: Id1b2babe420f4d10e564256fac5a54b60659c5e5
    Co-Authored-By: Paul Dardeau <paul.dardeau@intel.com>

commit 3218f8b064e462d901466b04a4813e15ec96da85
Author: Alistair Coles <alistair.coles@hpe.com>
Date:   Wed Oct 12 20:00:13 2016 +0100

Prevent ssync writing bad fragment data to diskfile
    
    Previously, if a reconstructor sync type job failed to provide
    sufficient bytes from a reconstructed fragment body iterator to match
    the content-length that the ssync sender had already sent to the ssync
    receiver, the sender would still proceed to send the next
    subrequest. The ssync receiver might then write the start of the next
    subrequest to the partially complete diskfile for the previous
    subrequest (including writing subrequest headers to that diskfile)
    until it has received content-length bytes.
    
    Since a reconstructor ssync job does not send an ETag header (it
    cannot because it does not know the ETag of a reconstructed fragment
    until it has been sent) then the receiving object server does not
    detect the "bad" data written to the fragment diskfile, and worse,
    will label it with an ETag that matches the md5 sum of the bad
    data. The bad fragment file will therefore appear good to the auditor.
    
    There is no easy way for the ssync sender to communicate a lack of
    source data to the receiver other than by disconnecting the
    session. So this patch adds a check in the ssync sender that the sent
    byte count is equal to the sent Content-Length header value for each
    subrequest, and disconnect if a mismatch is detected.
    
    The disconnect prevents the receiver finalizing the bad diskfile, but
    also prevents subsequent fragments in the ssync job being sync'd until
    the next cycle.
    
    Closes-Bug: #1631144
    Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
    
    Change-Id: I54068906efdb9cd58fcdc6eae7c2163ea92afb9d

commit 77f5b201248c0684d87289540d3f94873f5c5e38
Author: Christian Schwede <cschwede@redhat.com>
Date:   Fri Oct 7 12:17:08 2016 +0000

Throttle update_auditor_status calls
    
    If there are quite a few nearly empty partitions per disk you might see
    some write load even if your cluster is unused. The auditor will update
    the status file after every partition, and this might happen multiple
    times within a second if there is not much data stored yet.
    
    This patch throttles updates, and will only write out an updated status
    if the file was last updated more than a minute ago.
    
    Closes-Bug: 1631352
    Change-Id: Ib61ec9cd945e6b2d28756f6ca47801674a7e6060

commit f3be45c0598fc6f2afd51e8e993fa97a61cbb454
Author: zheng yin <yin.zheng@easystack.cn>
Date:   Tue Aug 16 19:17:34 2016 +0800

Simplify container_updater test code
    
    This patch simplifies the container_updater unit test code by
    refactoring the creation of the continer_updater into a helper method.
    This method by default will use a default configuration. However
    this configuration can be changed by passing in a dictionary of
    configuration elements to update, for example:
    
        self._get_container_updater({'mount_check': 'true'})
    
    Change-Id: Ide211d559bf9381c520a04cd90d278d9b2beae50

commit 19102c6e7f501241f36e5e40d27ab3ba5dcc6655
Author: Ondřej Nový <ondrej.novy@firma.seznam.cz>
Date:   Thu Aug 18 09:48:45 2016 +0200

Apply bash error handling consistently in all bash scripts
    
    Change-Id: I36dcff9cc2593bcfb6aebbe84b59d95c711eb3e4

commit b13b49a27caac17ae55b19f315d5ce31801c9522
Author: Alistair Coles <alistair.coles@hpe.com>
Date:   Tue Aug 9 16:09:38 2016 +0100

EC - eliminate .durable files
    
    Instead of using a separate .durable file to indicate
    the durable status of a .data file, rename the .data
    to include a durable marker in the filename. This saves
    one inode for every EC fragment archive.
    
    An EC policy PUT will, as before, first rename a temp
    file to:
    
       <timestamp>#<frag_index>.data
    
    but now, when the object is committed, that file will be
    renamed:
    
       <timestamp>#<frag_index>#d.data
    
    with the '#d' suffix marking the data file as durable.
    
    Diskfile suffix hashing returns the same result when the
    new durable-data filename or the legacy durable file is
    found in an object directory. A fragment archive that has
    been created on an upgraded object server will therefore
    appear to be in the same state, as far as the consistency
    engine is concerned, as the same fragment archive created
    on an older object server.
    
    Since legacy .durable files will still exist in deployed
    clusters, many of the unit tests scenarios have been
    duplicated for both new durable-data filenames and legacy
    durable files.
    
    Change-Id: I6f1f62d47be0b0ac7919888c77480a636f11f607

commit f0a8af293e1d30cbed4be89a0c0bbcb7b79db6b2
Author: Nicolas Helgeson <nh202b@att.com>
Date:   Thu Oct 6 11:06:49 2016 -0700

Update pyeclib dependency to 1.3.1
    
    Updated swift requirements.txt to change PyECLib version
    so that it fixes memory leak issue.
    
    Change-Id: Ifc069989b96188fd7abe0d8edc6b30266783ad79
    Partial-Bug: #1629410

commit 19426182af0064b8143ee57a29998523ad52273b
Author: Nguyen Phuong An <AnNP@vn.fujitsu.com>
Date:   Wed Sep 7 14:02:10 2016 +0700

[api-ref] Remove temporary block in conf.py
    
    The os-api-ref 1.0.0 is out. We can remove the support to older version now.
    Reference:
    https://github.com/openstack/senlin/commit/6d41feb58dd836d426c9b6f025846dbbf7295097
    
    Change-Id: I0b28cd3f6b5c393d63b9ab2b4dcd3b7b3c20f9b4

commit f62df7b80c083862c11cbef678f75cc09bb89e8f
Author: Pete Zaitcev <zaitcev@kotori.zaitcev.us>
Date:   Mon Oct 3 21:08:15 2016 -0600

Add a configurable URL base to staticweb
    
    This came to light because someone ran Tempest against a standard
    installation of RDO, which helpfuly terminates SSL for Swift in
    a pre-configured load-balancer. In such a case, staticweb has no
    way to know what scheme to use and guesses wrong, causing Tempest
    to fail.
    
    Related upstream bug:
     https://bugs.launchpad.net/mos/+bug/1537071
    
    Change-Id: Ie15cf2aff4f7e6bcf68b67ae733c77bb9353587a
    Closes-Bug: 1572011

commit 53aebba90324c0fdcb3ea61275b24db4d5a82110
Author: Luong Anh Tuan <tuanla@vn.fujitsu.com>
Date:   Tue Oct 4 10:06:23 2016 +0700

Fix a typo in documentation
    
    remove redundant 'this'
    
    Change-Id: I8860190d882b255a3d416de685f930d2b2c0ad17

commit 2ec4189e374d1f94009fc61c1b17d00d12f7e99b
Author: Tim Burke <tim.burke@gmail.com>
Date:   Mon Sep 19 17:37:47 2016 -0700

Fix when we set state in Spliterator
    
    Also clean up a comment and some exception text
    
    Change-Id: I1e7755cc0468f9a3ba96a0dd24868f09a10c3df0
    Related-Change: I24716e3271cf3370642e3755447e717fd7d9957c

commit a3dbc139c398be7f2c7af0f843b85d4d53eb81ae
Author: zheng yin <yin.zheng@easystack.cn>
Date:   Thu Sep 29 20:24:23 2016 +0800

add assert error information
    
    There are the same error resp.status_int in different operators of
    one function, then I can't point out which one returns this status_int.
    Therefore, I assert error information.
    
    Change-Id: I58bcba97a9f92e43e0f8ef26b801b69a6906af41

commit ddd40a0645b00b0300d134bbd22e636529fecb43
Author: zheng yin <yin.zheng@easystack.cn>
Date:   Wed Aug 10 20:21:34 2016 +0800

Add unit test content
    
    It lacks of some checks in unit test case. Therefore, I add them and
    make it completion.
    
    Change-Id: Id0d20158ae753d245a541fca5ac4fafeecc9790a

commit 45e4fd5f714326324e13d781673ff01666c1ab97
Author: Luong Anh Tuan <tuanla@vn.fujitsu.com>
Date:   Mon Oct 3 13:28:48 2016 +0700

Fix typo: remove redundant 'that'
    
    Change-Id: I9ed35d866ffb9faa41556c5c00f3b490d35412ab

commit 396baf4139f64c6127a01c4f3f228901a55b1c7d
Author: John Dickinson <me@not.mn>
Date:   Sat Sep 3 10:18:32 2016 -0700

Update test-requirements.txt
    
    The testing requirements should not cause extra packaging
    work for deployers (since these aren't required to deploy).
    This patch brings it in line with the latest global requirements,
    except for python-swiftclient itself (which I kept with no version
    specifier). By being in line with global requirements, this is a
    more accurate list with respect to how things are actually tested
    in the gate.
    
    Change-Id: Iddc92d9bb135144d38c3e5aec2e95df5787e9b5b

commit 7dcfe5a7ecb3f95f4556324dea0b8519b68d68dc
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Fri Sep 30 12:16:58 2016 -0700

Remove unneeded line from ring.utils.dispersion_report
    
    This line is render inert by the condition right above it.
    
    The behavior/correctness is already well vetted by
    ring.test_utils:TestUtils.test_dispersion_report
    
    Change-Id: I7d7a193c52d1005cecd9c70953d55e2b9755a64d

commit 9b98c89983821f709fbea3c14c264150d8613171
Author: Yushiro FURUKAWA <y.furukawa_2@jp.fujitsu.com>
Date:   Thu Sep 29 20:48:21 2016 +0900

Revises 'url' to 'URL' and 'json' to 'JSON'
    
    Change-Id: I44743fbb9bcbce3a50ed6770264ba0f4b17803d7

commit 1004509cab443a5cbb223ffc8258c45cb97ee5fb
Author: Alistair Coles <alistair.coles@hpe.com>
Date:   Thu Sep 29 19:22:36 2016 +0100

Fix failing cross-policy-versioning functional test
    
    Since [1] the functional tests for cross-policy
    object versioning fails with an error. These tests are
    skipped in all CI test jobs because none of them have
    more than one policy enabled. However, when running
    tests against a system with multiple policies the tests
    will not skip and the error is raised.
    
    [1] Related-Change: Ifebc1c3ce558b1df9e576a58a4100f2219dfc7e7
    
    Related-Bug: #1629234
    
    Change-Id: I48028928bc996252a31cbf3abfd2b5e67a74dc95

commit 05642d2958477354ce2807a93417524a11234639
Author: zheng yin <yin.zheng@easystack.cn>
Date:   Fri Sep 30 11:11:20 2016 +0800

fix word spelling mistake
    
    Change-Id: Ia7b03e52b8d6a334fc2b67c94912effe0e659941

commit 03b5e76d0942c075a0286db47bc9e0f029b1e231
Author: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Date:   Mon Sep 26 05:37:08 2016 -0700

Improve functional tests and test client
    
    This patch includes a couple of small functional test improvement.
    
    A. Change swift_test_client.File.sync_metadata to follow Swift object
    metadata semantics:
    
    swift_test_client.File.sync_metadata is designed to post object user
    metadata to an object. However, prior to this patch, the
    swift_test_client.File instance keeps the existing object metadata as
    its member attribute and if sync_metadata is called, it sends both
    existing metadata and incomming metadata from caller. It looks to result
    in the odd state as if Swift keeps the existing metadata when POST
    object requested.
    
    To tell the correct Swift object metadata semantics, when POST object
    requested, the existing metadata in the stored object should be gone
    even if no metadata is overwritten.
    
    i.e.
    if POST object with 'X-Object-Meta-Key: Val' to a stored object with
    'X-Object-Meta-foo: bar', it will result in an object with
    'X-Object-Meta-Key' (note that X-Object-Meta-Foo will be deleted)
    
    The prior behavior sometimes make us confused in the reviw [1] so that,
    this patch fixes it to send only incomming metadata if it's set.
    
    B. Check the response status code more strictly for ObjectVersioning case
    
    This patch fixes test_versioning_check_acl on both TestObjectVersioning and
    TestObjectVersioningHistoryMode to assert the response status code
    explisitly instead of asserting just "ResponseError". (e.g. 403 when trying
    to delete object from other account)
    
    1: https://review.openstack.org/#/c/360933/1/test/functional/tests.py@4142
    
    Change-Id: Ia3e5b40f17dc0f881b695aa4be39c98b91e2bb06

commit 3ae0b80bbd91e5ec3a64657913edc25b8728efcb
Author: Anh Tran <anhtt@vn.fujitsu.com>
Date:   Thu Sep 29 10:09:14 2016 +0700

Remove redundant 'the'
    
    Change-Id: I9f6a11d714e5e9f3c0c6609465225923495aa99a

commit 52473ea39277f6b3d8e7d77564189da6b1aec7c3
Author: Hanxi Liu <hanxi.liu@easystack.cn>
Date:   Wed Sep 28 10:58:30 2016 +0800

Use six.moves.urllib.parse instead of urllib
    
    Six urllib parse contains py3's urllib.parse and py2's urllib.
    Replace urllib with six.moves.urllib.parse to keep compatibility.
    
    Change-Id: Ie67987e4ffb981c2ee70360f7fa9b3fe873c2a96
    Closes-bug: 1280105

commit ccff3668116bd6107003e4f774ab136bb933eb77
Author: Mahati Chamarthy <mahati.chamarthy@gmail.com>
Date:   Mon Jul 25 20:10:44 2016 +0530

Update diskfile docstring for get_ondisk_files
    
    The return value for diskfile.BaseDiskFileManager.get_ondisk_files is
    a bit more rich than the current doc string describes; try to fill in
    some gaps.
    
    Co-Author: Alistair Coles <alistair.coles@hpe.com>
    
    Change-Id: Ief10644825ead124e58f3da5693bbdc1c6dc31ed

commit 2f880f04486ea1b1e15e8794707a7b61ae57d965
Author: Nam Nguyen Hoai <namnh@vn.fujitsu.com>
Date:   Wed Sep 28 15:53:08 2016 +0700

Fix typo in docstring
    
    This patch replaces type "tuble" to "tuple".
    
    Change-Id: I0a54f10df0f48fab9553e6438735e42ae1bb5b3b

commit a09e42732af4fec0a046e1ee402559e1a9e55487
Author: gecong1973 <ge.cong@zte.com.cn>
Date:   Tue Sep 27 17:14:13 2016 +0800

Fix a typo in proxy-server.conf-sample
    
    TrivialFix
    
    Change-Id: If650e25979a9488c93fe93621c905003946c27e5

commit 7d2f8e59ef801092878c92d752316bc74ab8eba8
Author: Ha Van Tu <tuhv@vn.fujitsu.com>
Date:   Tue Sep 27 16:13:05 2016 +0700

Fix typo in docstring
    
    This patch replace typo "contrain" by "constrain".
    
    Change-Id: If91550c4d55f4d072be59470b7f893d56ebd30ca

commit 31c92ac6dbaf843ec7aa3ed8dfa973356096ff5b
Author: Ha Van Tu <tuhv@vn.fujitsu.com>
Date:   Tue Sep 27 11:49:35 2016 +0700

Update link reference in Swift document
    
    The reference link for REST API [1] is out of date. We should change
    it to the new one [2].
    
    [1]:http://developer.openstack.org/api-ref-objectstorage-v1.html
    [2]:http://developer.openstack.org/api-ref/object-storage/
    
    Change-Id: Ib3e5a78ec2df0591f862671272f51ace5ac65ca7

commit 0c8c764547c78df0b00354bb06c5fb27d381647f
Author: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Date:   Fri Aug 26 00:15:45 2016 -0700

Add functional tests for new versioned_write mode
    
    This patch is follow up for [1] and [2] to add new functional
    tests for versioned_writes middlware 'history' mode.
    (i.e. using X-History-Location header to a container).
    
    The new test class, TestObjectHistoryModeVersioning, will use obvious
    setting the mode via new X-History-Location header, since the change [2],
    the setting X-Versions-Mode header added since [1] for incomming request has
    been deprecated. Hence, since [2], the syntax for stack mode is back to
    be same with older Swift than [1] so that the only thing we need now is
    just adding a test suite for the new X-History-location.
    
    It means the API has been changing like:
    ---------------
    For stack mode:
    ---------------
    Older than [1]:
    X-Versions-Location
    
    [1]~[2]:
    X-Vesions-Location (and X-Versions-Mode: 'stack' for obvious)
    
    Newer than [2]:
    X-Vesions-Location
    
    -----------------
    For history mode:
    -----------------
    Older than [1]:
    (Not supported)
    
    [1]~[2]:
    X-Vesions-Location and X-Versions-Mode: 'history'
    
    Newer than [2]:
    X-History-Location
    
    Note that this functional tests work on newer swift than [2].
    
    And then, this patch also sets allow_versioned_writes=True
    for in-process testing (the container server allow_versions
    option was already set, so this is just enabling in the middleware
    too). That means that in-process functional tests (such as run by
    the tox envs func-in-process-*) because history mode requires the
    middleware allow_versioned_writes option to be explicity set to True.
    
    1: https://review.openstack.org/#/c/214922/
    2: https://review.openstack.org/#/c/373537/
    
    Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>
    
    Related-Change: I555dc17fefd0aa9ade681aa156da24e018ebe74b
    Related-Change: Icfd0f481d4e40dd5375c737190aea7ee8dbc3bf9
    Change-Id: Ifebc1c3ce558b1df9e576a58a4100f2219dfc7e7

commit 6bf63d85a7c7c5e444a20176a6c51a7f3941c93f
Author: John Dickinson <me@not.mn>
Date:   Thu Sep 22 17:09:08 2016 -0400

authors/changelog updates for 2.10.0
    
    Added reno releasenotes file.
    
    Change-Id: Ia14875b992aabaf452242b74e9358cfd884adbcc

commit f07dfa202d98c0e9500b71a0040db32183a8106e
Author: zheng yin <yin.zheng@easystack.cn>
Date:   Fri Sep 23 18:07:28 2016 +0800

Add test_PUT/DELETE_bad_timestamp in obj/test_server
    
    It has valid_timestamp() to check timestamps in /POST/PUT/DELETE methods.
    It has test_POST_bad_timestamps in Post, but it has no test_PUT/DELETE_bad_timestamps
    int PUT/DELETE. Therefore,I add them
    
    Change-Id: Ib531f3b7819c1c8dc69a7b51d990581bf1e24dab

commit 3da65be4b3b47728833eb6976a33cd420c887aa8
Author: Alistair Coles <alistair.coles@hpe.com>
Date:   Thu Sep 22 16:56:36 2016 +0100

Cleanup tests for auditor invalidating hashes
    
    Cleans up some auditor tests added in the Related-Change.
    
    Make auditor NOT call invalidate_hash for a reclaimable
    tombstone in a zero bytes file process, so that
    invalidate_hash is only called once per reclaimable
    tombstone per auditor cycle.
    
    Previously each execution of 'swift-init object-auditor once'
    would result in two identical entries being appended to
    hashes.invalid for each reclaimable tombstone. With this change
    that unnecessary duplication is removed.
    
    Related-Change: I3e99dc702d55a7424c6482969e03cb4afac854a4
    Change-Id: I5dfaa8d74c07ca8a494c29159c1a2bed39499613

commit bfaa8e05834d4d3c78069af22387427c12e0e496
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Wed Sep 21 17:33:30 2016 -0700

Fix ChunkWriteError when running unittests
    
    I don't think this is a real bug - just that the mocked iter wasn't
    closing it subiters like the real iter does.
    
    Change-Id: I44c8159f9eea8737bc86b6c7eb59a512e57e86c1

commit 19a684ddedf9259bd389267071f19c1c65b2e836
Author: Luong Anh Tuan <tuanla@vn.fujitsu.com>
Date:   Mon Sep 12 15:49:38 2016 +0700

Fix using filter() to meet python2,3
    
    As mentioned in link[1], if we need filter on python3,
    Raplace filter(lambda obj: test(obj), data) with:
    [obj for obj in data if test(obj)].
    
    [1] https://wiki.openstack.org/wiki/Python3
    
    Change-Id: Ia1ea2ec89e4beb957a4cb358b0d0cef970f23e0a

commit 60a2fe0ba8e59f4abbb5e9c5a9cd093818d5b26c
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue Sep 20 13:54:55 2016 -0700

Use separate headers for versioned_writes' stack and history modes
    
    Now, instead of saying
    
       X-Versions-Location: <container>
       X-Versions-Mode: history
    
    clients should just say
    
       X-History-Location: <container>
    
    Since we've never had a release featuring a user-settable
    X-Versions-Mode header, support may be dropped and that is now ignored.
    
    Change-Id: Icfd0f481d4e40dd5375c737190aea7ee8dbc3bf9

commit 81d4673674febdbe0bba6e27a6d24739456eb3c4
Author: Mahati Chamarthy <mahati.chamarthy@gmail.com>
Date:   Mon Jul 25 20:10:44 2016 +0530

Delete old tombstones
    
    - Call invalidate_hash in auditor for reclaimable tombstones
    - assert changed auditor behavior with a unit test
    - driveby test: assert get_hashes behavior with a unit test
    
    Co-Authored-By: Pete Zaitcev <zaitcev@redhat.com>
    Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
    Closes-Bug: #1301728
    Change-Id: I3e99dc702d55a7424c6482969e03cb4afac854a4

commit 18bb99971f1a793dc75b6b3cb393d5503be43575
Author: Alistair Coles <alistair.coles@hpe.com>
Date:   Mon Sep 19 16:06:18 2016 +0100

Add more comment to authtoken sample options
    
    Prior to the Mitaka release the install guides showed
    services (including Swift) being in a default Keystone
    domain which existed by default and has id=default. This
    domain id is reflected in the proxy-server.conf-sample
    authtoken options and also shown in man page and auth docs.
    
    The Mitaka install guide shows a domain with *name* default
    being created, and having a random UUID assigned, in which
    services are created. This has caused confusion (see
    discussion on linked bug report).
    
    This patch does not change the sample options but does
    add to the comments in order to emphasize that a user
    may need to alter the options to match their Keystone
    configuration.
    
    Change-Id: I17bfcdbd983402eeb561bb704b8b1f1e27547c7d
    Partial-Bug: #1604674

commit 4f033a61b58d94bb63c14aa4a92949bf2c2793f0
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Wed Sep 21 00:28:39 2016 -0700

Fix unstable json.dumps ordering in obj-server test
    
    test_GET_HEAD_with_fragment_preferences seemed to fail on ocassion on
    my machine; the test seemed to be assuming the order of dictionary
    keys in json output sent in headers; the fix was convert back to a
    dictionary when making the assertion on the context of the json.
    
    Change-Id: I1ea1b734c2a9fb12b8f59262bb1229421803e48e
    Closes-Bug: #1625956

commit cbfa79a159aa741b2982e7c009241217dde45f92
Author: Alistair Coles <alistair.coles@hpe.com>
Date:   Tue Sep 6 16:36:36 2016 +0100

api-ref: fix some header definitions
    
    Clarify Content-Type header definition for listings.
    
    Distinguish between request and response definitions for
    X-Account-Meta-Temp-URL-Key* headers.
    
    Insert X-Container-Meta-Quota-* headers missing from some
    request/response definitions.
    
    Improve X-Container-Meta-Access-Control-Expose-Headers
    parameter formatting.
    
    Distinguish between request and response definitions for
    X-Container-Meta-Temp-URL-Key* headers.
    
    Co-Authored-By: Christian Schwede <cschwede@redhat.com>
    Change-Id: I8fc7610395973b520aa9ddd72c94e1eb75f602cd
    Related-Change: I315b4e550b7d10880fbc00fce9311127ba609c2d

commit d40ff7098b3dd13b5ea846b86b9e8461ac1875b1
Author: Tim Burke <tim.burke@gmail.com>
Date:   Fri Sep 16 17:21:00 2016 -0700

Clean up api-ref examples
    
    If we're going to talk about replacing an object, we should use the
    same object name as the previous example.
    
    Including a non-zero content-length on PUT but not providing a body
    will timeout.
    
    Not including the '-' in '-H' will make curl complain:
    
       Rebuilt URL to: H/
       * Could not resolve host: H
       * Closing connection 1
    
    201 Created responses have content-length of zero, not 116.
    
    Change-Id: Ifd878559ee4036e4893221c7968f53021f38e236

commit 9a81b9537f9b1e2d8aff7e237095b15f8cedabf3
Author: Ondřej Nový <ondrej.novy@firma.seznam.cz>
Date:   Sat Sep 17 15:42:52 2016 +0200

Fixed cut and pasted paragraph from commit message in manpage
    
    Change-Id: Iab839a917e407ce8119f0c6162751f11136cbf5b

commit 5ec9a62f4db57eb6ae86704bb987a83a8b0c6faa
Author: Alistair Coles <alistair.coles@hpe.com>
Date:   Tue Sep 6 16:24:13 2016 +0100

api-ref: clean up account listing examples
    
    Move the account listing sample responses to follow the sample
    requests, and to precede the request/response parameter definitions to
    be consistent with other parts of the doc.
    
    Related-Change: I315b4e550b7d10880fbc00fce9311127ba609c2d
    
    Change-Id: Ia20acacd238db4a423b8cd89af1658222b4c5828

commit 2355771d4bc87d003a7702405c0406363e0aa18b
Author: Alistair Coles <alistair.coles@hpe.com>
Date:   Thu Sep 8 16:41:26 2016 +0100

api-ref: Move repeated paragraph to an include file
    
    Move repeated test re metadata header syntax to an include
    file and make it be rendered as a note.
    
    Also make already included text about metadata header value
    encoding be a note.
    
    Change-Id: I4795836587492954ad24dd5baaa5d668746d6040

commit bb24695d04088c9032188285b890db27893e2e9b
Author: npraveen35 <npraveen35@gmail.com>
Date:   Sat Sep 17 20:48:00 2016 +0530

updated output for swift stat
    
    Change-Id: Iff30acb88124e6116803bb79d32d1fc8cbb2d0ce
    Closes-Bug: #1622138

commit d0144962f64bbe301c2e50425631efa839e1ebd4
Author: Tim Burke <tim.burke@gmail.com>
Date:   Fri Aug 5 16:19:30 2016 -0700

Include object sysmeta in POST responses
    
    This is *only* sysmeta, to ensure we neither change the semantics of the
    response (which could happen if we included allowed_headers) nor reveal
    previously-written metadata to write-only clients (which could happen if
    we ever get updateable object metadata and sent back user meta).
    
    We could conceivably send back transient sysmeta, though it seems
    better to wait for a use-case that demands it.
    
    Change-Id: I04fe0b36b85e546b5379bed12089ffd1bce01fcf
    Co-Authored-By: Thiago da Silva <thiago@redhat.com>

commit a741998bff925360e2d6a72b5ab53c38779f170a
Author: Tim Burke <tim.burke@gmail.com>
Date:   Fri Sep 16 13:31:35 2016 -0700

Turn on F812 check
    
       F812 list comprehension redefines <variable> from line ...
    
    While the current violations were benign, this sort of code can easily
    lead to subtle bugs. Seems worth checking, especially given how cheap it
    is to bring existing code in line with it.
    
    Change-Id: Ibdcf9f93b85a1f1411198001df6bdbfa8f92d114

commit 70e3cf15bf76b23c5661da8bf301ec7f054a492c
Author: John Dickinson <me@not.mn>
Date:   Fri Sep 16 13:12:51 2016 -0700

remove comment saying we ignore H233. we actually check it
    
    Change-Id: Ia78fbb32aa6f45aa6578734dc43f3262e717bab8

commit b587d1606c5bd524efc473182f6048580fae33c8
Author: Travis McPeak <tmcpeak@us.ibm.com>
Date:   Fri Sep 2 09:52:17 2016 -0700

Updating Bandit config file
    
    This commit updates the Swift Bandit file to the new style
    introduced in Bandit 1.0.  In response to the struggle with
    getting a Bandit config file working and kept up to date we
    introduced a simplified version in Bandit 1.0.
    
    This commit updates Swift's bandit.yaml to use the new version.
    
    Change-Id: Ida5dd08f4ea72a377346f2159caeb2f3741d4980

commit 44a861787a60346aded8be2a54ef4e16accccbb6
Author: Alistair Coles <alistair.coles@hp.com>
Date:   Mon Oct 5 17:40:06 2015 +0100

Enable object server to return non-durable data
    
    This patch improves EC GET response handling:
    
    - The proxy no longer requires all object servers to have a
      durable file for the fragment archive that they return in
      response to a GET. The proxy will now be satisfied if just
      one object server has a durable file at the same timestamp
      as fragments from other object servers.
    
      This means that the proxy can now successfully GET an
      object that had missing durable files when it was PUT.
    
    - The proxy will now ensure that it has a quorum of *unique*
      fragment indexes from object servers before considering a
      GET to be successful.
    
    - The proxy is now able to fetch multiple fragment archives
      having different indexes from the same node. This enables
      the proxy to successfully GET an object that has some
      fragments that have landed on the same node, for example
      after a rebalance.
    
    This new behavior is facilitated by an exchange of new
    headers on a GET request and response between the proxy and
    object servers.
    
    An object server now includes with a GET (or HEAD) response:
    
    - X-Backend-Fragments: the value of this describes all
      fragment archive indexes that the server has for the
      object by encoding a map of the form: timestamp -> <list
      of fragment indexes>
    
    - X-Backend-Durable-Timestamp: the value of this is the
      internal form of the timestamp of the newest durable file
      that was found, if any.
    
    - X-Backend-Data-Timestamp: the value of this is the
      internal form of the timestamp of the data file that was
      used to construct the diskfile.
    
    A proxy server now includes with a GET request:
    
    - X-Backend-Fragment-Preferences: the value of this
      describes the proxy's current preference with respect to
      those fragments that it would have object servers
      return. It encodes a list of timestamp, and for each
      timestamp a list of fragment indexes that the proxy does
      NOT require (because it already has them).
    
      The presence of a X-Backend-Fragment-Preferences header
      (even one with an empty list as its value) will cause the
      object server to search for the most appropriate fragment
      to return, disregarding the existence or not of any
      durable file. The object server assumes that the proxy
      knows best.
    
    Closes-Bug: 1469094
    Closes-Bug: 1484598
    
    Change-Id: I2310981fd1c4622ff5d1a739cbcc59637ffe3fc3
    Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
    Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>

commit acb8971c765fbc64e55f37923e8b16c0126598ac
Author: Ha Van Tu <tuhv@vn.fujitsu.com>
Date:   Thu Sep 15 14:31:55 2016 +0700

Update link reference
    
    This patch update the outdated link reference inside the doc.
    
    Change-Id: I9c5aabe869226489f361ff637e9dec731bbc3d7d

commit d062af836ce8a302aec1c43837289236289cefa6
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Wed Sep 14 10:12:38 2016 -0700

DRY out probe.common
    
    Specifically to facilitate the reuse of the retry check server
    function to fill in the creds for the test2 account which is required
    for probetests after the related change.
    
    Change-Id: I9729faa4c8c8d6d65a481bc2ea3f0566d511034c
    Related-Change: I8d503419b7996721a671ed6b2795224775a7d8c6

commit f679ed0cc8b21e8b71d3aacc34c2a22352c4c850
Author: Alistair Coles <alistair.coles@hpe.com>
Date:   Thu Jul 28 18:41:08 2016 +0100

Make container sync copy SLO manifests
    
    Currently the container sync daemon fails to copy
    an SLO manifest, and the error will stall progress
    of the sync process on that container. There are
    several reasons why the sync of an SLO manifest
    may fail:
    
    1. The GET of the manifest from the source
       container returns an X-Static-Large-Object header
       that is not allowed to be included with a PUT
       to the destination container.
    
    2. The format of the manifest object that is read
       from the source is not in the syntax required
       for a SLO manifest PUT.
    
    3. Assuming 2 were fixed, the PUT of the manifest
       includes an ETag header which will not match the
       md5 of the manifest generated by the receiving
       proxy's SLO middleware.
    
    4. If the manifest is being synced to a different
       account and/or cluster, then the SLO segments may
       not have been synced and so the validation of the
       PUT manifest will fail.
    
    This patch addresses all of these obstacles by
    enabling the destination container-sync middleware to
    cause the SLO middleware to be bypassed by setting a
    swift.slo_override flag in the request environ. This
    flag is only set for request that have been validated
    as originating from a container sync peer.
    
    This is justifed by noting that a SLO manifest PUT from
    a container sync peer can be assumed to have valid syntax
    because it was already been validated when written to
    the source container.
    
    Furthermore, we must allow SLO manifests to be synced
    without requiring the semantic of their content to be
    re-validated because we have no way to enforce or check
    that segments have been synced prior to the manifest, nor
    to check that the semantic of the manifest is still valid
    at the source.
    
    This does mean that GETs to synced SLO manifests may fail
    if segments have not been synced. This is however
    consistent with the expectation for synced DLO manifests
    and indeed for the source SLO manifest if segments have
    been deleted since it was written.
    
    Co-Authored-By: Oshrit Feder <oshritf@il.ibm.com>
    Change-Id: I8d503419b7996721a671ed6b2795224775a7d8c6
    Closes-Bug: #1605597

commit 9ce596a5a623348dee7ee5efb6bb2af16d8add97
Author: Donagh McCabe <donagh.mccabe@hp.com>
Date:   Fri Aug 12 15:52:55 2016 +0100

Corrections for the API specifications in api-ref
    
    Fixes a number of technical issues with the api-ref section
    including:
    
    - Added missing headers
    - The header descriptions were made specific to whether they
      are used in requests or responses and the verb in question
      (example: Content-Length in object HEAD is the object size,
      not the response body length).
    - Added references to API features such as bulk delete.
    - Many typographical fixes (e.g., spaces in the middle
      of header names)
    - Restored xml and json account/container listing
      examples.
    
    The following areas were not updated and it is proposed
    to defer them to a subsequent update. This is because
    I don't have time or their merit is debatable:
    
    - ACLs (as used in a Keystone context) are not
      described.
    - Account create/delete is not described.
    - I left List Endpoints as-is.
    
    Change-Id: I315b4e550b7d10880fbc00fce9311127ba609c2d

commit 2aa5811c7fdf86df302f5ce8b0413bf51e86e7ab
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Tue Sep 13 15:10:04 2016 -0700

fix string formatting in manager
    
    Change-Id: I0d5c351735bdb6249dd9ec3e0c8b7ebe709feea6

commit f35829c291277a997c45185cdfe0b701ec725374
Author: Tim Burke <tim.burke@gmail.com>
Date:   Mon Sep 12 15:02:38 2016 -0700

Bring "egregious range request" comment in line with reality
    
    With MAX_RANGES pulled out as a constant, developers are likely to
    just trust the comment rather than go verify the value.
    
    Change-Id: I47fbd4790492b272d53ee07fe5bc87170ce2c72b
    Releated-Change: I4dcedcaae6c3deada06a0223479e611094d57234

commit 1f690df60c0ce7b627c4ebceaecaa5370ff10042
Author: Luong Anh Tuan <tuanla@vn.fujitsu.com>
Date:   Mon Sep 12 15:01:29 2016 +0700

Use ConfigParser instead of SafeConfigParser
    
    SafeConfigParser supports interpolation on top of ConfigParser in
    Python 2, and SafeConfigParser is deprecated in Python 3.2 and log
    warning like " DeprecationWarning: The SafeConfigParser class has
    been renamed to ConfigParser in Python 3.2. This alias will be removed
    in future versions. Use ConfigParser directly instead." So we can use
    ConfigParser if we don't need interpolation.
    
    Change-Id: I7e399b3cb90ded909e0d777a4d10c44f1e9299da
    Closes-Bug: #1618666

commit 3f2a4b2d89d66daa64cd3cf9bf31efd25702a0a9
Author: zheng yin <yin.zheng@easystack.cn>
Date:   Fri Sep 9 11:13:15 2016 +0800

Add test_pop in the test_header_key_dict
    
    There is a pop() function in the header_key_dict, but it has no
    test_pop in the test_header_key_dict. Therefore, I add it.
    
    Change-Id: I97bf9729fae3f044bcc69014549137f45ecbe91b

commit f285cafaa7ffa9bdf3b0d93d5dd00038ed38d70b
Author: zheng yin <yin.zheng@easystack.cn>
Date:   Fri Sep 9 16:41:22 2016 +0800

delete TODO in test_manager
    
    After test_shutdown,every method has its test case. Therefore, it is
    not necessary to add more tests, then I remove TODO.
    
    Change-Id: I23c1344bb18a83b154f890be0baa12799db045f8

commit 51925e5e3496c725487c30132976c9e96732e17b
Author: Nandini Tata <nandini.tata.15@gmail.com>
Date:   Tue Aug 9 20:34:55 2016 +0000

Add man page for swift.conf
    
    Added man page for swift.conf
    
    Change-Id: I88235ed414dfc34cb46f2fea014f1e721962885a
    Closes-Bug: 1607027

commit 0900a2c2447eb5a6cc8b105b4331a638959dba5d
Author: Timur Alperovich <timuralp@swiftstack.com>
Date:   Tue Sep 6 17:26:40 2016 -0700

Fix error handling in proxy/controllers/base.py.
    
    There are a few issues going on in the
    controllers/base.py:_get_response_parts_iter(). One is that the
    "raise" statement that attempts to re-raise the GeneratorExit, may
    re-raise ValueError if that was the last exception caught.
    
    Secondly, the range may not actually be set in the backend_headers
    (need to investigate further, as that could actually be faulty tests,
    since learn_size_from_content_range should always set it). The patch
    changes the Range construction to throw a ValueError if None is passed
    in that case.
    
    Lastly, the range may be only half-defined, e.g. bytes=0-. In that
    case, the check of how many bytes are expected vs how many bytes have
    been sent does not make sense.
    
    Change-Id: Ida5adf3d33c736240b2c4bae5510b5289f03dee2

commit ba610e0d1158c6a5e1a116a85c6c3026ee41fc8e
Author: Thiago da Silva <thiago@redhat.com>
Date:   Wed Sep 7 13:32:06 2016 -0400

add missing partition directory to example
    
    Change-Id: I2e806432d7d49e8fa940598cc6be680a362dec8b

commit ee11955d3e0f46b47ba1f3aef3452f865ba851be
Author: Alistair Coles <alistair.coles@hpe.com>
Date:   Wed Sep 7 17:54:36 2016 +0100

Add troubleshooting tips to keystoneauth docs
    
    Documents two common mistakes that are made when
    deploying Swift with Keystone auth.
    
    Change-Id: Id81af09d1d21b5c8a3871bf0e20fe6c1448d0ad9

commit e28ff0629d72d7d2abb06f99d6a2a93a0e37d732
Author: Mohit Motiani <mohit.motiani@intel.com>
Date:   Tue Aug 2 23:16:43 2016 +0000

Add regions in ring-builder.py
    
    swift-ring-builder CLI usage does not specify about regions.
    This patch adds a description about it. It also adds
    a brief description of the flags (report parameter).
    
    Change-Id: If51a15c0904857abf62636b90ac0c7ed63300871
    Closes-Bug: #1583551

commit eea574a78adc54242a97ac3b5ed0b8deeea754bf
Author: Lokesh S <lokesh.s@hp.com>
Date:   Thu Jul 28 12:10:12 2016 +0000

Py3: Fixes eventlet ImportError: No module named urllib2
    
    Python3:
    from eventlet.green.urllib import request
    Python2:
    from eventlet.green import urllib2
    
    Change-Id: Ib38865e0b6a8f076b8a54de4fae221d49f315c91

commit dd5afdb6863d8e8e1a1cdfcd2ce5fe06bc45bc95
Author: zheng yin <yin.zheng@easystack.cn>
Date:   Fri Jul 29 12:12:55 2016 +0800

simple log information
    
    When it has the same log information, we can use a function to simple
    it, then it also improve code readability.
    
    Change-Id: I5bb901099dcc832fc0651b34707901849abb0c34

commit 1cfa335a354040bdba22f20aa6ca783428f1b7b1
Author: Mohit Motiani <mohit.motiani@intel.com>
Date:   Tue Sep 6 20:02:33 2016 +0000

Fixed Typo in proxy/controller/obj.py
    
    Change-Id: I2d7217c618371dce2dfa8ed273dd5e10cbeabe3f

commit 1751db201b4204ead8f1b91a8d3a388d54822610
Author: Alistair Coles <alistair.coles@hpe.com>
Date:   Thu Jun 2 16:09:18 2016 +0100

Add test for copy using a Range header
    
    Since COPY allows a copy of a ranged GET, add some more test
    coverage for that in both unit and functional tests.
    
    Drive-by fix to use better test assertion methods.
    
    Co-Authored-By: Thiago da Silva <thiago@redhat.com>
    
    Change-Id: I5cb202386df0862f953f7388107c4d3466e2e46c

commit ba912a10b0342c5ed25e67cbd5eedef5bda287ba
Author: zheng yin <yin.zheng@easystack.cn>
Date:   Tue Sep 6 11:55:41 2016 +0800

Add test_valid_api_version in the test_constraints
    
    There is a valid_api_version function in the constraints,but it has no
    tests in the test_constraints. Therefore, I add test_valid_api_version.
    
    Change-Id: Ic8dbe9c00f762513ec9a4ffc0b77e4e5d3e5d47b

commit 8b3c04a4aa7ccd554fb6460ec2222b0437514f4e
Author: Victor Stinner <vstinner@redhat.com>
Date:   Mon Aug 22 10:44:43 2016 +0200

py3: Port test_splice to Python 3
    
    * add NamedTemporaryFile() wrapper
    * use byte strings for binary data in tests
    * tox.ini: run test_splice on Python 3.4 and Python 3.5
    
    Change-Id: I042d739fbf29433733bf0c2154749bc90b210416

commit 0f761852f8aeca0081405323d1dc23ae08f35fc4
Author: Ondřej Nový <ondrej.novy@firma.seznam.cz>
Date:   Sat Aug 13 17:54:24 2016 +0200

Added missing manpages for all cli tools
    
    Change-Id: Ief3cf510d426a4eeefc8bb8d12f6fa9c8343f55b
    Closes-Bug: #1607005
    Closes-Bug: #1607007
    Closes-Bug: #1607010
    Closes-Bug: #1607012
    Closes-Bug: #1607014
    Closes-Bug: #1607016
    Closes-Bug: #1607019
    Closes-Bug: #1607021
    Closes-Bug: #1607023
    Closes-Bug: #1599923

commit 350f10bf3be9ce6c9eec4b3046afc9a05cf08454
Author: Christian Schwede <cschwede@redhat.com>
Date:   Thu Jul 28 08:32:17 2016 +0000

Deprecate swift-temp-url
    
    python-swiftclient includes an improved and tested method to generate
    tempurls. The command syntax is essentially the same, therefore we can
    deprecate this one by importing that method.
    
    python-swiftclient is not added as a requirement; if the import fails
    due to a missing swiftclient module it will just raise a deprecation
    warning.
    
    Closes-Bug: #1607523
    Closes-Bug: #1607519
    
    Change-Id: Ifa8bf636f20f82db4845b02d1b58699edaa39356

commit 84b264baa8d81ceafa1ed6beb840c6fb80b91cc3
Author: Kazuhiro MIYAHARA <miyahara.kazuhiro@lab.ntt.co.jp>
Date:   Thu Sep 1 14:59:56 2016 +0900

Fix api reference of object GET request with Range parameter
    
    In RFC 7233, response body size of range requests with parameter
    'bytes=N-M' is (M - N + 1). And response of object GET request with
    range parameter in current Swift implementation is according to the
    RFC.  However, in current api reference explains that response body
    size of object GET request with 'Range: bytes=10-15' is five
    ( != 15 - 10 + 1).
    This patch fixes the api reference explanation.
    
    Change-Id: I8371864f8e5adb42c1e56b7ea26c556ea1252728

commit 1883c1ec23b6dfa7a266ed36f905fbd0501cc87c
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue Apr 26 09:15:29 2016 -0500

Make get_info requests useful with recheck_*_existence == 0
    
    Before, when recheck_account_existence or recheck_container_existence
    was set to zero, get_info requests for accounts or containers wouldn't
    populate the env cache for the current request, so it wouldn't return
    the data *it just got*.
    
    Now, we'll still populate the env cache and memcache, as a cache time of
    zero means "keep it indefinitely". See the memcache docs at
    https://github.com/memcached/memcached/blob/1.4.25/doc/protocol.txt#L163
    
    Change-Id: Ia648263073bc88e35216cafb76821b7ce02c1d03
    Closes-Bug: 1224734

commit a4279112b3f3320f9b305051965a676df9dbce3e
Author: Kazuhiro MIYAHARA <miyahara.kazuhiro@lab.ntt.co.jp>
Date:   Thu Sep 1 21:24:52 2016 +0900

Remove meaningless lines in reconciler unit test
    
    There are two lines in which lists are sorted in test_reconciler.py
    However, the values in the lists will be same, then the two lines are
    meaningless.
    This patch removes the two lines from test_reconciler.py
    
    Change-Id: I7d72a7a4e9a27d87cd78c96f79d1b340b2ad23de

commit c9488027cb6db6aa3116fbcb9a9a8f9d294d52dc
Author: Christian Schwede <cschwede@redhat.com>
Date:   Thu Sep 1 11:42:09 2016 +0000

Make versioned_writes docstring more precise
    
    Add a note to the docstring that it is required to add a config section
    to the proxy-server.conf and an entry to the pipeline to support history
    mode.
    
    Closes-Bug: 1619261
    Change-Id: I888485ab4ece6f47db081a4d58c1aab24ce72a8a

commit 2a58b1efa4b70505bfa01b3fe410a25ea2fa799b
Author: Christian Schwede <cschwede@redhat.com>
Date:   Wed Aug 31 12:49:07 2016 +0000

More Test[Account|Container]Controller test updates
    
    Added another test that tries to replicate using an unknown sync method,
    with and without an existing DB file.
    
    Change-Id: I09a86615f012e4341268ea30cbc06099528f896b

commit ff52bb1b1c04cdad59f784f8222a4853e7a8304e
Author: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Date:   Wed Aug 31 23:01:05 2016 -0700

Avoid warnings for versioned_writes docs
    
    This includes a couple of fixes as follows:
    - Add one more "-" for mode example titles to adjust the title length
      (it seems needed to use verbatim expression)
    - Remove unnecessary back slash
    
    Change-Id: If41bba040ea31f2354642b2ec32bc92ece4f0971

commit 2825909d2553527283ac57de7bd9a842988e6624
Author: Timur Alperovich <timuralp@swiftstack.com>
Date:   Tue Aug 30 17:02:21 2016 -0700

Silence "Client disconnected" warnings on reads.
    
    When a client fully reads the content and closes the iterator, the
    Client disconnected warning is still generated, as there is no logic
    to check whether the GeneratorExit exception was raised after the
    client received all of the data. This can be observed when doing
    large object reads or using an InternalClient and reading exactly
    Content-Length bytes from the returned app_iter body.
    
    The patch amends the behavior to hoist how many bytes client read from
    a given part and only raise an exception if there are more parts left
    or a part was not fully read.
    
    Lastly, the GeneratorExit exception is no longer swallowed and is
    re-raised in the handling code.
    
    Change-Id: I879149897fdb25aae977b7f17e580610b188ce04

commit 671254224a4a4710e7556535ee68bd999536ab8d
Author: gengchc2 <geng.changcai2@zte.com.cn>
Date:   Fri Aug 26 14:08:08 2016 +0800

Correct swift reraising of exception
    
    When an exception was caught and rethrown, it should call 'raise'
    without any arguments because it shows the place where an exception
    occured initially instead of place where the exception re-raised.
    
    Change-Id: I783a2a65f3e38a23ba49b6ae5a2f56a5fdaac2fc

commit 0c22bc22573626dae0f81b976b56e82b265f77a2
Author: Tim Burke <tim.burke@gmail.com>
Date:   Wed Aug 31 14:10:27 2016 -0700

Include correct version in install-guide
    
    Change-Id: I8f9de7a824aa327d566c821ec05f97d0bc8931e9

commit 3f00148c6b350efc684250f3339fa5166704d5a3
Author: Nguyen Phuong An <AnNP@vn.fujitsu.com>
Date:   Wed Aug 31 14:37:11 2016 +0700

Config logABug feature for Swift api-ref
    
    Currently, Swift api-ref is not configured with logABug feature.
    When users click "Report bug" button, it leads to "openstack-manuals"
    which is default.
    
    Change-Id: I3b6fb410000637ff03d99110d440de88a5adc30c

commit 66c905e29458666cbd3ea6638a6a4c137e6a35a8
Author: Timur Alperovich <timuralp@swiftstack.com>
Date:   Mon Aug 29 17:27:50 2016 -0700

Close the iterators in string_along.
    
    Make sure to close the underlying iterator in string_along. What is
    currently happening when using the InternalClient is that "Client
    disconnected" warnings are generated and resources are tied up until
    GC runs.
    
    Change-Id: If1f6c0c756aee95f53f99371439533a97d347eab

commit cd8eb6db9188359039c1f13e34397cb9b24b8112
Author: zheng yin <yin.zheng@easystack.cn>
Date:   Fri Aug 26 14:04:46 2016 +0800

Add test to validate the error strings in test_expirer
    
    Improve test_get_process_values_* methods in obj/test_expirer in the
    form assertRaises(ValueError, x.get_process_values,{}/vals), to use
    the assertRaises context form.  This improves understandability by
    validateing the error strings in addition to the ValueError.
    
    Related-Change: I3d12b79470d122b2114f9ee486b15d381f290f95
    
    Change-Id: I1c66b8894cba8328d19cf99491a8ad18ded71078

commit 4d4885acdc8107c4f8f8cd10bc6d19ba3901eb25
Author: Tim Burke <tim.burke@gmail.com>
Date:   Fri Aug 5 18:05:45 2016 +0000

Tighten header checks for object PUT/POST paths
    
    Change-Id: If2cd059719fe5af1e73ecde5306e9f68d590831f

commit 773edb4a5dd22b22f5fcab57820bcbdd4176dfcd
Author: Prashanth Pai <ppai@redhat.com>
Date:   Thu Mar 5 18:18:25 2015 +0530

Make object creation more atomic in Linux
    
    Linux 3.11 introduced O_TMPFILE as a flag to open() sys call. This would
    enable users to get a fd to an unnamed temporary file. As it's unnamed,
    it does not require the caller to devise unique names. It is also not
    accessible through any path. Hence, file creation is race-free.
    
    This file is initially unreachable. It is then populated with data(write),
    metadata(fsetxattr) and fsync'd before being atomically linked into the
    filesystem in a fully formed state using linkat() sys call. Only after a
    successful linkat() will the object file will be available for reference.
    
    Caveats
    * Unlike os.rename(), linkat() cannot overwrite destination path if it
      already exists. If path exists, we unlink and try again.
    * XFS support for O_TMPFILE was only added in Linux 3.15.
    * If client disconnects during object upload, although there is no
      incomplete/stale file on disk, the object directory would persist
      and is not cleaned up immediately.
    
    Change-Id: I8402439fab3aba5d7af449b5e465f89332f606ec
    Signed-off-by: Prashanth Pai <ppai@redhat.com>

commit 06ff865d19b522c48e23637d1234fc8f6b9401ab
Author: Janie Richling <jrichli@us.ibm.com>
Date:   Wed Aug 17 00:54:09 2016 -0500

Allow DLO manifest to be updated when using post-as-copy
    
    Currently when using fast-post, the manifest is updated with the given
    'x-object-manifest' header on a POST.  If no such header is supplied,
    then the manifest will change to a regular object.
    This is not currently true when using post-as-copy.
    
    This patch changes the DLO POST using post-as-copy behavior to match
    that of using fast-post.  It was also documented that
    'x-object-manifest' must be provided on a POST to a manifest file.
    
    Change-Id: Ie1143ab1a2c8f8c21e258a36badbff5d947769d4
    Closes-bug: 1612991

commit 4bcd3d7f6de53ccd04d42005848060b92e7733ce
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Mon Aug 8 17:57:49 2016 -0700

Support multi-range GETs for static large objects.
    
    Bonus consistency: 416 responses now always have a body. Before, if
    you had "swob.HTTPRequestedRangeNotSatisfiable()", you'd get a body,
    but if you had "swob.Response(..., conditional_response=True)", then
    you'd get a length-0 response body. Now you always get a response
    body. It's just the default <html><h1>..., but at it's always there.
    
    Bonus efficiency: do a little caching of sub-SLO manifests to avoid
    needless re-fetches. This kicks in when there are multiple references
    to the same sub-SLO in a given manifest. The caching only holds 20
    sub-SLOs so that a malicious user can't build a giant SLO tree and use
    it to run the proxy out of memory (we're already holding up to 10
    manifests in memory at a time since a SLO can include another SLO to a
    depth of 10; this doesn't make the situation too much worse).
    
    Change-Id: I24716e3271cf3370642e3755447e717fd7d9957c

commit 1fc46a0e13eff5603ba0c26d98407e3012d9de42
Author: Lokesh S <lokesh.s@hp.com>
Date:   Thu Jul 21 15:41:58 2016 +0000

py3: Fixes encoding and type error
    
    Fixes encoding bytes to base64.
    Fixes unorderedtype int() and str()
    for python3.
    Fixes encoding issues.
    
    Change-Id: I6e7aaf21e080078d4b36562e41129804d71df2fc

commit 8d6bf368f710e1fdb1cf548ad3e40baddea3e7c7
Author: zheng yin <yin.zheng@easystack.cn>
Date:   Wed Aug 17 19:27:13 2016 +0800

Remove unnecessary variable
    
    It defines self.node information in setUp(),  it is unnecessary to
    define node in test_direct_delete_account. Because although self.node
    includes replication_ip and replication_port, direct_delete_account
    doesn't use both and uses ip,port and device. Therefore, I think self.node
    can instead of node.
    
    Change-Id: I6df7081280d2b540f984e4e688620931f5d4ac88

commit eadab846b32e78644925fec707c6a048d959c3a8
Author: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Date:   Thu Aug 4 23:34:40 2016 -0700

Remove unnecessary open call and related vars in unit tests
    
    Since the methods touching exisiting file is mocked, we don't have to
    open the file. (if it is needed, we should close at first anyway)
    
    And cleanup unecessary vars/imports that is used for making the file
    path.
    
    This patch is a followup on:
    https://review.openstack.org/#/c/350471/
    
    Change-Id: I8fbc0b5b9a01782b6da5a7dd674f52d4b566ca5c

commit b1b410902d3af011c65569a0ba94a39148f8265e
Author: zheng yin <yin.zheng@easystack.cn>
Date:   Wed Aug 3 17:11:25 2016 +0800

Add unit case
    
    "op" has three judgments in ReplicatorRpc.dispatch, rsync_then_merge,complete_rsync
    and other. But it only has two times judgments,including rsync_then_merge and other
    in test_REPLICATE_works, so I add complete_rsync in test_account and test_container
    
    Change-Id: I8277b556062dd6b30bf85dedd636d56517f10d8d

commit ab602a2711af2e09df1ffeb132f4a01e3fae5626
Author: Tim Burke <tim.burke@gmail.com>
Date:   Fri Jul 1 13:38:20 2016 -0700

Make load_crypto_meta more py3-friendly
    
    Change-Id: I1f2551630e7162f3b4efc8bf1f1a4d415da9d9a3

commit 88b14a0f803959ec5195d78a58b68dc2d5f30378
Author: Hisashi Osanai <osanai.hisashi@jp.fujitsu.com>
Date:   Fri Aug 14 06:49:18 2015 +0900

Add functional test for access control (RBAC INFO) with Keystone
    
    This patch adds test cases for info requests to accounts, containers
    and objects using various combinations of users/projects, roles and/or
    service tokens.
    
    Change-Id: Iece57b03a73b3cdf9762573678755f361e9b803e

commit 5448a55e37c409c099c2e0b1dfaf7f0473cc85df
Author: Hisashi Osanai <osanai.hisashi@jp.fujitsu.com>
Date:   Mon Aug 17 15:15:19 2015 +0900

Add functional test for access control (container ACL) with Keystone
    
    This patch adds test cases for requests to containers and objects using
    various combinations of users/projects, roles in order to check container
    ACL and referrer behavior.
    
    Co-Authored-By: Keisuke Yamamoto <keisuk.yamamoto@jp.fujitsu.com>
    Change-Id: I435655bc56e5088ba50a96a879a45c7e264e0d8d

tags:

added: in-feature-hummingbird

OpenStack Object Storage (swift)

EC missing durable can prevent reconstruction

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches