Bug #1491883 “Reconstructor complains 404 is invalid response wh...” : Bugs : OpenStack Object Storage (swift)

Revision history for this message

Minwoo Bae (minwoob) wrote on 2015-09-03:

#1

I agree that they are expected to happen from time to time, but I imagine that information regarding 404 frequencies would be useful to an operator. In my opinion, we should leave it, as it is.

Revision history for this message

Caleb Tennis (ctennis) wrote on 2015-09-03:

#2

For replication we don't additionally warn when we get 404s from the object server.

The request is logged by the object server, so you could easily track replicator requests that end with 404.

I'd say it's not optimal to get 404s, but it's certainly not "unexpected".

Every time someone changes their storage, we'll just fill up the logs with these until reconstruction stabilizes.

Minwoo Bae (minwoob) on 2015-09-09

Changed in swift:
assignee:	nobody → Minwoo Bae (minwoob)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-09-09: Fix proposed to swift (master)

#3

Fix proposed to branch: master
Review: https://review.openstack.org/221956

Changed in swift:
status:	New → In Progress

John Dickinson (notmyname) on 2015-09-09

Changed in swift:
importance:	Undecided → Low

Minwoo Bae (minwoob) on 2015-09-18

summary:

- Reconstructor complains 404 is invalid responce when building object
+ Reconstructor complains 404 is invalid response when building object

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-09-19: Fix merged to swift (master)

#4

Reviewed: https://review.openstack.org/221956
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=a63f70c17d392379ec75045a94c38a96878b4c5c
Submitter: Jenkins
Branch: master

commit a63f70c17d392379ec75045a94c38a96878b4c5c
Author: Minwoo Bae <email address hidden>
Date: Wed Sep 9 15:33:45 2015 -0500

Reconstructor logging to omit 404 warnings

    Currently, the replicator does not log warning messages
    for 404 responses. We would like the reconstructor to
    do the same, as 404s are not considered unusual, and
    are already handled by the object server.

Change-Id: Ia927bf30362548832e9f451923ff94053e11b758
Closes-Bug: #1491883

Changed in swift:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-09-21: Fix proposed to swift (feature/crypto)

#5

Fix proposed to branch: feature/crypto
Review: https://review.openstack.org/225540

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-09-21: Change abandoned on swift (feature/crypto)

#6

Change abandoned by John Dickinson (<email address hidden>) on branch: feature/crypto
Review: https://review.openstack.org/225540
Reason: jrichli will do this

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-09-21: Fix merged to swift (feature/crypto)

#7

Download full text (12.2 KiB)

Reviewed: https://review.openstack.org/225540
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=1936299943116abcedd8b36fb8b64c724e539ab7
Submitter: Jenkins
Branch: feature/crypto

commit 9e95613d717489da584531e220db20d1b8f43c51
Author: Alistair Coles <email address hidden>
Date: Mon Sep 21 10:01:54 2015 +0100

Fix .coveragrc to prevent nose tests error

    Since v4.0 the coverage package raises an error if unrecognized
    options are found in .coveragrc [1]. Previously they were ignored. The
    ignore-errors option therefore causes nosetests with coverage to error
    because the option should be ignore_errors (underscore not hyphen).

[1] https://bitbucket.org/ned/coveragepy/src/tip/CHANGES.rst

Change-Id: Ic488801b7cc43217f9e2a4ed945e06505b667608

commit 1fe8e4327b15f8933efef185d09b6067d1f28716
Author: Clay Gerrard <email address hidden>
Date: Fri Sep 18 13:54:52 2015 -0700

Fix recon tests on SAIO with multiple policies

    Recon middleware used to only look on rings that exist on disk when it was
    started, so if a test didn't create a ring in the temp swift_dir it can expect
    the middleware to not report it.

    However, after we started looking at policies to determine rings [1] - we need
    to be more careful to patch policies to match up with the test requirements.
    On development environments with only the legacy default polices the existing
    recon tests were passing by accident - but not in my environment.

    This change will patch policies for the TestCase so that tests will pass for
    me. Individual test methods that have more specific policy requirements for
    the test can continue to @patch_policies just for those tests but in general
    the existing test_methods all seem to expect legacy policies - so we just make
    the default for the TestCase legacy_only.

Change-Id: I778a0a59091ca8870e1cab985f3ec426eb045ab7

commit a63f70c17d392379ec75045a94c38a96878b4c5c
Author: Minwoo Bae <email address hidden>
Date: Wed Sep 9 15:33:45 2015 -0500

Reconstructor logging to omit 404 warnings

    Currently, the replicator does not log warning messages
    for 404 responses. We would like the reconstructor to
    do the same, as 404s are not considered unusual, and
    are already handled by the object server.

Change-Id: Ia927bf30362548832e9f451923ff94053e11b758
Closes-Bug: #1491883

commit 530102ae07ea27b4a994e4b1fb2f590700cfef0c
Author: Bill Huber <email address hidden>
Date: Mon Sep 14 16:01:39 2015 -0500

Update EC Support on how to build an EC ring with replicas count

    This doc is being updated to specify the replicas count parameter
    to build an EC ring that enforces both data and parity placements
    for each partition.

Change-Id: I770ad268e4017e610be3357e89b89f0b7d3c18af
Closes-Bug: 1487203

commit 0d9142abd45f189b15eaec1a25464ed7d1859f97
Author: Clay Gerrard <email address hidden>
Date: Mon Sep 14 17:17:29 2015 -0700

Fix typo in Deployment Guide and add some formatting

Change-Id: I58703162bf3e9f39656a5e511bd8fe845793bca2

commit 460a7e4b64d134d1fd47f09924d594196b6...

Reviewed:  https://review.openstack.org/225540
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=1936299943116abcedd8b36fb8b64c724e539ab7
Submitter: Jenkins
Branch:    feature/crypto

commit 9e95613d717489da584531e220db20d1b8f43c51
Author: Alistair Coles <alistair.coles@hp.com>
Date:   Mon Sep 21 10:01:54 2015 +0100

Fix .coveragrc to prevent nose tests error
    
    Since v4.0 the coverage package raises an error if unrecognized
    options are found in .coveragrc [1]. Previously they were ignored. The
    ignore-errors option therefore causes nosetests with coverage to error
    because the option should be ignore_errors (underscore not hyphen).
    
    [1] https://bitbucket.org/ned/coveragepy/src/tip/CHANGES.rst
    
    Change-Id: Ic488801b7cc43217f9e2a4ed945e06505b667608

commit 1fe8e4327b15f8933efef185d09b6067d1f28716
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Fri Sep 18 13:54:52 2015 -0700

Fix recon tests on SAIO with multiple policies
    
    Recon middleware used to only look on rings that exist on disk when it was
    started, so if a test didn't create a ring in the temp swift_dir it can expect
    the middleware to not report it.
    
    However, after we started looking at policies to determine rings [1] - we need
    to be more careful to patch policies to match up with the test requirements.
    On development environments with only the legacy default polices the existing
    recon tests were passing by accident - but not in my environment.
    
    This change will patch policies for the TestCase so that tests will pass for
    me.  Individual test methods that have more specific policy requirements for
    the test can continue to @patch_policies just for those tests but in general
    the existing test_methods all seem to expect legacy policies - so we just make
    the default for the TestCase legacy_only.
    
    Change-Id: I778a0a59091ca8870e1cab985f3ec426eb045ab7

commit a63f70c17d392379ec75045a94c38a96878b4c5c
Author: Minwoo Bae <minwoob@us.ibm.com>
Date:   Wed Sep 9 15:33:45 2015 -0500

Reconstructor logging to omit 404 warnings
    
    Currently, the replicator does not log warning messages
    for 404 responses. We would like the reconstructor to
    do the same, as 404s are not considered unusual, and
    are already handled by the object server.
    
    Change-Id: Ia927bf30362548832e9f451923ff94053e11b758
    Closes-Bug: #1491883

commit 530102ae07ea27b4a994e4b1fb2f590700cfef0c
Author: Bill Huber <wbhuber@us.ibm.com>
Date:   Mon Sep 14 16:01:39 2015 -0500

Update EC Support on how to build an EC ring with replicas count
    
    This doc is being updated to specify the replicas count parameter
    to build an EC ring that enforces both data and parity placements
    for each partition.
    
    Change-Id: I770ad268e4017e610be3357e89b89f0b7d3c18af
    Closes-Bug: 1487203

commit 0d9142abd45f189b15eaec1a25464ed7d1859f97
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Mon Sep 14 17:17:29 2015 -0700

Fix typo in Deployment Guide and add some formatting
    
    Change-Id: I58703162bf3e9f39656a5e511bd8fe845793bca2

commit 460a7e4b64d134d1fd47f09924d594196b6458d4
Author: Brian Cline <bcline@softlayer.com>
Date:   Wed Sep 2 04:13:29 2015 -0500

Fixes recon bug with initially missing rings
    
    Previously the recon middleware was doing a basic scan for object
    rings that exist at init time. In situations where an object-server
    was started without an object ring present, but received one shortly
    after, recon still would not report it in the /recon/ringmd5 response.
    This persists even when object-server gleefully chugs along after
    picking up the ring, and recon's behavior would only be corrected by
    an object-server reload/restart.
    
    This change brings the middleware a bit more up to date to use the
    common POLICIES instance to determine what policies were already loaded
    based on configuration, and derives the path for each ring.
    
    This effectively makes the config the source of truth for what rings
    *should* be present, rather than what's present at startup. Since we
    already dynamically check in ReconMiddleware.get_ring_md5 whether each
    of the predetermined ring files exist, recon now correctly reports a
    previously-missing ring whenever it falls into place.
    
    Change-Id: Ia079418e54ffac5e01ef6a15511f5069b7fe83ea

commit 9324ce83c6dcbf5fb4ffefdc781eec5db1b946c9
Author: Bill Huber <wbhuber@us.ibm.com>
Date:   Tue Sep 8 16:11:33 2015 -0500

Reconstructor GET excludes user_agent in log
    
    To make it easier for users to deduce in the log to find out
    where the request originates from, it is necessary to include
    the user_agent field in the reconstructor for a GET method
    and to have this particular log consistent with other servers'
    methods.
    
    Change-Id: I0ca7443436e97c2db64c966ab4d73c5c12a1f059
    Closes-Bug: 1491871
    Co-Authored-By: Kota Tsuyuzakai <tsuyuzaki.kota@lab.ntt.co.jp>

commit 369447ec47dc8df3e38046a305cf1aa0a6499ce9
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Thu Aug 27 15:09:29 2015 -0700

Fix purge for tombstone only REVERT job
    
    When we revert a partition we normally push it off to the specific
    primary node for the index of the data files in the partition.  However,
    when a partition is devoid of any data files (only tombstones) we build
    a REVERT job with a frag_index of None.
    
    This change updates the ECDiskFile's purge method to be robust to
    purging tombstones when the frag_index is None.
    
    Add probetest to validate tombstone only revert jobs will clean
    themselves up if they can validate they're in-sync with part-replica
    count nodes - even if one of the primaries is down (in which case they
    sync tombstones with other handoffs to fill in for the primaries)
    
    Change-Id: Ib9a42f412fb90d51959efce886c0f8952aba8d85

commit a3facce53cac0497edb326a93a97f24e02c603ab
Author: paul luse <paul.e.luse@intel.com>
Date:   Thu Aug 27 11:02:27 2015 -0700

Fix invalid frag_index header in ssync_sender when reverting EC tombstones
    
    Back in d124ce [1] we failed to recognize the situation where a revert
    job would have an explicit frag_index key wth the literal value None
    which would take precedence over the dict.get's default value of ''.
    Later in ssync_receiver we'd bump into the ValueError converting 'None'
    to an int (again).
    
    In ssync_sender we now handle literal None's correctly and should
    hopefully no longer put this invalid headers on the wire - but for belts
    and braces we'll also update ssync_receiver to raise a 400 series error
    and ssync_sender to better log the error messages.
    
    1. https://review.openstack.org/#/c/195457/
    
    Co-Author: Clay Gerrard <clay.gerrard@gmail.com>
    Co-Author: Alistair Coles <alistair.coles@hp.com>
    Change-Id: Ic71ba7cc82487773214030207bb193f425319449
    Closes-Bug: 1489546

commit 71f6fd025ea013bc457450a5cc4f8e92c6df92cc
Author: Romain LE DISEZ <romain.ledisez@ovh.net>
Date:   Tue Jun 16 12:47:26 2015 +0200

Allows to configure the rsync modules where the replicators will send data
    
    Currently, the rsync module where the replicators send data is static. It
    forbids administrators to set rsync configuration based on their current
    deployment or needs.
    
    As an example, the rsyncd configuration example encourages to set a connections
    limit for the modules account, container and object. It permits to protect
    devices from excessives parallels connections, because it would impact
    performances.
    
    On a server with many devices, it is tempting to increase this number
    proportionally, but nothing guarantees that the distribution of the connections
    will be balanced. In the worst scenario, a single device can receive all the
    connections, which is a severe impact on performances.
    
    This commit adds a new option named 'rsync_module' to the *-replicator sections
    of the *-server configuration file. This configuration variable can be
    extrapolated with device attributes like ip, port, device, zone, ... by using
    the format {NAME}. eg:
        rsync_module = {replication_ip}::object_{device}
    
    With this configuration, an administrators can solve the problem of connections
    distribution by creating one module per device in rsyncd configuration.
    
    The default values are backward compatible:
        {replication_ip}::account
        {replication_ip}::container
        {replication_ip}::object
    
    Option vm_test_mode is deprecated by this commit, but backward compatibility is
    maintained. The option is only effective when rsync_module is not set. In that
    case, {replication_port} is appended to the default value of rsync_module.
    
    Change-Id: Iad91df50dadbe96c921181797799b4444323ce2e

commit cb683d391cb66d0f52830de16760c80fd2afedf9
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Sat Sep 5 06:17:51 2015 +0000

Imported Translations from Transifex
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: I2d92b8e34a665fb0bb4c048cfb0c59de295dfce6

commit e4542455c8a07b7981c247df8b737816062c1655
Author: Emett Speer <speer.emett@gmail.com>
Date:   Wed Sep 2 17:18:03 2015 -0700

[Labs] Update links to Cloud Admin Guide
    
    Update links to the Cloud Admin Guide after the
    RST conversion of that book altered URLs.
    
    Change-Id: I899f8938498b744e62887968a65e58c00ef27f1b

commit 58fcc07523978306cd3889ada73af5d9e664cf59
Author: Christian Schwede <cschwede@redhat.com>
Date:   Wed Sep 2 10:52:34 2015 +0000

Test if container_sweep is executed on unmounted devices
    
    This change ensures that container_sweep is not run if a device is not mounted
    and mount_check is set to True.
    
    Change-Id: I823083c8431d9e61fd426508033ec9188503957b

commit 615c7a204b9386e05c5bab658bfe96766ad1e680
Author: Brian Cline <bcline@softlayer.com>
Date:   Tue Sep 1 10:51:20 2015 -0500

Adds useful dispersion info from changelog
    
    Change-Id: I1a45088fc32620b02ff9a754b02ec1eb75a59d6e

commit 3b8755098a1786c5447abf158bd686293a82977c
Author: janonymous <janonymous.codevulture@gmail.com>
Date:   Sun Aug 2 21:29:13 2015 +0530

Replace a / b with a // b to use integer division where needed
    
    Change-Id: I72c81faa62786e140b0de00e3a04934bf1b5adbd

commit 58a10a5fffec69304d7bcce0f1c43bd2a9f9ff52
Author: Alistair Coles <alistair.coles@hp.com>
Date:   Wed Aug 26 16:30:23 2015 +0100

Add test that a tempurl POST cannot set a DLO manifest header
    
    Follow up to [1] to add tests for tempurl POSTs not being allowed
    to set a DLO manifest header.
    
    [1] I11e68830009d3f6bff44ae4011a41b67139146f6
    
    Change-Id: I7c0ad5a936f71e56c599b8495a586913d3334422
    Related-Bug: 1453948

commit 4765189ef3f67b3f709c3141f222e15c6d89a62d
Author: Carlos Cavanna <ccavanna@ca.ibm.com>
Date:   Thu Jul 9 23:10:05 2015 -0400

Improving statistics sent to Graphite.
    
    Currently, statistics are organized by command. However, it would also be
    useful to display statistics organized by policy. Different policies may be
    based on different storage properties (ie, faster disks).
    With this change, all the statistics for object timers will be sent per policy
    as well.
    Policy statistics reporting will use policy index and the name in Graphite will
    show as proxy-server.object.policy.<policy-index>.<verb>, etc.
    Updated unit tests for per-policy stat reporting and added new unit tests for
    invalid cases.
    Updated documentation in the Administrator's Guide to reflect this new
    aggregation.
    
    Change-Id: Id70491e4833791a3fb8ff385953d69018514cd9c

commit 5b01c5eadc3a13607d2581a55007700fca0b00af
Author: Bill Huber <wbhuber@us.ibm.com>
Date:   Fri Aug 7 16:24:28 2015 -0500

Add unit test for container_update for unmounted device
    
    This unit test case gains coverage for container/update.py
    where a device (i.e. sda1) is not mounted and the code
    responds with a warning logger that the device is not
    mounted and continues.  Coverage increases 2% as a result.
    
    Change-Id: I33d247a930b28604093df4ade1ce7dbbd24aac54

tags:

added: in-feature-crypto

Thierry Carrez (ttx) on 2015-10-05

Changed in swift:
milestone:	none → 2.5.0
status:	Fix Committed → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-10-16: Fix proposed to swift (feature/hummingbird)

#8

Fix proposed to branch: feature/hummingbird
Review: https://review.openstack.org/236162

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-10-17: Fix merged to swift (feature/hummingbird)

#9

Download full text (52.0 KiB)

Reviewed: https://review.openstack.org/236162
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=18ddcaf0d6b67fcbb6b0a4cf4a9a99c72f6f3a08
Submitter: Jenkins
Branch: feature/hummingbird

commit a9ddc2d9ea402eaac7ccd8992387f77855968ab5
Author: Mahati Chamarthy <email address hidden>
Date: Fri Oct 16 18:18:33 2015 +0530

Hyperlink fix to first contribution doc

Change-Id: I19fc1abc89f888233b80a57c68a152c1c1758640

commit 83a1151d13e096b480aefe6ec18259f2d7d021db
Author: Pete Zaitcev <email address hidden>
Date: Fri Oct 9 16:45:20 2015 -0600

Interpolate the explanation string not whole HTML body

The only reason this exists is that I promised to do it.
But in our case, there's no big advantage, and here's why.

    The general thinking goes that strings must be interpolated
    first because the body may contain a syntax that confuses the
    interpolation. So this patch makes the code "more correct".
    However, our HTML template is tightly controlled. It's not
    like it contains additional percents.

So I'll just leave this here for now while I'm asking if
the content type is set correctly.

Change-Id: Ia18aeb0f94ef389f8b95450986a566e5fa06aa10

commit 384b91eb824376659989b904f9396cbf2e02d2bd
Author: asettle <email address hidden>
Date: Thu Sep 3 15:11:46 2015 +1000

Moving DLO functionality doc to the middleware code

    This change moves the RST DLO documentation from
    statically inside overview_large_objects.rst and moves it
    to middleware/dlo.py.
    This is where all middleware RST documentation is defined.

    The overview_large_objects.rst is still the main page
    for information on large objects, so now dynamically
    points to both the DLO and SLO middleware RST
    documentation and the relevant middleware.rst page
    simply points to it.

Change-Id: I40d918c8b7bc608ab945805d69fe359521df038a
Closes-bug: #1276852

commit 2996974e5d48b4efaa1b271b8fbd0387bced7242
Author: Ondřej Nový <email address hidden>
Date: Sat Oct 10 14:56:30 2015 +0200

Script for running unit, func and probe tests at once

When developing Swift it's often needed to run all tests.
This script makes it much simpler.

Change-Id: I67e6f7cc05ebd0475001c1b56e8f6fd09c8c644f

commit c2182fd4163050a5f76eb3dedb7703dc821fa83d
Author: janonymous <email address hidden>
Date: Fri Jul 17 20:20:15 2015 +0530

Python3: do not use im_self/im_func/func_closure

Use __self__, __func__ and __closure__ instead, as they work
with both Python 2 and 3.

Modifying usage of __func__ in codebase.

Change-Id: I57e907c28c1d4646605e70194ea3650806730b83

commit c0866ceaac2f69ae01345a795520141f59ec64f5
Author: Samuel Merritt <email address hidden>
Date: Fri Sep 25 17:26:37 2015 -0700

Improve SLO PUT error checking

    This commit tries to give the user a reason that their SLO manifest
    was invalid instead of just saying "Invalid SLO Manifest File". It
    doesn't get every error condition, but it's better than before.

Examples of things that now have real error...

Reviewed:  https://review.openstack.org/236162
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=18ddcaf0d6b67fcbb6b0a4cf4a9a99c72f6f3a08
Submitter: Jenkins
Branch:    feature/hummingbird

commit a9ddc2d9ea402eaac7ccd8992387f77855968ab5
Author: Mahati Chamarthy <mahati.chamarthy@gmail.com>
Date:   Fri Oct 16 18:18:33 2015 +0530

Hyperlink fix to first contribution doc
    
    Change-Id: I19fc1abc89f888233b80a57c68a152c1c1758640

commit 83a1151d13e096b480aefe6ec18259f2d7d021db
Author: Pete Zaitcev <zaitcev@kotori.zaitcev.us>
Date:   Fri Oct 9 16:45:20 2015 -0600

Interpolate the explanation string not whole HTML body
    
    The only reason this exists is that I promised to do it.
    But in our case, there's no big advantage, and here's why.
    
    The general thinking goes that strings must be interpolated
    first because the body may contain a syntax that confuses the
    interpolation. So this patch makes the code "more correct".
    However, our HTML template is tightly controlled. It's not
    like it contains additional percents.
    
    So I'll just leave this here for now while I'm asking if
    the content type is set correctly.
    
    Change-Id: Ia18aeb0f94ef389f8b95450986a566e5fa06aa10

commit 384b91eb824376659989b904f9396cbf2e02d2bd
Author: asettle <alexandra.settle@rackspace.com>
Date:   Thu Sep 3 15:11:46 2015 +1000

Moving DLO functionality doc to the middleware code
    
    This change moves the RST DLO documentation from
    statically inside overview_large_objects.rst and moves it
    to middleware/dlo.py.
    This is where all middleware RST documentation is defined.
    
    The overview_large_objects.rst is still the main page
    for information on large objects, so now dynamically
    points to both the DLO and SLO middleware RST
    documentation and the relevant middleware.rst page
    simply points to it.
    
    Change-Id: I40d918c8b7bc608ab945805d69fe359521df038a
    Closes-bug: #1276852

commit 2996974e5d48b4efaa1b271b8fbd0387bced7242
Author: Ondřej Nový <ondrej.novy@firma.seznam.cz>
Date:   Sat Oct 10 14:56:30 2015 +0200

Script for running unit, func and probe tests at once
    
    When developing Swift it's often needed to run all tests.
    This script makes it much simpler.
    
    Change-Id: I67e6f7cc05ebd0475001c1b56e8f6fd09c8c644f

commit c2182fd4163050a5f76eb3dedb7703dc821fa83d
Author: janonymous <janonymous.codevulture@gmail.com>
Date:   Fri Jul 17 20:20:15 2015 +0530

Python3: do not use im_self/im_func/func_closure
    
    Use __self__, __func__ and __closure__ instead, as they work
    with both Python 2 and 3.
    
    Modifying usage of __func__ in codebase.
    
    Change-Id: I57e907c28c1d4646605e70194ea3650806730b83

commit c0866ceaac2f69ae01345a795520141f59ec64f5
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Fri Sep 25 17:26:37 2015 -0700

Improve SLO PUT error checking
    
    This commit tries to give the user a reason that their SLO manifest
    was invalid instead of just saying "Invalid SLO Manifest File". It
    doesn't get every error condition, but it's better than before.
    
    Examples of things that now have real error messages include:
     * bad keys in manifest (e.g. using "name" instead of "path")
     * bogus range (e.g. "bytes=123-taco")
     * multiple ranges (e.g. "bytes=10-20,30-40")
     * bad JSON structure (i.e. not a list of objects)
     * non-integer size_bytes
    
    Also fixed an annoyance with unspecified-size segments that are too
    small. Previously, if you uploaded a segment reference with
    '{"size_bytes": null, ...}' in it and the referenced segment was less
    than 1 MiB, you'd get a response that looked like this:
    
        HTTP/1.1 400 Bad Request
        Content-Length: 62
        Content-Type: text/html; charset=UTF-8
        X-Trans-Id: txd9ee3b25896642098e4d9-0055dd095a
        Date: Wed, 26 Aug 2015 00:33:30 GMT
    
        Each segment, except the last, must be at least 1048576 bytes.
    
    This is true, but not particularly helpful, since it doesn't tell you
    which of your segments violated the rule.
    
    Now you get something more like this:
    
        HTTP/1.1 400 Bad Request
        Content-Length: 49
        Content-Type: text/plain
        X-Trans-Id: tx586e52580bac4956ad8e2-0055dd09c2
        Date: Wed, 26 Aug 2015 00:35:14 GMT
    
        Errors:
        /segs/small, Too Small; each segment, except the last...
    
    It's not exactly a tutorial on SLO manifests, but at least it names
    the problematic segment.
    
    This also changes the status code for a self-referential manifest from
    409 to 400. The rest of the error machinery was using 400, and
    special-casing self-reference would be really annoying. Besides, now
    that we're showing more than one error message at a time, what would
    the right status code be for a manifest with a self-referential
    segment *and* a segment with a bad range? 400? 409? 404.5? It's much
    more consistent to just say invalid manifest --> 400.
    
    Change-Id: I2275683230b36bc273319254e37c16b9e9b9d69c

commit 9cafa472a336f66d149a20c12f4251703d96f04d
Author: Ondřej Nový <ondrej.novy@firma.seznam.cz>
Date:   Sat Oct 10 20:57:07 2015 +0200

Autodetect systemctl in SAIO and use it on systemd distros
    
    Change-Id: I84a9b27baac89327749d8774032860f8ad5166f2

commit 92767f28d668643bc2affee7b2fd46fd9349656a
Author: Emile Snyder <emile.snyder@gmail.com>
Date:   Sun Oct 11 21:24:54 2015 -0700

Fix 'swift-ring-builder write_builder' after you remove a device
    
    clayg already posted the code fix in the bug, but noted it needs a test.
    
    Closes-Bug: #1487280
    Change-Id: I07317754afac7165baac4e696f07daeba2e72adc

commit a48649002970b2150d24d0622a100f54045443c5
Author: Lisak, Peter <peter.lisak@firma.seznam.cz>
Date:   Mon Oct 12 14:42:01 2015 +0200

swift-recon fails with socket.timeout exception
    
    If some server is overloaded or timeout set too low, swift-recon fails with
    raised socket.timeout exception.
    
    This error should be processed the same way as HTTPError/URLError.
    
    Change-Id: Ide8843977ab224fa866097d0f0b765d6899c66b8

commit 767fac8186ea4541f4466ac9a55c03abea6a878b
Author: Christian Schwede <cschwede@redhat.com>
Date:   Mon Oct 12 07:09:00 2015 +0000

Enable H234 check (assertEquals is deprecated, use assertEqual)
    
    All usages of assertEquals and assertNotEquals are fixed now, so let's enable
    the H234 check to avoid regressions in the future.
    
    Change-Id: I2c2ccb3b268cf9eb11f2db045378ab125a02bc31

commit 1882801be1d8983cd718786bd409cf09f65a00b0
Author: janonymous <janonymous.codevulture@gmail.com>
Date:   Mon Aug 31 21:49:49 2015 +0530

pep8 fix: assertNotEquals -> assertNotEqual
    
    assertNotEquals is deprecated in py3
    
    Change-Id: Ib611351987bed1199fb8f73a750955a61d022d0a

commit f5f9d791b0b8b32350bd9a47fbc00ff86a65f09d
Author: janonymous <janonymous.codevulture@gmail.com>
Date:   Wed Aug 5 23:58:14 2015 +0530

pep8 fix: assertEquals -> assertEqual
    
    assertEquals is deprecated in py3, replacing it.
    
    Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8
    Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>

commit 84f0a544452fefc65f6755fdd04cc2cc1eb11c88
Author: Victor Stinner <vstinner@redhat.com>
Date:   Thu Oct 8 15:42:27 2015 +0200

py3: Replace basestring with six.string_types
    
    The builtin basestring type was removed in Python 3. Replace it with
    six.string_types which works on Python 2 and Python 3.
    
    Change-Id: Ib92a729682322cc65b41050ae169167be2899e2c

commit 2447e0cf6ed502375a80e33e6aa4a5110305fa6f
Author: Victor Stinner <vstinner@redhat.com>
Date:   Thu Oct 8 15:41:02 2015 +0200

Replace itertools.ifilter with six.moves.filter
    
    Replace itertools.ifilter() with six.moves.filter(). The
    itertools.ifilter was removed in Python 3. This change makes modified
    code compatible with Python 2 and Python 3.
    
    Replace itertools.izip() with six.moves.zip().
    
    The patch was generated by the itertools operation of the sixer tool.
    
    Change-Id: Ie7f787cc6d66edfceb8fa2c1a906351a8c8c5fed

commit 1ba7641c794104de57e5010f76cecbf146a2a63b
Author: Zack M. Davis <zdavis@swiftstack.com>
Date:   Thu Oct 8 16:16:18 2015 -0700

minutæ: port ClientException tweaks from swiftclient; dict .pop
    
    openstack/python-swiftclient@5ae4b423 changed python-swiftclient's
    ClientException to have its http_status attribute default to
    None (rather than 0) and to use super in its __init__ method. For
    consistency's sake, it's nice for Swift's inlined copy of
    ClientException to receive the same patch. Also, the retry function in
    direct_client (a major user of ClientException) was using a somewhat
    awkward conditional-assignment-and-delete construction where the .pop
    method of dictionaries would be more idiomatic.
    
    Change-Id: I70a12f934f84f57549617af28b86f7f5637bd8fa

commit 01f9d15045129d094138d1ec6c84ee4df6944f82
Author: Alistair Coles <alistair.coles@hp.com>
Date:   Thu Oct 8 18:50:20 2015 +0100

Fix EC documentation of .durable quorum
    
    Update the doc to reflect the change [1] to ndata + 1
    .durable files being committed before a success response
    is returned for a PUT.
    
    [1] Ifd36790faa0a5d00ec79c23d1f96a332a0ca0f0b
    
    Change-Id: I1744d457bda8a52eb2451029c4031962e92c2bb7

commit a5d2faab900217914899af9b96d3ced0f321f56b
Author: Lisak, Peter <peter.lisak@firma.seznam.cz>
Date:   Thu Oct 8 14:39:32 2015 +0200

swift-ring-builder can't select id=0
    
    Currently, it is not possible to change weight of device with id=0
    by swift-ring-builder cli. Instead of change the help is shown.
    Example:
    $ swift-ring-builder object.builder set_weight --id 0 1.00
    
    But id=0 is generated by swift for the first device if not provided.
    Also --weight, --zone and --region cause the same bug.
    
    There is problem to detect new command format in validate_args
    function if zero is as valid value for some args.
    
    Change-Id: I4ee379c242f090d116cd2504e21d0e1904cdc2fc

commit 8f8542793913e87f41d4789aa5327d611101b2cb
Author: Victor Stinner <vstinner@redhat.com>
Date:   Thu Oct 8 15:38:36 2015 +0200

py3: Replace gen.next() with next(gen)
    
    The next() method of Python 2 generators was renamed to __next__().
    Call the builtin next() function instead which works on Python 2 and
    Python 3.
    
    The patch was generated by the next operation of the sixer tool.
    
    Change-Id: Id12bc16cba7d9b8a283af0d392188a185abe439d

commit 6a82097b0e92d0b9b70c52217322ca33b5da8b28
Author: Victor Stinner <vstinner@redhat.com>
Date:   Thu Oct 8 15:32:33 2015 +0200

py3: Use six.reraise() to reraise an exception
    
    Replace "raise exc_type, exc_value, exc_tb" with
    "six.reraise(exc_type, exc_value, exc_tb)".
    
    The patch was generated by the raise operation of the sixer tool on:
    bin/* swift/ test/.
    
    Change-Id: Ic4ca6d7f26d1e0075bd2a8a26d6e408b59b17fbb

commit c0af385173658fa149bddf155aeb1ae0bbd4eb7e
Author: Victor Stinner <vstinner@redhat.com>
Date:   Thu Oct 8 15:03:52 2015 +0200

py3: Replace urllib imports with six.moves.urllib
    
    The urllib, urllib2 and urlparse modules of Python 2 were reorganized
    into a new urllib namespace on Python 3. Replace urllib, urllib2 and
    urlparse imports with six.moves.urllib to make the modified code
    compatible with Python 2 and Python 3.
    
    The initial patch was generated by the urllib operation of the sixer
    tool on: bin/* swift/ test/.
    
    Change-Id: I61a8c7fb7972eabc7da8dad3b3d34bceee5c5d93

commit f2cac20d17b081e7b9b6285546414902aa2bdbec
Author: Victor Stinner <vstinner@redhat.com>
Date:   Thu Oct 8 13:08:45 2015 +0200

py3: Replace unicode with six.text_type
    
    The unicode type was renamed to str in Python 3. Use six.text_type to
    make the modified code compatible with Python 2 and Python 3.
    
    The initial patch was generated by the unicode operation of the sixer
    tool on: bin/* swift/ test/.
    
    Change-Id: I9e13748ccde36ee8110756202d55d3ae945d4860

commit c30ceec6f1cfde4da53ab1854d7b2d6c4d7f3a2b
Author: Christian Schwede <cschwede@redhat.com>
Date:   Wed Oct 7 19:48:17 2015 +0000

Fix ring device checks in probetests
    
    If a device has been removed from one of the rings, it actually is set as None
    within the ring. In that case the length of the devices is not True without
    filtering the None devices. However, if the length matched the condition but
    included a removed device the probetests would fail with a TypeError.
    
    This fix could be done also in swift/common/ring/ring.py, but it seems it only
    affects probetests right now, thus fixing it there and not changing the current
    behavior.
    
    Change-Id: I8ccf9b32a51957e040dd370bc9f711d4328d17b1

commit d01cd425094c2e56e4e89dbf3eaf887815dd5b62
Author: Charles Hsu <charles0126@gmail.com>
Date:   Tue Oct 6 16:28:34 2015 +0800

Fix replicator intersection exception when sync data to remote regions.
    
    This patch fixed the exception (AttributeError: 'list' object has no
    attribute 'intersection') when replicator try to sync data from
    handoff to primary partition in more than one remote region.
    
    Change-Id: I565c45dda8c99d36e24dbf1145f2d2527d593ac0
    Closes-Bug: 1503152

commit 06bede894227062eabceb0fdfac8146cdabf4c8b
Author: Zack M. Davis <zdavis@swiftstack.com>
Date:   Tue Sep 8 19:21:39 2015 -0700

replace use of deprecated rfc822.Message with a helper utility
    
    The rfc822 module has been deprecated since Python 2.3, and in
    particular is absent from the Python 3 standard library. However, Swift
    uses instances of rfc822.Message in a number of places, relying on its
    behavior of immediately parsing the headers of a file-like object
    without consuming the body, leaving the position of the file at the
    start of the body. Python 3's http.client has an undocumented
    parse_headers function with the same behavior, which inspired the new
    parse_mime_headers utility introduced here. (The HeaderKeyDict returned
    by parse_mime_headers doesn't have a `.getheader(key)` method like
    rfc822.Message did; the dictionary-like `[key]` or `.get(key)` interface
    should be used exclusively.)
    
    The implementation in this commit won't actually work with Python 3, the
    email.parser.Parser().parsestr of which expects a Unicode string, but it
    is believed that this can be addressed in followup work.
    
    Change-Id: Ia5ee2ead67e36e8c6416183667f64ae255887736

commit 47eb6a37f86f29c355297b556c2ff898c98da9b2
Author: John Dickinson <me@not.mn>
Date:   Fri Oct 2 17:06:49 2015 -0700

authors and changelog update for 2.5.0
    
    Change-Id: Id20b9340a017922b29d8bf9558825697a7f1f6f1

commit a8c2978707ac727c870a5707932f523a063d8b3c
Author: Victor Stinner <vstinner@redhat.com>
Date:   Thu Aug 27 01:03:27 2015 +0200

py3: Fix Python 3 issues in utils
    
    * Replace urllib imports with six.moves.urllib
    * Don't access private logging._levelNames attribute, but use the
      public function logging.addLevelName() instead.
    * Replace basestring with six.string_types
    
    Change-Id: I4cd5dd71ffb40f84e8844b5808b38630795ad520

commit 5070869ac0e6a2d577dd4054ffbcbffd06db3c5b
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Fri Sep 11 16:24:52 2015 -0700

Validate against duplicate device part replica assignment
    
    We should never assign multiple replicas of the same partition to the
    same device - our on-disk layout can only support a single replica of a
    given part on a single device.  We should not do this, so we validate
    against it and raise a loud warning if this terrible state is ever
    observed after a rebalance.
    
    Unfortunately currently there's a couple not necessarily uncommon
    scenarios which will trigger this observed state today:
    
     1. If we have less devices than replicas
     2. If a server or zones aggregate device weight make it the most
        appropriate candidate for multiple replicas and you're a bit unlucky
    
    Fixing #1 would be easy, we should just not allow that state anymore.
    Really we never did - if you have a 3 replica ring with one device - you
    have one replica.  Everything that iter_nodes'd would de-dupe.  We
    should just be insisting that you explicitly acknowledge your replica
    count with set_replicas.
    
    I have been lost in the abyss for days searching for a general solutions
    to #2.  I'm sure it exists, but I will not have wrestled it to
    submission by RC1.  In the meantime we can eliminate a great deal of the
    luck required simply by refusing to place more than one replica of a
    part on a device in assign_parts.
    
    The meat of the change is a small update to the .validate method in
    RingBuilder.  It basically unrolls a pre-existing (part, replica) loop
    so that all the replicas of the part come out in order so that we can
    build up the set of dev_id's for which all the replicas of a given part
    are assigned part-by-part.
    
    If we observe any duplicates - we raise a warning.
    
    To clean the cobwebs out of the rest of the corner cases we're going to
    delay get_required_overload from kicking in until we achive dispersion,
    and a small check was added when selecting a device subtier to validate
    if it's already being used - picking any other device in the tier works
    out much better.  If no other devices are available in the tier - we
    raise a warning.  A more elegant or optimized solution may exist.
    
    Many unittests did not meet the criteria #1, but the fix was straight
    forward after being identified by the pigeonhole check.
    
    However, many more tests were affected by #2 - but again the fix came to
    be simply adding more devices.  The fantasy that all failure domains
    contain at least replica count devices is prevalent in both our ring
    placement algorithm and it's tests.  These tests were trying to
    demonstrate some complex characteristics of our ring placement algorithm
    and I believe we just got a bit too carried away trying to find the
    simplest possible example to demonstrate the desirable trait.  I think
    a better example looks more like a real ring - with many devices in each
    server and many servers in each zone - I think more devices makes the
    tests better.  As much as possible I've tried to maintain the original
    intent of the tests - when adding devices I've either spread the weight
    out amongst them or added proportional weights to the other tiers.
    
    I added an example straw man test to validate that three devices with
    different weights in three different zones won't blow up.  Once we can
    do that without raising warnings and assigning duplicate device part
    replicas - we can add more.  And more importantly change the warnings to
    errors - because we would much prefer to not do that #$%^ anymore.
    
    Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
    Related-Bug: #1452431
    Change-Id: I592d5b611188670ae842fe3d030aa3b340ac36f9

commit a31ee07bda1a5ba8034277af9647aa21ad10be32
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Fri Oct 2 13:53:27 2015 -0700

Make sure we have enough .durable's for GETs
    
    Increase the number of nodes from which we require a final successful
    HTTP responses before we return success to the client on a write - to
    the same number of nodes we'll require successful responses from to
    service a client request for a read.
    
    Change-Id: Ifd36790faa0a5d00ec79c23d1f96a332a0ca0f0b
    Related-Bug: #1469094

commit 29c10db0cbb1369a99c3c63d6f583951ba828b8e
Author: Alistair Coles <alistair.coles@hp.com>
Date:   Wed Apr 22 12:56:50 2015 +0100

Add POST capability to ssync for .meta files
    
    ssync currently does the wrong thing when replicating object dirs
    containing both a .data and a .meta file. The ssync sender uses a
    single PUT to send both object content and metadata to the receiver,
    using the metadata (.meta file) timestamp. This results in the object
    content timestamp being advanced to the metadata timestamp,
    potentially overwriting newer object data on the receiver and causing
    an inconsistency with the container server record for the object.
    
    For example, replicating an object dir with {t0.data(etag=x), t2.meta}
    to a receiver with t1.data(etag=y) will result in the creation of
    t2.data(etag=x) on the receiver. However, the container server will
    continue to list the object as t1(etag=y).
    
    This patch modifies ssync to replicate the content of .data and .meta
    separately using a PUT request for the data (no change) and a POST
    request for the metadata. In effect, ssync replication replicates the
    client operations that generated the .data and .meta files so that
    the result of replication is the same as if the original client requests
    had persisted on all object servers.
    
    Apart from maintaining correct timestamps across sync'd nodes, this has
    the added benefit of not needing to PUT objects when only the metadata
    has changed and a POST will suffice.
    
    Taking the same example, ssync sender will no longer PUT t0.data but will
    POST t2.meta resulting in the receiver having t1.data and t2.meta.
    
    The changes are backwards compatible: an upgraded sender will only sync
    data files to a legacy receiver and will not sync meta files (fixing the
    erroneous behavior described above); a legacy sender will operate as
    before when sync'ing to an upgraded receiver.
    
    Changes:
    - diskfile API provides methods to get the data file timestamp
      as distinct from the diskfile timestamp.
    
    - diskfile yield_hashes return tuple now passes a dict mapping data and
      meta (if any) timestamps to their respective values in the timestamp
      field.
    
    - ssync_sender will encode data and meta timestamps in the
      (hash_path, timestamp) tuple sent to the receiver during
      missing_checks.
    
    - ssync_receiver compares sender's data and meta timestamps to any
      local diskfile and may specify that only data or meta parts are sent
      during updates phase by appending a qualifier to the hash returned
      in its 'wanted' list.
    
    - ssync_sender now sends POST subrequests when a meta file
      exists and its content needs to be replicated.
    
    - ssync_sender may send *only* a POST if the receiver indicates that
      is the only part required to be sync'd.
    
    - object server will allow PUT and DELETE with earlier timestamp than
      a POST
    
    - Fixed TODO related to replicated objects with fast-POST and ssync
    
    Related spec change-id: I60688efc3df692d3a39557114dca8c5490f7837e
    
    Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
    Closes-Bug: 1501528
    Change-Id: I97552d194e5cc342b0a3f4b9800de8aa6b9cb85b

commit 969f1ea9392589c22266ecd5cba326cbd4630fa3
Author: Tim Burke <tim.burke@gmail.com>
Date:   Thu Oct 1 21:10:43 2015 +0000

Fix slorange on-disk format when including whole object
    
    Not that the current implementation is broken, just wasteful.
    
    When a client specifies a range for an SLO segment that includes the
    entire referenced object, we should drop the 'range' key from the
    manifest that's stored on disk.
    
    Previously, we would do this if the uploaded manifest included the
    object-length for validation, but not if it didn't. Now we will
    always drop the 'range' key if the entire segment is being used.
    
    Change-Id: I69d2fff8c7c59b81e9e4777bdbefcd3c274b59a9
    Related-Change: Ia21d51c2cef4e2ee5162161dd2c1d3069009b52c

commit 8f6fd855a1ece5c23d9795a066e305532d56d304
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Wed Sep 23 16:58:54 2015 -0700

Add search filter examples to swift-ring-builder dispersion help
    
    ... because I always forget how to regex and have to independently
    discover it everytime I want to use the tool!
    
    Change-Id: I00d5ab6f573ef26e7e10502493c0066623583b00

commit 4b8f52b1536ad6ea36719b2bfd4cce29fbc843c6
Author: Christian Schwede <cschwede@redhat.com>
Date:   Mon Aug 10 08:44:20 2015 +0000

Fix copy requests to service accounts in Keystone
    
    In case of a COPY request the swift_owner was already set to True, and the
    following PUT request was granted access no matter if a service token was used
    or not.  This allowed to copy data to service accounts without any service
    token.
    
    Service token unit tests have been added to verify that when
    swift_owner is set to True in a request environ, this setting is
    ignored when authorizing another request based on the same
    environ. Applying only this test change on master fails currently, and
    only passes with the fix in this patch.
    
    Tempauth seems to be not affected, however a small doc update has been added to
    make it more clear that a service token is not needed to access a service account
    when an ACL is used.
    
    Further details with an example are available in the bug report
    (https://bugs.launchpad.net/swift/+bug/1483007).
    
    Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
    Co-Authored-By: Hisashi Osanai <osanai.hisashi@jp.fujitsu.com>
    Co-Authored-By: Donagh McCabe <donagh.mccabe@hp.com>
    
    Closes-Bug: 1483007
    Change-Id: I1207b911f018b855362b1078f68c38615be74bbd

commit 167f3c8cbdc09b8161d3a4fcc5b5c57091f0845c
Author: Alistair Coles <alistair.coles@hp.com>
Date:   Wed Sep 30 09:45:57 2015 +0100

Update EC overview doc for PUT path
    
    Update the EC overview docs 'under the hood' section to reflect the
    change in durable file parity from 2 to ec_nparity + 1 [1].
    
    Also fix some typos and cleanup the text.
    
    [1] change id I80d666f61273e589d0990baa78fd657b3470785d
    
    Change-Id: I23f6299da59ba8357da2bb5976d879d9a4bb173e

commit 590e80870c7b7eac6ee389b532bfb2c5029577b2
Author: John Dickinson <me@not.mn>
Date:   Tue Sep 29 15:00:53 2015 -0700

fix docstring: s/2xx/1xx/
    
    Change-Id: If863eb4e66e400081d2402ec8fbf0f9fe8f55b7c

commit 8f1c7409e7b6a854125a234b8a2b969075d26dae
Author: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Date:   Thu Sep 3 00:40:41 2015 -0700

Don't send commits for quorum *BAD* requests on EC
    
    In EC PUT request case, proxy-server may send commits to object-servers
    it may make .durable file even though the request failed due to a lack
    of quorum number.
    
    For example:
    - Considering the case that almost all object-servers fail by 422
      Unprocessable Entity
    - Using ec scheme 4 + 2
    - 5 (quorum size) object-server failed with 422, 1 object-servers
      succeeded as 201 created
    
    How it works:
    - Client creates a PUT request
    - Proxy will open connections to backend object-servers
    - Proxy will send whole encoded chunks to object-servers
    - Proxy will send content-md5 as footers.
    - Proxy will get responses [422, 422, 422, 422, 422, 201] (currently
      this list will be regarded as "we have quorum response")
    - And then proxy will send commits to object-servers (the only
      object-server with 201 will create .durable file)
    - Proxy will return 503 because the commits results in no response
      statuses from object-servers except the 201 node.
    
    This patch fixes the quorum handling at ObjectController to check
    that it has *successful* quorum responses before sending durable commits.
    
    Closes-Bug: #1491748
    Change-Id: Icc099993be76bcc687191f332db56d62856a500f

commit 224c40fa6730ed5f3ae9fa049959433901b23889
Author: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Date:   Mon Sep 28 04:50:46 2015 -0700

Fix inlines for test/unit/obj/test_server.py
    
    This patch fixes small nits for inline comments for
    
    https://review.openstack.org/#/c/211338
    
    as a follow-up patch, plus some other typos in comments.
    
    Change-Id: Ibf7dc5683b39d6662573dbb036da146174a965fd

commit be9a8fb56eb3579a20b18d2e0c8478561bea93ca
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Mon Sep 28 06:27:21 2015 +0000

Imported Translations from Zanata
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: I59e314778d95bce32ab05bfeca2067819180dd30

commit 3f943cfcf2de26e51f1ace96f2c28a36ab105887
Author: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Date:   Fri May 22 16:53:12 2015 -0700

Fix missing container update
    
    At PUT object request, proxy server makes backend headers (e.g.
    X-Container-Partition) which help object-servers to determine
    the container-server they should update. In addition, the backend
    headers are created as many as the number of container replicas.
    (i.e. 3 replica in container ring, 3 backend headers will be created)
    
    On EC case, Swift fans out fragment archives to backend object-servers.
    Basically the number of fragment archives will be more than the container
    replica number and proxy-server assumes a request as success when quorum
    number of object-server succeeded to store. That would cause to make an
    orphaned object which is stored but not container updated.
    
    For example, assuming k=10, m=4, container replica=3 case:
    
    Assuming, proxy-server attempts to make 14 backend streams but
    unfortunately first 3 nodes returns 507 (disk failure) and then
    the Swift doesn't have any other disks.
    
    In the case, proxy keeps 11 backend streams to store and current Swift
    assumes it as sufficient because it is more than or equals quorum (right
    now k+1 is sufficient i.e. 11 backend streams are enough to store)
    However, in the case, the 11 streams doesn't have the container update
    header so that the request will succeed but container will be never updated.
    
    This patch allows to extract container updates up to object quorum_size
    + 1 to more nodes to ensure the updates. This approach sacrifices the
    container update cost a bit because duplicated updates will be there but
    quorum sizes + 1 seems reasonable (even if it's reaplicated case) to pay
    to ensure that instead of whole objects incude the update headers.
    
    Now Swift will work like as follows:
    
    For example:
    k=10, m=4, qurum_size=11 (k+1), 3 replica for container.
    CU: container update
    CA: commit ack
    
    That result in like as
     CU   CU   CU   CU   CU   CU   CU   CU   CU   CU   CU   CU
    [507, 507, 507, 201, 201, 201, 201, 201, 201, 201, 201, 201, 201, 201]
                                                  CA   CA   CA   CA   CA
    
    In this case, at least 3 container updates are saved.
    
    For another example:
    7 replicated objects, qurum_size=4 (7//2+1), 3 replica for container.
    CU: container update
    CA: commit ack (201s for successful PUT on replicated)
    
     CU   CU   CU   CU   CU
    [507, 507, 507, 201, 201, 201, 201]
                     CA   CA   CA   CA
    
    In this replicated case, at least 2 container updates are saved.
    
    Cleaned up some unit tests so that modifying policies doesn't leak
    between tests.
    
    Co-Authored-By: John Dickinson <me@not.mn>
    Co-Authored-By: Sam Merritt <sam@swiftstack.com>
    
    Closes-Bug: #1460920
    Change-Id: I04132858f44b42ee7ecf3b7994cb22a19d001d70

commit 2a8b455c4a478f7e3e0fda13e6b86a90560ed6b8
Author: Tim Burke <tim.burke@gmail.com>
Date:   Thu Sep 24 21:42:20 2015 +0000

Only yield the pending segment on error if it's a SegmentListingError
    
    (This happens, for example, when we can't read a submanifest for SLO,
    or can't get the next page of objects for DLO)
    
    This should prevent warnings like "Exception RuntimeError: 'generator
    ignored GeneratorExit' in <generator object _coalesce_requests at
    0x4f9bf50> ignored" when running .unittests
    
    Change-Id: I5000053827369553669b1fb5fc9752b46c2cc782

commit 696186c680540606a8cea6495b39e23085b6863f
Author: paul luse <paul.e.luse@intel.com>
Date:   Mon Aug 10 14:37:10 2015 -0700

Better error handling for EC PUT path when client goes away
    
    There are a few places in the PUT path where the object server is
    reading WSGI input and can find that there's nothing there.  e.g. in the
    middle of a 2 phase commit and the proxy goes away for whatever reason,
    like maybe it timed out because things are really busy.  Anyway, this
    results in the ugly ValueError coming out of eventlet.wsgi about a
    zillion levels away from the PUT path.
    
    Expanding on the test cases from lp bug #1496205 and lp bug #1469094
    this change carefully narrows into our read/readline calls to
    wsgi_input and makes sure to tranlsate the ValueError to a
    ChunkReadError - which the object.server can handle along with
    ChunkReadTimeout.  When it made sense, this change attempts to stay
    consistent throughout the code path in logging/raising client disconnect
    instead of timeout.
    
    It's unfortunate the error coming out of eventlet is so generic, but
    that will be improved in future versions [1].
    
    1. https://github.com/eventlet/eventlet/commit/c3ce3eef0b4d0dfdbfb1ec0186d4bb204fb8ecd5
    
    Related-Bug: #1469094
    Related-Bug: #1496205
    Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
    Change-Id: I9e4dbf26623c0c6fc5c87afd14349466aa157385

commit 25d5e686a172a2418fc6e4028174a354df29e0f1
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue Aug 11 00:42:30 2015 -0500

Add the ability to specify ranges for SLO segments
    
    Users can now include an optional 'range' field in segment descriptions
    to specify which bytes from the underlying object should be used for the
    segment data. Only one range may be specified per segment. Note that the
    'etag' and 'size_bytes' fields still describe the backing object as a
    whole. So, if a user uploads a manifest like:
    
        [{"path": "/con/obj_seg_1", "etag": null, "size_bytes": 1048576,
          "range": "0-1023"},
         {"path": "/con/obj_seg_2", "etag": null, "size_bytes": 1048576,
          "range": "512-4095"},
         {"path": "/con/obj_seg_1", "etag": null, "size_bytes": 1048576,
          "range": "-2048"}]
    
    then the segment will consist of the first 1024 bytes of /con/obj_seg_1,
    followed by bytes 513 through 4096 (inclusive) of /con/obj_seg_2, and
    finally bytes 1046528 through 1048576 (i.e., the last 2048 bytes) of
    /con/obj_seg_1.
    
    ETag generation for SLOs had been updated to prevent collisions when
    using different ranges for the same set of objects.
    
    Additionally, there are two performance enhancements:
    
     * On download, multiple sequential requests for segments from the same
       underlying object will be coalesced into a single ranged request,
       provided it still does not meet Swift's "egregious range requests"
       critieria.
    
     * On upload, multiple sequential segments referencing the same object
       will be validated against the response from a single HEAD request.
    
    Change-Id: Ia21d51c2cef4e2ee5162161dd2c1d3069009b52c
    DocImpact

commit 3afdcf6b8fa74051fb6da0bf24799c80399624fd
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Thu Sep 17 09:54:30 2015 -0700

Fix proxy handling of EC client disconnect
    
    The ECObjectController was unconditionally sending down the frag archive
    commit document after the client source stream terminated - even if the
    client disconnected early.
    
    We can detect early disconnect in two ways:
    
      1. Content-Length and not enough bytes_transfered
    
         When eventlet.wsgi is reading from a Content-Length body the
         readable returns the empty string and our iterable raises
         StopIteration - but we can check content-length against
         bytes_transfered and know if the client disconnected.
    
      2. Transfer-Encoding: chunked - w/o a 0\r\n\r\n
    
         When eventlet.wsgi is reading from a Transfer-Encoding: chunked
         body the socket read returns the empty string, eventlet.wsgi's
         chunked parser raises ValueError (which we translate to
         ChunkReadError*) and we know we know the client disconnected.
    
    ... if we detect either of these conditions the proxy should:
    
      1. *not* send down the commit document to object servers
      2. disconnect from backend servers
      3. log the client disconnect
    
    Oddly the code on master was only messing up the first part.  Backend
    connections were terminated (gracefully after the commit document), and
    then the disconnect was being logged as 499.
    
    So now we only send down the commit document on a successful complete
    client HTTP request (either whole Content-Length, or clean
    Transfer-Encoding: chunked 0\r\n\r\n).
    
     * To detect the early disconnect on Transfer-Encoding: chunked a new
    swift.common.exceptions.ChunkReadError is used to translate
    eventlet.wsgi's more general IOError and ValueErrors into something
    more appropriate to catch and handle closer to our generic
    ChunkReadTimeout handling.
    
    Co-Author: Alistair Coles <alistair.coles@hp.com>
    Closes-Bug: #1496205
    Change-Id: I028a530aba82d50baa4ee1d05ddce18d4cce4e81

commit 371cd40f90e773ba74170c2000b1d9b4dc870c73
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Tue Sep 22 07:36:39 2015 +0000

Imported Translations from Zanata
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: I800cbf3d5cbc510b60cdf4f407d91b9216830094

commit 9046676968a27191a9a96b19924e3fd484b6653b
Author: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Date:   Wed Sep 16 06:54:57 2015 -0700

Fix client disconnect during multhiphase commit
    
    This patch add a test to figure out the failure case behavior of
    object-server when the connection from proxy-server disconnected
    during commit phase. Especially, this patch was made to focus on
    making sure whether or not contaienr updates occurs in the situation.
    
    In the process of working on that test we made the behavior of the
    object-server when the connection from the proxy-server disconnected
    during the commit phase - reasonable.
    
    We capture the IOError/ValueError's that eventlet.wsgi might barf out
    really close to the wsgi_input read and translate them to a
    swift.common.exceptions.ChunkReadError so we can handle them at a higher
    level in the ObjectController's generic PUT disconnect handling.
    
    Since that test went so well, we refactored the other ones to use some
    common context management and wrote a few more.
    
    Co-Author: Clay Gerrard <clay.gerrard@gmail.com>
    Change-Id: I60c98172e524869b06bdf23fd1c4e1bce7a98f80

commit db0e0b8f394d1714679e274fdad23225b6dcf716
Author: Alistair Coles <alistair.coles@hp.com>
Date:   Fri Sep 11 16:04:08 2015 +0100

diskfile: make get_ondisk_files return a dict
    
    get_ondisk_files gathers a dict with more info than just the set
    of data, ts and/or meta files, but only returns the
    set of files to the DiskFile class. The other info
    in the dict may be useful policy specific implementations, so
    return the whole dict.
    
    Also, refactor _get_ondisk_file to move duplicated code into the
    BaseDiskFile open() method.
    
    Change-Id: I7a17d26b7ed7f7c593f577332937793419c03cfa
    Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
    Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>