Bug #1722951 “Container updater may be stuck and not make progre...” : Series ocata : Bugs : Ubuntu Cloud Archive

Revision history for this message

Timur Alperovich (timur-alperovich) wrote on 2017-10-11:

#1

pipemutex_test.py Edit (2.2 KiB, text/x-python)

Revision history for this message

Samuel Merritt (torgomatic) wrote on 2017-10-11:

#2

The bug is real, but isn't PipeMutex's fault. We're letting eventlet pick the default hub sometimes. Eventlet chooses the epoll hub, and epoll doesn't play nicely with forking. (It's possible to use epoll and fork and still be correct, but eventlet doesn't make it so.)

Changed in swift:
importance:	Undecided → High
status:	New → Confirmed

Samuel Merritt (torgomatic) on 2017-10-12

Changed in swift:
assignee:	nobody → Samuel Merritt (torgomatic)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-10-12: Fix proposed to swift (master)

#3

Fix proposed to branch: master
Review: https://review.openstack.org/511550

Changed in swift:
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-10-16: Fix merged to swift (master)

#4

Reviewed: https://review.openstack.org/511550
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=dc8da5bb194d0a544fb8065aa9a4c7484f605715
Submitter: Zuul
Branch: master

commit dc8da5bb194d0a544fb8065aa9a4c7484f605715
Author: Samuel Merritt <email address hidden>
Date: Thu Oct 12 10:45:12 2017 -0700

Use "poll" or "selects" Eventlet hub for all Swift daemons.

    Previously, Swift's WSGI servers, the object replicator, and the
    object reconstructor were setting Eventlet's hub to either "poll" or
    "selects", depending on availability. Other daemons were letting
    Eventlet use its default hub, which is "epoll".

    In any daemons that fork, we really don't want to use epoll. Epoll
    instances end up shared between the parent and all children, and you
    get some awful messes when file descriptors are shared.

    Here's an example where two processes are trying to wait on the same
    file descriptor using the same epoll instance, and everything goes
    wrong:

[proc A] epoll_ctl(6, EPOLL_CTL_ADD, 3, ...) = 0

    [proc B] epoll_ctl(6, EPOLL_CTL_ADD, 3, ...) = -1 EEXIST (File exists)
    [proc B] epoll_wait(6, ...) = 1
    [proc B] epoll_ctl(6, EPOLL_CTL_DEL, 3, ...) = 0

[proc A] epoll_wait(6, ...)

    This primarily affects the container updater and object updater since
    they fork. I've decided to change the hub for all Swift daemons so
    that we don't add multiprocessing support to some other daemon someday
    and suffer through this same bug again.

    This problem was made more apparent by commit 6d16079, which made our
    logging mutex use file descriptors. However, it could have struck on
    any shared file descriptor on which a read or write returned EAGAIN.

Change-Id: Ic2c1178ac918c88b0b901e581eb4fab3b2666cfe
Closes-Bug: 1722951

Changed in swift:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-10-20: Fix proposed to swift (feature/s3api)

#5

Fix proposed to branch: feature/s3api
Review: https://review.openstack.org/513730

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-10-20: Fix merged to swift (feature/s3api)

#6

Download full text (8.6 KiB)

Reviewed: https://review.openstack.org/513730
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=bb064f0b30ec083cec85e8371fe12824804779cf
Submitter: Zuul
Branch: feature/s3api

commit 0bca0d4d2bf95a37e4a403457ee5c7db4c0b6290
Author: Pete Zaitcev <email address hidden>
Date: Wed Oct 18 18:15:11 2017 -0500

Be more tolerant of exception messages from sqlite

Parsing the text of error messages was always perilous, perhaps.
Either way, this should fix the problem of test_get() failing.

Change-Id: I929c2f6e60624241075a292d020c5707ea0ff1c3

commit e001c02ff938d9aea560970fa70f3d8884a8ea33
Author: Tim Burke <email address hidden>
Date: Tue Oct 17 22:49:39 2017 +0000

Stop logging tracebacks on bad xLOs

The error messages alone should provide plenty enough information.
Plus, running functional tests really shouldn't cause tracebacks.

Also, tighten up tests for log messages.

Change-Id: I55136484d342af756fa153d971dcb9159a435f13

commit 356b110229ee5e5be93ce1ec60ed2de8f75090bf
Author: Samuel Merritt <email address hidden>
Date: Wed Oct 18 15:09:12 2017 -0700

Remove swift-temp-url man page.

Commit 250da37a removed bin/swift-temp-url, but left the man page.

Change-Id: Ic518f23678e3c3134f02a46a51a2bcb90d92bdc2

commit 1aecd1dfc61988f12d4cdfda7110dd648133e4cf
Author: Alistair Coles <email address hidden>
Date: Wed Oct 18 10:52:52 2017 +0100

tighten up drop_privileges unit tests

add more assertions about args that are passed to
os module functions

Related-Change: Ida15e72ae4ecdc2d6ce0d37bd99c2d86bd4e5ddc
Change-Id: Iee483274aff37fc9930cd54008533de2917157f4

commit e6cf9b4758612f38f7317c52423f317b9a11bbc9
Author: Samuel Merritt <email address hidden>
Date: Tue Oct 17 15:16:43 2017 -0700

Fix some spelling in a docstring

Change-Id: I6b32238da3381848ae56aed1c570d299be72473e

commit 646f7507a1f5cf479b8d2023645009de0a28cd79
Author: Tim Burke <email address hidden>
Date: Tue Oct 17 21:17:57 2017 +0000

Quiet test output when running test_utils.py in isolation

Change-Id: I4cf85d3cd5a20424e9bbbdf0213120b3c3d4b837

commit 0f91c862e19bb8d81c20d03e65e6e6657f7265dc
Author: Corey Bryant <email address hidden>
Date: Tue Oct 17 15:04:45 2017 -0400

Drop group comparison from drop_privileges test

Drop the group comparison from drop_privileges test as it isn't
valid since os.setgroups() is mocked.

Change-Id: Ida15e72ae4ecdc2d6ce0d37bd99c2d86bd4e5ddc
Closes-Bug: #1724342

commit 5438fe7d9bcb7dc99118c12bdf1ccfd663390b08
Author: Samuel Merritt <email address hidden>
Date: Fri Oct 13 17:03:18 2017 -0700

Clean up some leftover Python 2.6-isms.

Change-Id: I24f0aa3bf7314a669c4cf44eadc46e874d6803f1

commit 250da37a7b279b05a1ac8954b506add1661f194d
Author: Tim Burke <email address hidden>
Date: Fri Oct 13 23:27:11 2017 +0000

Remove swift-temp-url script

This has been deprecated since Swift 2.10.0 (Newton) including a
message that it would go away. Let's actually remove it.

Change-Id: I7d3659761c71119363ff2c0c750e37b4c6374a39
Rela...

Reviewed:  https://review.openstack.org/513730
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=bb064f0b30ec083cec85e8371fe12824804779cf
Submitter: Zuul
Branch:    feature/s3api

commit 0bca0d4d2bf95a37e4a403457ee5c7db4c0b6290
Author: Pete Zaitcev <zaitcev@kotori.zaitcev.us>
Date:   Wed Oct 18 18:15:11 2017 -0500

Be more tolerant of exception messages from sqlite
    
    Parsing the text of error messages was always perilous, perhaps.
    Either way, this should fix the problem of test_get() failing.
    
    Change-Id: I929c2f6e60624241075a292d020c5707ea0ff1c3

commit e001c02ff938d9aea560970fa70f3d8884a8ea33
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue Oct 17 22:49:39 2017 +0000

Stop logging tracebacks on bad xLOs
    
    The error messages alone should provide plenty enough information.
    Plus, running functional tests really shouldn't cause tracebacks.
    
    Also, tighten up tests for log messages.
    
    Change-Id: I55136484d342af756fa153d971dcb9159a435f13

commit 356b110229ee5e5be93ce1ec60ed2de8f75090bf
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Wed Oct 18 15:09:12 2017 -0700

Remove swift-temp-url man page.
    
    Commit 250da37a removed bin/swift-temp-url, but left the man page.
    
    Change-Id: Ic518f23678e3c3134f02a46a51a2bcb90d92bdc2

commit 1aecd1dfc61988f12d4cdfda7110dd648133e4cf
Author: Alistair Coles <alistairncoles@gmail.com>
Date:   Wed Oct 18 10:52:52 2017 +0100

tighten up drop_privileges unit tests
    
    add more assertions about args that are passed to
    os module functions
    
    Related-Change: Ida15e72ae4ecdc2d6ce0d37bd99c2d86bd4e5ddc
    Change-Id: Iee483274aff37fc9930cd54008533de2917157f4

commit e6cf9b4758612f38f7317c52423f317b9a11bbc9
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Tue Oct 17 15:16:43 2017 -0700

Fix some spelling in a docstring
    
    Change-Id: I6b32238da3381848ae56aed1c570d299be72473e

commit 646f7507a1f5cf479b8d2023645009de0a28cd79
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue Oct 17 21:17:57 2017 +0000

Quiet test output when running test_utils.py in isolation
    
    Change-Id: I4cf85d3cd5a20424e9bbbdf0213120b3c3d4b837

commit 0f91c862e19bb8d81c20d03e65e6e6657f7265dc
Author: Corey Bryant <corey.bryant@canonical.com>
Date:   Tue Oct 17 15:04:45 2017 -0400

Drop group comparison from drop_privileges test
    
    Drop the group comparison from drop_privileges test as it isn't
    valid since os.setgroups() is mocked.
    
    Change-Id: Ida15e72ae4ecdc2d6ce0d37bd99c2d86bd4e5ddc
    Closes-Bug: #1724342

commit 5438fe7d9bcb7dc99118c12bdf1ccfd663390b08
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Fri Oct 13 17:03:18 2017 -0700

Clean up some leftover Python 2.6-isms.
    
    Change-Id: I24f0aa3bf7314a669c4cf44eadc46e874d6803f1

commit 250da37a7b279b05a1ac8954b506add1661f194d
Author: Tim Burke <tim.burke@gmail.com>
Date:   Fri Oct 13 23:27:11 2017 +0000

Remove swift-temp-url script
    
    This has been deprecated since Swift 2.10.0 (Newton) including a
    message that it would go away. Let's actually remove it.
    
    Change-Id: I7d3659761c71119363ff2c0c750e37b4c6374a39
    Related-Change: Ifa8bf636f20f82db4845b02d1b58699edaa39356

commit c118059719ded85381e9b134b091d04a7b5a3955
Author: Tim Burke <tim.burke@gmail.com>
Date:   Mon Sep 11 22:08:12 2017 +0000

Respond 400 Bad Request when Accept headers fail to parse
    
    Change-Id: I6eb4e4bca95e2ee4fecdb703394cb2419737922d
    Closes-Bug: 1716509

commit 03e8ab7171580f3fb93131e9ccfb69fd35364672
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Thu Oct 12 17:15:54 2017 -0700

functests: don't crash if no second account
    
    In test.functional.test_object.TestObject.setUp, we create a container
    in account 2. However, if we've only got one account, we don't skip
    this class, resulting in a TypeError down in requests somewhere and a
    stack trace. Since we're using account 2 in setup, we should skip the
    tests if account 2 is not configured.
    
    Change-Id: I569d98baf071d2dce7cf34a9538070f00afda388

commit eaea0c49332273085b17852e22582940ab059006
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Thu Oct 12 16:58:41 2017 -0700

Skip cross-account-copy functest if only one account
    
    This looks like a case of copy-paste-itis. The cross-account-copy
    functest is skipped if we have no test accounts configured, but not if
    we have only one.
    
    Change-Id: Ifbefdd9aeb98e3d02c536e9d29759f86ec9af6a1

commit dc8da5bb194d0a544fb8065aa9a4c7484f605715
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Thu Oct 12 10:45:12 2017 -0700

Use "poll" or "selects" Eventlet hub for all Swift daemons.
    
    Previously, Swift's WSGI servers, the object replicator, and the
    object reconstructor were setting Eventlet's hub to either "poll" or
    "selects", depending on availability. Other daemons were letting
    Eventlet use its default hub, which is "epoll".
    
    In any daemons that fork, we really don't want to use epoll. Epoll
    instances end up shared between the parent and all children, and you
    get some awful messes when file descriptors are shared.
    
    Here's an example where two processes are trying to wait on the same
    file descriptor using the same epoll instance, and everything goes
    wrong:
    
    [proc A] epoll_ctl(6, EPOLL_CTL_ADD, 3, ...) = 0
    
    [proc B] epoll_ctl(6, EPOLL_CTL_ADD, 3, ...) = -1 EEXIST (File exists)
    [proc B] epoll_wait(6, ...) = 1
    [proc B] epoll_ctl(6, EPOLL_CTL_DEL, 3, ...) = 0
    
    [proc A] epoll_wait(6, ...)
    
    This primarily affects the container updater and object updater since
    they fork. I've decided to change the hub for all Swift daemons so
    that we don't add multiprocessing support to some other daemon someday
    and suffer through this same bug again.
    
    This problem was made more apparent by commit 6d16079, which made our
    logging mutex use file descriptors. However, it could have struck on
    any shared file descriptor on which a read or write returned EAGAIN.
    
    Change-Id: Ic2c1178ac918c88b0b901e581eb4fab3b2666cfe
    Closes-Bug: 1722951

commit 37f23b072ee7a655ab9072f1d0bece0919dec61b
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Mon Oct 9 17:31:30 2017 -0700

Allow SLOs to have zero-byte last segments.
    
    Since we used to allow zero-byte last segments but now we don't, it
    can be difficult to deal with some old SLO manifests.
    
    Imagine you're writing some code to sync objects from Swift cluster A
    to Swift cluster B. You start off with just a GET from A piped into a
    PUT to B, and that works great until you hit a SLO manifest and B
    won't accept a 500GB object. So, you write some code to detect SLO
    manifests, sync their segments, then take the JSON manifest
    (?multipart-manifest=get) and sync *that* over.
    
    Now, life is good... until one day you get an exception notification
    that there's this manifest on cluster A that cluster B won't
    accept. Turns out that, back when Swift would take zero-byte final
    segments on SLOs (before commit 7f636a5), someone uploaded such a SLO
    to cluster A. Now, however, zero-byte final segments are invalid, so
    that SLO that exists over in cluster A can't just be copied to cluster
    B.
    
    A little coding later, your sync tool detects zero-byte final segments
    and removes them when copying a manifest. But now your ETags don't
    match between clusters, so you have to figure out some way to deal
    with that, and so you put it in metadata, but then you realize that
    your syncer might encounter a SLO which contains a sub-SLO which has a
    zero-byte final segment, and it's right about then that you start
    thinking about giving up on programming and getting a job as an
    elevator mechanic.
    
    This commit makes life easier for developers of such applications by
    allowing SLOs to have zero-byte segments again.
    
    Change-Id: Ia37880bbb435e269ec53b2963eb1b9121696d479

commit a959d24bf5733650c6005412382fe24c65897c8b
Author: Alistair Coles <alistairncoles@gmail.com>
Date:   Tue Aug 15 12:37:33 2017 +0100

Document keystone role element in container ACL
    
    The use of a keystone role name in container ACLs is supported
    and tested. This patch adds documentation.
    
    [1] https://github.com/openstack/swift/blob/fb3d01a974fb7df8cfadc56ff15bdc04b3c90759/swift/common/middleware/keystoneauth.py#L491-L497
    [2] test.unit.common.middleware.test_keystoneauth.TestAuthorize.test_authorize_succeeds_for_user_role_in_roles
    
    Change-Id: I77df27393a10f1d8c5a43161fdd4eb08be632566
    Closes-Bug: #1705300

tags:

added: in-feature-s3api

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-11-01: Fix proposed to swift (stable/ocata)

#7

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/517090

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-11-03: Fix proposed to swift (feature/deep)

#8

Fix proposed to branch: feature/deep
Review: https://review.openstack.org/517718

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-11-04: Fix merged to swift (feature/deep)

#9

Download full text (17.0 KiB)

Reviewed: https://review.openstack.org/517718
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=5c564e53966b74d6d7589c3505dd33816b3c19f1
Submitter: Zuul
Branch: feature/deep

commit 77a8a4455d4a2bd0a9b9146a01acfe0d93fe7550
Author: Tim Burke <email address hidden>
Date: Tue Oct 3 00:11:07 2017 +0000

Let clients request heartbeats during SLO PUTs

    An SLO PUT requires that we HEAD every referenced object; as a result, it
    can be a very time-intensive operation. This makes it difficult as a
    client to differentiate between a proxy-server that's still doing work and
    one that's crashed but left the socket open.

Now, clients can opt-in to receiving heartbeats during long-running PUTs
by including the query parameter

heartbeat=on

    With heartbeating turned on, the proxy will start its response immediately
    with 202 Accepted then send a single whitespace character periodically
    until the request completes. At that point, a final summary chunk will be
    sent which includes a "Response Status" key indicating success or failure
    and (if successful) an "Etag" key indicating the Etag of the resulting SLO.

This mechanism is very similar to the way bulk extractions and deletions
work, and even the way SLO behaves for ?multipart-manifest=delete requests.

    Note that this is opt-in: this prevents us from sending the 202 response
    to existing clients that may mis-interpret it as an immediate indication
    of success.

    Co-Authored-By: Alistair Coles <email address hidden>
    Related-Bug: 1718811
    Change-Id: I65cee5f629c87364e188aa05a06d563c3849c8f3

commit 92705bb36b4a771a757977d11875e7b929c41741
Author: David Rabel <email address hidden>
Date: Thu Nov 2 12:38:41 2017 +0100

Fix indent in overview_policies.rst

Change-Id: I7f070956d8b996db798837392adfca4483067aea

commit feee3998408e5ed03563c317ad9506ead92083a6
Author: Clay Gerrard <email address hidden>
Date: Fri Sep 1 14:15:45 2017 -0700

Use check_drive consistently

    We added check_drive to the account/container servers to unify how all
    the storage wsgi servers treat device dirs/mounts. Thus pushes that
    unification down into the consistency engine.

    Drive-by:
     * use FakeLogger less
     * clean up some repeititon in probe utility for device re-"mounting"

Related-Change-Id: I3362a6ebff423016bb367b4b6b322bb41ae08764
Change-Id: I941ffbc568ebfa5964d49964dc20c382a5e2ec2a

commit 29e9ae1cc5b74dce205643c101601555c06b67dc
Author: Tim Burke <email address hidden>
Date: Thu Oct 19 18:53:04 2017 +0000

Make xml responses less insane

Looking at bulk extractions:

       $ tar c *.py | curl http://saio:8090/v1/AUTH_test/c?extract-archive=tar \
          -H accept:application/xml -T -
       <?xml version="1.0" encoding="UTF-8"?>
       <delete>
       <number_files_created>2</number_files_created>
       <response_body></response_body>
       <response_status>201 Created</response_status>
       <errors>
       </errors>
       </delete>

Or SLO upload failures:

$ curl http://saio:8090/v1/AUTH_...

Reviewed:  https://review.openstack.org/517718
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=5c564e53966b74d6d7589c3505dd33816b3c19f1
Submitter: Zuul
Branch:    feature/deep

commit 77a8a4455d4a2bd0a9b9146a01acfe0d93fe7550
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue Oct 3 00:11:07 2017 +0000

Let clients request heartbeats during SLO PUTs
    
    An SLO PUT requires that we HEAD every referenced object; as a result, it
    can be a very time-intensive operation. This makes it difficult as a
    client to differentiate between a proxy-server that's still doing work and
    one that's crashed but left the socket open.
    
    Now, clients can opt-in to receiving heartbeats during long-running PUTs
    by including the query parameter
    
        heartbeat=on
    
    With heartbeating turned on, the proxy will start its response immediately
    with 202 Accepted then send a single whitespace character periodically
    until the request completes. At that point, a final summary chunk will be
    sent which includes a "Response Status" key indicating success or failure
    and (if successful) an "Etag" key indicating the Etag of the resulting SLO.
    
    This mechanism is very similar to the way bulk extractions and deletions
    work, and even the way SLO behaves for ?multipart-manifest=delete requests.
    
    Note that this is opt-in: this prevents us from sending the 202 response
    to existing clients that may mis-interpret it as an immediate indication
    of success.
    
    Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
    Related-Bug: 1718811
    Change-Id: I65cee5f629c87364e188aa05a06d563c3849c8f3

commit 92705bb36b4a771a757977d11875e7b929c41741
Author: David Rabel <rabel@b1-systems.de>
Date:   Thu Nov 2 12:38:41 2017 +0100

Fix indent in overview_policies.rst
    
    Change-Id: I7f070956d8b996db798837392adfca4483067aea

commit feee3998408e5ed03563c317ad9506ead92083a6
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Fri Sep 1 14:15:45 2017 -0700

Use check_drive consistently
    
    We added check_drive to the account/container servers to unify how all
    the storage wsgi servers treat device dirs/mounts.  Thus pushes that
    unification down into the consistency engine.
    
    Drive-by:
     * use FakeLogger less
     * clean up some repeititon in probe utility for device re-"mounting"
    
    Related-Change-Id: I3362a6ebff423016bb367b4b6b322bb41ae08764
    Change-Id: I941ffbc568ebfa5964d49964dc20c382a5e2ec2a

commit 29e9ae1cc5b74dce205643c101601555c06b67dc
Author: Tim Burke <tim.burke@gmail.com>
Date:   Thu Oct 19 18:53:04 2017 +0000

Make xml responses less insane
    
    Looking at bulk extractions:
    
       $ tar c *.py | curl http://saio:8090/v1/AUTH_test/c?extract-archive=tar \
          -H accept:application/xml -T -
       <?xml version="1.0" encoding="UTF-8"?>
       <delete>
       <number_files_created>2</number_files_created>
       <response_body></response_body>
       <response_status>201 Created</response_status>
       <errors>
       </errors>
       </delete>
    
    Or SLO upload failures:
    
       $ curl http://saio:8090/v1/AUTH_test/c/slo?multipart-manifest=put -X PUT \
          -d '[{"path": "not/found"}]' -H accept:application/xml
       <delete>
       <errors>
       <object><name>not/found</name><status>404 Not Found</status></object></errors>
       </delete>
    
    Why <delete>? Makes no sense.
    
    Drive-by: stop being so quadratic.
    
    Change-Id: I46b233864ba2815ac632d856b9f3c40cc9d0001a

commit 449d83fb0c8668f444baf3c74cf15c26a70bb02e
Author: Matthew Oliver <matt@oliver.net.au>
Date:   Tue Oct 31 15:55:13 2017 +1100

Doc uses alias instead of aliases
    
    The overview_policies doc makes reference to an `alias` option when in
    fact the option is `aliases`.
    The sample storage policy snippet is correct, it's just incorrect when
    listing the possible options.
    
    This change changes the listed option to `aliases`.
    
    Change-Id: Iddf0f19f4d50819ff6abd46e6a1156dc8e4a451d

commit 178d7f37cb80efa668bc81e79d1ee633204b8852
Author: HCLTech-SSW <hcl_ss_oss@hcl.com>
Date:   Wed Oct 25 03:23:47 2017 -0700

Added the man page for container-reconciler.conf
    
    Change-Id: Ic7fc6ddca2ab564b31156fa84b362bc9963825f1
    Closes-Bug: #1607025

commit 83be309627e79c2f4dd66faf17b5459fdf5bfc25
Author: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Date:   Thu Oct 26 18:41:56 2017 +0900

Fix a small typo
    
    That's redundant "to" in an inline comment.
    
    Change-Id: Idab1e11fbbd80a97b0e4ba1c8ab046be99e47d2d

commit 8d882095375dc53acdafafac40aa2efc3bafbcd4
Author: Thiago da Silva <thiago@redhat.com>
Date:   Thu Sep 15 16:45:06 2016 -0400

Return 404 on a GET if tombstone is newer
    
    Currently the proxy keeps iterating through
    the connections in hope of finding a success even
    if it already has found a tombstone (404).
    
    This change changes the code a little bit to compare
    the timestamp of a 200 and a 404, if the tombstone is
    newer, then it should be returned, instead of returning
    a stale 200.
    
    Closes-Bug: #1560574
    
    Co-Authored-By: Tim Burke <tim.burke@gmail.com>
    Change-Id: Ia81d6832709d18fe9a01ad247d75bf765e8a89f4
    Signed-off-by: Thiago da Silva <thiago@redhat.com>

commit e199192caefef068b5bf57da8b878e0bc82e3453
Author: Romain LE DISEZ <romain.le-disez@corp.ovh.com>
Date:   Wed Oct 26 10:53:46 2016 +0200

Replace replication_one_per_device by custom count
    
    This commit replaces boolean replication_one_per_device by an integer
    replication_concurrency_per_device. The new configuration parameter is
    passed to utils.lock_path() which now accept as an argument a limit for
    the number of locks that can be acquired for a specific path.
    
    Instead of trying to lock path/.lock, utils.lock_path() now tries to lock
    files path/.lock-X, where X is in the range (0, N), N being the limit for
    the number of locks allowed for the path. The default value of limit is
    set to 1.
    
    Change-Id: I3c3193344c7a57a8a4fc7932d1b10e702efd3572

commit 9c97c80b2631bedebbd3196e87f0696a6c67b8be
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Fri Oct 20 15:31:07 2017 -0700

Clean up a couple hand-rolled mocks.
    
    Change-Id: I6582985990e8b5e3a0c65bce5d3bb1e39d58dfb9

commit 9b2779ba131aa22893432b5c96fb19bf43656a4f
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Fri Oct 20 14:48:31 2017 -0700

Clean up memcache tests
    
    This is mostly just using mock.patch instead of doing it by hand. One
    test was doing some weird exception counting; now it just uses
    assertRaises in a loop.
    
    Change-Id: I54f903f170226720405df2c5f6845124909ab830