OpenStack Object Storage (swift)

Bug #1714274
Comment #6

Comment 6 for bug 1714274

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-11-06: Fix merged to swift (feature/s3api)

Reviewed: https://review.openstack.org/517838
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=76917dfb1dcc509ca9fa181639f37c50ade0434a
Submitter: Zuul
Branch: feature/s3api

commit 77a8a4455d4a2bd0a9b9146a01acfe0d93fe7550
Author: Tim Burke <email address hidden>
Date: Tue Oct 3 00:11:07 2017 +0000

Let clients request heartbeats during SLO PUTs

    An SLO PUT requires that we HEAD every referenced object; as a result, it
    can be a very time-intensive operation. This makes it difficult as a
    client to differentiate between a proxy-server that's still doing work and
    one that's crashed but left the socket open.

Now, clients can opt-in to receiving heartbeats during long-running PUTs
by including the query parameter

heartbeat=on

    With heartbeating turned on, the proxy will start its response immediately
    with 202 Accepted then send a single whitespace character periodically
    until the request completes. At that point, a final summary chunk will be
    sent which includes a "Response Status" key indicating success or failure
    and (if successful) an "Etag" key indicating the Etag of the resulting SLO.

This mechanism is very similar to the way bulk extractions and deletions
work, and even the way SLO behaves for ?multipart-manifest=delete requests.

    Note that this is opt-in: this prevents us from sending the 202 response
    to existing clients that may mis-interpret it as an immediate indication
    of success.

    Co-Authored-By: Alistair Coles <email address hidden>
    Related-Bug: 1718811
    Change-Id: I65cee5f629c87364e188aa05a06d563c3849c8f3

commit 92705bb36b4a771a757977d11875e7b929c41741
Author: David Rabel <email address hidden>
Date: Thu Nov 2 12:38:41 2017 +0100

Fix indent in overview_policies.rst

Change-Id: I7f070956d8b996db798837392adfca4483067aea

commit feee3998408e5ed03563c317ad9506ead92083a6
Author: Clay Gerrard <email address hidden>
Date: Fri Sep 1 14:15:45 2017 -0700

Use check_drive consistently

    We added check_drive to the account/container servers to unify how all
    the storage wsgi servers treat device dirs/mounts. Thus pushes that
    unification down into the consistency engine.

    Drive-by:
     * use FakeLogger less
     * clean up some repeititon in probe utility for device re-"mounting"

Related-Change-Id: I3362a6ebff423016bb367b4b6b322bb41ae08764
Change-Id: I941ffbc568ebfa5964d49964dc20c382a5e2ec2a

commit 29e9ae1cc5b74dce205643c101601555c06b67dc
Author: Tim Burke <email address hidden>
Date: Thu Oct 19 18:53:04 2017 +0000

Make xml responses less insane

Looking at bulk extractions:

       $ tar c *.py | curl http://saio:8090/v1/AUTH_test/c?extract-archive=tar \
          -H accept:application/xml -T -
       <?xml version="1.0" encoding="UTF-8"?>
       <delete>
       <number_files_created>2</number_files_created>
       <response_body></response_body>
       <response_status>201 Created</response_status>
       <errors>
       </errors>
       </delete>

Or SLO upload failures:

       $ curl http://saio:8090/v1/AUTH_test/c/slo?multipart-manifest=put -X PUT \
          -d '[{"path": "not/found"}]' -H accept:application/xml
       <delete>
       <errors>
       <object><name>not/found</name><status>404 Not Found</status></object></errors>
       </delete>

Why <delete>? Makes no sense.

Drive-by: stop being so quadratic.

Change-Id: I46b233864ba2815ac632d856b9f3c40cc9d0001a

commit 449d83fb0c8668f444baf3c74cf15c26a70bb02e
Author: Matthew Oliver <email address hidden>
Date: Tue Oct 31 15:55:13 2017 +1100

Doc uses alias instead of aliases

    The overview_policies doc makes reference to an `alias` option when in
    fact the option is `aliases`.
    The sample storage policy snippet is correct, it's just incorrect when
    listing the possible options.

This change changes the listed option to `aliases`.

Change-Id: Iddf0f19f4d50819ff6abd46e6a1156dc8e4a451d

commit 178d7f37cb80efa668bc81e79d1ee633204b8852
Author: HCLTech-SSW <email address hidden>
Date: Wed Oct 25 03:23:47 2017 -0700

Added the man page for container-reconciler.conf

Change-Id: Ic7fc6ddca2ab564b31156fa84b362bc9963825f1
Closes-Bug: #1607025

commit 83be309627e79c2f4dd66faf17b5459fdf5bfc25
Author: Kota Tsuyuzaki <email address hidden>
Date: Thu Oct 26 18:41:56 2017 +0900

Fix a small typo

That's redundant "to" in an inline comment.

Change-Id: Idab1e11fbbd80a97b0e4ba1c8ab046be99e47d2d

commit 8d882095375dc53acdafafac40aa2efc3bafbcd4
Author: Thiago da Silva <email address hidden>
Date: Thu Sep 15 16:45:06 2016 -0400

Return 404 on a GET if tombstone is newer

    Currently the proxy keeps iterating through
    the connections in hope of finding a success even
    if it already has found a tombstone (404).

    This change changes the code a little bit to compare
    the timestamp of a 200 and a 404, if the tombstone is
    newer, then it should be returned, instead of returning
    a stale 200.

Closes-Bug: #1560574

    Co-Authored-By: Tim Burke <email address hidden>
    Change-Id: Ia81d6832709d18fe9a01ad247d75bf765e8a89f4
    Signed-off-by: Thiago da Silva <email address hidden>

commit e199192caefef068b5bf57da8b878e0bc82e3453
Author: Romain LE DISEZ <email address hidden>
Date: Wed Oct 26 10:53:46 2016 +0200

Replace replication_one_per_device by custom count

    This commit replaces boolean replication_one_per_device by an integer
    replication_concurrency_per_device. The new configuration parameter is
    passed to utils.lock_path() which now accept as an argument a limit for
    the number of locks that can be acquired for a specific path.

    Instead of trying to lock path/.lock, utils.lock_path() now tries to lock
    files path/.lock-X, where X is in the range (0, N), N being the limit for
    the number of locks allowed for the path. The default value of limit is
    set to 1.

Change-Id: I3c3193344c7a57a8a4fc7932d1b10e702efd3572

commit 9c97c80b2631bedebbd3196e87f0696a6c67b8be
Author: Samuel Merritt <email address hidden>
Date: Fri Oct 20 15:31:07 2017 -0700

Clean up a couple hand-rolled mocks.

Change-Id: I6582985990e8b5e3a0c65bce5d3bb1e39d58dfb9

commit 9b2779ba131aa22893432b5c96fb19bf43656a4f
Author: Samuel Merritt <email address hidden>
Date: Fri Oct 20 14:48:31 2017 -0700

Clean up memcache tests

    This is mostly just using mock.patch instead of doing it by hand. One
    test was doing some weird exception counting; now it just uses
    assertRaises in a loop.

Change-Id: I54f903f170226720405df2c5f6845124909ab830

commit 348bd83b7eb638d3309ff6313d9a6501c540fa24
Author: Alistair Coles <email address hidden>
Date: Wed Aug 30 15:29:14 2017 +0100

Respect co-builder partition moves when min_part_hours is zero

    Repeated calls to each co-builder's _update_last_part_moves() are
    unnecessary and have the unfortunate side effect of resetting the
    _last_part_moved bitmap. When a component builder has zero
    min_part_hours this results in it not preventing its co-builders from
    moving parts that it has already moved.

    This patch changes the CompositeRingBuilder to call each component
    builder _update_last_part_moves() *once* before rebalancing any
    component builder. CooperativeRingBuilder's no longer forward calls to
    their _update_last_part_moves() method. Each component's
    _last_part_moved bitmap is therefore preserved until for the duration
    of the composite rebalance.

    The initialisation of the RingBuilder _last_part_moves array is moved
    to the RingBuilder __init__ method, so that calls to
    _update_last_part_moves() are effective even when rebalance() has
    never been called on that builder. Otherwise, during a composite
    rebalance, a component that has not previously been rebalanced will
    not have its _last_part_moves_epoch updated during rebalance and as a
    result may report erroneous min_part_seconds_left after its first
    rebalance.

    Related-Change: I1b30cb3d776be441346a4131007d2487a5440a81
    Closes-Bug: #1714274
    Change-Id: Ib165cf974c865d47c2d9e8f7b3641971d2e9f404

Reviewed:  https://review.openstack.org/517838
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=76917dfb1dcc509ca9fa181639f37c50ade0434a
Submitter: Zuul
Branch:    feature/s3api

commit 77a8a4455d4a2bd0a9b9146a01acfe0d93fe7550
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue Oct 3 00:11:07 2017 +0000

Let clients request heartbeats during SLO PUTs
    
    An SLO PUT requires that we HEAD every referenced object; as a result, it
    can be a very time-intensive operation. This makes it difficult as a
    client to differentiate between a proxy-server that's still doing work and
    one that's crashed but left the socket open.
    
    Now, clients can opt-in to receiving heartbeats during long-running PUTs
    by including the query parameter
    
        heartbeat=on
    
    With heartbeating turned on, the proxy will start its response immediately
    with 202 Accepted then send a single whitespace character periodically
    until the request completes. At that point, a final summary chunk will be
    sent which includes a "Response Status" key indicating success or failure
    and (if successful) an "Etag" key indicating the Etag of the resulting SLO.
    
    This mechanism is very similar to the way bulk extractions and deletions
    work, and even the way SLO behaves for ?multipart-manifest=delete requests.
    
    Note that this is opt-in: this prevents us from sending the 202 response
    to existing clients that may mis-interpret it as an immediate indication
    of success.
    
    Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
    Related-Bug: 1718811
    Change-Id: I65cee5f629c87364e188aa05a06d563c3849c8f3

commit 92705bb36b4a771a757977d11875e7b929c41741
Author: David Rabel <rabel@b1-systems.de>
Date:   Thu Nov 2 12:38:41 2017 +0100

Fix indent in overview_policies.rst
    
    Change-Id: I7f070956d8b996db798837392adfca4483067aea

commit feee3998408e5ed03563c317ad9506ead92083a6
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Fri Sep 1 14:15:45 2017 -0700

Use check_drive consistently
    
    We added check_drive to the account/container servers to unify how all
    the storage wsgi servers treat device dirs/mounts.  Thus pushes that
    unification down into the consistency engine.
    
    Drive-by:
     * use FakeLogger less
     * clean up some repeititon in probe utility for device re-"mounting"
    
    Related-Change-Id: I3362a6ebff423016bb367b4b6b322bb41ae08764
    Change-Id: I941ffbc568ebfa5964d49964dc20c382a5e2ec2a

commit 29e9ae1cc5b74dce205643c101601555c06b67dc
Author: Tim Burke <tim.burke@gmail.com>
Date:   Thu Oct 19 18:53:04 2017 +0000

Make xml responses less insane
    
    Looking at bulk extractions:
    
       $ tar c *.py | curl http://saio:8090/v1/AUTH_test/c?extract-archive=tar \
          -H accept:application/xml -T -
       <?xml version="1.0" encoding="UTF-8"?>
       <delete>
       <number_files_created>2</number_files_created>
       <response_body></response_body>
       <response_status>201 Created</response_status>
       <errors>
       </errors>
       </delete>
    
    Or SLO upload failures:
    
       $ curl http://saio:8090/v1/AUTH_test/c/slo?multipart-manifest=put -X PUT \
          -d '[{"path": "not/found"}]' -H accept:application/xml
       <delete>
       <errors>
       <object><name>not/found</name><status>404 Not Found</status></object></errors>
       </delete>
    
    Why <delete>? Makes no sense.
    
    Drive-by: stop being so quadratic.
    
    Change-Id: I46b233864ba2815ac632d856b9f3c40cc9d0001a

commit 449d83fb0c8668f444baf3c74cf15c26a70bb02e
Author: Matthew Oliver <matt@oliver.net.au>
Date:   Tue Oct 31 15:55:13 2017 +1100

Doc uses alias instead of aliases
    
    The overview_policies doc makes reference to an `alias` option when in
    fact the option is `aliases`.
    The sample storage policy snippet is correct, it's just incorrect when
    listing the possible options.
    
    This change changes the listed option to `aliases`.
    
    Change-Id: Iddf0f19f4d50819ff6abd46e6a1156dc8e4a451d

commit 178d7f37cb80efa668bc81e79d1ee633204b8852
Author: HCLTech-SSW <hcl_ss_oss@hcl.com>
Date:   Wed Oct 25 03:23:47 2017 -0700

Added the man page for container-reconciler.conf
    
    Change-Id: Ic7fc6ddca2ab564b31156fa84b362bc9963825f1
    Closes-Bug: #1607025

commit 83be309627e79c2f4dd66faf17b5459fdf5bfc25
Author: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Date:   Thu Oct 26 18:41:56 2017 +0900

Fix a small typo
    
    That's redundant "to" in an inline comment.
    
    Change-Id: Idab1e11fbbd80a97b0e4ba1c8ab046be99e47d2d

commit 8d882095375dc53acdafafac40aa2efc3bafbcd4
Author: Thiago da Silva <thiago@redhat.com>
Date:   Thu Sep 15 16:45:06 2016 -0400

Return 404 on a GET if tombstone is newer
    
    Currently the proxy keeps iterating through
    the connections in hope of finding a success even
    if it already has found a tombstone (404).
    
    This change changes the code a little bit to compare
    the timestamp of a 200 and a 404, if the tombstone is
    newer, then it should be returned, instead of returning
    a stale 200.
    
    Closes-Bug: #1560574
    
    Co-Authored-By: Tim Burke <tim.burke@gmail.com>
    Change-Id: Ia81d6832709d18fe9a01ad247d75bf765e8a89f4
    Signed-off-by: Thiago da Silva <thiago@redhat.com>

commit e199192caefef068b5bf57da8b878e0bc82e3453
Author: Romain LE DISEZ <romain.le-disez@corp.ovh.com>
Date:   Wed Oct 26 10:53:46 2016 +0200

Replace replication_one_per_device by custom count
    
    This commit replaces boolean replication_one_per_device by an integer
    replication_concurrency_per_device. The new configuration parameter is
    passed to utils.lock_path() which now accept as an argument a limit for
    the number of locks that can be acquired for a specific path.
    
    Instead of trying to lock path/.lock, utils.lock_path() now tries to lock
    files path/.lock-X, where X is in the range (0, N), N being the limit for
    the number of locks allowed for the path. The default value of limit is
    set to 1.
    
    Change-Id: I3c3193344c7a57a8a4fc7932d1b10e702efd3572

commit 9c97c80b2631bedebbd3196e87f0696a6c67b8be
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Fri Oct 20 15:31:07 2017 -0700

Clean up a couple hand-rolled mocks.
    
    Change-Id: I6582985990e8b5e3a0c65bce5d3bb1e39d58dfb9

commit 9b2779ba131aa22893432b5c96fb19bf43656a4f
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Fri Oct 20 14:48:31 2017 -0700

Clean up memcache tests
    
    This is mostly just using mock.patch instead of doing it by hand. One
    test was doing some weird exception counting; now it just uses
    assertRaises in a loop.
    
    Change-Id: I54f903f170226720405df2c5f6845124909ab830

commit 348bd83b7eb638d3309ff6313d9a6501c540fa24
Author: Alistair Coles <alistairncoles@gmail.com>
Date:   Wed Aug 30 15:29:14 2017 +0100

Respect co-builder partition moves when min_part_hours is zero
    
    Repeated calls to each co-builder's _update_last_part_moves() are
    unnecessary and have the unfortunate side effect of resetting the
    _last_part_moved bitmap. When a component builder has zero
    min_part_hours this results in it not preventing its co-builders from
    moving parts that it has already moved.
    
    This patch changes the CompositeRingBuilder to call each component
    builder _update_last_part_moves() *once* before rebalancing any
    component builder. CooperativeRingBuilder's no longer forward calls to
    their _update_last_part_moves() method. Each component's
    _last_part_moved bitmap is therefore preserved until for the duration
    of the composite rebalance.
    
    The initialisation of the RingBuilder _last_part_moves array is moved
    to the RingBuilder __init__ method, so that calls to
    _update_last_part_moves() are effective even when rebalance() has
    never been called on that builder. Otherwise, during a composite
    rebalance, a component that has not previously been rebalanced will
    not have its _last_part_moves_epoch updated during rebalance and as a
    result may report erroneous min_part_seconds_left after its first
    rebalance.
    
    Related-Change: I1b30cb3d776be441346a4131007d2487a5440a81
    Closes-Bug: #1714274
    Change-Id: Ib165cf974c865d47c2d9e8f7b3641971d2e9f404