Bug #1668399 “db replicator missing priority handoff options” : Bugs : OpenStack Object Storage (swift)

clayg (clay-gerrard) on 2017-02-27

Changed in swift:
importance:	Undecided → High
status:	New → Confirmed

clayg (clay-gerrard) on 2017-04-12

Changed in swift:
importance:	High → Medium
tags:	added: low-hanging-fruit

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-03-14: Fix merged to swift (master)

#1

Reviewed: https://review.openstack.org/545497
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=47fed6f2f960a2cd56960b8aeb2eeace405c3afa
Submitter: Zuul
Branch: master

commit 47fed6f2f960a2cd56960b8aeb2eeace405c3afa
Author: Samuel Merritt <email address hidden>
Date: Fri Feb 16 16:37:58 2018 -0800

Add handoffs-only mode to DB replicators.

    The object reconstructor has a handoffs-only mode that is very useful
    when a cluster requires rapid rebalancing, like when disks are nearing
    fullness. This mode's goal is to remove handoff partitions from disks
    without spending effort on primary partitions. The object replicator
    has a similar mode, though it varies in some details.

This commit adds a handoffs-only mode to the account and container
replicators.

Change-Id: I588b151ee65ae49d204bd6bf58555504c15edf9f
Closes-Bug: 1668399

Changed in swift:
status:	Confirmed → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-03-28: Fix proposed to swift (feature/s3api)

#2

Fix proposed to branch: feature/s3api
Review: https://review.openstack.org/557177

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-03-29: Fix merged to swift (feature/s3api)

#3

Download full text (22.5 KiB)

Reviewed: https://review.openstack.org/557177
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=8c17f610d49f3540159a80206fda10266cd2046b
Submitter: Zuul
Branch: feature/s3api

commit 9b4c978f99c993775e917f9ddec7cabb25c83c6d
Author: wanghui <email address hidden>
Date: Mon Mar 26 15:22:28 2018 +0800

Use http code constant instead of int

Change-Id: I1dce94ac6192746a7b90646fe4b4460a651bced1

commit 91e5cba03475fdb18a0bf21d9264628a97f8513a
Author: melissaml <ma.lei@99cloud.net>
Date: Sun Mar 25 18:15:14 2018 +0800

Use http code constant instead of int

To keep the consistency, using HTTP_NO_CONTENT instead
of 204

Change-Id: I3d4c0ab6851ce08d1ead0f3b275cb6b41fdb65ac

commit 3bc267d10c0b2a4499b76e4846f2df8e3777a361
Author: melissaml <ma.lei@99cloud.net>
Date: Fri Mar 23 07:29:02 2018 +0800

fix a typo in documentation

Change-Id: I0492ae1d50493585ead919904d6d9502b7738266

commit 8d7e245eb599b54d2bcc3c7a652b85987fdb36e1
Author: Alexandre Lécuyer <email address hidden>
Date: Thu Jul 13 15:44:56 2017 +0200

Change object_audit_location_generator() to yield for a single policy.

auditor.py currently relies on POLICY[0] object_audit_location_generator()
to yield an AuditLocation for all policies on the object-server.

    The changes in this patch are :
      - object_audit_location_generator() yields AuditLocation only for the
      requested policy
      - audit_all_objects() calls object_audit_location_generator() for each
        policy

Closes-Bug: 1534212
Change-Id: Ida92ba0a5e1e486a4f7132c6539460b38c34ec87

commit 9f4910f6b938aa1679571d4787bbdcfa5b5f3a19
Author: Alexandre Lécuyer <email address hidden>
Date: Tue Mar 20 15:18:58 2018 +0100

Add round_robin_iter function to common/utils

Move the existing code in db_replicator.py to utils.py

Change-Id: I04e3f30a82c89fb7e714c5eb7225374b9a5c76f9

commit 7e77bf34f2fdc43c935380064eb0d7045988eec0
Author: Kazuhiro MIYAHARA <email address hidden>
Date: Tue Mar 20 04:45:04 2018 +0000

Refactor tempurl functional test's try-finally

    This is a following patch of [1]. In [1] comment, tim suggested
    implementation for same behavior with less code.
    This change implemented the suggestion. See [1] for more details.

[1]: https://review.openstack.org/#/c/547306/

Change-Id: Ifd8a0534fbdf41837977028c0c6ef99f1f6ac0f0

commit 2c4c23defc615e0f672244457c4d15cd718971dd
Author: Clay Gerrard <email address hidden>
Date: Fri Mar 16 18:44:14 2018 -0700

Encapsulate some general utility in utils

... but either way works.

Change-Id: I56b7689270e528a1d2099ff96030ce660e588c94
Related-Change-Id: Ic108f5c38f700ac4c7bcf8315bf4c55306951361

commit bb3553d08428c2d2ea0a7d4029a3fecb97678446
Author: Clay Gerrard <email address hidden>
Date: Fri Mar 16 18:17:59 2018 -0700

Use state when you have it

... but either way works.

Change-Id: Ib66ee035229d0718b68450587550176c91ce9ca6
Related-Change-Id: Ic108f5c38f700ac4c7bcf8315bf4c55306951361

commit de38b727b475876b4a3e4dcf7dca21de...

Reviewed:  https://review.openstack.org/557177
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=8c17f610d49f3540159a80206fda10266cd2046b
Submitter: Zuul
Branch:    feature/s3api

commit 9b4c978f99c993775e917f9ddec7cabb25c83c6d
Author: wanghui <wang_hui@inspur.com>
Date:   Mon Mar 26 15:22:28 2018 +0800

Use http code constant instead of int
    
    Change-Id: I1dce94ac6192746a7b90646fe4b4460a651bced1

commit 91e5cba03475fdb18a0bf21d9264628a97f8513a
Author: melissaml <ma.lei@99cloud.net>
Date:   Sun Mar 25 18:15:14 2018 +0800

Use http code constant instead of int
    
    To keep the consistency, using HTTP_NO_CONTENT instead
    of 204
    
    Change-Id: I3d4c0ab6851ce08d1ead0f3b275cb6b41fdb65ac

commit 3bc267d10c0b2a4499b76e4846f2df8e3777a361
Author: melissaml <ma.lei@99cloud.net>
Date:   Fri Mar 23 07:29:02 2018 +0800

fix a typo in documentation
    
    Change-Id: I0492ae1d50493585ead919904d6d9502b7738266

commit 8d7e245eb599b54d2bcc3c7a652b85987fdb36e1
Author: Alexandre Lécuyer <alexandre.lecuyer@corp.ovh.com>
Date:   Thu Jul 13 15:44:56 2017 +0200

Change object_audit_location_generator() to yield for a single policy.
    
    auditor.py currently relies on POLICY[0] object_audit_location_generator()
    to yield an AuditLocation for all policies on the object-server.
    
    The changes in this patch are :
      - object_audit_location_generator() yields AuditLocation only for the
      requested policy
      - audit_all_objects() calls object_audit_location_generator() for each
        policy
    
    Closes-Bug: 1534212
    Change-Id: Ida92ba0a5e1e486a4f7132c6539460b38c34ec87

commit 9f4910f6b938aa1679571d4787bbdcfa5b5f3a19
Author: Alexandre Lécuyer <alexandre.lecuyer@corp.ovh.com>
Date:   Tue Mar 20 15:18:58 2018 +0100

Add round_robin_iter function to common/utils
    
    Move the existing code in db_replicator.py to utils.py
    
    Change-Id: I04e3f30a82c89fb7e714c5eb7225374b9a5c76f9

commit 7e77bf34f2fdc43c935380064eb0d7045988eec0
Author: Kazuhiro MIYAHARA <miyahara.kazuhiro@lab.ntt.co.jp>
Date:   Tue Mar 20 04:45:04 2018 +0000

Refactor tempurl functional test's try-finally
    
    This is a following patch of [1]. In [1] comment, tim suggested
    implementation for same behavior with less code.
    This change implemented the suggestion. See [1] for more details.
    
    [1]: https://review.openstack.org/#/c/547306/
    
    Change-Id: Ifd8a0534fbdf41837977028c0c6ef99f1f6ac0f0

commit 2c4c23defc615e0f672244457c4d15cd718971dd
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Fri Mar 16 18:44:14 2018 -0700

Encapsulate some general utility in utils
    
    ... but either way works.
    
    Change-Id: I56b7689270e528a1d2099ff96030ce660e588c94
    Related-Change-Id: Ic108f5c38f700ac4c7bcf8315bf4c55306951361

commit bb3553d08428c2d2ea0a7d4029a3fecb97678446
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Fri Mar 16 18:17:59 2018 -0700

Use state when you have it
    
    ... but either way works.
    
    Change-Id: Ib66ee035229d0718b68450587550176c91ce9ca6
    Related-Change-Id: Ic108f5c38f700ac4c7bcf8315bf4c55306951361

commit de38b727b475876b4a3e4dcf7dca21de8454722e
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Thu Mar 8 13:02:46 2018 -0800

Don't double-filter replication jobs
    
    ObjectReplicator.collect_jobs() takes and correctly applies the
    various overrides, so there's no need to check the returned jobs
    against the overrides.
    
    Change-Id: I2a59b26410d1732a5f2c8f1f32e397d77550860e

commit 22b9a4a943f5d49942b33e4901c90f5d52a09a7d
Author: Thomas Goirand <zigo@debian.org>
Date:   Mon Mar 12 18:07:37 2018 +0100

Fix tests using O_TMPFILE
    
    Unit tests using O_TMPFILE only rely on the kernel version to check
    for the feature. This is wrong, as some filesystem, like tmpfs, doesn't
    support O_TMPFILE.
    
    So, instead of checking kernel version, this patch actually attempts to
    open a file using O_TMPFILE and see if that's supported. If not, then
    the test is skipped.
    
    Change-Id: I5d652f1634b1ef940838573cfdd799ea17b8b572

commit e84ed57a729be2e95e7028bfdf2134d58e8a1e98
Author: Matthew Oliver <matt@oliver.net.au>
Date:   Tue Mar 13 06:19:44 2018 +0000

py3: port cli form_signature and unit tests
    
    This patch ports the form_signature cli `swift-form-signature`
    and it's corrisponding tests to py3.
    
    In essence in Py3 the HMAC function expects binary strings.
    
    Change-Id: I5dded4ceb80f0cc595403775e8f9c17873e1e37b

commit 1b546f6b7750fe9fd91f840f86a9fe5e370266ad
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue Mar 6 23:42:24 2018 +0000

Capture probe test logs
    
    Change-Id: Ib2f3aa1d70e6d28e00570fe21352031dc41325d0

commit e5ce83b9677db453a4c9f8551939a0dc36c880f2
Author: melissaml <ma.lei@99cloud.net>
Date:   Sun Mar 11 02:42:48 2018 +0800

Update links in README
    
    Change the outdated links to the latest links in README
    
    Change-Id: Ic68cc654722d47d761b308381f2ab489f3d13703

commit bac79f2daeb1a4f103bb81daf0b1a52fbce632c3
Author: Kazuhiro MIYAHARA <miyahara.kazuhiro@lab.ntt.co.jp>
Date:   Thu Mar 8 08:07:24 2018 +0000

Unify expirer unit test task queue
    
    To simplify unit tests for object-expirer, this patch unifies
    expirer's task queue for each unit tests.
    In this patch, following changes are applied:
        1: Unify expirer's task queue
        2: Remove redundant log checking, because some tests checks
           logs in their speciality
        3: Use mocked methods instead of dummy methods which
           raises specialized message exception
    
    Change-Id: I839f0bb43eb827384727356877e33a6be7d9b81d

commit e4640495d80133f09e7ff83b16894de389a271ef
Author: Tim Burke <tim.burke@gmail.com>
Date:   Fri Mar 2 12:24:09 2018 +0000

py3: have Timestamp accept bytestrings, better validate input
    
    Change-Id: I627894b9496cc85b3b1852e9df4812257df4ac6a

commit 335899f89fc7a3e9ede59b28cc2b775ff126541b
Author: Romain LE DISEZ <romain.le-disez@corp.ovh.com>
Date:   Sat Feb 25 18:50:50 2017 +0100

Functional tests for staticweb middleware
    
    Change-Id: I6bc70039c1b99b8a5bf5a30e9c6877d4f0aa638d

commit b640631dafefe4f03f6ce2f9fe723e262cf366e4
Author: Tim Burke <tim.burke@gmail.com>
Date:   Thu May 11 17:26:19 2017 -0400

Apply remote metadata in _handle_sync_response
    
    We've already got it in the response, may as well apply it now rather
    than wait for the other end to get around to running its replicators.
    
    Change-Id: Ie36a6dd075beda04b9726dfa2bba9ffed025c9ef

commit ca21d711525e4122a69ccac1e62fb64bced3f53e
Author: Thiago da Silva <thiago@redhat.com>
Date:   Fri Mar 2 10:00:59 2018 -0500

Add swift probe tests to gate
    
    This patch includes some generic playbooks to configure a SAIO[0]
    plus the playbook to run the probe tests.
    
    The current SAIO playbooks have only been tests on CentOS, they
    would need additional changes to run on Ubuntu.
    
    [0] - https://docs.openstack.org/swift/latest/development_saio.html
    
    Change-Id: Ib72c77021aeed20f2070d42cff0ea9b671a4317a

commit 63e6c75efbec739dd94846f826ae9e99dd0d7a4e
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Mon Mar 5 17:51:03 2018 -0800

Temporarily disable flaky test.
    
    test.unit.obj.test_replicator.TestObjectReplicator.test_replicate_lockup_detector
    is failing a lot in the gate. Let's disable it for now so that other
    patches can continue to land.
    
    Change-Id: I1790ebcbc0c8d075c2786aebca4e8ccf7547b178

commit b08c70d38e1d49b73c241b227e4fcc62fe164a04
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Wed Feb 28 15:37:04 2018 +0000

Support -d <devs> and -p <partitions> in DB replicators.
    
    Similar to the object replicator and reconstructor, these arguments
    are comma-separated lists of device names and partitions,
    respectively, on which the account or container replicator will
    operate. Other devices and partitions are ignored.
    
    Change-Id: Ic108f5c38f700ac4c7bcf8315bf4c55306951361

commit afb6cb583536e4070af881ae792011bcf3789f00
Author: Tim Burke <tim.burke@gmail.com>
Date:   Sat Mar 3 17:07:54 2018 +0000

Try to avoid leaving (killed) long-running rsyncs in the process table
    
    Also, add some guards against a NameError in particularly-bad races.
    
    Change-Id: If90662b6996e25bde74e0a202301b52a1d266e92
    Related-Change: Ifd14ce82de1f7ebb636d6131849e0fadb113a701

commit 57b632fbb52c84e02fefa307ada45081d86b23a6
Author: Tim Burke <tim.burke@gmail.com>
Date:   Thu Mar 1 13:01:49 2018 +0000

Fix object-server to not 400 all expirer DELETEs
    
    In the related changes, we switched to using
    Timestamp.normal representations for the X-If-Delete-At header.
    However, the object-server required that the header be an int,
    and the trailing '.00000' would cause trip the
    "Bad X-If-Delete-At header value" error handling.
    
    Now, we'll convert both the expirer header and the stored X-Delete-At to
    Timestamps, even though we expect them to have no fractional value.
    
    Note that we *could* have changed the expirer to continue sending
    headers that are valid ints, but Timestamps are already the normal
    Swift-way of passing and comparing times -- we should use that.
    
    Related-Change: Ibf61eb1f767a48cb457dd494e1f7c12acfd205de
    Related-Change: Ie82622625d13177e08a363686ec632f63d24f4e9
    Change-Id: Ida22c1c8c5bf21bdc72c33e225e75fb750f8444b

commit 9eeacbf150cdb0e1ee58a7a25fb875da0d955898
Author: Tim Burke <tim.burke@gmail.com>
Date:   Fri Mar 2 14:19:34 2018 +0000

Fix new doc warnings resulting from Sphinx 1.7.1
    
    Change-Id: Id6678dd7e59a85bc2a7b3ba3d41d28e08956ae3e

commit fdaf052d7316e131258017e6e9e49fe07948964a
Author: Pete Zaitcev <zaitcev@kotori.zaitcev.us>
Date:   Wed Feb 28 20:14:32 2018 -0600

Make test_greater_with_offset not fail on py36
    
    Reviewer, beware: we determined that the test was using the
    facilities improperly. This patch adjusts the test but does
    not fix the code under test.
    
    The time.time() output looks like this:
    
    [zaitcev@lembas swift-tsrep]$ python2
    Python 2.7.14 (default, Dec 11 2017, 14:52:53)
    [GCC 7.2.1 20170915 (Red Hat 7.2.1-2)] on linux2
    >>> import time
    >>> time.time()
    1519861559.96239
    >>> time.time()
    1519861561.046204
    >>> time.time()
    1519861561.732341
    >>>
    
    (it's never beyond 6 digits on py2)
    
    [zaitcev@lembas swift-tsrep]$ python3
    Python 3.6.3 (default, Oct  9 2017, 12:07:10)
    [GCC 7.2.1 20170915 (Red Hat 7.2.1-2)] on linux
    >>> import time
    >>> time.time()
    1519861541.7662468
    >>> time.time()
    1519861542.893482
    >>> time.time()
    1519861546.56222
    >>> time.time()
    1519861547.3297756
    >>>
    
    (can go beyond 6 digits on py3)
    
    When fraction is too long on py3, you get:
    
    >>> now = 1519830570.6949349
    >>> now
    1519830570.6949348
    >>> timestamp = Timestamp(now, offset=1)
    >>> timestamp
    1519830570.69493_0000000000000001
    >>> value = '%f' % now
    >>> value
    '1519830570.694935'
    >>> timestamp > value
    False
    >>>
    
    Note that the test fails in exactly the same way on py2, if time.time()
    returns enough digits. Therefore, rounding changes are not the culprit.
    
    The real problem is the assumption that you can take a float T, print
    it with '%f' into S, then do arithmetic on T to get O, convert S, T,
    and O into Timestamp, then make comparisons. This does not work,
    because rounding happens twice: once when you interpolate %f, and
    then when you construct a Timestamp. The only valid operation is
    to accept a timestamp (e.g. from X-Delete-At) as a floating point
    number as a decimal string, and convert it once. Only then you can
    do arithmetics to find the expiration.
    
    Change-Id: Ie3b002abbd4734c675ee48a7535b8b846032f9d1

commit 8b8a2a3406e4af2a8c61427f5ea43aa7ab161961
Author: Tim Burke <tim.burke@gmail.com>
Date:   Thu Mar 1 23:30:00 2018 +0000

Tolerate 404s during setUp/tearDown in func tests
    
    A couple times, I've seen tests fail in the gate because we got back a
    404 while trying to clean out the test account. The story that gets us
    here seems to be:
    
     - One or more object servers take too long to respond to the initial
       DELETE request, so the test client gets back a 503 and sleeps so
       it can retry.
     - Meanwhile, the servers finish writing their tombstones and want to
       respond 204 (but probably *actually* respond 408 because the proxy
       killed the connection).
     - The test client sends its retry, and since the object servers now
       have tombstones, it gets back a 404.
    
    But the thing is, this is *outside of the test scope* anyway, we're just
    trying to get back to a sane state. If it's gone, s much the better!
    
    For an example of this, see the failures on patchset 3 of
    https://review.openstack.org/#/c/534978 (which both failed for the same
    reason on different tests).
    
    Change-Id: I9ab2fd430d4800f9f55275959a20e30f09d9e1a4

commit 10943b89b97ba927e2d651eca454870bf4699f59
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Thu Mar 1 06:49:57 2018 +0000

Imported Translations from Zanata
    
    For more information about this automatic import see:
    https://docs.openstack.org/i18n/latest/reviewing-translation-import.html
    
    Change-Id: I5c423c5e838a3a152f767e6359e349a8f9833cbf

commit 36c42974d6746a306002f8f8e1b3dae664c783d6
Author: Tim Burke <tim.burke@gmail.com>
Date:   Thu Feb 22 22:48:55 2018 +0000

py3: Port more CLI tools
    
    Bring under test
    
     - test/unit/cli/test_dispersion_report.py
     - test/unit/cli/test_info.py and
     - test/unit/cli/test_relinker.py
    
    I've verified that swift-*-info (at least) behave reasonably under
    py3, even swift-object-info when there's non-utf8 metadata on the
    data/meta file.
    
    Change-Id: Ifed4b8059337c395e56f5e9f8d939c34fe4ff8dd

commit 624b5310b456e8bbf9cc729fe1d9a18aeb068ae1
Author: Tim Burke <tim.burke@gmail.com>
Date:   Thu Feb 1 15:25:53 2018 -0800

py3: port common/wsgi.py
    
    Note that we're punting on configuring socket buffer sizes (for now)
    
    Change-Id: I285a9b521fd0af381a227e0e824bc391817547f4

commit d09d60dfbb48da434ae432049c6d29d065d782b7
Author: OpenStack Release Bot <infra-root@openstack.org>
Date:   Wed Feb 7 17:46:44 2018 +0000

Update release notes for stable/queens
    
    Change-Id: I35581e63c9f23cc3632bb1391a79def18d86fbf8

commit 7eb9852cd4efd8903b486a0ceb58372c2fb3b6e3
Author: Tim Burke <tim.burke@gmail.com>
Date:   Mon Feb 26 10:48:43 2018 +0000

Don't log tracebacks when we know where they're going to end
    
    Change-Id: I789af9c2c02da8a895186581dbf8a2e3db8c5e9f

commit 1fadffeae0defb5c8956247e9118e74c8405f8ba
Author: Kazuhiro MIYAHARA <miyahara.kazuhiro@lab.ntt.co.jp>
Date:   Tue Feb 13 09:40:18 2018 +0000

Split expirer methods and parametrize task account
    
    To prepare for implement general task queue mode to expirer,
    this patch splits expirer's method into smaller ones and parametrize task
    account. This change will make expirer's general task queue patch [1] more
    simple.
    
    This patch has following approaches:
        1: Split methods into smaller ones
        2: Parameterize task account name to adapt many task accounts
           in general task queue
        3: Include task account names in log messages
        4: Skip task account when the account has no task containers
    
    [1]: https://review.openstack.org/#/c/517389/
    
    Change-Id: I907612f7c258495e9ccc53c1d57de4791b3e7ab7

commit 9e5f434574e6d65e97fbece69d5baa6a60d73418
Author: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Date:   Mon Feb 26 21:23:55 2018 +0900

Kill rsync coros when lockup detector tries to kill the process
    
    Because the replicator in the master doesn't propergate the kill
    signal to the subprocess in the coroutine. With the behavior, the lockup
    detector causes a lot of rsync processes even it tries to reset the process.
    
    This patch fixes the replicator kill rsync procs when the lockup detector
    calls kill of eventlet threads.
    
    Change-Id: Ifd14ce82de1f7ebb636d6131849e0fadb113a701

commit 5cb0869743e527026dad7c6691bda2e3f2174c07
Author: Tim Burke <tim.burke@gmail.com>
Date:   Thu Feb 1 14:30:32 2018 -0800

py3: port common/memcached.py
    
    Change-Id: I7f04b3977971f0581b04180e5372686d8186346f

commit 532ac9e1c7e883b0570a80e87d598a0ebf731e0b
Author: Kazuhiro MIYAHARA <miyahara.kazuhiro@lab.ntt.co.jp>
Date:   Fri Feb 23 07:24:12 2018 +0000

Ensure reverting test env if the env is temporarily changed
    
    test_tempurl_keys_hidden_from_acl_readonly changes test env parameter
    temporarily for container HEAD. After that the test reverts the change.
    But if the HEAD failed with exception, the change is not reverted.
    With the non reverted change, some other tests fails even if the test
    have no problems.
    
    This patch ensures the reversion by using try-finally.
    
    Change-Id: I8cd7928da6211e5516992fe9f2bc8e568bcab443

commit 47fed6f2f960a2cd56960b8aeb2eeace405c3afa
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Fri Feb 16 16:37:58 2018 -0800

Add handoffs-only mode to DB replicators.
    
    The object reconstructor has a handoffs-only mode that is very useful
    when a cluster requires rapid rebalancing, like when disks are nearing
    fullness. This mode's goal is to remove handoff partitions from disks
    without spending effort on primary partitions. The object replicator
    has a similar mode, though it varies in some details.
    
    This commit adds a handoffs-only mode to the account and container
    replicators.
    
    Change-Id: I588b151ee65ae49d204bd6bf58555504c15edf9f
    Closes-Bug: 1668399

commit 2bfd9c6a9b0d9ee5b547a31ba97be74b09ac3afc
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Thu Feb 15 15:42:09 2018 -0800

Make DB replicators ignore non-partition directories
    
    If a cluster operator has some tooling that makes directories in
    /srv/node/<disk>/accounts, then the account replicator will treat
    those directories as partition dirs and may remove empty
    subdirectories contained therein. This wastes time and confuses the
    operator.
    
    This commit makes DB replicators skip partition directories whose
    names don't look like positive integers. This doesn't completely avoid
    the problem since an operator can still use an all-digit name, but it
    will skip directories like "tmp21945".
    
    Change-Id: I8d6682915a555f537fc0ce8c39c3d52c99ff3056

commit 98d185905a173fd7cbcbe1f6a71b74eb7564851c
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Fri Jan 26 16:51:03 2018 -0800

Cleanup for iterators in SegmentedIterable
    
    We had a pair of large, complicated iterators to handle fetching all
    the segment data, and they were hard to read and think about. I tried
    to break them out into some simpler pieces:
    
     * one to handle coalescing multiple requests to the same segment
    
     * one to handle fetching the bytes from each segment
    
     * one to check that the download isn't taking too long
    
     * one to count the bytes and make sure we sent the right number
    
     * one to catch errors and handle cleanup
    
    It's more nesting, but each level now does just one thing.
    
    Change-Id: If6f5cbd79edeff6ecb81350792449ce767919bcc

commit 39910553df22a2664b1125a4d503d1e9149cf617
Author: vxlinux <yan.wei7@zte.com.cn>
Date:   Mon Jan 15 15:16:57 2018 +0800

Solve the zombie process problem of Auditor
    
    As the bug 1743310 reported,if we list the swift processes ,we will
    see a zombie process every one minute.The zombie processes numbers may
    be more than one.
    
    The related code as follows: swift/obj/auditor.py:386~397
    
            while pids:
                pid = os.wait()[0]
                # ZBF scanner must be restarted as soon as it finishes
                # unless we're in run-once mode
                if self.conf_zero_byte_fps and pid == zbf_pid and \
                   len(pids) > 1 and not once:
                    kwargs['device_dirs'] = override_devices
                    # sleep between ZBF scanner forks
                    self._sleep()
                    zbf_pid = self.fork_child(zero_byte_fps=True, **kwargs)
                    pids.add(zbf_pid)
                pids.discard(pid)
    
    The pids list includes a zbf pid and one or more pids with mode
    (ALL - parallel, objectXXX).If the zbf process is the last one
    finished，then the second zbf process will not be forked.Conversely，
    the zbf process will be forked again,before the fork procedure,the main
    process will sleep for self.interval seconds,30 seconds by default,
    during the self.interval time,if a non zbf process finished,this process
    will become a zombie,because the main process is in sleep.
    
    In this solution,we move the sleep from whileloop to the subprocess,and
    the main process will not be blocked in whileloop,so the subprocess
    will be recovered in time.
    
    Closes-Bug: #1743310
    
    Change-Id: I61c766aa2a1c4bad0247a44a8e78ef38d9f3ae47

commit 38ada71824f6094d428634a351229f1c0502aa4c
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Tue Jan 23 11:44:04 2018 -0800

Make statsd errors correspond to 5xx only
    
    The goal is to make the successful statsd buckets
    (e.g. "object-server.GET.timing") have timing information for all the
    requests that the server handled correctly, while the error buckets
    (e.g. "object-server.GET.errors.timing") have the rest.
    
    Currently, we don't do that great a job of it. We special-case a few
    4xx status codes (404, 412, 416) to not count as errors, but we leave
    some pretty large holes. If you're graphing errors, you'll see spikes
    when client is sending bogus requests (400) or failing to
    re-authenticate (403). You'll also see spikes when your drives are
    unmounted (507) and when there's bugs that need fixing (500).
    
    This commit makes .errors.timing be just 5xx in the hope that its
    graph will be more useful.
    
    Change-Id: I92b41bcbb880c0688c37ab231c19ebe984b18215

commit f95befb37fcb3563d3db965d72d1112f0e31bd1b
Author: Tim Burke <tim.burke@gmail.com>
Date:   Thu Jul 14 11:16:29 2016 -0500

Optionally drop common prefixes in swift-*-info output
    
    Add a --drop-prefixes flag to swift-account-info, swift-container-info,
    and swift-object-info. This makes the output between the three more
    consistent.
    
    Change-Id: I98252ff74c4983eaad0a93d9a9fc527c74ffce68

tags:

added: in-feature-s3api

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-04-05: Fix proposed to swift (feature/deep)

#4

Fix proposed to branch: feature/deep
Review: https://review.openstack.org/559175

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-05-30: Fix included in openstack/swift 2.18.0

#5

This issue was fixed in the openstack/swift 2.18.0 release.

OpenStack Object Storage (swift)

db replicator missing priority handoff options

Bug Description

Other bug subscribers

Remote bug watches