OpenStack Object Storage (swift)

object-replicator doesn't work while specifying partition number in once mode

Bug #1822731 reported by Hugo Kou on 2019-04-02

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Object Storage (swift)	Fix Released	Undecided	JoeYang

Bug Description

Swift version: 2.20

root@cloud-o01:/tmp# /usr/bin/python /opt/ss/bin/swift-object-replicator /tmp/test.conf -d d0 -p 73728 -i 0 -o -v
object-replicator: Starting 13490
object-replicator: Spawned worker 13566 with {'override_devices': ['d0'], 'override_partitions': [73728], 'override_policies': [0], 'have_overrides': True, 'multiprocess_worker_index': 0}
object-replicator: [worker 1/1 pid=13566] Running object replicator in script mode.
object-replicator: [worker 1/1 pid=13566] Nothing replicated for 0.280825138092 seconds.
object-replicator: [worker 1/1 pid=13566] Object replication complete (once). (0.00 minutes)
object-replicator: Forked worker 13566 finished
object-replicator: Worker 13566 exited
object-replicator: Finished 13490
object-replicator: Exited 13490

In this case, the object-replicator does nothing for partition 73728

Tags:

JoeYang (tokuzfunpi) on 2019-04-02

Changed in swift:
assignee:	nobody → JoeYang (tokuzfunpi)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-04-02: Fix proposed to swift (master)

Fix proposed to branch: master
Review: https://review.openstack.org/649240

Changed in swift:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-04-03:

Fix proposed to branch: master
Review: https://review.openstack.org/649502

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-04-03: Change abandoned on swift (master)

Change abandoned by JoeYang (<email address hidden>) on branch: master
Review: https://review.openstack.org/649502
Reason: squashed into 649240

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-04-09: Fix merged to swift (master)

Reviewed: https://review.openstack.org/649240
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=08d59ebcd4fa39085d069125fffc5af75e4d735b
Submitter: Zuul
Branch: master

commit 08d59ebcd4fa39085d069125fffc5af75e4d735b
Author: Joe Yang <email address hidden>
Date: Tue Apr 2 15:43:16 2019 +0800

Change partition name from string to integer

    Object-replicator let user to input target partition numbers and
    run the replicator job, and partition input will be stored as
    integer list. But in build_replication_jobs method, it use
    os.listdir method to retreve partitions in string list, so it is
    failed to compare with target partition numbers.

Change-Id: I85dcaf65b9f19ac4659fa5937f9b0b0e804fc54e
Closes-Bug: #1822731

Changed in swift:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-04-25: Fix proposed to swift (feature/losf)

Fix proposed to branch: feature/losf
Review: https://review.opendev.org/655630

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-04-29: Fix merged to swift (feature/losf)

Download full text (11.3 KiB)

Reviewed: https://review.opendev.org/655630
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=d391957fd70a02866615c9c5edc070dbde2b23ea
Submitter: Zuul
Branch: feature/losf

commit 32bf43990c3c9ce771fca5968242f102155467c1
Author: ZhongShengping <chdzsp@163.com>
Date: Mon Apr 22 11:05:10 2019 +0800

Replace git.openstack.org URLs with opendev.org URLs

Thorough replacement of git.openstack.org URLs with their opendev.org
counterparts.

Change-Id: I5e77307de6a484fd69b2a5863423ceb357e8601f

commit 89eced960c5bf5c2e14b6245c70b615dc23d45a6
Author: OpenDev Sysadmins <email address hidden>
Date: Fri Apr 19 19:28:47 2019 +0000

OpenDev Migration Patch

    This commit was bulk generated and pushed by the OpenDev sysadmins
    as a part of the Git hosting and code review systems migration
    detailed in these mailing list posts:

http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html

    Attempts have been made to correct repository namespaces and
    hostnames based on simple pattern matching, but it's possible some
    were updated incorrectly or missed entirely. Please reach out to us
    via the contact information listed at https://opendev.org/ with any
    questions you may have.

commit b6ebabee78b77108b2a3e5261873f038772d304d
Author: Tim Burke <email address hidden>
Date: Wed Apr 17 09:15:27 2019 -0700

Clean up dlo unit tests

We don't actually need so many py2/py3 branches, and once we
clean those up, there's hardly any reason to import six.

Change-Id: Ia3b4f02e7eb99ad1a76aa35c39dc198528fd39ad

commit 893acffbc01ba817361f8d8735513dc784fb92c0
Author: Pete Zaitcev <email address hidden>
Date: Wed Feb 20 17:09:08 2019 -0600

py3: port dlo

Change-Id: I7236ddea0acde93d0789ad8affa76df0097a86aa

commit 7cb276da425592280cd3f522c06cfb7b02311bde
Author: Tim Burke <email address hidden>
Date: Tue Apr 2 17:42:21 2019 -0700

Move from py35 to py37 for gating py3 jobs

    Looking at https://governance.openstack.org/tc/reference/runtimes/train.html
    OpenStack is no longer targetting Python 3.5, so it doesn't really make sense
    to gate on it. Move to Python 3.7, just to make it so we can go longer before
    having to do this shuffle again.

    Presumably, once we actually declare support for Python 3.6, we should
    make that job voting, too? We've had enough gate flaky-ness lately that
    I'm not keen on it.

Change-Id: I9c9e5df76a6592ab9c512b7c1ba807f9a0053059

commit 65d3cb75f817b3683aa85b73239d606e29c73d1c
Author: zhufl <email address hidden>
Date: Mon Apr 15 15:12:30 2019 +0800

Use assertIn to check whether substring is in str

This is to use assertIn to check whether substring is in str,
go get clearer failure message.

Change-Id: I40aff29454c423389755a5330751d2f69a227a05

commit e5ff405ec967bfe19b3eaeaac5b5a32c95218f9d
Author: Tim Burke <email address hidden>
Date: Fri Jun 1 15:59:25 2018 -0700

Make staticweb return URL-encoded Location headers

...

Reviewed:  https://review.opendev.org/655630
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=d391957fd70a02866615c9c5edc070dbde2b23ea
Submitter: Zuul
Branch:    feature/losf

commit 32bf43990c3c9ce771fca5968242f102155467c1
Author: ZhongShengping <chdzsp@163.com>
Date:   Mon Apr 22 11:05:10 2019 +0800

Replace git.openstack.org URLs with opendev.org URLs
    
    Thorough replacement of git.openstack.org URLs with their opendev.org
    counterparts.
    
    Change-Id: I5e77307de6a484fd69b2a5863423ceb357e8601f

commit 89eced960c5bf5c2e14b6245c70b615dc23d45a6
Author: OpenDev Sysadmins <openstack-infra@lists.openstack.org>
Date:   Fri Apr 19 19:28:47 2019 +0000

OpenDev Migration Patch
    
    This commit was bulk generated and pushed by the OpenDev sysadmins
    as a part of the Git hosting and code review systems migration
    detailed in these mailing list posts:
    
    http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
    http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html
    
    Attempts have been made to correct repository namespaces and
    hostnames based on simple pattern matching, but it's possible some
    were updated incorrectly or missed entirely. Please reach out to us
    via the contact information listed at https://opendev.org/ with any
    questions you may have.

commit b6ebabee78b77108b2a3e5261873f038772d304d
Author: Tim Burke <tim.burke@gmail.com>
Date:   Wed Apr 17 09:15:27 2019 -0700

Clean up dlo unit tests
    
    We don't actually need so many py2/py3 branches, and once we
    clean those up, there's hardly any reason to import six.
    
    Change-Id: Ia3b4f02e7eb99ad1a76aa35c39dc198528fd39ad

commit 893acffbc01ba817361f8d8735513dc784fb92c0
Author: Pete Zaitcev <zaitcev@kotori.zaitcev.us>
Date:   Wed Feb 20 17:09:08 2019 -0600

py3: port dlo
    
    Change-Id: I7236ddea0acde93d0789ad8affa76df0097a86aa

commit 7cb276da425592280cd3f522c06cfb7b02311bde
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue Apr 2 17:42:21 2019 -0700

Move from py35 to py37 for gating py3 jobs
    
    Looking at https://governance.openstack.org/tc/reference/runtimes/train.html
    OpenStack is no longer targetting Python 3.5, so it doesn't really make sense
    to gate on it. Move to Python 3.7, just to make it so we can go longer before
    having to do this shuffle again.
    
    Presumably, once we actually declare support for Python 3.6, we should
    make that job voting, too? We've had enough gate flaky-ness lately that
    I'm not keen on it.
    
    Change-Id: I9c9e5df76a6592ab9c512b7c1ba807f9a0053059

commit 65d3cb75f817b3683aa85b73239d606e29c73d1c
Author: zhufl <zhu.fanglei@zte.com.cn>
Date:   Mon Apr 15 15:12:30 2019 +0800

Use assertIn to check whether substring is in str
    
    This is to use assertIn to check whether substring is in str,
    go get clearer failure message.
    
    Change-Id: I40aff29454c423389755a5330751d2f69a227a05

commit e5ff405ec967bfe19b3eaeaac5b5a32c95218f9d
Author: Tim Burke <tim.burke@gmail.com>
Date:   Fri Jun 1 15:59:25 2018 -0700

Make staticweb return URL-encoded Location headers
    
    Also, enhance tests to verify proper encoding in generated responses.
    
    Change-Id: Ib696b1c3a34722f3a6de010973b90ef8b1917012
    Partial-Bug: 1774238

commit 049e56a5d03fb8b801bbd31557fc559912cd799e
Author: Tim Burke <tim.burke@gmail.com>
Date:   Mon Apr 8 14:52:42 2019 -0700

Remove our urlparse wrapper
    
    It has not been necessary since we dropped support for Python 2.6.
    See https://github.com/python/cpython/commit/8c6d9d7 and
    https://bugs.python.org/issue2987.
    
    Be sure to keep a `urlparse` name in utils, though; swauth (at least)
    still expects there to be a swift.common.utils.urlparse.
    
    Change-Id: If2502868f251b8a83aa929ee22b10046e708d111

commit 5b40f69ede905bfd9413c12b6705fef6d809a929
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Tue Apr 9 14:15:32 2019 -0500

Add tests for unexpected names in part dir
    
    An earlier revision of the related change had a bug that this test would
    have discovered.
    
    Related-Change-Id: I85dcaf65b9f19ac4659fa5937f9b0b0e804fc54e
    Change-Id: Iedd85ec65a11de189ce73e650d674aee0dc7e402

commit b86bc51607a6855deaa5a365d705174836151e8b
Author: Matthew Oliver <matt@oliver.net.au>
Date:   Mon Mar 18 16:43:29 2019 +1100

py3: port staticweb middleware
    
    This patch is porting the staticweb middleware to py3.
    
    Change-Id: I5d9a13baecedd13d2b7a8ae3dd639eaff0894441

commit 621e92488342f41b9cc32461b9ff99a3fabfcb4c
Author: Thiago da Silva <thiagodasilva@gmail.com>
Date:   Mon Apr 8 16:49:29 2019 -0400

Update openstack repo for libec install
    
    Update to rocky release
    
    Change-Id: Id35a41d689a24bf68d17647879780f5daaecdf37

commit 08d59ebcd4fa39085d069125fffc5af75e4d735b
Author: Joe Yang <jyang@swiftstack.com>
Date:   Tue Apr 2 15:43:16 2019 +0800

Change partition name from string to integer
    
    Object-replicator let user to input target partition numbers and
    run the replicator job, and partition input will be stored as
    integer list. But in build_replication_jobs method, it use
    os.listdir method to retreve partitions in string list, so it is
    failed to compare with target partition numbers.
    
    Change-Id: I85dcaf65b9f19ac4659fa5937f9b0b0e804fc54e
    Closes-Bug: #1822731

commit e5e22ebeba2140b1da2e9d52d6b8ecd92bc75c88
Author: Tim Burke <tim.burke@gmail.com>
Date:   Fri Jun 1 15:38:10 2018 -0700

Make symlink work with Unicode account names
    
    Also, ensure that the stored symlink headers really *are* url-encoded as
    we've been assuming.
    
    Change-Id: I1f300d69bec43f0deb430294da05a4ec04308040
    Related-Bug: 1774238
    Closes-Bug: #1821240

commit 52bc11f097f0a583b8c7ec70703a337f0acf00f4
Author: Pete Zaitcev <zaitcev@kotori.zaitcev.us>
Date:   Tue Apr 2 22:45:27 2019 -0500

Use labels to mount filesystems in the guide
    
    It was a recommended practice for years to get away from straight
    names in /dev, like /dev/sda1, when mounting filesystems. The
    man page for mount(8) says:
    
       The device name of disk partitions are unstable; hardware
       reconfiguration, adding or removing a device can cause change
       in names. This is reason why it's strongly recommended to use
       filesystem or partition identificators like UUID or LABEL.
    
    Nonetheless, novice operators sometimes follow our deployment
    guide to the letter and then get into trouble when device names
    shift from under their deployments. This patch fixes the problem
    without bloating up the guide with general explanations.
    
    Change-Id: I5faae158b62e0395d6e774cd67bd868c785c2186

commit 0294836d9b800de860e36699986f4de0fb1d2141
Author: Tim Burke <tim.burke@gmail.com>
Date:   Fri Mar 29 14:50:06 2019 -0700

py3: test/unit/common/middleware/s3api/test_s3token.py
    
    Change-Id: Ibbff151703395d517452c3d6927652506a9eab66

commit adc826f376e9336394fff858d0ea7d13c8693201
Author: Tim Burke <tim.burke@gmail.com>
Date:   Tue Apr 2 16:48:29 2019 -0700

Clean up more things before running unit tests
    
    Apparently some versions of py3 will drop symlinks in the venv.
    
    Change-Id: I77c52b6ad776ab75c4f9713290cc42a1fcb6e0f7

commit 9d9f8b0ff2865f5b9899f2a60b1bd8bead31186f
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Tue Apr 2 07:06:09 2019 -0500

add more detail to s3api test failure
    
    Change-Id: I89d19c299cf67c459ee893cde7ed6f6716c04f9b

commit e3e79a6e48c572e045380f7110d9ef83deb55707
Author: Tim Burke <tim.burke@gmail.com>
Date:   Fri Mar 29 17:18:49 2019 -0700

Remove "gross hack" for py3pep8 env; run releasenotes under py3
    
    Change-Id: Ib853f79bc062ec550627342d0fc5b2b92029f25c

commit 6c93c5768547a3ca75865e7675ed1efd4db7e2cc
Author: Tim Burke <tim.burke@gmail.com>
Date:   Fri Mar 29 16:14:58 2019 -0700

Run docs tox env under py3
    
    ...and fix this one warning-treated-as-error that crept in since
    https://github.com/openstack/swift/commit/d185b60
    
    Change-Id: Id46ee3ab23e2703170191528427aaa2788aba1ee

commit ffe51501c21c33d65907178511e8d8a747fd442d
Author: Tim Burke <tim.burke@gmail.com>
Date:   Fri Mar 29 16:06:31 2019 -0700

api-ref: add link to SLO docs from multipart-manifest=put param
    
    Drive-by: run api-ref tox env under py3.
    
    Change-Id: Iebce47eabcb3b198d4aa7c1b8cfaf8f53300f7f9

commit 9b694a332436041a96b5211ff9329b562b3a8c6d
Author: Tim Burke <tim.burke@gmail.com>
Date:   Fri Nov 9 15:00:02 2018 -0800

s3token: Add an underscore to the end of reseller_prefix if not present
    
    This matches the behavior of tempauth/keystoneauth with regard to reseller_prefix.
    
    Change-Id: I10f8bc354d0f45dd692915de86857679e9ca3446

commit 8f23dbad6c357a3d10641018a5858d03e3f58a0d
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Tue Mar 26 12:05:10 2019 -0500

Make reconciler test more stable in the gate
    
    Change-Id: I3a831ac2cd2ef7ef83445b8d374d1435ab7292ab

commit 10cb205fe0556769dc688c9059d8802f49a4ee07
Author: Sean McGinnis <sean.mcginnis@gmail.com>
Date:   Fri Mar 22 10:23:36 2019 -0500

Fix lower-constraint deps handling
    
    When the lower-constraints tox target was added, it was assumed the
    install_command was just running the install and that the dependencies
    and constraints were being set using "deps = ".
    
    This fixed the install_command and deps to follow the expected pattern
    so the lower-constraints job actual does install the lower constraints.
    
    Also:
      * raise the minimum for netifaces as the existing version would fail
        to load
      * pin oslo.log to the minimum required by our minimum
        keystonemiddleware
      * fix up some error handling for ancient versions of pastedeploy
      * specify that we shouldn't install xattr on a platform we don't
        support anyway, because reasons
    
    Change-Id: Ie78c0dabe12e01109db2b6412166c3564b87ee96
    Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>

commit a4298fb3ab071b1284fb7080f92c305c6b1cb445
Author: Wang Yuxin <wang.yuxin@ostorage.com.cn>
Date:   Mon Apr 23 17:44:30 2018 +0800

s3api: Fix get_container_info missing some sysmeta info.
    
    When get_container_info is called and not authenticated, it will
    make a HEAD subrequest and get info by passing resp.sw_headers to
    headers_to_container_info. This will lose all sysmeta stored in
    resp.sysmeta_headers.
    
    The patch fixes this by passing all sw_headers and
    sysmeta_headers to headers_to_container_info.
    
    Change-Id: I6e538ed7a748b60bdb9db7e894eaedc9d72559c1
    Closes-Bug: #1765679

commit 29b877de712cc6baf0cfb41bfa53ae018b04de8b
Author: Tim Burke <tim.burke@gmail.com>
Date:   Thu Jan 10 01:35:05 2019 +0000

s3api: fix partition between S3 and Swift headers
    
    In swift3 [1], we partitioned between swift3-specific sysmeta headers and
    all other swift headers. During the transition to s3api [2], we were
    sure to include a support for reading swift3 sysmeta as s3api sysmeta,
    but we forgot to add all *other* sysmeta to the swift headers!
    
    Fix that, and rename sw_sysmeta_headers to s3_sysmeta_headers to
    more-accurately reflect its purpose.
    
    [1] https://github.com/openstack/swift3/blob/1.12/swift3/response.py#L90-L96
    [2] https://review.openstack.org/#/c/557623/
    
    Change-Id: Id6de002b18ba784ad53bda38855376159c7d2ba7

tags:

added: in-feature-losf

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-07-22: Fix included in openstack/swift 2.22.0

This issue was fixed in the openstack/swift 2.22.0 release.