Either delimiter option is not properly documented or has a bug

Bug #1475018 reported by Timur Alperovich
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
Fix Released
Undecided
Unassigned

Bug Description

Setting just the delimiter (without a prefix) results in behavior that is different from setting both parameters.

Example to illustrate the problem:
1. Create the following objects in a container: /dir, /dir/subdir/foo, dir, dir/subdir/foo
2. Issue a query that sets the delimiter to "/" and prefix to "/"
    Expected result: subdir: "/dir/", blobs: "/dir"
    Actual result: subdir: "/dir/", blobs: "/dir" -- this is correct
3. Issue a query that sets the prefix to "" and delimiter to "/"
    Expected result: subdir: ["/", "dir/"]
    Actual result: subdir: ["dir/"], blobs: ["/dir", "/dir/subdir/foo", "dir"]

This is really puzzling, as for keys that start with the delimiter, _all_ keys are listed, even if they include multiple occurrences of the delimiter. This behavior seems wrong to me.

This is also interesting given the documentation for the path option: "Equivalent to setting delimiter to / and prefix to the path with a / at the end." However, this is not correct. Setting path to "" should then be the same as setting delimiter to "/" and prefix to "/", however, the actual results from setting path to "" are: "dir".

I am observing this behavior with Swift 2.2.1 (Rackspace), 2.2.2 (docker-swift), and 2.3.0-rc2 (also, as docker-swift). Note: this problem is not specific to using "/" as a delimiter. The same issue appears if "-" is used, for example.

I'm also not clear (from the documentation), what the expected output should be when keys start with the delimiter and there is no prefix.

Revision history for this message
Timur Alperovich (timur-alperovich) wrote :

I submitted a commit for review that adds a unit test for this behavior and changes the delimiter handling when the key begins with the delimiter character.

Revision history for this message
John Dickinson (notmyname) wrote :
Changed in swift:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on swift (master)

Change abandoned by John Dickinson (<email address hidden>) on branch: master
Review: https://review.openstack.org/202302
Reason: Abanoning based on the lack of activity since the last negative review. If you want to continue working on this, please reopen this patch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to swift (master)

Reviewed: https://review.openstack.org/202302
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=432e280aef1ba08bbb8dc239260604024325c9f6
Submitter: Jenkins
Branch: master

commit 432e280aef1ba08bbb8dc239260604024325c9f6
Author: Timur Alperovich <email address hidden>
Date: Wed Jul 15 14:22:45 2015 -0700

    Correctly handle keys starting with the delimiter.

    When processing keys where the names start with the delimiter
    character, swift should list only the delimiter character. To get the
    list of nested keys, the caller should also supply the prefix which is
    equal to the delimiter.

    Added a functional test and unit tests to verify this behavior.

    Fixes Bug: 1475018

    Change-Id: I27701a31bfa22842c272b7781738e8c546b82cbc

Changed in swift:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to swift (feature/crypto)

Fix proposed to branch: feature/crypto
Review: https://review.openstack.org/304242

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: feature/crypto
Review: https://review.openstack.org/304251

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on swift (feature/crypto)

Change abandoned by Alistair Coles (<email address hidden>) on branch: feature/crypto
Review: https://review.openstack.org/304242

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to swift (feature/crypto)
Download full text (7.3 KiB)

Reviewed: https://review.openstack.org/304251
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=248b9c6679911261efec5a05c92e9c97216080ee
Submitter: Jenkins
Branch: feature/crypto

commit 33f06dc48f7bec2e128b44427fb429ad640cd486
Author: Ondřej Nový <email address hidden>
Date: Sat Apr 9 18:47:58 2016 +0200

    Fixed Sphinx errors

    doc/source/deployment_guide.rst:1372: ERROR: Malformed table.
    swift/obj/diskfile.py:docstring of swift.obj.diskfile.BaseDiskFileManager.yield_hashes:13: ERROR: Unexpected indentation.
    doc/source/ops_runbook/diagnose.rst:188: WARNING: Inline emphasis start-string without end-string.

    Change-Id: Id20eb62eb5baebb3814e7af5676badb94f17dee5

commit a057c409ec8a23290bc72c4fa45d55a1178f4828
Author: OpenStack Proposal Bot <email address hidden>
Date: Fri Apr 8 07:02:33 2016 +0000

    Imported Translations from Zanata

    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure

    Change-Id: I9f4330ec20463e4d303e8ba3b67f86813a914ac5

commit d09ef0da62b64067b04a980c643f77526a9078ac
Author: Alistair Coles <email address hidden>
Date: Wed Apr 6 15:40:42 2016 +0100

    Assert that ChunkWriteTimouts are not raised

    Follow up for change Ibbc89449e7878fc4215e47e3f7dfe4ae58a2d638
    to add a test assertion that the ChunkWriteTimeout contexts are
    exited without raising the timeout exception in
    iter_bytes_from_response_part().

    Change-Id: I6d323cb26779e457fb5940093a81b349b333a0af

commit 7c0f58ec2ed020186ca3f269153b184fc02bf37a
Author: OpenStack Proposal Bot <email address hidden>
Date: Thu Apr 7 07:00:08 2016 +0000

    Imported Translations from Zanata

    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure

    Change-Id: Ib80e3a759fa1e4a99576710607ad07fc5f259527

commit edc413b85ec2b703d7506be9c4801eb347611c58
Author: Nguyen Hung Phuong <email address hidden>
Date: Thu Apr 7 13:31:26 2016 +0700

    Fix typos in Swift files

    Change-Id: I39dbf55c094c42347b57ef67520abff9e6fc24bc

commit 95efd3f9035ec4141e1b182516f040a59a3e5aa6
Author: Samuel Merritt <email address hidden>
Date: Wed Mar 23 13:51:47 2016 -0700

    Fix infinite recursion during logging when syslog is down

    Change-Id: Ia9ecffc88ce43616977e141498e5ee404f2c29c4

commit 0bf518e3b0eeaf66653db6972525701cacfe6333
Author: Thiago da Silva <email address hidden>
Date: Wed Apr 6 16:58:36 2016 -0400

    remove unused current_status method

    Change-Id: I574919eaa14cadc800f3a1f6014221ee382ee7e0
    Signed-off-by: Thiago da Silva <email address hidden>

commit e15bceaa7e541c77f26a1f11ee2cbddbc871cbf1
Author: Kota Tsuyuzaki <email address hidden>
Date: Mon Dec 21 03:13:50 2015 -0800

    Refactor CORS unit tests

    This is a follow-up patch for https://review.openstack.org/#/c/258392/
    That one added good unit test cases for various kinds of
    allowe_origin like '*' or ''(empty). However, the result of handling
    in Swift proxy will depend on strict_cors_mode option configur...

Read more...

tags: added: in-feature-crypto
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to swift (feature/hummingbird)

Fix proposed to branch: feature/hummingbird
Review: https://review.openstack.org/323599

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to swift (feature/hummingbird)
Download full text (84.7 KiB)

Reviewed: https://review.openstack.org/323599
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=0330478b70d0a699a0f9c21ef87c7e639d92564b
Submitter: Jenkins
Branch: feature/hummingbird

commit 5fe392b562de3baed080704df433fb392cb4fb31
Author: Ondřej Nový <email address hidden>
Date: Tue May 31 16:25:50 2016 +0200

    Fixed typo

    Change-Id: I7a35c0076360c7a23cf405189828d3c252ec6708

commit b52eccb3b1ea0591f0040587228d3705b5d3f68d
Author: Clay Gerrard <email address hidden>
Date: Wed May 25 11:21:25 2016 -0700

    Clarify overload best practices in admin guide

    Change-Id: Ib7c08bdeab6374771bb8e2b05053e7e16973524d

commit f1fd50723bb84c4941e949895576733f6eb67793
Author: Christian Schwede <email address hidden>
Date: Wed May 25 09:53:31 2016 +0200

    Add dispersion --verbose example to admin guide

    Change-Id: I5f9cacedde2a329332ccf744800b6f2453e8b28e

commit b3ab715c055283ccfea9a504d6da20741d82e7ad
Author: Matthew Oliver <email address hidden>
Date: Wed May 25 14:35:54 2016 +1000

    Add ring-builder dispersion command to admin guide

    This change updates the admin guide to point out the dispersion command
    in swift-ring-builder and mentions the dispersion verbose table to make
    it more obvious to operators.

    Change-Id: I72b4c8b2d718e6063de0fdabbaf4f2b73694e0a4

commit fb7a8e9ab7596a36a6992a3a8f8c6d005a2c2829
Author: Tim Burke <email address hidden>
Date: Tue May 24 13:37:58 2016 -0700

    Add links to mitaka install guides

    Change-Id: I62331923751c521daded4468b5cc5f03655226bc

commit e09c4ee7800e82aa09ca2f6ae375420b766182a4
Author: Tim Burke <email address hidden>
Date: Fri Apr 29 12:12:00 2016 -0500

    Allow concurrent bulk deletes

    Before, server-side deletes of static large objects could take a long
    time to complete since the proxy would wait for a response to each
    segment DELETE before starting the next DELETE request.

    Now, operators can configure a concurrency factor for the slo and bulk
    middlewares to allow up to N concurrent DELETE requests. By default, two
    DELETE requests will be allowed at a time.

    Note that objects and containers are now deleted in separate passes, to
    reduce the likelihood of 409 Conflict responses when deleting
    containers.

    Upgrade Consideration
    =====================
    If operators have enabled the bulk or slo middlewares and would like to
    preserve the prior (single-threaded) DELETE behavior, they must add the
    following line to their [filter:slo] and [filter:bulk] proxy config
    sections:

       delete_concurrency = 1

    This may be done prior to upgrading Swift.

    UpgradeImpact
    Closes-Bug: 1524454
    Change-Id: I128374d74a4cef7a479b221fd15eec785cc4694a

commit 226557afc42c245e050d84162497f46341407ef7
Author: Tim Burke <email address hidden>
Date: Thu May 19 18:55:40 2016 -0700

    Turn on H703, so our translators don't punch us

    Change-Id: I4ce3068f79563e4d4296c6e1078bc12f0cf84c96
    Related-Bug: 1559431

commit 7b706926a8ed5bbcec3a678e868e301c9a6ed8f1
Author: Alistair Coles <email address hidden>
Date: Mon May ...

tags: added: in-feature-hummingbird
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to swift (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/700964

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to swift (stable/train)

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/700965

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to swift (master)

Reviewed: https://review.opendev.org/700964
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=55ab08a5fadda87ffc904d9fe0ba1d1e09612ae3
Submitter: Zuul
Branch: master

commit 55ab08a5fadda87ffc904d9fe0ba1d1e09612ae3
Author: Tim Burke <email address hidden>
Date: Thu Jan 2 16:21:19 2020 -0800

    account-server: Add test for leading delimiter

    Related-Change: I27701a31bfa22842c272b7781738e8c546b82cbc
    Related-Change: If912f71d8b0d03369680374e8233da85d8d38f85
    Change-Id: I0c850b8ae40d1ab477d0f5d18f92579c457da54e
    Related-Bug: 1475018

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to swift (stable/train)

Reviewed: https://review.opendev.org/700965
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=d1d0b706e4ac57b1fd4bbbce386e5d8dadf171b8
Submitter: Zuul
Branch: stable/train

commit d1d0b706e4ac57b1fd4bbbce386e5d8dadf171b8
Author: Tim Burke <email address hidden>
Date: Thu Jan 2 16:21:19 2020 -0800

    account-server: Correctly handle containers starting with delimiter

    Related-Change: I27701a31bfa22842c272b7781738e8c546b82cbc
    Related-Change: If912f71d8b0d03369680374e8233da85d8d38f85
    Change-Id: I0c850b8ae40d1ab477d0f5d18f92579c457da54e
    Related-Bug: 1475018

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to swift (feature/losf)

Related fix proposed to branch: feature/losf
Review: https://review.opendev.org/713632

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to swift (feature/losf)
Download full text (40.5 KiB)

Reviewed: https://review.opendev.org/713632
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=79bd2e59e5c15ee84814ec1c4f0893176ba79412
Submitter: Zuul
Branch: feature/losf

commit f2ffd900593db2829a39a073f0c033d82985c40f
Author: Clay Gerrard <email address hidden>
Date: Fri Feb 28 11:09:51 2020 -0600

    Apply limit to list versioned containers

    Change-Id: I28e062273d673c4f07cd3c5da088aa790b77a599
    Closes-Bug: #1863841

commit dc40779307095b8d0b2761b77b9cb2904ec721ae
Author: Clay Gerrard <email address hidden>
Date: Fri Feb 28 10:00:25 2020 -0600

    Use float consistently for proxy timeout settings

    Change-Id: I433c97df99193ec31c863038b9b6fd20bb3705b8

commit 55049beda5b9d7038a3604a87f28312d7702ccb2
Author: Tim Burke <email address hidden>
Date: Fri Feb 28 18:59:32 2020 -0800

    tests: Use timedelta to adjust dates, not string manipulations

    Change-Id: I8f65ccd7f2a79d5b877bfbef0274fb7857e21391

commit 3b65a5998cc921d2763cf1a9ec1e40b88491262d
Author: Tim Burke <email address hidden>
Date: Wed Jan 10 06:16:41 2018 +0000

    Fix up some Content-Type handling in account/container listings

    Update content type on 204 (not just 200) to properly handle HEAD
    requests from xml/txt listings.

    Add "Vary: Accept" header to listings, since otherwise, browsers may
    serve the wrong content type from cache (even though we *would have*
    sent the *right* type if it actually sent the request).

    Change-Id: Iaa333aaca36a8dc2df65d38ef2173e3a6e2000ee

commit ecca23eb806e11cf6517f0456483da7a065350a8
Author: Clay Gerrard <email address hidden>
Date: Fri Feb 21 15:33:21 2020 -0600

    Extend eventlet_debug logging to GreenAsyncPile

    Change-Id: Ibd9fe5c9a1e75b86eb7d540594d5cf516758e17e

commit 0fb3371484f1d0f629d0b0e33f6aafbff0e43ee9
Author: Sam Morrison <email address hidden>
Date: Tue Feb 18 10:17:50 2020 +1100

    Delay importing swiftclient until after monkey-patching

    Commit message below partly copied from nova:

    Eventlet monkey patching should be as early as possible

    We were seeing infinite recursion opening an ssl socket when running
    various combinations of python3, eventlet, and urllib3. It is not
    clear exactly what combination of versions are affected, but for
    background there is an example of this issue documented here:

    https://github.com/eventlet/eventlet/issues/371

    The immediate cause in swift's case was that we were calling
    eventlet.monkey_patch() after importing swiftclient (which imported
    requests, which finally imported urllib3).

    We only use the imported function in one place, however; hold off on
    importing until we actually need it to ensure that monkey patching
    happens first. Note that we *don't* want to monkey-patch at import time,
    as we've previously had bugs related to import-time side-effects.

    Change-Id: I24f4bcc3d62dc37fd9559032bfd25f5b15f98745
    Closes-bug: #1863680
    Related-bug: #1380815

commit a5afe767581d2cb97cf3690067e6d626c7682c2c
Author: Tim Burke <email address hidden>
Date: Wed Feb 19 10:09:49 2020 -0800

    Revert "Make roll...

tags: added: in-feature-losf
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.