Dynamic Large Object does not work with "?" in prefix

Bug #1598093 reported by Dariusz Działak on 2016-07-01
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
Undecided
Unassigned

Bug Description

It's about storing dlo manifest with header X-Object-Manifest: <container>/<prefix>

I've tried to put segments to virtual directory (prefix) with char "?" but reading such object failed with "500 Internal Server Error"

Tim Burke (1-tim-z) wrote :

Does it work if you replace '?' with '%3F' (so the question mark is URL-encoded)?

Either way, the 500 is definitely a bug.

Download full text (4.5 KiB)

The header is always url-encoded, so in the request there is already "%3F".
Do you want to uncode it once again?
For the next week I'll be unavailable but for now I reproduced that bug
with python-swiftclient.

manifest location: container/file?2
prefix for segments: segments/file?2/1467410424.120684/123000000/6000000/

Looks like dlo is formatting wrong request to the proxy and try to get
"segments/file" which does not exists (and there is 404 response):

Jul 1 22:40:08 108253809cd3 object-server: 127.0.0.1 - -
[01/Jul/2016:22:40:08 +0000] "GET /sdb1/104/AUTH_test/container/file%3F2"
200 - "GET http://127.0.0.1:8080/v1/AUTH_test/container/file%3F2"
"tx61bc267352de45129421f-005776f148" "proxy-server 544" 0.0009
Jul 1 22:40:08 108253809cd3 container-server: 127.0.0.1 - -
[01/Jul/2016:22:40:08 +0000] "GET /sdb1/64/AUTH_test/segments" 200 4578
"tx61bc267352de45129421f-005776f148" "GET
http://127.0.0.1:8080/v1/AUTH_test/segments?format=json&prefix=file%3F2/1467410424.120684/123000000/6000000/"
"proxy-server 544" 0.0042
Jul 1 22:40:08 108253809cd3 proxy-server: - - 01/Jul/2016/22/40/08 GET
/v1/AUTH_test/segments%3Fformat%3Djson%26prefix%3Dfile%253F2/1467410424.120684/123000000/6000000/
HTTP/1.0 200 -
python-requests/2.2.1%20CPython/2.7.6%20Linux/4.4.0-24-generic%20DLO%20MultipartGET
AUTH_tk53ba255234764381902cbddfcd64bdc1 - 4578 -
tx61bc267352de45129421f-005776f148 - 0.0085 DLO - 1467412808.672676086
1467412808.681154013
Jul 1 22:40:08 108253809cd3 object-server: 127.0.0.1 - -
[01/Jul/2016:22:40:08 +0000] "GET /sdb1/95/AUTH_test/segments/file" 404 -
"GET
http://127.0.0.1:8080/v1/AUTH_test/segments/file?2/1467410424.120684/123000000/6000000/00000000?multipart-manifest=get"
"tx61bc267352de45129421f-005776f148" "proxy-server 544" 0.0004

It's even more funny, I've stored object: segments/file and received log:
Jul 1 22:47:46 108253809cd3 proxy-server: - - 01/Jul/2016/22/47/46 GET
/v1/AUTH_test/segments/file%3F2/1467410424.120684/123000000/6000000/00000020%3Fmultipart-manifest%3Dget
HTTP/1.0 499 -
python-requests/2.2.1%20CPython/2.7.6%20Linux/4.4.0-24-generic%20%20DLO%20MultipartGET
AUTH_tk53ba255234764381902cbddfcd64bdc1 - 3014656 -
txa172304a884d4e8585287-005776f308 - 0.0303 DLO - 1467413266.517827034
1467413266.548119068
Jul 1 22:47:46 108253809cd3 proxy-server: Client disconnected on read
(txn: txa172304a884d4e8585287-005776f308)
Jul 1 22:47:46 108253809cd3 proxy-server: Too many bytes for
/v1/AUTH_test/container/file%3F2; truncating in /v1/AUTH_test/segments/file
with -14656 bytes left: #012Traceback (most recent call last):#012 File
"/usr/lib/python2.7/dist-packages/swift/common/request_helpers.py", line
346, in __iter__#012 'left': bytes_left})#012SegmentError: Too many
bytes for /v1/AUTH_test/container/file%3F2; truncating in
/v1/AUTH_test/segments/file with -14656 bytes left (txn:
txa172304a884d4e8585287-005776f308)

BUT client left that file:
root@108253809cd3:/# swift download container 'file?2'
file?2 [auth 0.016s, headers 0.066s, total 10.564s, 11.661 MB/s]
root@108253809cd3:/# lswift stat container 'file?2'
           URL: http://127.0.0.1:8080/v1/AUTH_test/container/file?2
    Auth Token: AUTH_tk53ba255234764381902cbddfcd64...

Read more...

Tim Burke (1-tim-z) on 2016-08-22
Changed in swift:
status: New → Confirmed
assignee: nobody → Tim Burke (1-tim-z)
Dariusz Działak (ddzialak) wrote :

any update?

Tim Burke (1-tim-z) on 2017-09-11
Changed in swift:
assignee: Tim Burke (1-tim-z) → nobody

Reviewed: https://review.openstack.org/571905
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=fa678949ae310aa0499938fef788ec04409625d9
Submitter: Zuul
Branch: master

commit fa678949ae310aa0499938fef788ec04409625d9
Author: Tim Burke <email address hidden>
Date: Wed May 30 11:43:40 2018 -0700

    Fix quoting for large objects

    Change-Id: I46bdb6da8f778a6c86e0f8e883b52fc31e9fd44e
    Partial-Bug: 1774238
    Closes-Bug: 1678022
    Closes-Bug: 1598093
    Closes-Bug: 1762997

Changed in swift:
status: Confirmed → Fix Released

This issue was fixed in the openstack/swift 2.21.0 release.

Download full text (11.9 KiB)

Reviewed: https://review.openstack.org/648245
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=6afc1130fd753306d64745c9bee7712182b273d3
Submitter: Zuul
Branch: feature/losf

commit 89e5927f7dd94fc28b3847944eb7dd227d516fa8
Author: Thiago da Silva <email address hidden>
Date: Tue Mar 26 10:46:02 2019 -0400

    Fix mocking time

    When running on Centos the side_effect was returning a MagicMock
    object instead of the intended int.

    Change-Id: I73713a9a96dc415073a637d85a40304021f76072

commit 50715acb1838fbde628e447e7b02545ce8469180
Author: OpenStack Release Bot <email address hidden>
Date: Mon Mar 25 17:07:54 2019 +0000

    Update master for stable/stein

    Add file to the reno documentation build to show release notes for
    stable/stein.

    Use pbr instruction to increment the minor version number
    automatically so that master versions are higher than the versions on
    stable/stein.

    Change-Id: I6109bff3227f87d914abf7bd1d76143aaf91419d
    Sem-Ver: feature

commit 179fa7ccd4d6faeacc989715887b69f9422a17b2
Author: John Dickinson <email address hidden>
Date: Mon Mar 18 17:09:31 2019 -0700

    authors/changelog update for 2.21.0 release

    Change-Id: Iac51a69c71491e5a8db435aae396178a6c592c73

commit 64eec5fc93eb670e581cbb3a6dedb6a7aa501e99
Author: Tim Burke <email address hidden>
Date: Thu Mar 7 14:36:02 2019 -0800

    Fix how we UTF-8-ify func tests

    I noticed while poking at the DLO func tests that we don't actually use
    non-ascii chars when we set up the test env.

    By patching the create name function earlier (in SetUpClass) we can
    ensure we get some more interesting characters in our object names.

    Change-Id: I9480ddf74463310aeb11ad876b79527888d8c871

commit fe3a20f2e4b745bf7d81f9bda97082b593e8794a
Author: Tim Burke <email address hidden>
Date: Tue Mar 19 14:52:19 2019 -0700

    Remove uncalled function

    Change-Id: Ica67815f0ddf4b00bce1ffe183735490c7f7c0b5
    Related-Change: I5629de9f2e9b2331ed3f455d253efc69d030df72

commit adc568c97f5b30d9d4628eaf448f81d736ad4e51
Author: John Dickinson <email address hidden>
Date: Fri Mar 15 15:18:36 2019 -0700

    Fix bulk responses when using xml and Expect 100-continue

    When we fixed bulk response heartbeating in https://review.openstack.org/#/c/510715/,
    code review raised the issue of moving the xml header down to after the
    early-exit clauses. At the time, it didn't seem to break anything, so
    it was left in place. However, that insight was correct.

    The purpose of the earlier patch was to force eventlet to use chunked
    transfer encoding on the response in order to prevent eventlet from
    buffering the whole response, thus defeating the purpose of the
    heartbeat responses.

    Moving the first line of the body lower (ie after the early exit
    checks), allows other headers in a chunked transfer encoding response
    to be appropriately processed before sending the headers. Sending the
    xml declaration early causes it to get intermingled in the 100-continue
    protocol, thus breaking the chunked transfer encoding semantics.

    Closes-Bug: #1819...

tags: added: in-feature-losf
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers