HEAD requests on DLOs break when segment container is missing

Bug #1613316 reported by Stefan Majewsky
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
Fix Released
Undecided
Unassigned

Bug Description

I'm trying to use container-sync inside a single Liberty Swift cluster to move data from a Docker Registry to a different container in a different project. The sync itself seems to be progressing well, but I'm running into bizarre errors when trying to configure the Registry application against the new container.

The problem seems to be with DLOs. On the receiving side, the DLO shows up in the container listing, but 404 is returned when trying to GET it.

I'll grab an example. Here's what I can see in the source container:

1. GET on the container finds the object. (It's listed with zero size, which is wrong, but whatever.)

    $ curl -i -H 'X-Auth-Token: <redacted>' 'https://swift.local/v1/AUTH_<source-project-id>/<source-container>?prefix=files/docker/registry/v2/blobs/sha256/03/0374098d7d826c67f/data&format=json'
    HTTP/1.1 200 OK
    X-Container-Sync-Key: <redacted>
    Content-Length: 271
    X-Container-Object-Count: 499476
    X-Storage-Policy: default
    Accept-Ranges: bytes
    X-Container-Sync-To: <redacted>
    X-Container-Bytes-Used: 1108641002465
    X-Timestamp: 1447781107.80138
    Content-Type: application/json; charset=utf-8
    X-Trans-Id: tx387d34644f584abcafa16-0057b1d093
    Date: Mon, 15 Aug 2016 14:24:20 GMT

    [{"hash": "d41d8cd98f00b204e9800998ecf8427e", "last_modified": "2016-07-19T11:31:30.437110", "bytes": 0, "name": "files/docker/registry/v2/blobs/sha256/03/0374098d7d826c67f/data", "content_type": "application/octet-stream"}]

2. When I get the DLO, the content is returned.

    $ curl --head -H 'X-Auth-Token: <redacted>' https://swift.local/v1/AUTH_<source-project-id>/<source-container>/files/docker/registry/v2/blobs/sha256/03/0374098d7d826c67f/data
    HTTP/1.1 200 OK
    Content-Length: 1905
    Etag: "d3120bba48f0479789dcefb1e26a03ce"
    Accept-Ranges: bytes
    Last-Modified: Tue, 19 Jul 2016 11:31:31 GMT
    X-Object-Manifest: <source-container>/segments/2f6/46f636b25690827fec7a67d8
    X-Timestamp: 1468927890.43711
    Content-Type: application/octet-stream
    X-Trans-Id: tx1ee45a0383e445b8b3c46-0057b1cb7b
    Date: Mon, 15 Aug 2016 14:02:35 GMT

3. And of course, since this is a DLO, I can GET on the container to list the segments (just one in this case).

    $ curl -i -H 'X-Auth-Token: <redacted>' 'https://swift.local/v1/AUTH_<source-project-id>/<source-container>?prefix=segments/2f6/46f636b25690827fec7a67d8&format=json'
    HTTP/1.1 200 OK
    X-Container-Sync-Key: <redacted>
    Content-Length: 499
    X-Container-Object-Count: 499426
    X-Storage-Policy: default
    Accept-Ranges: bytes
    X-Container-Sync-To: <redacted>
    X-Container-Bytes-Used: 1108364443373
    X-Timestamp: 1447781107.80138
    Content-Type: application/json; charset=utf-8
    X-Trans-Id: txb1369af939e5499c8c8cf-0057b1cc2b
    Date: Mon, 15 Aug 2016 14:05:31 GMT

    [{"hash": "293fa5ac84ff3c10d690e192cbcfa95f", "last_modified": "2016-07-19T11:31:28.772710", "bytes": 1905, "name": "segments/2f6/46f636b25690827fec7a67d8/0000000000000001", "content_type": "application/octet-stream"}]

Nothing fancy until here. Let's see how things look in the target container:

1. GET on the container finds the object (again, with zero size).

    $ curl -i -H 'X-Auth-Token: <redacted>' 'https://swift.local/v1/AUTH_<target-project-id>/<target-container>?prefix=files/docker/registry/v2/blobs/sha256/03/0374098d7d826c67f/data&format=json'
    HTTP/1.1 200 OK
    X-Container-Sync-Key: <redacted>
    Content-Length: 271
    X-Container-Object-Count: 485432
    Accept-Ranges: bytes
    X-Storage-Policy: default
    X-Container-Meta-Temp-Url-Key: <redacted>
    X-Container-Bytes-Used: 1102059424351
    X-Timestamp: 1469003815.67509
    Content-Type: application/json; charset=utf-8
    X-Trans-Id: tx85710b95e51245848dde0-0057b1c857
    Date: Mon, 15 Aug 2016 13:49:11 GMT

    [{"hash": "d41d8cd98f00b204e9800998ecf8427e", "last_modified": "2016-07-19T11:31:30.437110", "bytes": 0, "name": "files/docker/registry/v2/blobs/sha256/03/0374098d7d826c67f/data", "content_type": "application/octet-stream"}]

2. Which means that I can GET the object, too. Except I can't. Wait, what?

    $ curl -i -H 'X-Auth-Token: <redacted>' https://swift.local/v1/AUTH_<target-project-id>/<target-container>/files/docker/registry/v2/blobs/sha256/03/0374098d7d826c67f/data; echo
    HTTP/1.1 404 Not Found
    Content-Length: 70
    Content-Type: text/html; charset=UTF-8
    X-Trans-Id: tx01b439f8583b470fbfb03-0057b1c859
    Date: Mon, 15 Aug 2016 13:49:13 GMT

    <html><h1>Not Found</h1><p>The resource could not be found.</p></html>

3. The segment has been synced correctly, though.

    $ curl -i -H 'X-Auth-Token: <redacted>' 'https://swift.local/v1/AUTH_<target-project-id>/<target-container>?prefix=segments/2f6/46f636b25690827fec7a67d8&format=json'
    HTTP/1.1 200 OK
    X-Container-Sync-Key: <redacted>
    Content-Length: 499
    X-Container-Object-Count: 485457
    Accept-Ranges: bytes
    X-Storage-Policy: default
    X-Container-Meta-Temp-Url-Key: <redacted>
    X-Container-Bytes-Used: 1102135036266
    X-Timestamp: 1469003815.67509
    Content-Type: application/json; charset=utf-8
    X-Trans-Id: tx348d8516cb0e487f89e56-0057b1d3da
    Date: Mon, 15 Aug 2016 14:38:18 GMT

    [{"hash": "293fa5ac84ff3c10d690e192cbcfa95f", "last_modified": "2016-07-19T11:31:28.772710", "bytes": 1905, "name": "segments/2f6/46f636b25690827fec7a67d8/0000000000000001", "content_type": "application/octet-stream"}]

description: updated
Revision history for this message
Stefan Majewsky (stefan-majewsky) wrote :

After further inspection, I now see what is going on. The DLO is copied correctly, but the manifest is not adjusted for the new container name. When the DLO is queried (whether with HEAD or GET), Swift tries to find the segments, but croaks because the container mentioned in the manifest does not exist.

So the actual bug here is that a DLO referring to segments in a non-existing container cannot be interacted with in a meaningful way. (At least HEAD should work IMO.)

summary: - container sync produces broken DLOs
+ HEAD requests on DLOs break when segment container is missing
Revision history for this message
Stefan Majewsky (stefan-majewsky) wrote :

For the other issue that's hidden in here, I reported https://bugs.launchpad.net/swift/+bug/1613681 separately.

Revision history for this message
Alistair Coles (alistair-coles) wrote :

You can retrieve the manifest object itself (as opposed to the concatenated segments that it points to) by using a multipart-manifest=get query string with a GET or HEAD request. See [1].

Otherwise, RFC 7231 [2] disallows sending different response headers for a HEAD than a GET so if the GET for a DLO does not resolve the segments and returns 404, the HEAD must do the same.

There is a note here [3] that segments must be sync'd as well as manifests [3], but it could be clearer that the source and target container names must be the same.

[1] http://developer.openstack.org/api-ref-objectstorage-v1.html#showObject
[2] https://tools.ietf.org/html/rfc7231#section-4.3.2
[3] http://docs.openstack.org/developer/swift/overview_container_sync.html

Revision history for this message
Alistair Coles (alistair-coles) wrote :

I have improved the API doc here [1] for using multipart-manifest=get with a HEAD

[1] https://review.openstack.org/356012

Revision history for this message
Stefan Majewsky (stefan-majewsky) wrote :

I can see your reasoning. Would there be any recommended way to sync a container containing DLOs to another location where the container name must be different because of business requirements?

Revision history for this message
Alistair Coles (alistair-coles) wrote :

I can't think of a way to do that. You can of course use a different account, but if the container name chanegs then unfortunately the DLO manifest would need to change.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to swift (master)

Reviewed: https://review.openstack.org/356012
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=cc2b2cf9c8a5cb913817e2f900c676809cd2e027
Submitter: Jenkins
Branch: master

commit cc2b2cf9c8a5cb913817e2f900c676809cd2e027
Author: Alistair Coles <email address hidden>
Date: Tue Aug 16 16:21:59 2016 +0100

    Improve doc for using container-sync with large objects

    Clarify that synced segment container names must be the same
    when syncing large objects.

    Also add multipart-menifest query string option to API ref
    for object GETs.

    Change-Id: Ib2d2a1e6c1e5eff215fc75c2b49e7d6758b17b7e
    Partial-Bug: #1613681
    Closes-Bug: #1613316

Changed in swift:
status: New → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to swift (feature/hummingbird)

Fix proposed to branch: feature/hummingbird
Review: https://review.openstack.org/363111

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to swift (feature/hummingbird)
Download full text (84.1 KiB)

Reviewed: https://review.openstack.org/363111
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=1ab2a296f58ae76aeffef9f3f0fb90e15358be27
Submitter: Jenkins
Branch: feature/hummingbird

commit 3b5850836c59c46f2507a7f62aceccf4c37e5d41
Author: gecong1973 <email address hidden>
Date: Tue Aug 30 15:08:49 2016 +0800

    Remove white space between print and ()

    There is a white space between print and ()
    in /tempauth.py, This patch fix it

    Change-Id: Id3493bdef12223aa3a2bffc200db8710f5949101

commit f88e7fc0da2ed6a63e0ea3c3459d80772b3068cd
Author: zheng yin <email address hidden>
Date: Mon Aug 29 20:21:44 2016 +0800

    Clarify test case in common/ring/test_builder

    They use a bare assertRaises(ValueError, ring.RingBuilder, *,*,*), but
    it's not clear which one raises which ValueError(), so I extend them to
    validate the error strings as well.

    Change-Id: I63280a9fc47ff678fe143e635046a0b402fd4506

commit d68b1bd6ddf44c5088e9d02dcb2f1b802c71411b
Author: zhufl <email address hidden>
Date: Mon Aug 29 14:31:27 2016 +0800

    Remove unnecessary tearDown

    This is to remove unnecessary tearDown to keep code clean.

    Change-Id: Ie70e40d6b55f379b0cc9bc372a35705462cade8b

commit d2fc2614575b04fd9cab5ae589880b92eee9b186
Author: Matthew Oliver <email address hidden>
Date: Fri Aug 19 16:17:31 2016 +1000

    Authorise versioned write PUTs before copy

    Currently a versioned write PUT uses a pre-authed request to move
    it into the versioned container before checking whether the
    user is authorised. This can lead to some interesting behaviour
    whereby a user can select a versioned object path that it does not
    have access to, request a put on that versioned object, and this
    request will execute the copy part of the request before it fails
    due to lack of permissions.

    This patch changes the behaviour to be the same as versioned DELETE
    where the request is authorised before anything is moved.

    Change-Id: Ia8b92251718d10b1eb44a456f28d3d2569a30003
    Closes-Bug: #1562175

commit c1ef6539b6ba9d2e4354c9cd2eec8a0195cdb19f
Author: Clay Gerrard <email address hidden>
Date: Thu Aug 25 11:00:49 2016 -0700

    add test for expirer processes == process

    This is a follow up from a change that improved the error message.

    Related-Change: I3d12b79470d122b2114f9ee486b15d381f290f95

    Change-Id: I093801f3516a60b298c13e2aa026c11c68a63792

commit 01477c78c1163822de41484e914a0736e622085b
Author: zheng yin <email address hidden>
Date: Thu Aug 25 15:37:42 2016 +0800

    Fix ValueError information in obj/expirer

    I fix error information in raise ValueError(...)
    For example:
        if a>=b:
            # It should be under below and not 'a must be less than or equal to b'
            raise ValueError('a must be less than b')

    Change-Id: I3d12b79470d122b2114f9ee486b15d381f290f95

commit b81f53b964fdb8f3b50dd369ce2e194ee4dbb0b7
Author: zheng yin <email address hidden>
Date: Tue Aug 23 14:26:47 2016 +0800

    Improve readability in the obj server's unit tests

    This change improves the reada...

tags: added: in-feature-hummingbird
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/swift 2.10.0

This issue was fixed in the openstack/swift 2.10.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.