Comment 0 for bug 1613316

I'm trying to use container-sync inside a single Liberty Swift cluster to move
data from a Docker Registry to a different container in a different project.
The sync itself seems to be progressing well, but I'm running into bizarre
errors when trying to configure the Registry application against the new
container.

The problem seems to be with DLOs. On the receiving side, the DLO shows up in
the container listing, but 404 is returned when trying to GET it.

I'll grab an example. Here's what I can see in the source container:

1. GET on the container finds the object. (It's listed with zero size, which is wrong, but whatever.)

    $ curl -i -H 'X-Auth-Token: <redacted>' 'https://swift.local/v1/AUTH_<source-project-id>/<source-container>?prefix=files/docker/registry/v2/blobs/sha256/03/0374098d7d826c67f/data&format=json'
    HTTP/1.1 200 OK
    X-Container-Sync-Key: <redacted>
    Content-Length: 271
    X-Container-Object-Count: 499476
    X-Storage-Policy: default
    Accept-Ranges: bytes
    X-Container-Sync-To: <redacted>
    X-Container-Bytes-Used: 1108641002465
    X-Timestamp: 1447781107.80138
    Content-Type: application/json; charset=utf-8
    X-Trans-Id: tx387d34644f584abcafa16-0057b1d093
    Date: Mon, 15 Aug 2016 14:24:20 GMT

    [{"hash": "d41d8cd98f00b204e9800998ecf8427e", "last_modified": "2016-07-19T11:31:30.437110", "bytes": 0, "name": "files/docker/registry/v2/blobs/sha256/03/0374098d7d826c67f/data", "content_type": "application/octet-stream"}]

2. When I get the DLO, the content is returned.

    $ curl --head -H 'X-Auth-Token: <redacted>' https://swift.local/v1/AUTH_<source-project-id>/<source-container>/files/docker/registry/v2/blobs/sha256/03/0374098d7d826c67f/data
    HTTP/1.1 200 OK
    Content-Length: 1905
    Etag: "d3120bba48f0479789dcefb1e26a03ce"
    Accept-Ranges: bytes
    Last-Modified: Tue, 19 Jul 2016 11:31:31 GMT
    X-Object-Manifest: <source-container>/segments/2f6/46f636b25690827fec7a67d8
    X-Timestamp: 1468927890.43711
    Content-Type: application/octet-stream
    X-Trans-Id: tx1ee45a0383e445b8b3c46-0057b1cb7b
    Date: Mon, 15 Aug 2016 14:02:35 GMT

3. And of course, since this is a DLO, I can GET on the container to list the segments (just one in this case).

    $ curl -i -H 'X-Auth-Token: <redacted>' 'https://swift.local/v1/AUTH_<source-project-id>/<source-container>?prefix=segments/2f6/46f636b25690827fec7a67d8&format=json'
    HTTP/1.1 200 OK
    X-Container-Sync-Key: <redacted>
    Content-Length: 499
    X-Container-Object-Count: 499426
    X-Storage-Policy: default
    Accept-Ranges: bytes
    X-Container-Sync-To: <redacted>
    X-Container-Bytes-Used: 1108364443373
    X-Timestamp: 1447781107.80138
    Content-Type: application/json; charset=utf-8
    X-Trans-Id: txb1369af939e5499c8c8cf-0057b1cc2b
    Date: Mon, 15 Aug 2016 14:05:31 GMT

    [{"hash": "293fa5ac84ff3c10d690e192cbcfa95f", "last_modified": "2016-07-19T11:31:28.772710", "bytes": 1905, "name": "segments/2f6/46f636b25690827fec7a67d8/0000000000000001", "content_type": "application/octet-stream"}]

Nothing fancy until here. Let's see how things look in the target container:

1. GET on the container finds the object (again, with zero size).

    $ curl -i -H 'X-Auth-Token: <redacted>' 'https://swift.local/v1/AUTH_<target-project-id>/<target-container>?prefix=files/docker/registry/v2/blobs/sha256/03/0374098d7d826c67f/data&format=json'
    HTTP/1.1 200 OK
    X-Container-Sync-Key: <redacted>
    Content-Length: 271
    X-Container-Object-Count: 485432
    Accept-Ranges: bytes
    X-Storage-Policy: default
    X-Container-Meta-Temp-Url-Key: <redacted>
    X-Container-Bytes-Used: 1102059424351
    X-Timestamp: 1469003815.67509
    Content-Type: application/json; charset=utf-8
    X-Trans-Id: tx85710b95e51245848dde0-0057b1c857
    Date: Mon, 15 Aug 2016 13:49:11 GMT

    [{"hash": "d41d8cd98f00b204e9800998ecf8427e", "last_modified": "2016-07-19T11:31:30.437110", "bytes": 0, "name": "files/docker/registry/v2/blobs/sha256/03/0374098d7d826c67f/data", "content_type": "application/octet-stream"}]

2. Which means that I can GET the object, too. Except I can't. Wait, what?

    $ curl -i -H 'X-Auth-Token: <redacted>' https://swift.local/v1/AUTH_<target-project-id>/<target-container>/files/docker/registry/v2/blobs/sha256/03/0374098d7d826c67f/data; echo
    HTTP/1.1 404 Not Found
    Content-Length: 70
    Content-Type: text/html; charset=UTF-8
    X-Trans-Id: tx01b439f8583b470fbfb03-0057b1c859
    Date: Mon, 15 Aug 2016 13:49:13 GMT

    <html><h1>Not Found</h1><p>The resource could not be found.</p></html>

3. The segment has been synced correctly, though.

    $ curl -i -H 'X-Auth-Token: <redacted>' 'https://swift.local/v1/AUTH_<target-project-id>/<target-container>?prefix=segments/2f6/46f636b25690827fec7a67d8&format=json'
    HTTP/1.1 200 OK
    X-Container-Sync-Key: <redacted>
    Content-Length: 499
    X-Container-Object-Count: 485457
    Accept-Ranges: bytes
    X-Storage-Policy: default
    X-Container-Meta-Temp-Url-Key: <redacted>
    X-Container-Bytes-Used: 1102135036266
    X-Timestamp: 1469003815.67509
    Content-Type: application/json; charset=utf-8
    X-Trans-Id: tx348d8516cb0e487f89e56-0057b1d3da
    Date: Mon, 15 Aug 2016 14:38:18 GMT

    [{"hash": "293fa5ac84ff3c10d690e192cbcfa95f", "last_modified": "2016-07-19T11:31:28.772710", "bytes": 1905, "name": "segments/2f6/46f636b25690827fec7a67d8/0000000000000001", "content_type": "application/octet-stream"}]