A complete multipart upload request verification failed due to object-server unable to update container DB

[Charles Hsu]

This issue happens in some high latency backend network between object-server and container-server.

We saw client got 400 errors due to a complete multipart-upload request failed because there is one part isn't exist when Swift3 middleware try to get a list of all parts. And it caused by object-server unable to reach container-server in 0.5s (a connection timeout is 0.5s in default). So that's why swift3 unable to see the part in a container listing.

After 5 mins, object-updater try to get all async_pendings and sync to container DB and the client should able to see the part when doing container listing.

It causes random errors `One or more of the specified parts could not be found. The part might not have been uploaded, or the specified entity tag might not have matched the part's entity tag.` when clients try to upload object with multipart-upload. You won't see this in every multipart upload, but you'll see it sometimes.

[Tim Burke (1-tim-z)]

I think following Swift 2.4.0 [1], the container listing should be unnecessary. We should be able to generate the manifest to PUT based off of our segment naming conventions and the client-provided etags. I need to check what happens if the client uploads more parts than are referenced in the finalization request, though.

[1] https://github.com/openstack/swift/commit/bb71657

[Tim Burke (1-tim-z)]

:-( There's still the need to check that all segments (except the last) meet the minimum size.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/609860

[Tim Burke (1-tim-z)]

For some reason I can't tag this Won't Fix for swift3? Weird. swift3 development's moved to upstream swift, though -- and I don't expect any fix in swift to get backported to swift3.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/609860
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=284bbdd391b24823f1f1154577b2b68575e7f7e9
Submitter: Zuul
Branch: master

commit 284bbdd391b24823f1f1154577b2b68575e7f7e9
Author: Tim Burke <email address hidden>
Date: Thu Oct 11 23:25:32 2018 +0000

    Add slo_manifest_hook callback

    ... to allow other middlewares to impose additional constraints on
    or make edits to SLO manifests before being written.

    The callback takes a single argument: the python list that represents
    the manifest to be written. All the normal list operations listed at
    https://docs.python.org/2/library/stdtypes.html#mutable-sequence-types
    are available to make changes to that before SLO serializes it as JSON.

    The callback may return a list of problematic segments; each item in the
    list should be a tuple of

        (quoted object name, description of problem)

    This will be useful both for s3api minimum segment size validation and
    creating tar large objects.

    Change-Id: I198c5196e0221a72b14597a06e5ce3c4b2bbf436
    Related-Bug: #1636663

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/610747
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=fcccb52ff492f59bc26cc650761a3597216dd143
Submitter: Zuul
Branch: master

commit fcccb52ff492f59bc26cc650761a3597216dd143
Author: Tim Burke <email address hidden>
Date: Mon Oct 15 15:44:03 2018 -0700

    s3api: Stop relying on container listings during multipart uploads

    Previously, we would list the segments container before completing a
    multipart upload so that we could verify ETags and sizes before
    attempting to create the SLO. However, container listings are only
    eventually-consistent, which meant that clients could receive a 400
    response complaining that parts could not be found, even though all
    parts were uploaded successfully.

    Now, use the new SLO validator callback to validate segment sizes, and
    use the existing SLO checks to validate ETags.

    Change-Id: I57ae6756bd5f06b80cf03a6b40bf58c845f710fe
    Closes-Bug: #1636663

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/swift 2.21.0 release.