Check container existence before creating one in s3api

[Yuxin Wang]

When initiating multiupload in s3api, we should check container existence before creating one.

A container PUT request will create a container db or update the existing one if the container's already there. When the disk that the container db landed is under heavy load, such container PUT will fail due to LockTimeout. It's worse if you are initiating lots of multiuploads (which was one of our clients' use cases). For avoiding this, we'd better first check its existence using HEAD, then PUT.

Code in:
https://github.com/openstack/swift/blob/2.18.0/swift/common/middleware/s3api/controllers/multi_upload.py#L363-L366

        try:
            req.get_response(self.app, 'PUT', container, '')
        except BucketAlreadyExists:
            pass

Error logs:

Jun 8 10:20:49 master proxy-server: ERROR with Container server 10.10.130.2:6001/hdd003 re: Trying to PUT /AUTH_ec9c9d94cf834818bf1a9d712ca30914/testbucket+segments: Timeout (10.0s) (txn: tx2156f533b1bc4a7095f3b-005b19e7f7) (client_ip: 10.10.130.13)
Jun 8 10:20:49 master container-server: ERROR __call__ error with PUT /hdd003/1333/AUTH_ec9c9d94cf834818bf1a9d712ca30914/testbucket%2Bsegments : LockTimeout (10s) /srv/node/hdd003/containers/1333/fea/535fb93d4f130a3ea3646ff1b76fdfea/.lock (txn: tx2156f533b1bc4a7095f3b-005b19e7f7)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/580333

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/580333
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=e02b9b798007b9f8c70e33c22fa6189d81a41ca3
Submitter: Zuul
Branch: master

commit e02b9b798007b9f8c70e33c22fa6189d81a41ca3
Author: Yuxin Wang <email address hidden>
Date: Thu Jul 5 17:46:50 2018 +0800

    Use get_container_info to check existence before container PUT

    PUT request on an existing container will trigger an update on
    container db. When disks where container db landed are under
    heavy loads, update on the container db may fail due to LockTimout.

    Hence, we first check existence, if it's not there, we PUT.

    Change-Id: Ic61153948e35f1c09b05bfc97dfac3fb487b0898
    Closes-Bug: 1780204

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/668007
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=aca7474ca3c7bba54473fca35b6d88a3a1efdca4
Submitter: Zuul
Branch: feature/losf

commit 9d1b7497400e8c3a7159b11dd8455c55a31db985
Author: Tim Burke <email address hidden>
Date: Tue Mar 26 13:02:24 2019 -0700

    py3: port staticweb and domain_remap func tests

    Drive-by: Tighten domain_remap assertions on listings, which required
    that we fix proxy pipeline placement. Add a note about it to the sample
    config.

    Change-Id: I41835148051294088a2c0fb4ed4e7a7b61273e5f

commit 38a24571ad5c192bacaf60a5634ea66164dbbb71
Author: Tim Burke <email address hidden>
Date: Wed Jul 10 09:13:44 2019 -0700

    functests: make container creation less flakey in test_object

    Change-Id: If62d82beb202dea553776920a95c177518b162ab

commit 4c4bd778ea8fe8d02a2892524c7918da0ca25ea9
Author: Tim Burke <email address hidden>
Date: Wed Jul 3 09:52:41 2019 -0700

    container-replicator: Add a timeout for get_shard_ranges

    Previously this had no timeout, which meant that the replicator might
    hang and fail to make progress indefinitely while trying to receive
    shard ranges.

    While we're at it, only call get_shard_ranges when the remote indicates
    that it has shard ranges for us to sync -- this reduces the number of
    requests necessary to bring unsharded replicas in sync.

    Change-Id: I32f51f42d76db38271442a261600089404a00f91
    Closes-Bug: #1835260

commit 345f577ff191ee01c3cb4626805338028815c2b4
Author: Tim Burke <email address hidden>
Date: Wed Jul 3 07:28:14 2019 -0700

    s3token: fix conf option name

    Related-Change: Ica740c28b47aa3f3b38dbfed4a7f5662ec46c2c4
    Change-Id: I71f411a2e99fa8259b86f11ed29d1b816ff469cb

commit 76fde89261e1940daadb720c41df1a3595314a97
Author: Tim Burke <email address hidden>
Date: Mon Jun 17 09:25:52 2019 -0700

    py3: Be able to read and write non-ASCII headers

    Apparently Python's stdlib got more picky about what a header should
    look like. As a result, if an account, container, or object had a
    non-ASCII metadata name (values were fine), the proxy-server wouldn't
    parse all of the headers. See https://bugs.python.org/issue37093 for
    more information.

    This presented several problems:
    - Since the non-ASCII header aborts parsing, we may lose important
      HTTP-level information like Content-Length or Transfer-Encoding.
    - Since the offending header wouldn't get parsed, the client wouldn't
      even know what the problem was.
    - Even if the client knew what the bad header was, it would have no way
      to clear it, as the server uses the same logic to parse incoming
      requests.

    So, hack in our own header parsing if we detect that parsing was
    aborted. Note that we also have to mangle bufferedhttp's putheader so we
    can get non-ASCII headers to the backend servers.

    Now, we can run the test_unicode_metadata tests in
    test/functional/test_account.py and test/functional/test_container.py
    under py2 against services running under py3.

    Change-Id: I0f03c211f35a9a49e047a571...

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/swift 2.22.0 release.