s3api claims to delete an object when it hasn't actually done it

[Tim Burke]

Funny story time:

An S3 client creates a whole bunch (~12k) of objects. This puts load on disks, to the point that some start getting error limited. A handful of objects get pretty unlucky: not only are 2/3 primaries error-limited, but so are the first three handoffs! That's OK, Swift can manage -- since they were error limited, we just skip over them and try to PUT the data on that last primary and the fourth and fifth handoffs. PUT to the primary fails (something about BadStatusLine -- coulda been a timeout or a restart or something) but the two deep handoffs succeed so we can reply 201.

A little while later (long enough that error-limiting's worn off, but not so long that replication's had a chance to do its job), the client comes through and deletes everything. The way s3api does that is to first do a HEAD to determine whether it's a multipart upload or not, then do a DELETE either with or without a ?multipart-manifest=delete query param. So, we go do our HEAD -- all of the primaries respond 404, as do the first three handoffs. s3api gets back a 404, skips the delete, and calls it success (because AWS only 404s a DELETE if the *bucket* doesn't exist).

"Well *there's* your bug," you say. "We should have still issued the DELETE!" However:

- We just did a whole bunch of work to look for the object -- do we
  really want to go to disk *again* to write down something everyone
  already seems to know?
- That HEAD was important -- if the object was a multi-part upload,
  all of the segments are in a +segments container where *the client
  can't see them*. If we issue the DELETE, the object would get
  cleared out of listings and requests for it would eventually 404,
  but the user's account would still be using up space (and getting
  billed for it) while we actively prevent their client from telling
  them why.

[Tim Burke (1-tim-z)]

One more argument in favor of issuing the delete: container listings may never have gotten the message about object deletes. This may lead to containers you can't easily delete via the S3 API (if, say, your s3 client is opinionated and only wants to issue multi-deletes). May be the root cause for https://ask.openstack.org/en/question/114715/s3-method-post-on-bucket-with-query-delete-doesnt-work/

[Bhaskar Singhal (bhaskarsinghal)]

Tim, with reference to the ask.openstack query, the MultiPart object never received the delete, hence the container was not updated.

root@VM:~# s3cmd del --recursive --force --region us-east-1 s3://bucket9
delete: 's3://bucket9/2M_file.file'
delete: 's3://bucket9/firmimg.d7'
root@VM:~# s3cmd ls --region us-east-1 s3://bucket9
2019-01-03 08:21 105639142 s3://bucket9/firmimg.d7

Proxy-server logs:
root@VM:~# grep bucket9 /var/log/swift/proxy.log
Jan 4 05:43:40.891106 hostname proxy-server: {'access_key': 'xxxxxxxxxxxxxxxxxxxxxxxf1717b67f', 'container_name': 'bucket9', '_timestamp': 1546580620.00000_0000000000000000, 'user_id': None, 'slo_enabled': True, '_signed_headers': set(['x-amz-content-sha256', 'host', 'x-amz-date']), 'string_to_sign': 'AWS4-HMAC-SHA256\n20190104T054340Z\n20190104/us-east-1/s3/aws4_request\n324616a0858b1da3e27453fa34401678f3c58e131dfbff23b99678bd493c3d0d', '_params_cache': {}, 'headers': <swift.common.swob.HeaderEnvironProxy object at 0x7f4e3fc97d50>, 'object_name': '', 'bucket_in_host': None, 'environ': {'HTTP_AUTHORIZATION': 'AWS xxxxxxxxxxxxxxxxxxxxxxxf1717b67f:c83ebdfe2e77e1a2c199611359e89a5309196b29c987ddd13dfe046f83339fa1', 'SCRIPT_NAME': '', 'keystone.token_auth': <keystonemiddleware.auth_token._user_plugin.UserAuthPlugin object at 0x7f4e3fb585d0>, 'webob.adhoc_attrs': {'response': <_AuthTokenResponse at 0x7f4e3fafcf50 200 OK>}, 'REQUEST_METHOD': 'GET', 'HTTP_X_AMZ_DATE': '20190104T054340Z', 'PATH_INFO': '/bucket9/', 'SERVER_PROTOCOL': 'HTTP/1.0', 'REMOTE_ADDR': '150.50.2.135', 'CONTENT_LENGTH': '0', 'eventlet.posthooks': [], 'RAW_PATH_INFO': '/bucket9/', 'REMOTE_PORT': '49285', 'eventlet.input': <eventlet.wsgi.Input object at 0x7f4e3fc53fd0>, 'HTTP_X_IDENTITY_STATUS': 'Invalid', 'wsgi.url_scheme': 'http', 'SERVER_PORT': '80', 'HTTP_X_AMZ_CONTENT_SHA256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'swift.leave_relative_location': True, 'wsgi.input': <eventlet.wsgi.Input object at 0x7f4e3fc53fd0>, 'HTTP_HOST': 'vsa-00000ae2-zadara-dev2.zadarazios.com', 'swift.cache': <swift.common.memcached.MemcacheRing object at 0x7f4e3fc6bf10>, 'wsgi.multithread': True, 'wsgi.version': (1, 0), 'SERVER_NAME': '150.50.2.135', 'GATEWAY_INTERFACE': 'CGI/1.1', 'wsgi.run_once': False, 'wsgi.errors': <swift.common.utils.LoggerFileObject object at 0x7f4e3fbf9fd0>, 'wsgi.multiprocess': False, 'swift.trans_id': 'tx79aefa29e485496fa408b-005c2ef28c', 'CONTENT_TYPE': None, 'swift3.auth_details': {'access_key': 'xxxxxxxxxxxxxxxxxxxxxxxf1717b67f', 'check_signature': <bound method SigV4Request.check_signature of <swift3.request.SigV4Request object at 0x7f4e3fc97090>>, 'string_to_sign': 'AWS4-HMAC-SHA256\n20190104T054340Z\n20190104/us-east-1/s3/aws4_request\n324616a0858b1da3e27453fa34401678f3c58e131dfbff23b99678bd493c3d0d', 'signature': 'c83ebdfe2e77e1a2c199611359e89a5309196b29c987ddd13dfe046f83339fa1'}, 'HTTP_ACCEPT_ENCODING': 'identity'}, 'token': None, 'signature': 'c83ebdfe2e77e1a2c199611359e89a5309196b29c987ddd13dfe046f83339fa1', 'account': None}
Jan 4 05:43:41.105368 hostname proxy-server: - - 04/Jan/2019/05/43/41 HEAD /v1/AUTH_e99dfde4680f4b2f8a270f66ab3d2cf1/bucket9 HTTP/1.0 204 - Swift - - - - tx79ae...

[Tim Burke (1-tim-z)]

Thanks for the extra info Bhaskar! I wrote up https://bugs.launchpad.net/swift/+bug/1810567 to describe your trouble, and https://review.openstack.org/#/c/628701/ to fix it.

[Bhaskar Singhal (bhaskarsinghal)]

Thanks a lot, Tim!