Objects stored on PUT even when etags don't match on object-servers

[Sam Morrison]

We run a multi region cluster (version 2.7.1 although have also replicated in test with version 2.15.1)

One of our regions has a firewall/anti-malware appliance thingy and we have found that every so often uploading objects would fail. The error would be that the etags didn't match on all returning object servers and a 500 is returned to the user. (we tracked this down to it changing lines of http header data in the objects themselves but that is another story.....)

Even though the object PUT returned a 500 the object is stored and is listed in the container etc.

When querying all object servers for the object however we get a different etag for the dodgy region:

root@swift:~# curl -g -I -XHEAD "http://172.22.59.46:6000/sdh/256733/AUTH_0bdf024c921848c4b74d9e69af9edf08/etag-test/test-file"
HTTP/1.1 200 OK
Content-Length: 10
X-Backend-Timestamp: 1516754468.33842
Content-Type: application/octet-stream
Last-Modified: Wed, 24 Jan 2018 00:41:09 GMT
Etag: "b2cfa4183267af678ea06c7407d4d6d8"
X-Timestamp: 1516754468.33842
X-Backend-Data-Timestamp: 1516754468.33842
X-Backend-Durable-Timestamp: 1516754468.33842
X-Object-Meta-Mtime: 1516754458.990904
Date: Wed, 24 Jan 2018 22:47:37 GMT

root@swift:~# curl -g -I -XHEAD "http://172.22.59.43:6000/sdf/256733/AUTH_0bdf024c921848c4b74d9e69af9edf08/etag-test/test-file"
HTTP/1.1 200 OK
Content-Length: 10
X-Backend-Timestamp: 1516754468.33842
Content-Type: application/octet-stream
Last-Modified: Wed, 24 Jan 2018 00:41:09 GMT
Etag: "9d240a40422483b268a24371f9e357cf"
X-Timestamp: 1516754468.33842
X-Backend-Data-Timestamp: 1516754468.33842
X-Backend-Durable-Timestamp: 1516754468.33842
X-Object-Meta-Mtime: 1516754458.990904
Date: Wed, 24 Jan 2018 22:47:37 GMT

root@swift:~# curl -g -I -XHEAD "http://172.22.59.45:6000/sdc/256733/AUTH_0bdf024c921848c4b74d9e69af9edf08/etag-test/test-file"
HTTP/1.1 200 OK
Content-Length: 10
X-Backend-Timestamp: 1516754468.33842
Content-Type: application/octet-stream
Last-Modified: Wed, 24 Jan 2018 00:41:09 GMT
Etag: "b2cfa4183267af678ea06c7407d4d6d8"
X-Timestamp: 1516754468.33842
X-Backend-Data-Timestamp: 1516754468.33842
X-Backend-Durable-Timestamp: 1516754468.33842
X-Object-Meta-Mtime: 1516754458.990904
Date: Wed, 24 Jan 2018 22:47:38 GMT

Also upon inspecting the container dbs I see that one of the dbs contains the bad etag, the rest show the good etag so there are inconsistencies there too.

Should some swift process be fixing these inconstancies up somehow? or possibly not allowing the data to be stored at all?

[clayg (clay-gerrard)]

For replicated objects this isn't currently easy to solve. For EC and encryption the MIME/chunked packing protocol supports the proxy sending metadata after the object body which the object-server should consider before the finalize.

Moving forward with https://review.openstack.org/#/c/427911/ may offer a framework by which we can have a common pattern between unencrypted replicated and other storage formats that allows us to always send a proxy computed etag behind a object PUT.

[John Dickinson (notmyname)]

The triage for now is to ensure that clients send an etag with the request. This will enable the object servers to validate the computed checksum against a known baseline and fail if it doesn't match.