proxy handling of fast-POST meta is inconsistent with replication

[clayg]

Replication seems to be ok with a lonely .meta, or even a .meta sitting over a .ts - assuming you managed to get a .meta written down over a .data before a .ts reaped it

However the object-server will NOT write down a .meta from a POST request if it doesn't have a .data file [1]

This means if you have multiple replicas rebalancing and the proxy's POST responses look like [202, 404, 404] it returns 404 even though the first primary did successfully create the .meta and replication will make it fully durable.

When the object-server hits DoesNotExist and refuses to write down the .meta we introduce a durability gap. When the proxy recieves a single 202, but returns 404 (even tho no node reported a tombstone timestamp) we're not being as accurate as we could be given the information at hand.

I think replication is doing the right thing - the proxy and object-server handling should be more consistent with replication.

1. maybe because it wouldn't have enough info to make the container update, but we could jin up an infinately old placeholder if needed

[clayg (clay-gerrard)]

https://review.opendev.org/c/openstack/swift/+/798022