Comment 1 for bug 1882247

This has passed both through OSCI and manual testing. Will continue try re-runs until it fails enough to introspect live. Agreed, seems a brittle test.

The test as written writes an object to region1 and then once that is acked, immediately grabs the object header from region2 to determine number of copies. Unanswered questions are:

1. How does write_affinity work when noting both zones in the configuration?
2. Does each region report back how many copies are only local to the region, or respond with how many copies there are across the two regions?
3. I believe there is a mechanism within swift that a third copy can be made asynchronously if there is immediate denial of service from the third-copy object server which may be the brittleness that we're experiencing. This should be investigated and handled as an allowable exception in the testing code if true.

That we set write_affinity to include both zones seems inconsistent with the advice of the Global Cluster documentation:

https://docs.openstack.org/swift/pike/overview_global_cluster.html#write-affinity

I'm wondering if using both zones in write_affinity is creating a situation in swift where writes to all 6 zones are not happening at once, and there's a race between requesting the header of the object and the async replication. There should be 3 copies of the data across the two zones immediately, but all 6 copies are not necessarily synchronous with our current config.

I'll see if I can keep re-running to get this to fail for further introspection, but this issue appears in-line with the eventual-consistency model of Swift that writes can be acked after only one write has been completed. If we want to cover the assurance of the third async copy better in testing, we could check both region's swift-storage servers for oldest replication completion time (swift-recon -r from each region's proxy), and when that time is less than the time since object write, then we can test for number of replicas.