Comment 23 for bug 1700585

In response to comment #17:

Do you have any dispersion populate/report data you can query [1]?

# swift-dispersion-populate
Created 10 containers for dispersion reporting, 1s, 0 retries
Created 10 objects for dispersion reporting, 2s, 0 retries

# swift-dispersion-report
Queried 11 containers for dispersion reporting, 1s, 0 retries
100.00% of container copies found (33 of 33)
Sample represents 1.07% of the container partition space
Queried 10 objects for dispersion reporting, 2s, 0 retries
There were 10 partitions missing 0 copy.
100.00% of object copies found (30 of 30)
Sample represents 0.98% of the object partition space

Can you describe your device failure/replacement procedure [2]?
For a drive being replaced immediately with drive of same size:
We have only had 1 failed drive so far, and the following procedure was used:
1. make sure it's unmounted
2. remove from ring and rebalance
3. add new drive, format, mount in same place, and change permissions
4. add to ring, and rebalance

For quarantine info:
[root@swift-r1p1 ~]# swift-recon --all | grep quarantined
[ALL_quarantined_last_path] low: 0, high: 0, avg: 0.0, total: 0, Failed: 0.0%, no_result: 0, reported: 3
[ZBF_quarantined_last_path] low: 0, high: 0, avg: 0.0, total: 0, Failed: 0.0%, no_result: 0, reported: 3
[quarantined_objects] low: 0, high: 0, avg: 0.0, total: 0, Failed: 0.0%, no_result: 0, reported: 3
[quarantined_accounts] low: 0, high: 0, avg: 0.0, total: 0, Failed: 0.0%, no_result: 0, reported: 3
[quarantined_containers] low: 0, high: 0, avg: 0.0, total: 0, Failed: 0.0%, no_result: 0, reported: 3

For this one: Can you turn your proxies request_node_count [3] >= 84 (3 nodes * 28 devices) and check again for the 404'd objects.

What exactly should that line look like? If we have 85 devices, should it read:
request_node_count = 85

We have not changed that yet, because we don't want to pile on a misconfiguration if we get it wrong.