We use separate drives for account and container servers, one per server. When 2 drives crashed we noticed that used space on all rest account and container DB storages increased 2 times.
Space was consumed slowly - every replication cycle was consumed 1/server_count of space. There was increase from 15% to 30% used space where trends stopped and then increased to 45% when 2 drives were failed. When drive was replaced after few hours, everything returned to normal state.
------------------------------------------
How to reproduce with SAIO:
* configure 2 replica with 4 servers or increase container servers count to have drives count >= replicas + 2,
* create some containers
for c in {1..50} ; do swift -A http://127.0.0.1:8080/auth/v1.0 -U test:tester -K testing post cont$c ; done
* configure one (or more) container servers to see device as unmounted: set mount_check = true in /etc/swift/container-server/2.conf (check_mount fails on directory that is not mounted device) and restart container server(s),
* replicate, replicate - every replicator cycle replicates databases of unmounted device into next servers
for n in {1..2} ; do for i in 1 3 4 ; do swift-container-replicator /etc/swift/container-server/$i.conf -o ; done ; done
What to expect with 4 devices, 2 replica and 1 "unmounted" drive: databases from unmounted drive is replicated on 2 more devices (1 is unmounted and 1 is primary location, 2 are handoff) but it should be only 1 handoff.
With 8 devices you should have 6 more replicas (8 = 1 unmounted, 1 is primary location, 6 handoffs).
Issue is in review: https:/