empty shards fail audit with reclaimed root db

[clayg]

When a sharded database is empty, it can be deleted - but the empty shards do not appear to be marked as deleted. When a non-deleted shard is unable to find (404) the root db, it logs a warning. Even if the shard is empty; once the deleted root db is reclaimed it creates a storm of log warnings and won't ever fix itself.

To work around manually you can issue DELETE requests to the shard containers in the .shard_AUTHXXX account

[clayg (clay-gerrard)]

having talked it through with Al & Tim - they think it might be reasonable for the sharder, during audit to detect the root db has reclaimed (404 w/o a backend-timestamp) and then mark itself as deleted

[Matthew Oliver (matt-0)]

in an ideal world, aka a world of auto-sharding I'd hope the empty shard containers would shrink and callapse back into the root container before reclaim_age. But yeha we need a backup to make sure they don't stick around.

[clayg (clay-gerrard)]

https://review.opendev.org/c/openstack/swift/+/771086