object-replicator using rsync needs a post-rsync command that is cheaper

[clayg]

To summarize ongoing work on rebalance improvements [1] REPLICATE requests have a very course API, they are ostensibly designed to speed up replication, but are known to cause a significant amount of IO contention and can be slow UNDER SOME CIRCUMSTANCES.

One such circumstance is when using rsync replication under heavy IO contention

If we implement lp bug #1820893 we're going to have a problem if lots of post-rsync REPLICATE requests get rejected for concurrency [1]. But as described in lp bug #1818709 we need to fire the post-rsync REPLICATE request to invalidate suffixes at a MINIMUM.

Luckily suffix invalidation is cheap (just append to .invalid file) so we should be able to make sure that request is made.

1. It would be nice if we could move forward with https://review.openstack.org/#/c/578213/ to reduce contention from primary REPLICATE calls with tuning.

[Tim Burke (1-tim-z)]

Done in https://github.com/openstack/swift/commit/5eaf1548 -- now, we just invalidate suffixes after rsync and delay rehashing until it's actually needed.