container-reconciler needs to scale better

[Tim Burke]

Currently, the reconciler tries to process every misplaced object on every pass. This is a pain: you either put it on a single node, in which case you've got a single point of failure; or multiple nodes, in which case you get a ton of duplicated work.

We should slice up the work like we do for the expirer.

[Tim Burke (1-tim-z)]

Clay had a start on this: https://review.opendev.org/#/c/103779/

[Tim Burke (1-tim-z)]

Note that the "ton of duplicate work" approach can end horribly if you've got encryption enabled. We've seen three separate reconcilers try to move an object to an 8+4 EC policy at approximately the same time, leading to

* 7 frags stored with key A,
* 3 frags stored with key B, and
* 1 frag stored with key C

all with the same timestamp -- so now data can't be reconstructed.

[Matthew Oliver (matt-0)]

The reconsiler should do a put with a newer timestamp, well using the source but with the timestamp offset incremented. If it's encrytion causing this issue then does that mean it's not honouring timestamp offsets?

[Matthew Oliver (matt-0)]

Oh and what is it reconciling here. From from 1 EC container to a slightly different EC policy?

[Matthew Oliver (matt-0)]

oh this is discussed here: https://bugs.launchpad.net/swift/+bug/1910804