Comment 7 for bug 1464022

Indeed the intention of container mirroring is mirroring between regions.

A multi-site storage policy has been considered, and was overruled due to isolation requirements coming from a
concern that problems happening in one region affect other regions when there is a shared ring between them.

Two other concerns have to do with policy management and resource utilization:
Suppose that to reduce the isolation problem mentioned above, there is a policy for each two sites (or for each 3 sites if you want 3X 'multi-region replication'). This leads to O(N^2) or O(M^m) policies for m replicas over N sites. Having so many policies, or groups of policies also brings the question of resource utilization. With one ring resources usage is pretty much evenly distributed, the more rings we have to work harder to globally share the resources.

Other then that, I do believe that replicating the metadata is a natural extension to the existing container sync generic feature.

The scale problem is a concern we also share. While we try to measure the 'limits' of container sync, we can think of short term and long term solutions for this:
1. Short term solution - Add more md servers to be able to deal with larger sync BW.
2. Long term solution, which you pretty much suggest above - Rely on the notification mechanism to 'decouple' the container db from the container 'sync' node so as to distribute the work amongst more nodes. BTW From a more theoretical P.O.V. I wonder if this can be done today using a 'remote' broker.