Post-deploy charm support for failure domains

[Peter Sabaini]

The ceph-{mon,osd} charms support distributing replicas across AZ via the custom-failure-domain flag. When turned on, this will update the crush map to ensure distribution across Juju AZs by:

a) Setting crush location on osds' /etc/ceph/ceph.conf to rack=AZ

b) Editing crush rules to essentiall do a "sed 's/step chooseleaf firstn 0 type host/step chooseleaf firstn 0 type rack/'"

However this will, afaict, only work on install time. When set post-deploy, it

a) won't move existing OSDs into the rack bucket in the crushmap.

b) the crush rule editing ^^ is only performed in the mon-relation-*, ie. at the time the MON cluster is updated. This might actually be mildly dangerous to run post-deploy if there are existing pg w/o OSDs in the rack bucket, as they won't have anywhere to move to

It would be great to have support to have charm support for distributing replicas across Juju AZs post deploy, either by config or maybe an action