ceph-mgr balancer module support for "upmap"

[Dan Hill]

[Impact]
OSD utilization variance can adversely impact cluster performance (hot-spotting individual OSDs) and will dictate the maximum usable capacity of the cluster. A cluster is as full as its fullest OSD!

CRUSH will place data in a normal distribution across OSDs within the cluster. The number of PGs per OSD heavily influences the variance. Achieving a tighter distribution is at odds with keeping ceph's overhead low. At the recommended 100 PGs per OSD, there can be outliers that differ from the mean by more than 20%.

This problem can be more pronounced with smaller clusters, and when individual hosts or AZs do not contain homogenous storage device configurations.

The upmap balancer module adds a CRUSH exception table that allows for a significant reduction in OSD variance. Using the upmap balancer improves performance and reduces the need for administrators to repeatedly reweight full/nearfull OSD outliers as the cluster fills.

[Test Case]
# Make sure the balancer module is enabled
ceph mgr module ls
ceph mgr module enable balancer

# Restrict all clients to Luminous+ (must support CRUSH upmap exceptions)
ceph osd set-require-min-compat-client luminous

# Configure the percentage to rebalance each iteration
ceph config set mgr target_max_misplaced_ratio .07

# Set upmap mode, and enable automatic balancing
ceph balancer mode upmap
ceph balancer on

[Other Info]
The balancer module supports several undocumented configuration options [0] that may be useful for administrators who want rebalancing to only occur during off-peak hours. Example below:
ceph config set mgr mgr/balancer/begin_time 1800
ceph config set mgr mgr/balancer/end_time 0500

[0] https://github.com/ceph/ceph/blob/cc96a4cb0cc65ee0d761656af3689c5740c2c4a9/src/pybind/mgr/balancer/module.py#L201
[1] https://docs.ceph.com/docs/master/rados/operations/balancer/
[2] https://docs.ceph.com/docs/master/rados/operations/upmap/
[3] "CRUSHing the OSD Variance Problem" https://www.youtube.com/watch?v=niFNZN5EKvE

[Dan Hill (hillpd)]

Just a quick note that balancer scheduling options were added in Nautilus.

[Dan Hill (hillpd)]

Note: CRUSH buckets must be straw2 to enable upmap. There needs to be a sanity check to ensure there are no old hammer-era straw buckets that have yet to be converted.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-ceph-mon/+/807943

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-mon/+/807943
Committed: https://opendev.org/openstack/charm-ceph-mon/commit/691605e6fc6d61b22b18c9e43df424d822dc93e9
Submitter: "Zuul (22348)"
Branch: master

commit 691605e6fc6d61b22b18c9e43df424d822dc93e9
Author: Luciano Lo Giudice <email address hidden>
Date: Wed Sep 8 17:16:36 2021 -0300

    Add balancer module support for 'upmap'

    This allows the user to change the configuration parameter
    'balancer-mode' via Juju in order to set the balancer mode for Ceph.

    Change-Id: I60dbd5f163e0c9d004275eff65db7ada41ad2660
    Closes-Bug: #1888914