ceph-mgr balancer module support for "upmap"
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ceph Monitor Charm |
Fix Released
|
Wishlist
|
Luciano Lo Giudice |
Bug Description
[Impact]
OSD utilization variance can adversely impact cluster performance (hot-spotting individual OSDs) and will dictate the maximum usable capacity of the cluster. A cluster is as full as its fullest OSD!
CRUSH will place data in a normal distribution across OSDs within the cluster. The number of PGs per OSD heavily influences the variance. Achieving a tighter distribution is at odds with keeping ceph's overhead low. At the recommended 100 PGs per OSD, there can be outliers that differ from the mean by more than 20%.
This problem can be more pronounced with smaller clusters, and when individual hosts or AZs do not contain homogenous storage device configurations.
The upmap balancer module adds a CRUSH exception table that allows for a significant reduction in OSD variance. Using the upmap balancer improves performance and reduces the need for administrators to repeatedly reweight full/nearfull OSD outliers as the cluster fills.
[Test Case]
# Make sure the balancer module is enabled
ceph mgr module ls
ceph mgr module enable balancer
# Restrict all clients to Luminous+ (must support CRUSH upmap exceptions)
ceph osd set-require-
# Configure the percentage to rebalance each iteration
ceph config set mgr target_
# Set upmap mode, and enable automatic balancing
ceph balancer mode upmap
ceph balancer on
[Other Info]
The balancer module supports several undocumented configuration options [0] that may be useful for administrators who want rebalancing to only occur during off-peak hours. Example below:
ceph config set mgr mgr/balancer/
ceph config set mgr mgr/balancer/
[0] https:/
[1] https:/
[2] https:/
[3] "CRUSHing the OSD Variance Problem" https:/
summary: |
- ceph-mgr balancer module support + ceph-mgr balancer module support for "upmap" |
tags: | added: sts |
Changed in charm-ceph-mon: | |
assignee: | nobody → Luciano Lo Giudice (lmlogiudice) |
Changed in charm-ceph-mon: | |
status: | New → Triaged |
importance: | Undecided → Wishlist |
Changed in charm-ceph-mon: | |
milestone: | none → 21.10 |
Changed in charm-ceph-mon: | |
status: | Fix Committed → Fix Released |
Just a quick note that balancer scheduling options were added in Nautilus.