mon_pg_warn_max_object_skew setting not exposed via charm

[Drew Freiberger]

In newly built openstack environments with ceph backed glance/cinder/nova with charm default %util settings, after relating glance-simplestreams-sync to glance and importing standard ubuntu images, the skew of the %objects in the glance pool trips ceph HEALTH_WARN state.

We find that this issue happens until there is some level of adoption of the cloud by end-users, and we manually tune the setting mon_pg_warn_max_object_skew on each of the ceph mon.* objects to alleviate the HEALTH_WARN state at cloud initialization.

It would be extremely helpful to have mon_pg_warn_max_object_skew to be exposed via the charm to allow for ease of managing this setting, or potentially to have a time-based change of this setting automated in the charm to allow for the first X days of cloud operation for this setting to be quite high until greater adoption and import of data into the ceph environment.

Current resolution process is to login to the unit and run:

ceph tell mon.* injectargs '--mon_pg_warn_max_object_skew 20'

[Drew Freiberger (afreiberger)]

Was thinking about creating an action for this, but the action wouldn't persist the setting if we extended to another unit.

[James Hilling (jamesh5979)]

It would be useful if we could also have the following exposed to the charm:

mon data avail warn = X
mon data avail crit = Y

This becomes especially important, when ceph starts to alert on disk space, at a different set of values to whatever your monitoring system is set to alert at.

For example: when using the nagios, in conjunction to deploying the nrpe subordinate charm to a ceph-mon unit, the deployed nrpe unit will have a set of alerting thresholds for the 'check_disk_root' check. These will be different to the values that ceph will alert at usually.

We want to be able to tune ceph to alert at a value that will not interfere with monitoring.

[Edward Hope-Morley (hopem)]

Hi James/Drew, couple of points here;

The ceph charms have a config-flags option that allows you to set any config option within the [global], [mon], [osd] or [rgw] sections. It was added so that people could try out different configs then suggest them for native inclusion in the charm if appropriate. I think the scenario you mention here is valid since the warning is displayed despite a perfectly healthy cloud and ceph. I suggest that you leverage the config-flags opt to apply the config until this feature lands.

[Edward Hope-Morley (hopem)]

One more thing, it appears that setting mon_pg_warn_max_object_skew to 0 will disable the check [1].

[1] https://github.com/ceph/ceph/blob/master/src/mon/PGMap.cc#L2631

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/615600
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-mon/commit/?id=7a362ff0a5b355f55c430e867ebbf4f1303faf2a
Submitter: Zuul
Branch: master

commit 7a362ff0a5b355f55c430e867ebbf4f1303faf2a
Author: Shane Peters <email address hidden>
Date: Mon Nov 5 11:45:32 2018 -0500

    Add disable-pg-max-object-skew option

    Openstack clouds that use ceph will typically start their life with
    at least one pool (glance) loaded with a disproportionately high
    amount of data/objects where other pools may remain empty. This can
    trigger a HEALTH_WARN if mon_pg_warn_max_object_skew is exceeded
    but that is actually a false positive.

    Change-Id: I5a535dbb17db2149630d971d85ac311f14298b09
    Closes-Bug: 1720374

[Chris MacNaughton (chris.macnaughton)]

Marking the charm-ceph task wontfix as the ceph charm has been removed from support for a while now