Allow scrub/deep-scrub interval configuration

[Chris Sanders]

With Bluestore data now has CRC checksums and the aggressive deep-scrubbing that Ceph does is not as important as it once was. Combined with deep-scrubs having a high impact to a cluster some configurations need to be added to tune them.

I've been setting the following configuration values via the config-flags on ceph-osd:
osd deep scrub interval = 2592000
osd scrub sleep = 0.5
osd scrub begin hour = 7
osd scrub end hour = 14

This lets me set scrubs to happen less frequently (monthly instead of weekly), not be as impactful to clients during a scrub, and restrict scrubbing to off-hours. You may want to review other scrub settings as well but this seems like the minimum that should be provided.

Starting in Nautilus, this becomes even more important because using config-flags is no longer sufficient any longer. With the introducing of the Mgr daemon configuration isn't just on the osd any more. The Mgr now needs to be informed either under the [global] or [mgr] sections in addition to setting the [osd] settings. The charm config options only allow setting [osd] and [mon] sections.

See [0] and [1] for examples of how people misconfigure this and produce invalid ceph warnings.

[0]: https://tracker.ceph.com/issues/44959
[1]: https://tracker.ceph.com/issues/41054

[Diko Parvanov (dparv)]

subscribing field-high as this is impacting a customer where this needs to be configured to run with a lower interval to not impact performance

[Diko Parvanov (dparv)]

Unsubscribing field-high, as permanent workaround can be done via:

ceph config set osd osd_scrub_min_interval X
ceph config set osd osd_scrub_max_interval Y
ceph config set osd osd_deep_scrub_interval Z

ceph config set mgr osd_scrub_min_interval X
ceph config set mgr osd_scrub_max_interval Y
ceph config set mgr osd_deep_scrub_interval Z

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-ceph-osd/+/844111