Allow "mds cache memory limit" tuning

[Dan Hill]

[Impact]
In the luminous release, ceph added support for modifying the metadata server cache memory limits [0].

On larger deployments, adjusting this setting upwards from the 1GB default is necessary to meet ceph fs client needs.

By default, ceph reserves 5% of the cache limit for the creation of new metadata. Make sure to account for this reserved amount when calculating suitable cache sizings.

[Example]
# Query the active configuration on the mds host
sudo ceph daemon mds.$HOSTNAME config show | grep memory_limit

# Add the following lines to ceph.conf persist a new 8GB cache limit
[mds]
mds cache memory limit = 8589934592

# Run the following command to apply the new run-time setting
sudo ceph daemon mds.$HOSTNAME config set

[Other Info]
The "mds cache memory limit" setting is configurable at run-time [1]. Apply through `ceph config` (Mimic+) or through `ceph tell` or `ceph daemon` (Luminous+). A service restart is not necessary to pick up the change.

[0] https://ceph.io/community/new-luminous-cephfs-metadata-server-memory-limits/
[1] https://github.com/ceph/ceph/blob/028eb3fdfb2346b3ed1ebfc3c3198ec99e21154c/src/mds/MDCache.cc#L241

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-ceph-fs/+/842555

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-fs/+/842555
Committed: https://opendev.org/openstack/charm-ceph-fs/commit/8f31a33a0fd032b079f76abb20f68c4e71e3f50c
Submitter: "Zuul (22348)"
Branch: master

commit 8f31a33a0fd032b079f76abb20f68c4e71e3f50c
Author: Ethan Myers <email address hidden>
Date: Thu May 19 09:44:34 2022 -0600

    Enable charm to configure mds cache options.

    Closes-Bug: #1891409
    Func-Test-PR: https://github.com/openstack-charmers/zaza-openstack-tests/pull/774
    Change-Id: If2bdd5c0f2afa1843e686cf69570a50901c85875