Configurable osd_memory_target

[Ponnuvel Palaniyappan]

Can we have an option to configure `osd_memory_target` via the charm?
This value defaults to 4G. But the ideal value depends on the host machine's available memory.

We can also consider calculating it automatically, similar to what ceph-ansible does [0]. There's been some upstream discussion [1] about doing it in cephadm. So we can do it in charms as well.

[0] https://github.com/ceph/ceph-ansible/blob/c878e99589bde0eecb8ac72a7ec8bc1f66403eeb/roles/ceph-config/templates/ceph.conf.j2#L103
[1] https://<email address hidden>/thread/Z77XO23JPXDNHKM7IG6UN4URYKA6L7VH/

[Alex Kavanagh (ajkavanagh)]

This is a nice idea. I'm guessing that a config parameter that also has an "auto" option for calculation would be idea, with the default of not being set staying at 4GB for backwards compatibility.

[Nobuto Murata (nobuto)]

100% agreed. The default 4GB is fine, but we definitely need a way to bump it particularly for dedicated storage nodes instead of hyper-converged.

As stated in the bug description, cephadm has a logic trying to use 70% of host memory with OSDs. We could have a similar option and suggest setting 70% (but with an empty value for backward compatibility).
https://docs.ceph.com/en/latest/cephadm/services/osd/#automatically-tuning-osd-memory
https://github.com/ceph/ceph/blob/08718ae283f6a62612c74b10d5da7efd2ea5f896/src/pybind/mgr/cephadm/serve.py#L172

[Samuel Allan (samuelallan)]

Submitted patch (work in progress): https://review.opendev.org/c/openstack/charm-ceph-osd/+/869619

[Samuel Allan (samuelallan)]

For the logic in the patch I've gone with:

- setting the value to an int in bytes -> templated out directly as is
- setting the value to "auto" -> automatically calculated based on the algorithm that ceph-ansible uses
- unset -> value not set in ceph.conf (ie. the default, backwards compatible)

Please let me know any feedback or alternate solutions required. :)

[Samuel Allan (samuelallan)]

Logic has been updated since; please see the patches for latest info:

- https://review.opendev.org/c/openstack/charm-ceph-osd/+/869619
- https://review.opendev.org/c/openstack/charm-ceph-mon/+/869896

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-ceph-mon/+/886254

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-mon/+/886254
Committed: https://opendev.org/openstack/charm-ceph-mon/commit/b31de1f027b32fa2ebe8ab62756819de466bbbf2
Submitter: "Zuul (22348)"
Branch: master

commit b31de1f027b32fa2ebe8ab62756819de466bbbf2
Author: Nobuto Murata <email address hidden>
Date: Fri Jun 16 21:21:39 2023 +0900

    Don't clear osd_memory_target unconditionally

    The charm can now set osd_memory_target, but it's not per device class
    or type by the nature of how the charm works. Resetting
    osd_memory_target always when osd_memory_target is not passed over the
    relation is a bit risky behavior since operators may have set
    osd_memory_target by hand with `ceph config` command out side of the
    charm. Let's be less disruptive on the charm upgrade.

    Closes-Bug: #1934143
    Change-Id: I34dd33e54193a9ebdbc9571d153aa6206c85a067

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-osd/+/869619
Committed: https://opendev.org/openstack/charm-ceph-osd/commit/ba6186e5de0ad82250599e90347a4c8cf121475e
Submitter: "Zuul (22348)"
Branch: master

commit ba6186e5de0ad82250599e90347a4c8cf121475e
Author: Samuel Walladge <email address hidden>
Date: Tue Jan 10 12:01:32 2023 +1030

    Add config option for tuning osd memory target

    Closes-Bug: #1934143

    Depends-On: https://review.opendev.org/c/openstack/charm-ceph-mon/+/869896

    Change-Id: I22dfc25c4ac2737f5d872ca2bdab3c533533dbff

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/quincy.2
Review: https://review.opendev.org/c/openstack/charm-ceph-osd/+/897725

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/quincy.2
Review: https://review.opendev.org/c/openstack/charm-ceph-mon/+/897731

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-mon/+/897731
Committed: https://opendev.org/openstack/charm-ceph-mon/commit/bacf46d3a568c70cd74d7359f9565123463b4ddb
Submitter: "Zuul (22348)"
Branch: stable/quincy.2

commit bacf46d3a568c70cd74d7359f9565123463b4ddb
Author: Nobuto Murata <email address hidden>
Date: Fri Jun 16 21:21:39 2023 +0900

    Don't clear osd_memory_target unconditionally

    The charm can now set osd_memory_target, but it's not per device class
    or type by the nature of how the charm works. Resetting
    osd_memory_target always when osd_memory_target is not passed over the
    relation is a bit risky behavior since operators may have set
    osd_memory_target by hand with `ceph config` command out side of the
    charm. Let's be less disruptive on the charm upgrade.

    Closes-Bug: #1934143
    Change-Id: I34dd33e54193a9ebdbc9571d153aa6206c85a067
    (cherry picked from commit b31de1f027b32fa2ebe8ab62756819de466bbbf2)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-osd/+/897725
Committed: https://opendev.org/openstack/charm-ceph-osd/commit/e852a99c33659b9eaf7fdb212d5bae647d267f50
Submitter: "Zuul (22348)"
Branch: stable/quincy.2

commit e852a99c33659b9eaf7fdb212d5bae647d267f50
Author: Samuel Walladge <email address hidden>
Date: Tue Jan 10 12:01:32 2023 +1030

    Add config option for tuning osd memory target

    Closes-Bug: #1934143

    Depends-On: https://review.opendev.org/c/openstack/charm-ceph-mon/+/897724

    Change-Id: I22dfc25c4ac2737f5d872ca2bdab3c533533dbff
    (cherry picked from commit ba6186e5de0ad82250599e90347a4c8cf121475e)