Config change doesn't restart ceph-mon service

[Facundo Ciccioli]

A production ceph cluster was in HEALTH_WARN state showing a mon low on available space. Both warn and crit thresholds were at their defaults, 30 and 5 resp.

We appled a config change via juju:

        juju config ceph-mon monitor-data-available-warning=25
        juju config ceph-mon monitor-data-available-critical=20

The ceph.conf file reflected the changes but we noticed that ceph was still reporting HEALTH_WARN.

Looking at ceph-mon@*.service we noticed that it hadn't been restarted. After roll-restarting all three mons, the cluster went back to HEALTH_OK.

Shouldn't the ceph-mon services be restarted after ceph.conf is modified?

[Chris MacNaughton (chris.macnaughton)]

Give that we render the ceph.conf in one place [1], it's easy to confirm that we don't restart the mons. That said, we'd have to be _very_ careful in restarting the mons due to a config=changed as we have to ensure that we maintain quorum during a rolling restart. There is some code in the charm, today, to handle a rolling upgrade & restart of the charm that could be leveraged to do this.

1: https://github.com/openstack/charm-ceph-mon/blob/d3b2494ee8a23fe58cff4eb3308306a24a8f1434/hooks/ceph_hooks.py#L225

[Facundo Ciccioli (fandanbango)]

If restarts are a delicate matter, maybe for certain configuration keys (ie, known ones like the ones involved in this bug and not arbitrary ones like whatever you may specify on config-flags) they could be injected live without restarting. I believe ceph allows such a thing to be done, though we could be introducing further complexity.

[Nobuto Murata (nobuto)]

I'm speaking only for myself, but
`ceph config assimilate-conf -i /etc/ceph/ceph.conf`
could be executed whenever ceph.conf is changed. So any appropriate config can be reflected on the fly.

https://docs.ceph.com/en/latest/cephadm/adoption/?highlight=assimilate-conf#adoption-process
https://docs.ceph.com/en/latest/rados/configuration/ceph-conf/?highlight=assimilate-conf#commands

[Nobuto Murata (nobuto)]

Hmm, never mind. It looks like assimilate-conf is not idempotent but only effective once for the initial migration to mon config database. After that, explicit `ceph config set mon <key> <value>` seems necessary.