configuration database support

[Dan Hill]

[Impact]
In the mimic release, ceph introduced a new configuration database feature [0] with a simply stated goal: "avoid the need for external tools to manage ceph.conf configuration file."

On an initial deployment, nothing has changed. The ceph.conf file is parsed and all values are read into the configuration database.

The core issue arises with day-2 operations. Any value that already exists in the configuration database will no longer receive updates from ceph.conf. This breaks a number of settings in the ceph-* charms that rely on ceph.conf to propagate changes.

[Test Case]
# change loglevel to a new value
juju config ceph-mon loglevel 20

# verify ceph.conf values were updated
juju run --application ceph-mon 'sudo cat /etc/ceph/ceph.conf'

# dump config
juju run --application ceph-mon 'sudo ceph config dump'

# restart ceph-mon service to pick up ceph.conf changes
juju run --application ceph-mon 'sudo systemctl restart ceph-mon'

# dump config
juju run --application ceph-mon 'sudo ceph config dump'

The debug log levels have not changed. The log levels were set in the config db during deployment, so any further changes are not picked up from ceph.conf.

[Other Info]
The upstream expectation is for configuration changes to flow through the new ceph config CLI. They provide an 'assimilate-config' command to migrate to the new minimal ceph.conf that just includes keys and mon ips. All that is required to connect to the mon config database.

This is an issue with all ceph releases since mimic.

[0] https://ceph.io/community/new-mimic-centralized-configuration-management/

[David Ames (thedac)]

TRIAGE:

As Liam pointed out we actually already have the infrastructure to execute ceph config set [0][1][2][3].

We will need to add all ceph-mon charm configuration parameters to the send_osd_settings [2].

[0] https://github.com/openstack/charm-ceph-mon/blob/master/hooks/ceph_hooks.py#L688
[1] https://github.com/openstack/charms.ceph/blob/master/charms_ceph/utils.py#L3303
[2] https://github.com/juju/charm-helpers/blob/master/charmhelpers/contrib/storage/linux/ceph.py#L165
[3] https://github.com/openstack/charm-ceph-osd/blob/master/hooks/ceph_hooks.py#L666

[Dmitrii Shcherbakov (dmitriis)]

The commands that are already supported by the ceph charms are specific to a local daemon. As such, they provide local runtime overrides and do not get committed to the centralized Ceph mon config DB.

The config flow is as follows:

ceph-mon charm -> client relation -> ceph osd charm -> ceph.conf
                                                    -> ceph daemon osd.<osd_id> config set <key> <value>

Overall an OSD can get settings from:

  * charm-ceph-osd config values directly translated into ceph.conf options;
  * hard-coded template values (mostly enabled via high-level charm-ceph-osd options);
  * the config-flags setting (arbitrary overrides provided by an operator);
  * config values obtained via the "client" relation with ceph-mon.
Also:
  * via ceph-mon config DB (global and osd-specific options). In other words: a ceph client changes daemon-specific settings via Ceph mon and they are distributed by Ceph internally - there is no need for Juju to render config or apply runtime-only overrides.

From the point of view of Ceph docs (https://ceph.io/community/new-mimic-centralized-configuration-management/) the config is applied in this order:

1) Compiled-in default values;
2) Cluster configuration database (the new thing!);
3) Local ceph.conf file;
4) Runtime override (via “ceph daemon <daemon> config set …” or “ceph tell <daemon> injectargs …”)

"config set" commands we have supported are related to p.4 above:
https://github.com/openstack/charms.ceph/blob/af0eac506dadb4504f7f2428608860a00d29b987/charms_ceph/utils.py#L3314
    base_cmd = 'ceph daemon osd.{osd_id} config --format=json'
    get_cmd = base_cmd + ' get {key}'
    set_cmd = base_cmd + ' set {key} {value}'

`ceph daemon osd.<osd_id> config set <key> <value>` is documented as "not persistent" as it is a local runtime override.
https://github.com/ceph/ceph/blob/v13.2.9/src/osd/OSD.cc#L6144-L6155
COMMAND("config set " \
 "name=key,type=CephString name=value,type=CephString",
 "Set a configuration option at runtime (not persistent)",
 "osd", "rw", "cli,rest")
https://github.com/ceph/ceph/blob/v13.2.9/src/osd/OSD.cc#L6273-L6307

[Dmitrii Shcherbakov (dmitriis)]

To expand on #2, there are a couple of challenges here:

1) we need to support both the old and new versions of Ceph in our charms - so we can't just drop the config rendering code paths;
2) Some settings are not possible to update without a restart at runtime. Even if we do all config updates centrally through ceph-mon, some settings will not be applied until a ceph-osd restart. We will need a mechanism to trigger an osd restart from a mon unit based on whether a setting is possible to apply at runtime or not.

$ ceph config help ms_type
...
  Default: async+posix
  Can update at runtime: false

As a suggestion, a mon leader unit could apply an OSD-specific setting and check whether an OSD restart is needed and then do an update of a nonce value on a relation with OSDs so that restarts of OSD services are performed.

In this case, all config-flag overrides (runtime-only) would need to be reapplied following an OSD service restart which introduces a potential for race conditions.

[Dmitrii Shcherbakov (dmitriis)]

All of those commands:

config get <who> {<key>} Show configuration option(s) for an entity
config reset <int> Revert configuration to previous state
config rm <who> <name> Clear a configuration option for one or more
config set <who> <name> <value> Set a configuration option for one or more

and the `assimilate-conf` command (regardless of where it is executed) will update the mon DB.

* `ceph config assimilate-conf -i <input file> -o <output file>`
  will injest a configuration file from *input file* and move any
  valid options into the monitors' configuration database. Any
  settings that are unrecognized, invalid, or cannot be controlled by
  the monitor will be returned in an abbreviated config file stored in
  *output file*. This command is useful for transitioning from legacy
  configuration files to centralized monitor-based configuration.

So, there is no code path that would do something like this:

ceph.conf -> osd -> mon config database

rather

ceph.conf -> client -> mon config db

[Dmitrii Shcherbakov (dmitriis)]

Looking through the Ceph docs and its code again I begin to doubt that our current approach in charms themselves is a problem and that the issue described in the original report is the case under normal circumstances:

"Any value that already exists in the configuration database will no longer receive updates from ceph.conf. This breaks a number of settings in the ceph-* charms that rely on ceph.conf to propagate changes."

After all, this would be a major breaking change upstream. I think we need more data, namely we need to know if it affects mons only or other daemons too due to the following:

1) Ceph docs clearly state that config files are still being read and interpreted and values there override those that come from sources earlier in the list:

https://docs.ceph.com/docs/mimic/rados/configuration/ceph-conf/#config-sources
"Each Ceph daemon, process, and library will pull its configuration from several sources, listed below. Sources later in the list will override those earlier in the list when both are present.

1) the compiled-in default value
2) the monitor cluster’s centralized configuration database
3) a configuration file stored on the local host
4) environment variables
5) command line arguments
6) runtime overrides set by an administrator

2) I could find the implementation for the behavior mentioned in (1):

There are config source priority levels defined:
https://github.com/ceph/ceph/blob/v13.2.9/src/common/config.cc#L54-L66
const char *ceph_conf_level_name(int level)
{
  switch (level) {
  case CONF_DEFAULT: return "default"; // built-in default
  case CONF_MON: return "mon"; // monitor config database
  case CONF_ENV: return "env"; // process environment (CEPH_ARGS)
  case CONF_FILE: return "file"; // ceph.conf file
  case CONF_CMDLINE: return "cmdline"; // process command line args
  case CONF_OVERRIDE: return "override"; // injectargs or 'config set' at runtime
  case CONF_FINAL: return "final";
  default: return "???";
  }

Each call to _set_val includes an integer "level" passed by the caller:
https://github.com/ceph/ceph/blob/v13.2.9/src/common/config.cc#L1347-L1385
int md_config_t::_set_val(
  const std::string &raw_val,
  const Option &opt,
  int level,
  std::string *error_message)
{
    // ...
    if (p->second.rbegin()->first > level) {
      // there was a higher priority value; no effect
   // ...

Examples (mon settings and settings from a file):

https://github.com/ceph/ceph/blob/v13.2.9/src/common/config.cc#L263-L292
int md_config_t::set_mon_vals(CephContext *cct,
// ...
    int r = _set_val(i.second, *o, CONF_MON, &err);

https://github.com/ceph/ceph/blob/v13.2.9/src/common/config.cc#L440
int md_config_t::parse_config_files(const char *conf_files_str,
        std::ostream *warnings,
        int flags)
// ...
      int r = _set_val(val, opt, CONF_FILE, &error_message);

3) the same priority-based behavior is documented in the commit about a rewrite of common/config
https://github.com/ceph/ceph/commit/f8d4e5b5e9e540b16a0ed1e5617986046a7b2d25
"- for every option, we track values from any inputs (config, mon, override). At get_val() time, we pick the highest priority one."

...

[Billy Olsen (billy-olsen)]

Based on the last comment and analysis provided by Dmitrii, I am unsubscribing the field-high team to this bug.

[Billy Olsen (billy-olsen)]

It is possible that we may want to consider updating the charms to leverage the config database as suggested, however the current method of modifying ceph based configuration seems sufficient.

I am re-triaging this as a wishlist item for future consideration if we want to move to use the config database. Based on the current research, a change such as this will need to be carefully thought through to consider migration/upgrade paths and should be properly spec'd as it needs to consider the older ceph releases as well as current ceph releases.