With pool-prefix RGW creates proper pools but doesn't adjust the zone

[Andrey Grebennikov]

Standalone ceph and a single unit of RGW, charm revision 283:
  ceph-radosgw:
    annotations:
      gui-x: '1000'
      gui-y: '250'
    charm: cs:ceph-radosgw
    num_units: 1
    options:
      source: cloud:bionic-stein
      pool-prefix: gravit
    bindings:
      "": default
    to:
    - lxd:1

The customer requests S3/Swift object to be stored in the pools with pre-defined names, decided to set "pool-prefix" to "gravit".
Once deployed, all necessary pools have been created:
root@juju-33c46d-0-lxd-0:~# rados lspools
.rgw.root
gravit.rgw.buckets.data
gravit.rgw.control
gravit.rgw.data.root
gravit.rgw.gc
gravit.rgw.log
gravit.rgw.intent-log
gravit.rgw.meta
gravit.rgw.usage
gravit.rgw.users.keys
gravit.rgw.users.email
gravit.rgw.users.swift
gravit.rgw.users.uid
gravit.rgw.buckets.extra
gravit.rgw.buckets.index
default.rgw.control
default.rgw.meta
default.rgw.log

However the single zone is still called "default" and uses improper pools:
root@juju-33c46d-0-lxd-0:~# radosgw-admin zone get default
{
    "id": "f0e44e84-c623-439a-a969-7d3c8271d33b",
    "name": "default",
    "domain_root": "default.rgw.meta:root",
    "control_pool": "default.rgw.control",
    "gc_pool": "default.rgw.log:gc",
    "lc_pool": "default.rgw.log:lc",
    "log_pool": "default.rgw.log",
    "intent_log_pool": "default.rgw.log:intent",
    "usage_log_pool": "default.rgw.log:usage",
    "reshard_pool": "default.rgw.log:reshard",
    "user_keys_pool": "default.rgw.meta:users.keys",
    "user_email_pool": "default.rgw.meta:users.email",
    "user_swift_pool": "default.rgw.meta:users.swift",
    "user_uid_pool": "default.rgw.meta:users.uid",
    "otp_pool": "default.rgw.otp",
    "system_key": {
        "access_key": "",
        "secret_key": ""
    },
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "default.rgw.buckets.index",
                "data_pool": "default.rgw.buckets.data",
                "data_extra_pool": "default.rgw.buckets.non-ec",
                "index_type": 0,
                "compression": ""
            }
        }
    ],
    "metadata_heap": "",
    "realm_id": ""
}

[James Page (james-page)]

Confirmed - setting pool-prefix creates the required pools, but the default zone is misconfigured

[James Page (james-page)]

I don't think this option has ever worked AFAICT - the original commit makes no attempt to reconfigure the default zone with the required pool names.

[Frode Nordahl (fnordahl)]

It does indeed raise a few questions, would you even be interested in having a zone named default if you use the configuration option? It would also be problematic to handle any changes to the config post-deploy.

FWIW, if a immutable config type approach would be on the table we have an example of how that can be handled here: https://review.opendev.org/#/c/701476/

[Dmitrii Shcherbakov (dmitriis)]

Looked at this in detail and I think there are a couple of things to look at here.

The Rados Gateway daemon itself has some initialization logic on first startup that depends on the value of the rgw_zone configuration option (which is affected by the "zone" config key on the charm).

* If a zone group is not yet initialized, it will attempt to use local configuration to initialize some state;
* It will unconditionally create a zone named "default" (even if rgw_zone is set in the config on startup)

https://github.com/ceph/ceph/blob/v16.0.0/src/rgw/rgw_zone.cc#L31 default_zone_name == "default"
https://github.com/ceph/ceph/blob/v16.0.0/src/rgw/rgw_zone.cc#L81-L103 (creates a RGWZoneParams and unconditionally passes "default" for the zone name, calls RGWZoneParams::create_default -> RGWZoneParams::create -> fix_pool_names)
https://github.com/ceph/ceph/blob/v16.0.0/src/rgw/rgw_zone.cc#L105-L106 (creates a default zone RGWZone instance and passed the "default" zone name into it)

https://github.com/ceph/ceph/blob/v16.0.0/src/rgw/rgw_zone.cc#L752-L762

* It will set the zone parameters for the internal zone service to use "default" for the zone name, unless a zone name is provided via a the "rgw_zone" config:

https://github.com/ceph/ceph/blob/v16.0.0/src/rgw/services/svc_zone.h#L39
  RGWZoneParams *zone_params{nullptr}; /* internal zone params, e.g., rados pools */

https://github.com/ceph/ceph/blob/v16.0.0/src/rgw/services/svc_zone.cc#L135-L138
  if (creating_defaults && cct->_conf->rgw_zone.empty()) {
    ldout(cct, 10) << " Using default name "<< default_zone_name << dendl;
    zone_params->set_name(default_zone_name);
  }

* Will create a log pool https://github.com/ceph/ceph/blob/v16.0.0/src/rgw/rgw_log.cc#L452
* Will create a control pool https://github.com/ceph/ceph/blob/v16.0.0/src/rgw/rgw_zone.cc#L752-L762

The control and log pools will have names like this:

1) if zone config on the charm is set (hence the rgw_zone config is set):
<rgw_zone>.rgw.log
<rgw_zone>.rgw.control

2) if zone config is not set and whether or not pool-prefix is set:
default.rgw.log
default.rgw.control

The "pool-prefix" variable is charm-specific and doesn't get propagated to the rados gateway config file (does not get used to populate the rgw_zone config option):

* "zone" config option has a priority over "pool-prefix"
https://github.com/openstack/charm-ceph-radosgw/blob/0f51adbba096c054979349ae86adfd2ce602bf87/hooks/hooks.py#L249-L250

* if neither is specified, "default" is used as a default pool prefix name for pool creation by the charm, otherwise it will either be set to "zone" (in priority) or "pool-prefix" (if "zone" is not set):

https://github.com/openstack/charm-ceph-radosgw/blob/0f51adbba096c054979349ae86adfd2ce602bf87/hooks/ceph_rgw.py#L114

-----

Also, for non-multi-site deployments the charm does not create a zone specified in the "zone" config. I think we need to fix this and do that if the "zone" config is set.

As a result, I think we need to:

* deprecate the pool-prefix config;
* create a zone with a name provided via the "zone" charm config. This might require some clever initialization logic since we cannot create a zone before the A...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/754338
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-radosgw/commit/?id=2466691f854a04d3c74b6d9a07dfb912972bedd5
Submitter: Zuul
Branch: master

commit 2466691f854a04d3c74b6d9a07dfb912972bedd5
Author: Dan Ardelean <email address hidden>
Date: Tue Sep 22 14:30:19 2020 +0000

    Fix pool creation for single zone setups.
    Deprecate 'pool-prefix' charm config.

    Change-Id: I34079d8975d995ea958f219e0516a972d73319f7
    Closes-Bug: #1856106
    Co-Authored-By: Andrei Bacos <email address hidden>