Adding storage-backend relation to subordinate changes `host` identifier of cinder globally

[Frode Nordahl]

The `host` option is an opaque identifier. It is not necessarily a host name, FQDN, or IP address. It can also be set individually per backend. Cinder will use the value of this configuration option in combination with `volume_backend_name` to create the opaque identifier stored in its database as `host`@`volume_backend_name`.

The default value of `host` is the system hostname. The default value of `volume_backend_name` is the same as the value of the backend section name.

Once a value for the `host` identifier has been chosen it should not be changed lightly as it is referenced by consumers. In the event it is changed any already created volumes will be invalidated until their host reference is updated accordingly.

For direct attached stroage backends like `LVM` the `host` identifier should be unique and persistent on a per unit+backend basis.

For distributed storage backends like `ceph` the `host` identifier should be equal for all cinder unit+backend combinations serving the same view of the distributed storage backend.

Currently when you add a storage-backend subordinate to cinder the principal unit will set a new value for the `host` option globally. This prevents a user from using the same cinder application for serving requests for multiple backends. Examples: directly attached `LVM` and `ceph`, or multiple `ceph` clusters simultaneously.

We should stop doing this and make it the subordinates responsibility to set its per-backend `host` option when sharing configuration data with the principal unit.

[Dmitrii Shcherbakov (dmitriis)]

From the cinder perspective "host" option is a global option.

cinder/common/config.py
    cfg.HostAddressOpt('host',
                       default=socket.gethostname(),
                       help='Name of this node. This can be an opaque '
                            'identifier. It is not necessarily a host name, '
                            'FQDN, or IP address.'),

However, it can be overridden on a per-backend basis via backend_host option:

https://github.com/openstack/cinder/blob/stable/queens/cinder/cmd/volume.py#L61-L62
host_opt = cfg.StrOpt('backend_host', help='Backend override of host value.')
CONF.register_cli_opt(host_opt)

https://github.com/openstack/cinder/blob/stable/queens/cinder/cmd/volume.py#L88-L91

def _launch_service(launcher, backend):
    CONF.register_opt(host_opt, group=backend)
    backend_host = getattr(CONF, backend).backend_host
    host = "%s@%s" % (backend_host or CONF.host, backend)

The current charm implementation relies on cinder-ceph charm populating a context and sending it to cinder via relation data. The cinder charm then renders sections received from a subordinate in a generic way - this could be utilized to properly set backend_host config in the cinder-ceph charm without modifying the cinder charm.

[Dmitrii Shcherbakov (dmitriis)]

Subscribed ~field-high as this issue affects a deployment with multiple ceph backends used.

The fix is proposed here: https://review.openstack.org/#/c/571762/

[Frode Nordahl (fnordahl)]

Ah, perfect. Thank you for weeding out the devil in the details there and proposing a fix @dmitriis.

I still think we have a bug in the cinder charm for the LVM case or indeed the vintage case of direct ceph relation to cinder charm because of the unnecessary global hostname change when adding the relation to cinder-ceph subordinate.

So retaining original global `host` setting would still be a task for cinder charm.

[Dmitrii Shcherbakov (dmitriis)]

Frode, you are right about LVM and the old ceph handling case.

I think if a fix for cinder-ceph lands we can at least "lift" the field-high on this and triage the cinder work appropriately.

The proposed fix does not touch the global setting so it will still be used as before while for storage-backend relation it will provide the necessary override.

[Dmitrii Shcherbakov (dmitriis)]

If only storage-backend relation is used multiple cinder-ceph <-> cinder relations seem to be handled properly (Queens, 18.02 charms).

https://paste.ubuntu.com/p/YYkm9GgW6S/

I also looked at libvirt domain xml and tried using dd on 2 volumes from different cinder-ceph pools attached to a VM - this worked just fine.

Therefore, I am unsubscribing ~field-high from this.

[Frode Nordahl (fnordahl)]

Great, lowering severity of the tasks in this bug then. Keeping bug open as there is some untangling to be done wrt. who sets the hostname, whether it is in global or section config etc.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Frode Nordahl (<email address hidden>) on branch: master
Review: https://review.openstack.org/571762
Reason: Marking this review as abandoned due to lack of activity in the past 6 months.
Feel free to restore it again if you want to pick up and continue the work.