Ceph Broker Conversation does not complete with CMR

[Nobuto Murata]

With CMR (Cross Model Relation), Ceph Broker Conversation does not complete with:
"Ignoring legacy broker_rsp without unit key as remote service supports unit specific replies"
Because it assumes "broker-rsp-APP_NAME-0" in the relation data, but CMR replaces "APP_NAME" with "remote-UUID".

https://github.com/juju/charm-helpers/blob/e1dc8165b906846e7aa62c4e4d6672d78bb65542/charmhelpers/contrib/storage/linux/ceph.py#L1339-L1369

How to reproduce:

1. deploy ceph-mon in default model
2. offer ceph-mon for CMR

$ juju offer ceph-mon:client

3. deploy cinder-backup for example in another model.

$ juju add-model another-model
$ juju deploy cinder
$ juju config cinder block-device=None
$ juju deploy cinder-backup
$ juju add-relation cinder cinder-backup

4. add CMR

$ juju add-relation admin/default.ceph-mon cinder-backup:ceph

[model log]
unit-cinder-backup-0: 13:39:09 INFO unit.cinder-backup/0.juju-log ceph:2: Making dir /var/lib/charm/cinder-backup root:root 555
unit-cinder-backup-0: 13:39:09 INFO unit.cinder-backup/0.juju-log ceph:2: Making dir /etc/ceph root:root 555
unit-cinder-backup-0: 13:39:09 INFO unit.cinder-backup/0.juju-log ceph:2: Registered config file: /var/lib/charm/cinder-backup/ceph.conf
unit-cinder-backup-0: 13:39:09 DEBUG unit.cinder-backup/0.juju-log ceph:2: Generating template context for ceph
unit-cinder-backup-0: 13:39:13 WARNING unit.cinder-backup/0.juju-log ceph:2: Ceph keyring exists at /etc/ceph/ceph.client.cinder-backup.keyring.
unit-cinder-backup-0: 13:39:13 DEBUG unit.cinder-backup/0.juju-log ceph:2: Ignoring legacy broker_rsp without unit key as remote service supports unit specific replies
unit-cinder-backup-0: 13:39:14 DEBUG unit.cinder-backup/0.juju-log ceph:2: Ignoring legacy broker_rsp without unit key as remote service supports unit specific replies
unit-cinder-backup-0: 13:39:14 DEBUG unit.cinder-backup/0.juju-log ceph:2: Request already sent but not complete, not sending new request
unit-cinder-backup-0: 13:39:14 INFO unit.cinder-backup/0.juju-log ceph:2: Unit is ready

[relation data]
$ juju run -m another-model --unit cinder-backup/0 -- relation-get -r ceph:2 - ceph-mon/0
auth: cephx
broker-rsp-remote-938f0ad2ef134a9e84aa3ac5cc5c9e48-0: '{"exit-code": 0, "request-id":
  "e4b59d93-8331-11e8-86bc-00163e8f9cfb"}'
broker_rsp: '{"exit-code": 0, "request-id": "e4b59d93-8331-11e8-86bc-00163e8f9cfb"}'
ceph-public-address: 10.0.8.155
egress-subnets: 10.0.8.155/32
ingress-address: 10.0.8.155
key: AQCE5kJbbD0SGBAAqeyGsoSDJhF4VpEnoTv0Gg==
private-address: 10.0.8.155

$ juju run -m default --unit ceph-mon/0 -- relation-get -r client:7 - remote-938f0ad2ef134a9e84aa3ac5cc5c9e48/0
broker_req: '{"api-version": 1, "request-id": "e4b59d93-8331-11e8-86bc-00163e8f9cfb",
  "ops": [{"group": null, "name": "cinder-backup", "weight": null, "replicas": 3,
  "pg_num": null, "group-namespace": null, "op": "create-pool"}]}'
egress-subnets: 10.0.8.131/32
ingress-address: 10.0.8.131
private-address: 10.0.8.131

[Cory Johns (johnsca)]

This also impacts the Kubernetes charms, and leads us to use the admin key for everything, rather than the client key, since we don't know (at least with CMR) what username the client key is associated with.

Perhaps it would be sufficient to echo back the service_name that was used over the relation in a new key. Then charmhelpers could be updated to look for that key, and if found, use it instead of local_unit() when calculating the broker_rsp key; and Kubernetes could also use it as the userID value.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/736709
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-mon/commit/?id=d875568ad78cb7e6caeccfe2c43a4f8819b3ab5a
Submitter: Zuul
Branch: master

commit d875568ad78cb7e6caeccfe2c43a4f8819b3ab5a
Author: Liam Young <email address hidden>
Date: Thu Jun 18 13:08:14 2020 +0000

    Support ceph client over CMRs

    Support ceph client over CMRs of and only if permit-insecure-cmr
    config option has been set to true, otherwise go into a blocked
    state.

    To support CMR clients try and get client service name from relation
    data first before falling back to using the remote unit name. Using
    the remote unit name fails when the clients are connecting via a
    cross-model relation.

    The clients side change is here: https://github.com/juju/charm-helpers/pull/481

    Change-Id: If9616170b8af9eac309dc6e8edd670fb5cfd8e0f
    Closes-Bug: #1780712