when relating ceph-proxy with kubernetes-master, no StorageClass of rbd is created

[Yoshi Kadokawa]

For using external Ceph cluster(not deployed by Juju charms) as backend storage for Kubernetes,
I have deployed CDK with ceph-proxy charm.
However, no StorageClass will be created as it would with ceph-mon charm.
According to kubernetes-master charm, it will only create Ceph storageclass when "ceph-storage.available" flag is set.[0]
However, only ceph-client.available is set with ceph-proxy charm.

$ juju run --unit kubernetes-master/0 -- 'charms.reactive -p get_flags' | grep ceph
 'ceph-client.available',
 'ceph-client.connected',

I believe because of this, StorageClass, secrets and all other related resources for ceph are not configured in Kubernetes.

[0] https://github.com/charmed-kubernetes/charm-kubernetes-master/blob/8b365975b12d09984a99fc8852f04d9b2bdf4256/reactive/kubernetes_master.py#L994

[Alexey Tsitsin (cicin)]

Hello, are there any updates regarding the issue. We've got the same problems with Kubernetes 1.21.
SC can't be created.

Can we provide additional info to help resolve the issue or test the bug?

[Camille Rodriguez (camille.rodriguez)]

Is there any update on this ? It impairs all deployments with external ceph clusters

[Camille Rodriguez (camille.rodriguez)]

This is still relevant with charmed kubernetes 1.21 (charm rev 974) and ceph-proxy charm rev 44

[George Kraft (cynerva)]

It looks like the gap here is that the ceph-proxy charm doesn't have a relation with the ceph-admin interface, which kubernetes-master is dependent on.

Per https://bugs.launchpad.net/bugs/1840492 back in 2019, the ceph-admin interface was considered experimental and it was recommended that we remove the need for it from kubernetes-master. I would appreciate clarification from the OpenStack team if that is still the case.

[George Kraft (cynerva)]

I'm marking this as "Won't Fix" for kubernetes-master. In the upcoming CK 1.24 release, installing Ceph CSI via kubernetes-master relation to ceph will be deprecated. The recommendation will be to use the Ceph CSI charm instead.

However, it looks like the Ceph CSI charm is also dependent on the ceph-admin interface, so I've added it as an affected project.

[George Kraft (cynerva)]

To fix this, either the ceph-proxy charm will need to be updated to provide a relation with the ceph-admin interface, or the ceph-csi charm will need to be updated to work without it.

I'm not sure if the latter is doable, but it's worth looking into.

[Camille Rodriguez (camille.rodriguez)]

From conversation with the openstack team:
kubernetes-master should relate only to ceph-client, which is implemented by ceph-proxy and ceph-mon. Kubernetes-master is currently dependent on ceph-admin, which is untested & unmaintained by Canonical.

This change would enable the ceph-proxy relation to work as expected

[Camille Rodriguez (camille.rodriguez)]

Subscribing field-high as this issue essentially makes the relation between ceph-proxy and kubernetes-master useless. Workaround is to remove ceph-proxy and implement the ceph integration manually following the upstream instructions https://docs.ceph.com/en/latest/rbd/rbd-kubernetes/

[George Kraft (cynerva)]

After looking further into the Ceph CSI charm, it's clear that it's not ready for production use. We'll need to continue supporting Ceph integration via kubernetes-control-plane and cdk-addons through the CK 1.24 release.

[George Kraft (cynerva)]

It looks like the ceph-client relation provides an auth key that doesn't have the capabilities that Ceph CSI needs.

Output of `ceph auth list`:

client.kubernetes-control-plane
        key: ...
        caps: [mon] allow r; allow command "osd blacklist"
        caps: [osd] allow rwx

The capabilities required for Ceph CSI are documented here: https://github.com/ceph/ceph-csi/blob/devel/docs/capabilities.md

I'm still looking to see if there's a way to get the capabilities we need without using the ceph-admin relation.

[Chris MacNaughton (chris.macnaughton)]

it's very possible to get almost any permissions one needs via the ceph-client relation, see: https://github.com/ChrisMacNaughton/charm-ceph-nfs/blob/master/src/charm.py#L117-L125

[George Kraft (cynerva)]

Ah perfect, thanks Chris. I'll give that a go today.

With that, I'm reasonably confident we should be able to resolve this in the kubernetes-control-plane and ceph-csi charms in time for the Charmed Kubernetes 1.24 release in early May.

[George Kraft (cynerva)]

I didn't quite get to the capabilities today, but I was able to work through the more subtle differences between the ceph-client and ceph-admin interfaces.

This will need changes to cdk-addons as well, to remove a hard-coded "admin" username.

[George Kraft (cynerva)]

PRs for kubernetes-control-plane and cdk-addons:
https://github.com/charmed-kubernetes/charm-kubernetes-control-plane/pull/219
https://github.com/charmed-kubernetes/cdk-addons/pull/212

I'm working on fixing this for the Ceph CSI charm as well.

[George Kraft (cynerva)]

PR for Ceph CSI charm: https://github.com/charmed-kubernetes/ceph-csi-operator/pull/1