K8s 1.18
Ceph 12.2.12

Ceph-mon related to k8s master as admin and client

Creating PVCs with ReadWriteOnce and volumeMode as Filesystem works as expected (with 1 pod at a time accessing the volume)

Creating PVCs with ReadWriteOnce and volumeMode as Block (so multiple mounts can happen), an error message is generated indicating that the /etc/ceph/ceph.conf (and other client files) are not provided to the pod.  The PVC and the PV get created and bound to each other.

---

$ kubectl describe pod/cdk-cats-block-c446cdfcc-nrhzm
Name:           cdk-cats-block-c446cdfcc-nrhzm
Namespace:      default
Priority:       0
Node:           stha8p0pm/10.109.14.69
Start Time:     Wed, 08 Jan 2020 18:22:21 +0000
Labels:         app=cdk-cats-block
                pod-template-hash=c446cdfcc
Annotations:    <none>
Status:         Pending
IP:
IPs:            <none>
Controlled By:  ReplicaSet/cdk-cats-block-c446cdfcc
Containers:
  cdk-cats-block:
    Container ID:
    Image:          calvinhartwell/cdk-cats-block:latest
    Image ID:
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Liveness:       http-get http://:80/ delay=5s timeout=30s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-p4qq6 (ro)
    Devices:
      /dev/xvda from cats-pv-block
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  cats-pv-block:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  cats-pv-block-claim
    ReadOnly:   false
  default-token-p4qq6:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-p4qq6
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                  Age        From                     Message
  ----     ------                  ----       ----                     -------
  Warning  FailedScheduling        <unknown>  default-scheduler        error while running "VolumeBinding" filter plugin for pod "cdk-cats-block-c446cdfcc-nrhzm": pod has unbound immediate PersistentVolumeClaims
  Warning  FailedScheduling        <unknown>  default-scheduler        error while running "VolumeBinding" filter plugin for pod "cdk-cats-block-c446cdfcc-nrhzm": pod has unbound immediate PersistentVolumeClaims
  Normal   Scheduled               <unknown>  default-scheduler        Successfully assigned default/cdk-cats-block-c446cdfcc-nrhzm to stha8p0pm
  Normal   SuccessfulAttachVolume  33s        attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae"
  Warning  FailedMapVolume         29s        kubelet, stha8p0pm       MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:25.811 7fab3bb12b00 -1 Errors while parsing config file!
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 Errors while parsing config file!
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.831 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.831 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.831 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.835 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.835 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.835 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
  Warning  FailedMapVolume  28s  kubelet, stha8p0pm  MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:26.503 7f1961217b00 -1 Errors while parsing config file!
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 Errors while parsing config file!
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.523 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.523 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.523 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.527 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.527 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.527 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
  Warning  FailedMapVolume  27s  kubelet, stha8p0pm  MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 Errors while parsing config file!
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 Errors while parsing config file!
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.723 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.727 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.727 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.731 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.731 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.731 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
  Warning  FailedMapVolume  25s  kubelet, stha8p0pm  MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:29.923 7f00195e1b00 -1 Errors while parsing config file!
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 Errors while parsing config file!
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.939 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.943 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.943 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.947 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.947 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.947 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
  Warning  FailedMapVolume  20s  kubelet, stha8p0pm  MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:34.047 7f456cd18b00 -1 Errors while parsing config file!
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 Errors while parsing config file!
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.063 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.067 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.067 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.071 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.071 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.071 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
  Warning  FailedMapVolume  12s  kubelet, stha8p0pm  MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 Errors while parsing config file!
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 Errors while parsing config file!
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.211 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.215 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.215 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.215 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.219 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.219 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory

---

Here's the configMap for the deployment requesting this.

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cats-pv-block-claim
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Block
  resources:
    requests:
      storage: 3Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: cdk-cats-block
  name: cdk-cats-block
spec:
  replicas: 2
  selector:
    matchLabels:
      app: cdk-cats-block
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: cdk-cats-block
    spec:
      volumes:
        - name: cats-pv-block
          persistentVolumeClaim:
            claimName: cats-pv-block-claim
      containers:
      - image: calvinhartwell/cdk-cats-block:latest
        imagePullPolicy: ""
        name: cdk-cats-block
        ports:
        - containerPort: 80
        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 5
          timeoutSeconds: 30
        resources: {}
        volumeDevices:
          - devicePath: "/dev/xvda"
            name: cats-pv-block
      restartPolicy: Always
      serviceAccountName: ""
status: {}
---
apiVersion: v1
kind: Service
metadata:
  name: cdk-cats-block
spec:
  type: LoadBalancer
  selector:
     app: cdk-cats-block
  ports:
  - name: cdk-cats-block
    protocol: TCP
    port: 80
    targetPort: 80

---

Also, some descriptions of the pvc and pv associated with this.

---

$ kubectl describe persistentvolumeclaim/cats-pv-block-claim
Name:          cats-pv-block-claim
Namespace:     default
StorageClass:  ceph-xfs
Status:        Bound
Volume:        pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae
Labels:        <none>
Annotations:   kubectl.kubernetes.io/last-applied-configuration:
                 {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"cats-pv-block-claim","namespace":"default"},"spec":...
               pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: rbd.csi.ceph.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      3Gi
Access Modes:  RWX
VolumeMode:    Block
Mounted By:    cdk-cats-block-c446cdfcc-nrhzm
               cdk-cats-block-c446cdfcc-vx4sg
Events:
  Type    Reason                 Age                  From                                                                               Message
  ----    ------                 ----                 ----                                                                               -------
  Normal  ExternalProvisioning   2m3s (x2 over 2m3s)  persistentvolume-controller                                                        waiting for a volume to be created, either by external provisioner "rbd.csi.ceph.com" or manually created by system administrator
  Normal  Provisioning           2m3s                 rbd.csi.ceph.com_csi-rbdplugin-provisioner-0_69327706-3240-11ea-8efe-f6a7fad2373c  External provisioner is provisioning volume for claim "default/cats-pv-block-claim"
  Normal  ProvisioningSucceeded  2m2s                 rbd.csi.ceph.com_csi-rbdplugin-provisioner-0_69327706-3240-11ea-8efe-f6a7fad2373c  Successfully provisioned volume pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae


$ kubectl describe persistentvolume/pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae
Name:            pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae
Labels:          <none>
Annotations:     pv.kubernetes.io/provisioned-by: rbd.csi.ceph.com
Finalizers:      [kubernetes.io/pv-protection external-attacher/rbd-csi-ceph-com]
StorageClass:    ceph-xfs
Status:          Bound
Claim:           default/cats-pv-block-claim
Reclaim Policy:  Delete
Access Modes:    RWX
VolumeMode:      Block
Capacity:        3Gi
Node Affinity:   <none>
Message:
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            rbd.csi.ceph.com
    VolumeHandle:      csi-rbd-vol-cea8bce3-3243-11ea-b2fe-f6a7fad2373c
    ReadOnly:          false
    VolumeAttributes:      adminid=admin
                           fsType=xfs
                           imageFeatures=layering
                           imageFormat=2
                           monitors=10.109.12.10 10.109.12.8 10.109.12.9
                           pool=xfs-pool
                           storage.kubernetes.io/csiProvisionerIdentity=1578506281220-8081-
                           userid=admin
Events:                <none>


---

I followed the ceph documentation on setting up this type of config by following https://docs.ceph.com/docs/master/rbd/rbd-kubernetes/#create-a-persistentvolumeclaim and it in no way mentions the need to manually configure the ceph client bits in the pod.