when specifying volumeMode Block with Ceph related ceph client config isn't being written to pods

[Jeff Hillman]

K8s 1.17
Ceph 12.2.12

Ceph-mon related to k8s master as admin and client

Creating PVCs with ReadWriteOnce and volumeMode as Filesystem works as expected (with 1 pod at a time accessing the volume)

Creating PVCs with ReadWriteMany and volumeMode as Block (so multiple mounts can happen), an error message is generated indicating that the /etc/ceph/ceph.conf (and other client files) are not provided to the pod. The PVC and the PV get created and bound to each other.

---

$ kubectl describe pod/cdk-cats-block-c446cdfcc-nrhzm
Name: cdk-cats-block-c446cdfcc-nrhzm
Namespace: default
Priority: 0
Node: stha8p0pm/10.109.14.69
Start Time: Wed, 08 Jan 2020 18:22:21 +0000
Labels: app=cdk-cats-block
                pod-template-hash=c446cdfcc
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/cdk-cats-block-c446cdfcc
Containers:
  cdk-cats-block:
    Container ID:
    Image: calvinhartwell/cdk-cats-block:latest
    Image ID:
    Port: 80/TCP
    Host Port: 0/TCP
    State: Waiting
      Reason: ContainerCreating
    Ready: False
    Restart Count: 0
    Liveness: http-get http://:80/ delay=5s timeout=30s period=10s #success=1 #failure=3
    Environment: <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-p4qq6 (ro)
    Devices:
      /dev/xvda from cats-pv-block
Conditions:
  Type Status
  Initialized True
  Ready False
  ContainersReady False
  PodScheduled True
Volumes:
  cats-pv-block:
    Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName: cats-pv-block-claim
    ReadOnly: false
  default-token-p4qq6:
    Type: Secret (a volume populated by a Secret)
    SecretName: default-token-p4qq6
    Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Warning FailedScheduling <unknown> default-scheduler error while running "VolumeBinding" filter plugin for pod "cdk-cats-block-c446cdfcc-nrhzm": pod has unbound immediate PersistentVolumeClaims
  Warning FailedScheduling <unknown> default-scheduler error while running "VolumeBinding" filter plugin for pod "cdk-cats-block-c446cdfcc-nrhzm": pod has unbound immediate PersistentVolumeClaims
  Normal Scheduled <unknown> default-scheduler Successfully assigned default/cdk-cats-block-c446cdfcc-nrhzm to stha8p0pm
  Normal SuccessfulAttachVolume 33s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae"
  Warning FailedMapVolume 29s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:25.811 7fab3bb12b00 -1 Errors while parsing config file!
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 Errors while parsing config file!
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.831 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.831 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.831 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.835 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.835 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.835 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
  Warning FailedMapVolume 28s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:26.503 7f1961217b00 -1 Errors while parsing config file!
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 Errors while parsing config file!
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.523 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.523 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.523 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.527 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.527 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.527 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
  Warning FailedMapVolume 27s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 Errors while parsing config file!
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 Errors while parsing config file!
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.723 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.727 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.727 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.731 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.731 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.731 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
  Warning FailedMapVolume 25s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:29.923 7f00195e1b00 -1 Errors while parsing config file!
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 Errors while parsing config file!
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.939 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.943 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.943 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.947 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.947 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.947 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
  Warning FailedMapVolume 20s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:34.047 7f456cd18b00 -1 Errors while parsing config file!
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 Errors while parsing config file!
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.063 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.067 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.067 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.071 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.071 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.071 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
  Warning FailedMapVolume 12s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 Errors while parsing config file!
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 Errors while parsing config file!
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.211 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.215 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.215 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.215 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.219 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.219 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory

---

Here's the configMap for the deployment requesting this.

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cats-pv-block-claim
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Block
  resources:
    requests:
      storage: 3Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: cdk-cats-block
  name: cdk-cats-block
spec:
  replicas: 2
  selector:
    matchLabels:
      app: cdk-cats-block
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: cdk-cats-block
    spec:
      volumes:
        - name: cats-pv-block
          persistentVolumeClaim:
            claimName: cats-pv-block-claim
      containers:
      - image: calvinhartwell/cdk-cats-block:latest
        imagePullPolicy: ""
        name: cdk-cats-block
        ports:
        - containerPort: 80
        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 5
          timeoutSeconds: 30
        resources: {}
        volumeDevices:
          - devicePath: "/dev/xvda"
            name: cats-pv-block
      restartPolicy: Always
      serviceAccountName: ""
status: {}
---
apiVersion: v1
kind: Service
metadata:
  name: cdk-cats-block
spec:
  type: LoadBalancer
  selector:
     app: cdk-cats-block
  ports:
  - name: cdk-cats-block
    protocol: TCP
    port: 80
    targetPort: 80

---

Also, some descriptions of the pvc and pv associated with this.

---

$ kubectl describe persistentvolumeclaim/cats-pv-block-claim
Name: cats-pv-block-claim
Namespace: default
StorageClass: ceph-xfs
Status: Bound
Volume: pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
                 {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"cats-pv-block-claim","namespace":"default"},"spec":...
               pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: rbd.csi.ceph.com
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 3Gi
Access Modes: RWX
VolumeMode: Block
Mounted By: cdk-cats-block-c446cdfcc-nrhzm
               cdk-cats-block-c446cdfcc-vx4sg
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal ExternalProvisioning 2m3s (x2 over 2m3s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "rbd.csi.ceph.com" or manually created by system administrator
  Normal Provisioning 2m3s rbd.csi.ceph.com_csi-rbdplugin-provisioner-0_69327706-3240-11ea-8efe-f6a7fad2373c External provisioner is provisioning volume for claim "default/cats-pv-block-claim"
  Normal ProvisioningSucceeded 2m2s rbd.csi.ceph.com_csi-rbdplugin-provisioner-0_69327706-3240-11ea-8efe-f6a7fad2373c Successfully provisioned volume pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae

$ kubectl describe persistentvolume/pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae
Name: pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae
Labels: <none>
Annotations: pv.kubernetes.io/provisioned-by: rbd.csi.ceph.com
Finalizers: [kubernetes.io/pv-protection external-attacher/rbd-csi-ceph-com]
StorageClass: ceph-xfs
Status: Bound
Claim: default/cats-pv-block-claim
Reclaim Policy: Delete
Access Modes: RWX
VolumeMode: Block
Capacity: 3Gi
Node Affinity: <none>
Message:
Source:
    Type: CSI (a Container Storage Interface (CSI) volume source)
    Driver: rbd.csi.ceph.com
    VolumeHandle: csi-rbd-vol-cea8bce3-3243-11ea-b2fe-f6a7fad2373c
    ReadOnly: false
    VolumeAttributes: adminid=admin
                           fsType=xfs
                           imageFeatures=layering
                           imageFormat=2
                           monitors=10.109.12.10 10.109.12.8 10.109.12.9
                           pool=xfs-pool
                           storage.kubernetes.io/csiProvisionerIdentity=1578506281220-8081-
                           userid=admin
Events: <none>

---

I followed the ceph documentation on setting up this type of config by following https://docs.ceph.com/docs/master/rbd/rbd-kubernetes/#create-a-persistentvolumeclaim and it in no way mentions the need to manually configure the ceph client bits in the pod.

[George Kraft (cynerva)]

The Ceph CSI support matrix[1] suggests that RBD supports Block mode RWX, so we should expect this case to work.

The bit about missing /etc/ceph/ceph.conf is a red herring. It's a Ceph CSI quirk. I see that message all the time in csi-rbdplugin logs even with successful PVs. The actual failure comes from the last line of the rbd output, which is:

rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory

Unfortunately, that error message is unclear, and likely distant from the root of the problem. This will need further investigation.

[1]: https://github.com/ceph/ceph-csi/tree/4b8b52e0de568002420fa38e8dfdb5db4b714bc9#ceph-csi-features-and-available-versions

[Jeff Hillman (jhillman)]

I'm open to suggestions on how to dig further into this. The container/pod never starts on any node so there are no logs to look at.

Right now, there are 2 pods, and each have a different uuid for the missing image.

I can easily recreate this at the moment FYI.

[Jeff Hillman (jhillman)]

So, as a test, I switched the configmap to ReadWriteOnce, and set to 1 replica, and I'm getting the exact same error.

Because of this, I changed the title of this bug. Basically, just doing a simple block device via ceph appears to have issues.

[George Kraft (cynerva)]

Knobby apparently can't comment on this bug, but wanted me to pass along that RWX Block mode is not supported unless you have Ceph version >= Mimic (v13). You can see that in the support matrix I linked above.

Of course, the support matrix also says Mimic is required for RWO File mode PVs but those seem to work on v12 anyway. Maybe there's some way to get RWX Block mode to work on v12, too, but I'm not very optimistic.

[Jeff Hillman (jhillman)]

I updated ceph-mon units and ceph-osd units to ceph 13.2.6 and the issue persists. Exact same error.

For the record, this was an in-place upgrade and not a reinstall.

[George Kraft (cynerva)]

I'm able to reproduce this easily enough. It looks like a bug in Ceph CSI v1.0.2's getVolumeName function[1], which takes the target_path from kubelet's NodePublishVolume request:

{ ... "target_path": "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-c5a78d65-52a7-4cd2-92c9-d2a250db5089/0a023bbf-4302-4385-8cad-ff7446692518" ... }

and takes the last directory to be the volume name:

0a023bbf-4302-4385-8cad-ff7446692518

When in fact, the actual volume name is:

pvc-c5a78d65-52a7-4cd2-92c9-d2a250db5089

The volume name and RBD image name are the same, so csi-rbdplugin ends up looking for the RBD image with the wrong name, and fails to find it. The bug only occurs with block volumes because the target_path for block volumes is slightly different and has its own code path in getVolumeName.

Later versions of Ceph CSI (v1.1.0+) appear to use a different approach for determining the volume name[2]. So, I suspect we can fix this by updating to a more recent version of Ceph CSI.

[1]: https://github.com/ceph/ceph-csi/blob/a4dd8457350b4c4586743d78cbd5776437e618b6/pkg/rbd/nodeserver.go#L107-L109
[2]: https://github.com/ceph/ceph-csi/blob/c7ba26d23d784f7e21d850c6d014897f25a9868d/pkg/rbd/nodeserver.go#L132-L142

[George Kraft (cynerva)]

Slight correction to the above. It looks like we're running csi-rbdplugin v1.0.0, not v1.0.2.

[George Kraft (cynerva)]

Removing 1.18 milestone since we had to revert ceph-csi back to v1.0.0 for 1.18. Details here: https://bugs.launchpad.net/cdk-addons/+bug/1867940

[Cory Johns (johnsca)]

Fixed with https://github.com/charmed-kubernetes/cdk-addons/pull/190