when specifying volumeMode Block with Ceph related ceph client config isn't being written to pods
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
CDK Addons |
Fix Released
|
High
|
Joseph Borg |
Bug Description
K8s 1.17
Ceph 12.2.12
Ceph-mon related to k8s master as admin and client
Creating PVCs with ReadWriteOnce and volumeMode as Filesystem works as expected (with 1 pod at a time accessing the volume)
Creating PVCs with ReadWriteMany and volumeMode as Block (so multiple mounts can happen), an error message is generated indicating that the /etc/ceph/ceph.conf (and other client files) are not provided to the pod. The PVC and the PV get created and bound to each other.
---
$ kubectl describe pod/cdk-
Name: cdk-cats-
Namespace: default
Priority: 0
Node: stha8p0pm/
Start Time: Wed, 08 Jan 2020 18:22:21 +0000
Labels: app=cdk-cats-block
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/
Containers:
cdk-cats-block:
Container ID:
Image: calvinhartwell/
Image ID:
Port: 80/TCP
Host Port: 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Liveness: http-get http://
Environment: <none>
Mounts:
/
Devices:
/dev/xvda from cats-pv-block
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
cats-pv-block:
Type: PersistentVolum
ClaimName: cats-pv-block-claim
ReadOnly: false
default-
Type: Secret (a volume populated by a Secret)
SecretName: default-token-p4qq6
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling <unknown> default-scheduler error while running "VolumeBinding" filter plugin for pod "cdk-cats-
Warning FailedScheduling <unknown> default-scheduler error while running "VolumeBinding" filter plugin for pod "cdk-cats-
Normal Scheduled <unknown> default-scheduler Successfully assigned default/
Normal SuccessfulAttac
Warning FailedMapVolume 29s kubelet, stha8p0pm MapVolume.
2020-01-08 18:22:25.811 7fab3bb12b00 -1 Errors while parsing config file!
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open /etc/ceph/
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open /root/.
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 Errors while parsing config file!
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open /etc/ceph/
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open /root/.
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.831 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:25.831 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:25.831 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:25.835 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:25.835 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:25.835 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/
rbd: error opening image 80daf51d-
)
Warning FailedMapVolume 28s kubelet, stha8p0pm MapVolume.
2020-01-08 18:22:26.503 7f1961217b00 -1 Errors while parsing config file!
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open /etc/ceph/
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open /root/.
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 Errors while parsing config file!
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open /etc/ceph/
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open /root/.
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.523 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:26.523 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:26.523 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:26.527 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:26.527 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:26.527 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/
rbd: error opening image 80daf51d-
)
Warning FailedMapVolume 27s kubelet, stha8p0pm MapVolume.
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 Errors while parsing config file!
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open /etc/ceph/
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open /root/.
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 Errors while parsing config file!
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open /etc/ceph/
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open /root/.
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.723 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:27.727 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:27.727 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:27.731 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:27.731 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:27.731 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/
rbd: error opening image 80daf51d-
)
Warning FailedMapVolume 25s kubelet, stha8p0pm MapVolume.
2020-01-08 18:22:29.923 7f00195e1b00 -1 Errors while parsing config file!
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open /etc/ceph/
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open /root/.
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 Errors while parsing config file!
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open /etc/ceph/
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open /root/.
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.939 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:29.943 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:29.943 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:29.947 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:29.947 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:29.947 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/
rbd: error opening image 80daf51d-
)
Warning FailedMapVolume 20s kubelet, stha8p0pm MapVolume.
2020-01-08 18:22:34.047 7f456cd18b00 -1 Errors while parsing config file!
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open /etc/ceph/
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open /root/.
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 Errors while parsing config file!
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open /etc/ceph/
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open /root/.
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.063 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:34.067 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:34.067 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:34.071 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:34.071 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:34.071 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/
rbd: error opening image 80daf51d-
)
Warning FailedMapVolume 12s kubelet, stha8p0pm MapVolume.
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 Errors while parsing config file!
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open /etc/ceph/
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open /root/.
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 Errors while parsing config file!
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open /etc/ceph/
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open /root/.
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.211 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:42.215 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:42.215 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:42.215 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:42.219 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/
2020-01-08 18:22:42.219 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/
rbd: error opening image 80daf51d-
---
Here's the configMap for the deployment requesting this.
---
apiVersion: v1
kind: PersistentVolum
metadata:
name: cats-pv-block-claim
spec:
accessModes:
- ReadWriteMany
volumeMode: Block
resources:
requests:
storage: 3Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimes
labels:
app: cdk-cats-block
name: cdk-cats-block
spec:
replicas: 2
selector:
matchLabels:
app: cdk-cats-block
strategy: {}
template:
metadata:
creationT
labels:
app: cdk-cats-block
spec:
volumes:
- name: cats-pv-block
containers:
- image: calvinhartwell/
name: cdk-cats-block
ports:
- containerPort: 80
httpGet:
path: /
port: 80
resources: {}
- devicePath: "/dev/xvda"
name: cats-pv-block
restartPo
serviceAc
status: {}
---
apiVersion: v1
kind: Service
metadata:
name: cdk-cats-block
spec:
type: LoadBalancer
selector:
app: cdk-cats-block
ports:
- name: cdk-cats-block
protocol: TCP
port: 80
targetPort: 80
---
Also, some descriptions of the pvc and pv associated with this.
---
$ kubectl describe persistentvolum
Name: cats-pv-block-claim
Namespace: default
StorageClass: ceph-xfs
Status: Bound
Volume: pvc-fe4ade0b-
Labels: <none>
Annotations: kubectl.
Finalizers: [kubernetes.
Capacity: 3Gi
Access Modes: RWX
VolumeMode: Block
Mounted By: cdk-cats-
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ExternalProvisi
Normal Provisioning 2m3s rbd.csi.
Normal ProvisioningSuc
$ kubectl describe persistentvolum
Name: pvc-fe4ade0b-
Labels: <none>
Annotations: pv.kubernetes.
Finalizers: [kubernetes.
StorageClass: ceph-xfs
Status: Bound
Claim: default/
Reclaim Policy: Delete
Access Modes: RWX
VolumeMode: Block
Capacity: 3Gi
Node Affinity: <none>
Message:
Source:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: rbd.csi.ceph.com
VolumeHandle: csi-rbd-
ReadOnly: false
VolumeAttri
Events: <none>
---
I followed the ceph documentation on setting up this type of config by following https:/
summary: |
- when specifying volumeType Block for ReadWriteMany with Ceph related + when specifying volumeMode Block for ReadWriteMany with Ceph related ceph client config isn't being written to pods |
description: | updated |
description: | updated |
Changed in cdk-addons: | |
milestone: | none → 1.18 |
Changed in cdk-addons: | |
milestone: | 1.19 → none |
importance: | Undecided → Critical |
status: | Confirmed → Triaged |
Changed in cdk-addons: | |
importance: | Critical → High |
Changed in cdk-addons: | |
status: | Fix Committed → Fix Released |
The Ceph CSI support matrix[1] suggests that RBD supports Block mode RWX, so we should expect this case to work.
The bit about missing /etc/ceph/ceph.conf is a red herring. It's a Ceph CSI quirk. I see that message all the time in csi-rbdplugin logs even with successful PVs. The actual failure comes from the last line of the rbd output, which is:
rbd: error opening image 80daf51d- 2443-42ae- 9283-6f5fc27268 a3: (2) No such file or directory
Unfortunately, that error message is unclear, and likely distant from the root of the problem. This will need further investigation.
[1]: https:/ /github. com/ceph/ ceph-csi/ tree/4b8b52e0de 568002420fa38e8 dfdb5db4b714bc9 #ceph-csi- features- and-available- versions