| 2020-01-08 18:26:48 |
Jeff Hillman |
bug |
|
|
added bug |
| 2020-01-08 18:27:05 |
Jeff Hillman |
summary |
when specifying volumeType Block for ReadWriteMany with Ceph related ceph client config isn't being written to pods |
when specifying volumeMode Block for ReadWriteMany with Ceph related ceph client config isn't being written to pods |
|
| 2020-01-08 18:31:13 |
Jeff Hillman |
description |
K8s 1.18
Ceph 12.2.12
Ceph-mon related to k8s master as admin and client
Creating PVCs with ReadWriteOnce and volumeMode as Filesystem works as expected (with 1 pod at a time accessing the volume)
Creating PVCs with ReadWriteOnce and volumeMode as Block (so multiple mounts can happen), an error message is generated indicating that the /etc/ceph/ceph.conf (and other client files) are not provided to the pod. The PVC and the PV get created and bound to each other.
---
$ kubectl describe pod/cdk-cats-block-c446cdfcc-nrhzm
Name: cdk-cats-block-c446cdfcc-nrhzm
Namespace: default
Priority: 0
Node: stha8p0pm/10.109.14.69
Start Time: Wed, 08 Jan 2020 18:22:21 +0000
Labels: app=cdk-cats-block
pod-template-hash=c446cdfcc
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/cdk-cats-block-c446cdfcc
Containers:
cdk-cats-block:
Container ID:
Image: calvinhartwell/cdk-cats-block:latest
Image ID:
Port: 80/TCP
Host Port: 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Liveness: http-get http://:80/ delay=5s timeout=30s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-p4qq6 (ro)
Devices:
/dev/xvda from cats-pv-block
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
cats-pv-block:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: cats-pv-block-claim
ReadOnly: false
default-token-p4qq6:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-p4qq6
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling <unknown> default-scheduler error while running "VolumeBinding" filter plugin for pod "cdk-cats-block-c446cdfcc-nrhzm": pod has unbound immediate PersistentVolumeClaims
Warning FailedScheduling <unknown> default-scheduler error while running "VolumeBinding" filter plugin for pod "cdk-cats-block-c446cdfcc-nrhzm": pod has unbound immediate PersistentVolumeClaims
Normal Scheduled <unknown> default-scheduler Successfully assigned default/cdk-cats-block-c446cdfcc-nrhzm to stha8p0pm
Normal SuccessfulAttachVolume 33s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae"
Warning FailedMapVolume 29s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:25.811 7fab3bb12b00 -1 Errors while parsing config file!
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 Errors while parsing config file!
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.831 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.831 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.831 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.835 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.835 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.835 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
Warning FailedMapVolume 28s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:26.503 7f1961217b00 -1 Errors while parsing config file!
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 Errors while parsing config file!
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.523 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.523 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.523 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.527 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.527 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.527 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
Warning FailedMapVolume 27s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 Errors while parsing config file!
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 Errors while parsing config file!
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.723 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.727 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.727 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.731 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.731 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.731 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
Warning FailedMapVolume 25s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:29.923 7f00195e1b00 -1 Errors while parsing config file!
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 Errors while parsing config file!
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.939 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.943 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.943 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.947 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.947 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.947 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
Warning FailedMapVolume 20s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:34.047 7f456cd18b00 -1 Errors while parsing config file!
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 Errors while parsing config file!
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.063 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.067 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.067 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.071 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.071 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.071 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
Warning FailedMapVolume 12s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 Errors while parsing config file!
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 Errors while parsing config file!
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.211 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.215 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.215 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.215 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.219 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.219 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
---
Here's the configMap for the deployment requesting this.
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: cats-pv-block-claim
spec:
accessModes:
- ReadWriteMany
volumeMode: Block
resources:
requests:
storage: 3Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: cdk-cats-block
name: cdk-cats-block
spec:
replicas: 2
selector:
matchLabels:
app: cdk-cats-block
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: cdk-cats-block
spec:
volumes:
- name: cats-pv-block
persistentVolumeClaim:
claimName: cats-pv-block-claim
containers:
- image: calvinhartwell/cdk-cats-block:latest
imagePullPolicy: ""
name: cdk-cats-block
ports:
- containerPort: 80
livenessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 5
timeoutSeconds: 30
resources: {}
volumeDevices:
- devicePath: "/dev/xvda"
name: cats-pv-block
restartPolicy: Always
serviceAccountName: ""
status: {}
---
apiVersion: v1
kind: Service
metadata:
name: cdk-cats-block
spec:
type: LoadBalancer
selector:
app: cdk-cats-block
ports:
- name: cdk-cats-block
protocol: TCP
port: 80
targetPort: 80
---
Also, some descriptions of the pvc and pv associated with this.
---
$ kubectl describe persistentvolumeclaim/cats-pv-block-claim
Name: cats-pv-block-claim
Namespace: default
StorageClass: ceph-xfs
Status: Bound
Volume: pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"cats-pv-block-claim","namespace":"default"},"spec":...
pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: rbd.csi.ceph.com
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 3Gi
Access Modes: RWX
VolumeMode: Block
Mounted By: cdk-cats-block-c446cdfcc-nrhzm
cdk-cats-block-c446cdfcc-vx4sg
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ExternalProvisioning 2m3s (x2 over 2m3s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "rbd.csi.ceph.com" or manually created by system administrator
Normal Provisioning 2m3s rbd.csi.ceph.com_csi-rbdplugin-provisioner-0_69327706-3240-11ea-8efe-f6a7fad2373c External provisioner is provisioning volume for claim "default/cats-pv-block-claim"
Normal ProvisioningSucceeded 2m2s rbd.csi.ceph.com_csi-rbdplugin-provisioner-0_69327706-3240-11ea-8efe-f6a7fad2373c Successfully provisioned volume pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae
$ kubectl describe persistentvolume/pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae
Name: pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae
Labels: <none>
Annotations: pv.kubernetes.io/provisioned-by: rbd.csi.ceph.com
Finalizers: [kubernetes.io/pv-protection external-attacher/rbd-csi-ceph-com]
StorageClass: ceph-xfs
Status: Bound
Claim: default/cats-pv-block-claim
Reclaim Policy: Delete
Access Modes: RWX
VolumeMode: Block
Capacity: 3Gi
Node Affinity: <none>
Message:
Source:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: rbd.csi.ceph.com
VolumeHandle: csi-rbd-vol-cea8bce3-3243-11ea-b2fe-f6a7fad2373c
ReadOnly: false
VolumeAttributes: adminid=admin
fsType=xfs
imageFeatures=layering
imageFormat=2
monitors=10.109.12.10 10.109.12.8 10.109.12.9
pool=xfs-pool
storage.kubernetes.io/csiProvisionerIdentity=1578506281220-8081-
userid=admin
Events: <none>
---
I followed the ceph documentation on setting up this type of config by following https://docs.ceph.com/docs/master/rbd/rbd-kubernetes/#create-a-persistentvolumeclaim and it in no way mentions the need to manually configure the ceph client bits in the pod. |
K8s 1.18
Ceph 12.2.12
Ceph-mon related to k8s master as admin and client
Creating PVCs with ReadWriteOnce and volumeMode as Filesystem works as expected (with 1 pod at a time accessing the volume)
Creating PVCs with ReadWriteMany and volumeMode as Block (so multiple mounts can happen), an error message is generated indicating that the /etc/ceph/ceph.conf (and other client files) are not provided to the pod. The PVC and the PV get created and bound to each other.
---
$ kubectl describe pod/cdk-cats-block-c446cdfcc-nrhzm
Name: cdk-cats-block-c446cdfcc-nrhzm
Namespace: default
Priority: 0
Node: stha8p0pm/10.109.14.69
Start Time: Wed, 08 Jan 2020 18:22:21 +0000
Labels: app=cdk-cats-block
pod-template-hash=c446cdfcc
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/cdk-cats-block-c446cdfcc
Containers:
cdk-cats-block:
Container ID:
Image: calvinhartwell/cdk-cats-block:latest
Image ID:
Port: 80/TCP
Host Port: 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Liveness: http-get http://:80/ delay=5s timeout=30s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-p4qq6 (ro)
Devices:
/dev/xvda from cats-pv-block
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
cats-pv-block:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: cats-pv-block-claim
ReadOnly: false
default-token-p4qq6:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-p4qq6
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling <unknown> default-scheduler error while running "VolumeBinding" filter plugin for pod "cdk-cats-block-c446cdfcc-nrhzm": pod has unbound immediate PersistentVolumeClaims
Warning FailedScheduling <unknown> default-scheduler error while running "VolumeBinding" filter plugin for pod "cdk-cats-block-c446cdfcc-nrhzm": pod has unbound immediate PersistentVolumeClaims
Normal Scheduled <unknown> default-scheduler Successfully assigned default/cdk-cats-block-c446cdfcc-nrhzm to stha8p0pm
Normal SuccessfulAttachVolume 33s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae"
Warning FailedMapVolume 29s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:25.811 7fab3bb12b00 -1 Errors while parsing config file!
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 Errors while parsing config file!
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.831 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.831 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.831 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.835 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.835 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.835 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
Warning FailedMapVolume 28s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:26.503 7f1961217b00 -1 Errors while parsing config file!
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 Errors while parsing config file!
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.523 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.523 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.523 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.527 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.527 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.527 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
Warning FailedMapVolume 27s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 Errors while parsing config file!
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 Errors while parsing config file!
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.723 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.727 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.727 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.731 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.731 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.731 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
Warning FailedMapVolume 25s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:29.923 7f00195e1b00 -1 Errors while parsing config file!
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 Errors while parsing config file!
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.939 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.943 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.943 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.947 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.947 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.947 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
Warning FailedMapVolume 20s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:34.047 7f456cd18b00 -1 Errors while parsing config file!
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 Errors while parsing config file!
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.063 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.067 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.067 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.071 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.071 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.071 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
Warning FailedMapVolume 12s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 Errors while parsing config file!
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 Errors while parsing config file!
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.211 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.215 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.215 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.215 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.219 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.219 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
---
Here's the configMap for the deployment requesting this.
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: cats-pv-block-claim
spec:
accessModes:
- ReadWriteMany
volumeMode: Block
resources:
requests:
storage: 3Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: cdk-cats-block
name: cdk-cats-block
spec:
replicas: 2
selector:
matchLabels:
app: cdk-cats-block
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: cdk-cats-block
spec:
volumes:
- name: cats-pv-block
persistentVolumeClaim:
claimName: cats-pv-block-claim
containers:
- image: calvinhartwell/cdk-cats-block:latest
imagePullPolicy: ""
name: cdk-cats-block
ports:
- containerPort: 80
livenessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 5
timeoutSeconds: 30
resources: {}
volumeDevices:
- devicePath: "/dev/xvda"
name: cats-pv-block
restartPolicy: Always
serviceAccountName: ""
status: {}
---
apiVersion: v1
kind: Service
metadata:
name: cdk-cats-block
spec:
type: LoadBalancer
selector:
app: cdk-cats-block
ports:
- name: cdk-cats-block
protocol: TCP
port: 80
targetPort: 80
---
Also, some descriptions of the pvc and pv associated with this.
---
$ kubectl describe persistentvolumeclaim/cats-pv-block-claim
Name: cats-pv-block-claim
Namespace: default
StorageClass: ceph-xfs
Status: Bound
Volume: pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"cats-pv-block-claim","namespace":"default"},"spec":...
pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: rbd.csi.ceph.com
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 3Gi
Access Modes: RWX
VolumeMode: Block
Mounted By: cdk-cats-block-c446cdfcc-nrhzm
cdk-cats-block-c446cdfcc-vx4sg
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ExternalProvisioning 2m3s (x2 over 2m3s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "rbd.csi.ceph.com" or manually created by system administrator
Normal Provisioning 2m3s rbd.csi.ceph.com_csi-rbdplugin-provisioner-0_69327706-3240-11ea-8efe-f6a7fad2373c External provisioner is provisioning volume for claim "default/cats-pv-block-claim"
Normal ProvisioningSucceeded 2m2s rbd.csi.ceph.com_csi-rbdplugin-provisioner-0_69327706-3240-11ea-8efe-f6a7fad2373c Successfully provisioned volume pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae
$ kubectl describe persistentvolume/pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae
Name: pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae
Labels: <none>
Annotations: pv.kubernetes.io/provisioned-by: rbd.csi.ceph.com
Finalizers: [kubernetes.io/pv-protection external-attacher/rbd-csi-ceph-com]
StorageClass: ceph-xfs
Status: Bound
Claim: default/cats-pv-block-claim
Reclaim Policy: Delete
Access Modes: RWX
VolumeMode: Block
Capacity: 3Gi
Node Affinity: <none>
Message:
Source:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: rbd.csi.ceph.com
VolumeHandle: csi-rbd-vol-cea8bce3-3243-11ea-b2fe-f6a7fad2373c
ReadOnly: false
VolumeAttributes: adminid=admin
fsType=xfs
imageFeatures=layering
imageFormat=2
monitors=10.109.12.10 10.109.12.8 10.109.12.9
pool=xfs-pool
storage.kubernetes.io/csiProvisionerIdentity=1578506281220-8081-
userid=admin
Events: <none>
---
I followed the ceph documentation on setting up this type of config by following https://docs.ceph.com/docs/master/rbd/rbd-kubernetes/#create-a-persistentvolumeclaim and it in no way mentions the need to manually configure the ceph client bits in the pod. |
|
| 2020-01-08 19:00:26 |
Jeff Hillman |
description |
K8s 1.18
Ceph 12.2.12
Ceph-mon related to k8s master as admin and client
Creating PVCs with ReadWriteOnce and volumeMode as Filesystem works as expected (with 1 pod at a time accessing the volume)
Creating PVCs with ReadWriteMany and volumeMode as Block (so multiple mounts can happen), an error message is generated indicating that the /etc/ceph/ceph.conf (and other client files) are not provided to the pod. The PVC and the PV get created and bound to each other.
---
$ kubectl describe pod/cdk-cats-block-c446cdfcc-nrhzm
Name: cdk-cats-block-c446cdfcc-nrhzm
Namespace: default
Priority: 0
Node: stha8p0pm/10.109.14.69
Start Time: Wed, 08 Jan 2020 18:22:21 +0000
Labels: app=cdk-cats-block
pod-template-hash=c446cdfcc
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/cdk-cats-block-c446cdfcc
Containers:
cdk-cats-block:
Container ID:
Image: calvinhartwell/cdk-cats-block:latest
Image ID:
Port: 80/TCP
Host Port: 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Liveness: http-get http://:80/ delay=5s timeout=30s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-p4qq6 (ro)
Devices:
/dev/xvda from cats-pv-block
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
cats-pv-block:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: cats-pv-block-claim
ReadOnly: false
default-token-p4qq6:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-p4qq6
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling <unknown> default-scheduler error while running "VolumeBinding" filter plugin for pod "cdk-cats-block-c446cdfcc-nrhzm": pod has unbound immediate PersistentVolumeClaims
Warning FailedScheduling <unknown> default-scheduler error while running "VolumeBinding" filter plugin for pod "cdk-cats-block-c446cdfcc-nrhzm": pod has unbound immediate PersistentVolumeClaims
Normal Scheduled <unknown> default-scheduler Successfully assigned default/cdk-cats-block-c446cdfcc-nrhzm to stha8p0pm
Normal SuccessfulAttachVolume 33s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae"
Warning FailedMapVolume 29s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:25.811 7fab3bb12b00 -1 Errors while parsing config file!
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 Errors while parsing config file!
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.831 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.831 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.831 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.835 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.835 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.835 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
Warning FailedMapVolume 28s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:26.503 7f1961217b00 -1 Errors while parsing config file!
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 Errors while parsing config file!
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.523 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.523 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.523 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.527 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.527 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.527 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
Warning FailedMapVolume 27s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 Errors while parsing config file!
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 Errors while parsing config file!
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.723 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.727 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.727 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.731 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.731 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.731 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
Warning FailedMapVolume 25s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:29.923 7f00195e1b00 -1 Errors while parsing config file!
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 Errors while parsing config file!
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.939 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.943 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.943 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.947 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.947 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.947 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
Warning FailedMapVolume 20s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:34.047 7f456cd18b00 -1 Errors while parsing config file!
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 Errors while parsing config file!
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.063 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.067 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.067 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.071 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.071 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.071 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
Warning FailedMapVolume 12s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 Errors while parsing config file!
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 Errors while parsing config file!
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.211 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.215 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.215 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.215 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.219 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.219 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
---
Here's the configMap for the deployment requesting this.
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: cats-pv-block-claim
spec:
accessModes:
- ReadWriteMany
volumeMode: Block
resources:
requests:
storage: 3Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: cdk-cats-block
name: cdk-cats-block
spec:
replicas: 2
selector:
matchLabels:
app: cdk-cats-block
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: cdk-cats-block
spec:
volumes:
- name: cats-pv-block
persistentVolumeClaim:
claimName: cats-pv-block-claim
containers:
- image: calvinhartwell/cdk-cats-block:latest
imagePullPolicy: ""
name: cdk-cats-block
ports:
- containerPort: 80
livenessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 5
timeoutSeconds: 30
resources: {}
volumeDevices:
- devicePath: "/dev/xvda"
name: cats-pv-block
restartPolicy: Always
serviceAccountName: ""
status: {}
---
apiVersion: v1
kind: Service
metadata:
name: cdk-cats-block
spec:
type: LoadBalancer
selector:
app: cdk-cats-block
ports:
- name: cdk-cats-block
protocol: TCP
port: 80
targetPort: 80
---
Also, some descriptions of the pvc and pv associated with this.
---
$ kubectl describe persistentvolumeclaim/cats-pv-block-claim
Name: cats-pv-block-claim
Namespace: default
StorageClass: ceph-xfs
Status: Bound
Volume: pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"cats-pv-block-claim","namespace":"default"},"spec":...
pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: rbd.csi.ceph.com
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 3Gi
Access Modes: RWX
VolumeMode: Block
Mounted By: cdk-cats-block-c446cdfcc-nrhzm
cdk-cats-block-c446cdfcc-vx4sg
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ExternalProvisioning 2m3s (x2 over 2m3s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "rbd.csi.ceph.com" or manually created by system administrator
Normal Provisioning 2m3s rbd.csi.ceph.com_csi-rbdplugin-provisioner-0_69327706-3240-11ea-8efe-f6a7fad2373c External provisioner is provisioning volume for claim "default/cats-pv-block-claim"
Normal ProvisioningSucceeded 2m2s rbd.csi.ceph.com_csi-rbdplugin-provisioner-0_69327706-3240-11ea-8efe-f6a7fad2373c Successfully provisioned volume pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae
$ kubectl describe persistentvolume/pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae
Name: pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae
Labels: <none>
Annotations: pv.kubernetes.io/provisioned-by: rbd.csi.ceph.com
Finalizers: [kubernetes.io/pv-protection external-attacher/rbd-csi-ceph-com]
StorageClass: ceph-xfs
Status: Bound
Claim: default/cats-pv-block-claim
Reclaim Policy: Delete
Access Modes: RWX
VolumeMode: Block
Capacity: 3Gi
Node Affinity: <none>
Message:
Source:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: rbd.csi.ceph.com
VolumeHandle: csi-rbd-vol-cea8bce3-3243-11ea-b2fe-f6a7fad2373c
ReadOnly: false
VolumeAttributes: adminid=admin
fsType=xfs
imageFeatures=layering
imageFormat=2
monitors=10.109.12.10 10.109.12.8 10.109.12.9
pool=xfs-pool
storage.kubernetes.io/csiProvisionerIdentity=1578506281220-8081-
userid=admin
Events: <none>
---
I followed the ceph documentation on setting up this type of config by following https://docs.ceph.com/docs/master/rbd/rbd-kubernetes/#create-a-persistentvolumeclaim and it in no way mentions the need to manually configure the ceph client bits in the pod. |
K8s 1.17
Ceph 12.2.12
Ceph-mon related to k8s master as admin and client
Creating PVCs with ReadWriteOnce and volumeMode as Filesystem works as expected (with 1 pod at a time accessing the volume)
Creating PVCs with ReadWriteMany and volumeMode as Block (so multiple mounts can happen), an error message is generated indicating that the /etc/ceph/ceph.conf (and other client files) are not provided to the pod. The PVC and the PV get created and bound to each other.
---
$ kubectl describe pod/cdk-cats-block-c446cdfcc-nrhzm
Name: cdk-cats-block-c446cdfcc-nrhzm
Namespace: default
Priority: 0
Node: stha8p0pm/10.109.14.69
Start Time: Wed, 08 Jan 2020 18:22:21 +0000
Labels: app=cdk-cats-block
pod-template-hash=c446cdfcc
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/cdk-cats-block-c446cdfcc
Containers:
cdk-cats-block:
Container ID:
Image: calvinhartwell/cdk-cats-block:latest
Image ID:
Port: 80/TCP
Host Port: 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Liveness: http-get http://:80/ delay=5s timeout=30s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-p4qq6 (ro)
Devices:
/dev/xvda from cats-pv-block
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
cats-pv-block:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: cats-pv-block-claim
ReadOnly: false
default-token-p4qq6:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-p4qq6
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling <unknown> default-scheduler error while running "VolumeBinding" filter plugin for pod "cdk-cats-block-c446cdfcc-nrhzm": pod has unbound immediate PersistentVolumeClaims
Warning FailedScheduling <unknown> default-scheduler error while running "VolumeBinding" filter plugin for pod "cdk-cats-block-c446cdfcc-nrhzm": pod has unbound immediate PersistentVolumeClaims
Normal Scheduled <unknown> default-scheduler Successfully assigned default/cdk-cats-block-c446cdfcc-nrhzm to stha8p0pm
Normal SuccessfulAttachVolume 33s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae"
Warning FailedMapVolume 29s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:25.811 7fab3bb12b00 -1 Errors while parsing config file!
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 Errors while parsing config file!
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.811 7fab3bb12b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:25.831 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.831 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.831 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.835 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.835 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:25.835 7fab3bb12b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
Warning FailedMapVolume 28s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:26.503 7f1961217b00 -1 Errors while parsing config file!
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 Errors while parsing config file!
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.503 7f1961217b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:26.523 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.523 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.523 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.527 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.527 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:26.527 7f1961217b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
Warning FailedMapVolume 27s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 Errors while parsing config file!
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 Errors while parsing config file!
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.707 7f44a6b4ab00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:27.723 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.727 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.727 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.731 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.731 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:27.731 7f44a6b4ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
Warning FailedMapVolume 25s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:29.923 7f00195e1b00 -1 Errors while parsing config file!
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 Errors while parsing config file!
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.923 7f00195e1b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:29.939 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.943 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.943 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.947 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.947 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:29.947 7f00195e1b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
Warning FailedMapVolume 20s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:34.047 7f456cd18b00 -1 Errors while parsing config file!
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 Errors while parsing config file!
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.047 7f456cd18b00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:34.063 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.067 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.067 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.071 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.071 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:34.071 7f456cd18b00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
)
Warning FailedMapVolume 12s kubelet, stha8p0pm MapVolume.MapPodDevice failed for volume "pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae" : kubernetes.io/csi: blockMapper.publishVolumeForBlock failed: rpc error: code = Unknown desc = fail to check rbd image status with: (exit status 2), rbd output: (did not load config file, using default settings.
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 Errors while parsing config file!
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 Errors while parsing config file!
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.195 7fbb35d6ab00 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2020-01-08 18:22:42.211 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.215 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.215 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.215 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.219 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2020-01-08 18:22:42.219 7fbb35d6ab00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
rbd: error opening image 80daf51d-2443-42ae-9283-6f5fc27268a3: (2) No such file or directory
---
Here's the configMap for the deployment requesting this.
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: cats-pv-block-claim
spec:
accessModes:
- ReadWriteMany
volumeMode: Block
resources:
requests:
storage: 3Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: cdk-cats-block
name: cdk-cats-block
spec:
replicas: 2
selector:
matchLabels:
app: cdk-cats-block
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: cdk-cats-block
spec:
volumes:
- name: cats-pv-block
persistentVolumeClaim:
claimName: cats-pv-block-claim
containers:
- image: calvinhartwell/cdk-cats-block:latest
imagePullPolicy: ""
name: cdk-cats-block
ports:
- containerPort: 80
livenessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 5
timeoutSeconds: 30
resources: {}
volumeDevices:
- devicePath: "/dev/xvda"
name: cats-pv-block
restartPolicy: Always
serviceAccountName: ""
status: {}
---
apiVersion: v1
kind: Service
metadata:
name: cdk-cats-block
spec:
type: LoadBalancer
selector:
app: cdk-cats-block
ports:
- name: cdk-cats-block
protocol: TCP
port: 80
targetPort: 80
---
Also, some descriptions of the pvc and pv associated with this.
---
$ kubectl describe persistentvolumeclaim/cats-pv-block-claim
Name: cats-pv-block-claim
Namespace: default
StorageClass: ceph-xfs
Status: Bound
Volume: pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"cats-pv-block-claim","namespace":"default"},"spec":...
pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: rbd.csi.ceph.com
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 3Gi
Access Modes: RWX
VolumeMode: Block
Mounted By: cdk-cats-block-c446cdfcc-nrhzm
cdk-cats-block-c446cdfcc-vx4sg
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ExternalProvisioning 2m3s (x2 over 2m3s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "rbd.csi.ceph.com" or manually created by system administrator
Normal Provisioning 2m3s rbd.csi.ceph.com_csi-rbdplugin-provisioner-0_69327706-3240-11ea-8efe-f6a7fad2373c External provisioner is provisioning volume for claim "default/cats-pv-block-claim"
Normal ProvisioningSucceeded 2m2s rbd.csi.ceph.com_csi-rbdplugin-provisioner-0_69327706-3240-11ea-8efe-f6a7fad2373c Successfully provisioned volume pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae
$ kubectl describe persistentvolume/pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae
Name: pvc-fe4ade0b-5b41-48de-9950-6e4760eb10ae
Labels: <none>
Annotations: pv.kubernetes.io/provisioned-by: rbd.csi.ceph.com
Finalizers: [kubernetes.io/pv-protection external-attacher/rbd-csi-ceph-com]
StorageClass: ceph-xfs
Status: Bound
Claim: default/cats-pv-block-claim
Reclaim Policy: Delete
Access Modes: RWX
VolumeMode: Block
Capacity: 3Gi
Node Affinity: <none>
Message:
Source:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: rbd.csi.ceph.com
VolumeHandle: csi-rbd-vol-cea8bce3-3243-11ea-b2fe-f6a7fad2373c
ReadOnly: false
VolumeAttributes: adminid=admin
fsType=xfs
imageFeatures=layering
imageFormat=2
monitors=10.109.12.10 10.109.12.8 10.109.12.9
pool=xfs-pool
storage.kubernetes.io/csiProvisionerIdentity=1578506281220-8081-
userid=admin
Events: <none>
---
I followed the ceph documentation on setting up this type of config by following https://docs.ceph.com/docs/master/rbd/rbd-kubernetes/#create-a-persistentvolumeclaim and it in no way mentions the need to manually configure the ceph client bits in the pod. |
|
| 2020-01-09 15:57:52 |
Jeff Hillman |
summary |
when specifying volumeMode Block for ReadWriteMany with Ceph related ceph client config isn't being written to pods |
when specifying volumeMode Block with Ceph related ceph client config isn't being written to pods |
|
| 2020-01-09 22:17:51 |
George Kraft |
cdk-addons: status |
New |
Confirmed |
|
| 2020-01-10 16:04:28 |
George Kraft |
cdk-addons: milestone |
|
1.18 |
|
| 2020-03-31 15:16:01 |
George Kraft |
cdk-addons: milestone |
1.18 |
1.19 |
|
| 2020-05-15 15:33:29 |
George Kraft |
cdk-addons: milestone |
1.19 |
|
|
| 2020-05-15 15:36:17 |
George Kraft |
cdk-addons: importance |
Undecided |
Critical |
|
| 2020-05-15 15:36:20 |
George Kraft |
cdk-addons: status |
Confirmed |
Triaged |
|
| 2020-05-20 15:29:46 |
George Kraft |
cdk-addons: importance |
Critical |
High |
|
| 2020-08-05 19:12:32 |
Cory Johns |
cdk-addons: status |
Triaged |
Fix Committed |
|
| 2020-08-05 19:12:40 |
Cory Johns |
cdk-addons: assignee |
|
Joseph Borg (joeborg) |
|
| 2020-08-05 19:12:41 |
Cory Johns |
cdk-addons: milestone |
|
1.19 |
|
| 2020-10-02 19:33:05 |
Tim Van Steenburgh |
cdk-addons: status |
Fix Committed |
Fix Released |
|
