pvc charm-multus-operator-0 stuck in pending state

[kashif nawaz]

contrail@maas:~$ kubectl get sc
2022/02/22 18:39:18.907688 system_key.go:253: running from non-installed location /home/contrail/bin/kubectl: ignoring system-key
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
ceph-ext4 rbd.csi.ceph.com Delete Immediate true 3h
ceph-xfs rbd.csi.ceph.com Delete Immediate true 3h
cephfs (default) cephfs.csi.ceph.com Delete Immediate true 3h
contrail@maas:~$ kubectl get po
2022/02/22 18:39:22.688507 system_key.go:253: running from non-installed location /home/contrail/bin/kubectl: ignoring system-key
NAME READY STATUS RESTARTS AGE
csi-cephfsplugin-2fwgp 3/3 Running 0 169m
csi-cephfsplugin-67pdf 3/3 Running 0 170m
csi-cephfsplugin-provisioner-6f7ffd894d-hdrkj 6/6 Running 0 179m
csi-cephfsplugin-provisioner-6f7ffd894d-lv8w5 6/6 Running 0 179m
csi-cephfsplugin-provisioner-6f7ffd894d-rqlwn 0/6 Pending 0 179m
csi-rbdplugin-5887t 3/3 Running 0 169m
csi-rbdplugin-84x4z 3/3 Running 0 170m
csi-rbdplugin-provisioner-7dffff7f4d-4ftk9 0/7 Pending 0 179m
csi-rbdplugin-provisioner-7dffff7f4d-7llfs 7/7 Running 0 179m
csi-rbdplugin-provisioner-7dffff7f4d-v4btq 7/7 Running 0 179m
contrail@maas:~$

contrail@maas:~$ kubectl get pv
2022/02/22 18:38:53.254479 system_key.go:253: running from non-installed location /home/contrail/bin/kubectl: ignoring system-key
No resources found

contrail@maas:~$ kubectl get pvc --all-namespaces
2022/02/22 18:39:04.108593 system_key.go:253: running from non-installed location /home/contrail/bin/kubectl: ignoring system-key
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
my-k8s-model charm-multus-operator-0 Pending cephfs 168m

contrail@maas:~$ kubectl get pods --all-namespaces
2022/02/22 18:37:55.747946 system_key.go:253: running from non-installed location /home/contrail/bin/kubectl: ignoring system-key
NAMESPACE NAME READY STATUS RESTARTS AGE
default csi-cephfsplugin-2fwgp 3/3 Running 0 168m
default csi-cephfsplugin-67pdf 3/3 Running 0 169m
default csi-cephfsplugin-provisioner-6f7ffd894d-hdrkj 6/6 Running 0 177m
default csi-cephfsplugin-provisioner-6f7ffd894d-lv8w5 6/6 Running 0 177m
default csi-cephfsplugin-provisioner-6f7ffd894d-rqlwn 0/6 Pending 0 177m
default csi-rbdplugin-5887t 3/3 Running 0 168m
default csi-rbdplugin-84x4z 3/3 Running 0 169m
default csi-rbdplugin-provisioner-7dffff7f4d-4ftk9 0/7 Pending 0 177m
default csi-rbdplugin-provisioner-7dffff7f4d-7llfs 7/7 Running 0 177m
default csi-rbdplugin-provisioner-7dffff7f4d-v4btq 7/7 Running 0 177m
ingress-nginx-kubernetes-worker default-http-backend-kubernetes-worker-6cd58d8886-sqrrj 1/1 Running 0 177m
ingress-nginx-kubernetes-worker nginx-ingress-controller-kubernetes-worker-hbm8z 1/1 Running 0 177m
ingress-nginx-kubernetes-worker nginx-ingress-controller-kubernetes-worker-j4b5v 1/1 Running 0 177m
kube-system coredns-5564855696-w978n 1/1 Running 0 177m
kube-system kube-state-metrics-5ccbcf64d5-fbvw7 1/1 Running 0 177m
kube-system metrics-server-v0.5.1-79b4746b65-4krk4 2/2 Running 0 177m
kubernetes-dashboard dashboard-metrics-scraper-5cd54464bf-qdr97 1/1 Running 0 177m
kubernetes-dashboard kubernetes-dashboard-55796c99c-5gnb8 1/1 Running 0 177m
my-k8s-model modeloperator-c7b46b94b-plkfl 1/1 Running 0 167m
my-k8s-model multus-operator-0 0/1 Pending 0 167m

contrail@maas:~$ kubectl describe pvc charm-multus-operator-0 -n my-k8s-model
2022/02/22 18:36:36.590891 system_key.go:253: running from non-installed location /home/contrail/bin/kubectl: ignoring system-key
Name: charm-multus-operator-0
Namespace: my-k8s-model
StorageClass: cephfs
Status: Pending
Volume:
Labels: operator.juju.is/name=multus
               operator.juju.is/target=application
Annotations: controller.juju.is/id: bf08dd9d-e631-4bb2-80f8-16a186427ca1
               model.juju.is/id: ffaddd5f-e77a-4488-8e59-b7f4f57a3c63
               volume.beta.kubernetes.io/storage-provisioner: cephfs.csi.ceph.com
               volume.kubernetes.io/storage-provisioner: cephfs.csi.ceph.com
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: multus-operator-0
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal ExternalProvisioning 64s (x682 over 166m) persistentvolume-controller waiting for a volume to be created, either by external provisioner "cephfs.csi.ceph.com" or manually created by system administrator
  Normal Provisioning 15s (x42 over 166m) cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-6f7ffd894d-hdrkj_909f245f-3d16-419d-b18a-fa97c65f2304 External provisioner is provisioning volume for claim "my-k8s-model/charm-multus-operator-0"

contrail@maas:~$ kubectl describe pod multus-operator-0 -n my-k8s-model
2022/02/22 18:38:31.390847 system_key.go:253: running from non-installed location /home/contrail/bin/kubectl: ignoring system-key
Name: multus-operator-0
Namespace: my-k8s-model
Priority: 0
Node: <none>
Labels: controller-revision-hash=multus-operator-5cfbcd7468
                operator.juju.is/name=multus
                operator.juju.is/target=application
                statefulset.kubernetes.io/pod-name=multus-operator-0
Annotations: apparmor.security.beta.kubernetes.io/pod: runtime/default
                controller.juju.is/id: bf08dd9d-e631-4bb2-80f8-16a186427ca1
                juju.is/version: 2.9.22
                kubernetes.io/psp: privileged
                model.juju.is/id: ffaddd5f-e77a-4488-8e59-b7f4f57a3c63
                seccomp.security.beta.kubernetes.io/pod: docker/default
Status: Pending
IP:
IPs: <none>
Controlled By: StatefulSet/multus-operator
Containers:
  juju-operator:
    Image: jujusolutions/jujud-operator:2.9.22
    Port: <none>
    Host Port: <none>
    Command:
      /bin/sh
    Args:
      -c
      export JUJU_DATA_DIR=/var/lib/juju
      export JUJU_TOOLS_DIR=$JUJU_DATA_DIR/tools

      mkdir -p $JUJU_TOOLS_DIR
      cp /opt/jujud $JUJU_TOOLS_DIR/jujud

      $JUJU_TOOLS_DIR/jujud caasoperator --application-name=multus --debug

    Environment:
      JUJU_APPLICATION: multus
      JUJU_OPERATOR_SERVICE_IP: 10.104.163.187
      JUJU_OPERATOR_POD_IP: (v1:status.podIP)
      JUJU_OPERATOR_NAMESPACE: my-k8s-model (v1:metadata.namespace)
    Mounts:
      /var/lib/juju/agents from charm (rw)
      /var/lib/juju/agents/application-multus/operator.yaml from multus-operator-config (rw,path="operator.yaml")
      /var/lib/juju/agents/application-multus/template-agent.conf from multus-operator-config (rw,path="template-agent.conf")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lx8s9 (ro)
Conditions:
  Type Status
  PodScheduled False
Volumes:
  charm:
    Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName: charm-multus-operator-0
    ReadOnly: false
  multus-operator-config:
    Type: ConfigMap (a volume populated by a ConfigMap)
    Name: multus-operator-config
    Optional: false
  kube-api-access-lx8s9:
    Type: Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds: 3607
    ConfigMapName: kube-root-ca.crt
    ConfigMapOptional: <nil>
    DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Warning FailedScheduling 2m29s (x136 over 168m) default-scheduler 0/2 nodes are available: 2 pod has unbound immediate PersistentVolumeClaims.

[kashif nawaz (knawaz)]

contrail@maas:~$
contrail@maas:~$ kubectl describe pvc charm-multus-operator-0 -n my-k8s-model
2022/02/22 19:24:24.455510 system_key.go:253: running from non-installed location /home/contrail/bin/kubectl: ignoring system-key
Name: charm-multus-operator-0
Namespace: my-k8s-model
StorageClass: cephfs
Status: Pending
Volume:
Labels: operator.juju.is/name=multus
               operator.juju.is/target=application
Annotations: controller.juju.is/id: bf08dd9d-e631-4bb2-80f8-16a186427ca1
               model.juju.is/id: ffaddd5f-e77a-4488-8e59-b7f4f57a3c63
               volume.beta.kubernetes.io/storage-provisioner: cephfs.csi.ceph.com
               volume.kubernetes.io/storage-provisioner: cephfs.csi.ceph.com
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: multus-operator-0
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Warning ProvisioningFailed 7m9s cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-6f7ffd894d-hdrkj_909f245f-3d16-419d-b18a-fa97c65f2304 failed to provision volume with StorageClass "cephfs": rpc error: code = DeadlineExceeded desc = context deadline exceeded
  Normal ExternalProvisioning 3m52s (x26 over 9m39s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "cephfs.csi.ceph.com" or manually created by system administrator
  Normal Provisioning 2m55s (x10 over 9m39s) cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-6f7ffd894d-hdrkj_909f245f-3d16-419d-b18a-fa97c65f2304 External provisioner is provisioning volume for claim "my-k8s-model/charm-multus-operator-0"
  Warning ProvisioningFailed 2m55s (x9 over 7m9s) cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-6f7ffd894d-hdrkj_909f245f-3d16-419d-b18a-fa97c65f2304 failed to provision volume with StorageClass "cephfs": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-1f09ceae-f78b-4825-b76a-2668437bc5c5 already exists
contrail@maas:~$

[kashif nawaz (knawaz)]

issue was related to ceph.. once ceph cluster got fixed ,, this issue also got resolved..