Next iteration of the test case (apply RR patch), in one subcloud, stx-oidc-client pod is in "unknown" state. kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE cert-manager cm-cert-manager-7b8b94bf9f-qjc7x 1/1 Running 2 7h10m cert-manager cm-cert-manager-cainjector-56b68989b5-qqq2p 1/1 Running 6 7h10m cert-manager cm-cert-manager-webhook-7d5c897795-kpzgx 1/1 Running 2 7h10m default small-7497946d9-24rlb 1/1 Running 3 2d4h default small-7497946d9-2cn2r 1/1 Running 3 2d4h default small-7497946d9-2jvfc 1/1 Running 3 2d4h default small-7497946d9-492nk 1/1 Running 3 2d4h default small-7497946d9-4jvtv 1/1 Running 3 2d4h default small-7497946d9-4lzbc 1/1 Running 3 2d4h default small-7497946d9-4nc99 1/1 Running 3 2d4h default small-7497946d9-5mcvf 1/1 Running 3 2d4h default small-7497946d9-64xqv 1/1 Running 3 2d4h default small-7497946d9-6tmb7 1/1 Running 3 2d4h default small-7497946d9-84khl 1/1 Running 3 2d4h default small-7497946d9-895qk 1/1 Running 3 2d4h default small-7497946d9-8ks9m 1/1 Running 3 2d4h default small-7497946d9-922xz 1/1 Running 3 2d4h default small-7497946d9-b58bl 1/1 Running 3 2d4h default small-7497946d9-bhw82 1/1 Running 3 2d4h default small-7497946d9-cdrnr 1/1 Running 3 2d4h default small-7497946d9-cr7br 1/1 Running 3 2d4h default small-7497946d9-jwgvl 1/1 Running 3 2d4h default small-7497946d9-l745j 1/1 Running 3 2d4h default small-7497946d9-lfhvs 1/1 Running 3 2d4h default small-7497946d9-m6hvj 1/1 Running 3 2d4h default small-7497946d9-mjdrj 1/1 Running 3 2d4h default small-7497946d9-mkzmn 1/1 Running 3 2d4h default small-7497946d9-mmh69 1/1 Running 3 2d4h default small-7497946d9-mzvf6 1/1 Running 3 2d4h default small-7497946d9-pflxz 1/1 Running 4 2d4h default small-7497946d9-rs9wb 1/1 Running 3 2d4h default small-7497946d9-s2st4 1/1 Running 3 2d4h default small-7497946d9-vm99s 1/1 Running 3 2d4h kube-system calico-kube-controllers-5cd4695574-h5mkx 1/1 Running 8 3d9h kube-system calico-node-wt2cq 1/1 Running 5 3d9h kube-system coredns-7fc965fbd7-hbh92 1/1 Running 4 3d9h kube-system ic-nginx-ingress-controller-rhw4d 1/1 Running 2 7h10m kube-system ic-nginx-ingress-default-backend-5ffcfd7744-56kfq 1/1 Running 2 7h10m kube-system kube-apiserver-controller-0 1/1 Running 3 2d23h kube-system kube-controller-manager-controller-0 1/1 Running 7 3d9h kube-system kube-multus-ds-amd64-pb9nb 1/1 Running 6 3d9h kube-system kube-proxy-qdrpd 1/1 Running 4 3d9h kube-system kube-scheduler-controller-0 1/1 Running 7 3d9h kube-system kube-sriov-cni-ds-amd64-22xtq 1/1 Running 4 3d9h kube-system kube-sriov-device-plugin-amd64-m8f9b 1/1 Running 2 7h10m kube-system oidc-dex-6585f5f9bc-r2rh6 1/1 Running 2 7h10m kube-system stx-oidc-client-6c8cfc5f65-8bbwc 0/1 Unknown 0 2d23h kube-system tiller-deploy-5c8dd9fb56-x9cq4 1/1 Running 5 3d9h monitor mon-filebeat-fhr8t 1/1 Running 3 2d5h monitor mon-kube-state-metrics-7c7f79f45c-542xc 1/1 Running 4 7h10m monitor mon-logstash-0 1/1 Running 3 7h10m monitor mon-metricbeat-l5ph4 1/1 Running 3 2d5h monitor mon-metricbeat-metrics-679dddb9fc-z2rcc 1/1 Running 2 7h10m platform-deployment-manager platform-deployment-manager-0 2/2 Running 13 3d9h Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning ProbeWarning 7h22m (x23189 over 2d23h) kubelet, controller-0 Readiness probe warning: See Other. Warning NodeAffinity 7h11m kubelet, controller-0 Predicate NodeAffinity failed Warning FailedMount 7h11m kubelet, controller-0 MountVolume.SetUp failed for volume "dex-client-secret-volume" : failed to sync secret cache: timed out waiting for the condition Warning FailedMount 7h11m kubelet, controller-0 MountVolume.SetUp failed for volume "https-tls" : failed to sync secret cache: timed out waiting for the condition Warning FailedMount 7h11m (x2 over 7h11m) kubelet, controller-0 MountVolume.SetUp failed for volume "config" : failed to sync configmap cache: timed out waiting for the condition Warning FailedMount 6h58m (x3 over 7h7m) kubelet, controller-0 Unable to attach or mount volumes: unmounted volumes=[config], unattached volumes=[config default-token-snk48 dex-client-secret-volume https-tls]: timed out waiting for the condition Warning FailedMount 6h56m (x2 over 7h) kubelet, controller-0 Unable to attach or mount volumes: unmounted volumes=[config], unattached volumes=[default-token-snk48 dex-client-secret-volume https-tls config]: timed out waiting for the condition Warning FailedMount 6h25m (x12 over 7h9m) kubelet, controller-0 Unable to attach or mount volumes: unmounted volumes=[config], unattached volumes=[dex-client-secret-volume https-tls config default-token-snk48]: timed out waiting for the condition Warning FailedMount 6h5m (x38 over 7h11m) kubelet, controller-0 MountVolume.SetUp failed for volume "config" : stat /var/lib/kubelet/pods/758ce0d3-867b-43ab-a683-8b5ac29d5547/volumes/kubernetes.io~configmap/config: no such file or directory Warning FailedMount 5h55m kubelet, controller-0 MountVolume.SetUp failed for volume "dex-client-secret-volume" : failed to sync secret cache: timed out waiting for the condition Warning FailedMount 5h55m kubelet, controller-0 MountVolume.SetUp failed for volume "https-tls" : failed to sync secret cache: timed out waiting for the condition Warning FailedMount 5h55m kubelet, controller-0 MountVolume.SetUp failed for volume "config" : failed to sync configmap cache: timed out waiting for the condition Warning FailedMount 5h55m kubelet, controller-0 MountVolume.SetUp failed for volume "config" : stat /var/lib/kubelet/pods/758ce0d3-867b-43ab-a683-8b5ac29d5547/volumes/kubernetes.io~configmap/config: no such file or directory Warning FailedMount 5h55m (x2 over 5h55m) kubelet, controller-0 MountVolume.SetUp failed for volume "dex-client-secret-volume" : failed to sync secret cache: timed out waiting for the condition Warning FailedMount 5h55m (x2 over 5h55m) kubelet, controller-0 MountVolume.SetUp failed for volume "https-tls" : failed to sync secret cache: timed out waiting for the condition Warning FailedMount 5h55m (x2 over 5h55m) kubelet, controller-0 MountVolume.SetUp failed for volume "config" : failed to sync configmap cache: timed out waiting for the condition Warning FailedMount 99m (x22 over 5h44m) kubelet, controller-0 Unable to attach or mount volumes: unmounted volumes=[config], unattached volumes=[https-tls config default-token-snk48 dex-client-secret-volume]: timed out waiting for the condition Warning FailedMount 49m (x156 over 5h55m) kubelet, controller-0 MountVolume.SetUp failed for volume "config" : stat /var/lib/kubelet/pods/758ce0d3-867b-43ab-a683-8b5ac29d5547/volumes/kubernetes.io~configmap/config: no such file or directory Warning FailedMount 44m (x32 over 5h42m) kubelet, controller-0 Unable to attach or mount volumes: unmounted volumes=[config], unattached volumes=[default-token-snk48 dex-client-secret-volume https-tls config]: timed out waiting for the condition Warning FailedMount 29m (x22 over 5h53m) kubelet, controller-0 Unable to attach or mount volumes: unmounted volumes=[config], unattached volumes=[config default-token-snk48 dex-client-secret-volume https-tls]: timed out waiting for the condition Warning FailedMount 19m (x80 over 5h48m) kubelet, controller-0 Unable to attach or mount volumes: unmounted volumes=[config], unattached volumes=[dex-client-secret-volume https-tls config default-token-snk48]: timed out waiting for the condition Warning FailedMount 8m42s kubelet, controller-0 MountVolume.SetUp failed for volume "https-tls" : failed to sync secret cache: timed out waiting for the condition Warning FailedMount 8m42s kubelet, controller-0 MountVolume.SetUp failed for volume "config" : failed to sync configmap cache: timed out waiting for the condition Warning FailedMount 8m42s kubelet, controller-0 MountVolume.SetUp failed for volume "dex-client-secret-volume" : failed to sync secret cache: timed out waiting for the condition Warning FailedMount 8m9s kubelet, controller-0 MountVolume.SetUp failed for volume "dex-client-secret-volume" : failed to sync secret cache: timed out waiting for the condition Warning FailedMount 8m8s (x2 over 8m9s) kubelet, controller-0 MountVolume.SetUp failed for volume "https-tls" : failed to sync secret cache: timed out waiting for the condition Warning FailedMount 8m8s (x2 over 8m9s) kubelet, controller-0 MountVolume.SetUp failed for volume "config" : failed to sync configmap cache: timed out waiting for the condition Warning FailedMount 6m7s kubelet, controller-0 Unable to attach or mount volumes: unmounted volumes=[config], unattached volumes=[default-token-snk48 dex-client-secret-volume https-tls config]: timed out waiting for the condition Warning FailedMount 4m4s kubelet, controller-0 Unable to attach or mount volumes: unmounted volumes=[config], unattached volumes=[https-tls config default-token-snk48 dex-client-secret-volume]: timed out waiting for the condition Warning FailedMount 2m kubelet, controller-0 Unable to attach or mount volumes: unmounted volumes=[config], unattached volumes=[dex-client-secret-volume https-tls config default-token-snk48]: timed out waiting for the condition Warning FailedMount 116s (x9 over 8m7s) kubelet, controller-0 MountVolume.SetUp failed for volume "config" : stat /var/lib/kubelet/pods/758ce0d3-867b-43ab-a683-8b5ac29d5547/volumes/kubernetes.io~configmap/config: no such file or directory