Seen the issue with stx-monitor pods after RR patch removed. Seen it in few subclouds: kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE cert-manager cm-cert-manager-7b8b94bf9f-khwbt 1/1 Running 1 6h11m cert-manager cm-cert-manager-cainjector-56b68989b5-fp75x 1/1 Running 3 6h11m cert-manager cm-cert-manager-webhook-7d5c897795-pqflz 1/1 Running 1 6h11m default small-7497946d9-2k4wd 1/1 Running 2 2d3h default small-7497946d9-58qmd 1/1 Running 2 2d3h default small-7497946d9-5jvqx 1/1 Running 2 2d3h default small-7497946d9-5p7ng 1/1 Running 2 2d3h default small-7497946d9-7cpzb 1/1 Running 2 2d3h default small-7497946d9-88hwf 1/1 Running 2 2d3h default small-7497946d9-8p75z 1/1 Running 2 2d3h default small-7497946d9-9sv8m 1/1 Running 2 2d3h default small-7497946d9-b7h8g 1/1 Running 2 2d3h default small-7497946d9-bxpdn 1/1 Running 2 2d3h default small-7497946d9-dljff 1/1 Running 2 2d3h default small-7497946d9-fvmf2 1/1 Running 2 2d3h default small-7497946d9-fxq9s 1/1 Running 2 2d3h default small-7497946d9-h7rj8 1/1 Running 2 2d3h default small-7497946d9-jgxcm 1/1 Running 2 2d3h default small-7497946d9-k8ntz 1/1 Running 2 2d3h default small-7497946d9-kk2jh 1/1 Running 2 2d3h default small-7497946d9-mtt4m 1/1 Running 2 2d3h default small-7497946d9-p42l7 1/1 Running 2 2d3h default small-7497946d9-pjq57 1/1 Running 2 2d3h default small-7497946d9-pp5dm 1/1 Running 2 2d3h default small-7497946d9-q4vrs 1/1 Running 2 2d3h default small-7497946d9-rnt24 1/1 Running 2 2d3h default small-7497946d9-rppsj 1/1 Running 2 2d3h default small-7497946d9-vmmnw 1/1 Running 2 2d3h default small-7497946d9-wnzql 1/1 Running 2 2d3h default small-7497946d9-xpt94 1/1 Running 2 2d3h default small-7497946d9-xtm2c 1/1 Running 2 2d3h default small-7497946d9-z9gq9 1/1 Running 2 2d3h default small-7497946d9-zz4zj 1/1 Running 2 2d3h kube-system calico-kube-controllers-5cd4695574-kg45c 1/1 Running 6 4d kube-system calico-node-9h2t5 1/1 Running 3 4d kube-system coredns-7fc965fbd7-gjddv 1/1 Running 3 4d kube-system ic-nginx-ingress-controller-xbls8 1/1 Running 3 4d kube-system ic-nginx-ingress-default-backend-5ffcfd7744-gvssw 1/1 Running 1 6h11m kube-system kube-apiserver-controller-0 1/1 Running 2 2d23h kube-system kube-controller-manager-controller-0 1/1 Running 5 4d kube-system kube-multus-ds-amd64-gbz62 1/1 Running 3 4d kube-system kube-proxy-t4wgw 1/1 Running 3 4d kube-system kube-scheduler-controller-0 1/1 Running 5 4d kube-system kube-sriov-cni-ds-amd64-5bk79 1/1 Running 3 4d kube-system kube-sriov-device-plugin-amd64-g99rj 1/1 Running 2 4d kube-system oidc-dex-6585f5f9bc-vpg2w 1/1 Running 1 6h11m kube-system stx-oidc-client-6c8cfc5f65-rdv5v 1/1 Running 5 6h11m kube-system tiller-deploy-5c8dd9fb56-kplts 1/1 Running 1 6h11m monitor mon-filebeat-9nd4w 0/1 Running 2 2d4h monitor mon-kube-state-metrics-7c7f79f45c-dl7f9 1/1 Running 1 6h11m monitor mon-logstash-0 0/1 Unknown 1 2d4h monitor mon-metricbeat-bbl9p 0/1 Running 2 2d4h monitor mon-metricbeat-metrics-6d4fdf4567-ns9zx 0/1 Running 1 6h11m platform-deployment-manager platform-deployment-manager-0 2/2 Running 11 4d Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Pulled 9h (x2 over 2d4h) kubelet, controller-0 Container image "registry.local:9001/docker.elastic.co/logstash/logstash:7.6.0" already present on machine Normal Created 9h (x2 over 2d4h) kubelet, controller-0 Created container logstash Normal Started 9h (x2 over 2d4h) kubelet, controller-0 Started container logstash Warning NodeAffinity 6h15m kubelet, controller-0 Predicate NodeAffinity failed Warning FailedMount 6h15m (x2 over 6h15m) kubelet, controller-0 MountVolume.SetUp failed for volume "logstashconfig" : failed to sync configmap cache: timed out waiting for the condition Warning FailedMount 6h15m (x2 over 6h15m) kubelet, controller-0 MountVolume.SetUp failed for volume "default-token-z7n6z" : failed to sync secret cache: timed out waiting for the condition Warning FailedMount 6h15m (x2 over 6h15m) kubelet, controller-0 MountVolume.SetUp failed for volume "logstashsetup" : failed to sync configmap cache: timed out waiting for the condition Warning FailedMount 6h15m (x2 over 6h15m) kubelet, controller-0 MountVolume.SetUp failed for volume "logstashpipeline" : failed to sync configmap cache: timed out waiting for the condition Warning FailedMount 6h14m (x5 over 6h15m) kubelet, controller-0 MountVolume.SetUp failed for volume "logstashpipeline" : stat /var/lib/kubelet/pods/b0bb523c-7d9a-449e-86a0-01fc96a551c5/volumes/kubernetes.io~configmap/logstashpipeline: no such file or directory Warning FailedMount 6h8m kubelet, controller-0 Unable to attach or mount volumes: unmounted volumes=[logstashsetup logstashconfig logstashpipeline], unattached volumes=[logstashsetup logstashconfig logstashpipeline monenv default-token-z7n6z]: timed out waiting for the condition Warning FailedMount 6h4m kubelet, controller-0 (combined from similar events): Unable to attach or mount volumes: unmounted volumes=[logstashsetup logstashconfig logstashpipeline], unattached volumes=[default-token-z7n6z logstashsetup logstashconfig logstashpipeline monenv]: timed out waiting for the condition Warning FailedMount 5h29m (x3 over 5h38m) kubelet, controller-0 Unable to attach or mount volumes: unmounted volumes=[logstashconfig logstashpipeline logstashsetup], unattached volumes=[logstashconfig logstashpipeline monenv default-token-z7n6z logstashsetup]: timed out waiting for the condition Warning FailedMount 5h24m (x31 over 6h15m) kubelet, controller-0 MountVolume.SetUp failed for volume "logstashsetup" : stat /var/lib/kubelet/pods/b0bb523c-7d9a-449e-86a0-01fc96a551c5/volumes/kubernetes.io~configmap/logstashsetup: no such file or directory Warning FailedMount 5h9m (x38 over 6h15m) kubelet, controller-0 MountVolume.SetUp failed for volume "logstashconfig" : stat /var/lib/kubelet/pods/b0bb523c-7d9a-449e-86a0-01fc96a551c5/volumes/kubernetes.io~configmap/logstashconfig: no such file or directory Warning FailedMount 4h59m kubelet, controller-0 MountVolume.SetUp failed for volume "default-token-z7n6z" : failed to sync secret cache: timed out waiting for the condition Warning FailedMount 4h59m kubelet, controller-0 MountVolume.SetUp failed for volume "logstashconfig" : failed to sync configmap cache: timed out waiting for the condition Warning FailedMount 4h59m kubelet, controller-0 MountVolume.SetUp failed for volume "logstashsetup" : failed to sync configmap cache: timed out waiting for the condition Warning FailedMount 4h59m kubelet, controller-0 MountVolume.SetUp failed for volume "logstashpipeline" : failed to sync configmap cache: timed out waiting for the condition Warning FailedMount 4h59m kubelet, controller-0 MountVolume.SetUp failed for volume "logstashpipeline" : stat /var/lib/kubelet/pods/b0bb523c-7d9a-449e-86a0-01fc96a551c5/volumes/kubernetes.io~configmap/logstashpipeline: no such file or directory Warning FailedMount 4h59m (x2 over 4h59m) kubelet, controller-0 MountVolume.SetUp failed for volume "logstashconfig" : failed to sync configmap cache: timed out waiting for the condition Warning FailedMount 4h59m (x2 over 4h59m) kubelet, controller-0 MountVolume.SetUp failed for volume "logstashsetup" : failed to sync configmap cache: timed out waiting for the condition Warning FailedMount 4h59m (x2 over 4h59m) kubelet, controller-0 MountVolume.SetUp failed for volume "logstashpipeline" : failed to sync configmap cache: timed out waiting for the condition Warning FailedMount 4h59m (x2 over 4h59m) kubelet, controller-0 MountVolume.SetUp failed for volume "default-token-z7n6z" : failed to sync secret cache: timed out waiting for the condition Warning FailedMount 4h58m (x5 over 4h59m) kubelet, controller-0 MountVolume.SetUp failed for volume "logstashsetup" : stat /var/lib/kubelet/pods/b0bb523c-7d9a-449e-86a0-01fc96a551c5/volumes/kubernetes.io~configmap/logstashsetup: no such file or directory Warning FailedMount 4h58m (x6 over 4h59m) kubelet, controller-0 MountVolume.SetUp failed for volume "logstashconfig" : stat /var/lib/kubelet/pods/b0bb523c-7d9a-449e-86a0-01fc96a551c5/volumes/kubernetes.io~configmap/logstashconfig: no such file or directory Warning FailedMount 4h53m kubelet, controller-0 (combined from similar events): Unable to attach or mount volumes: unmounted volumes=[logstashconfig logstashpipeline logstashsetup], unattached volumes=[logstashconfig logstashpipeline monenv default-token-z7n6z logstashsetup]: timed out waiting for the condition Warning FailedMount 128m (x12 over 4h26m) kubelet, controller-0 Unable to attach or mount volumes: unmounted volumes=[logstashconfig logstashpipeline logstashsetup], unattached volumes=[logstashconfig logstashpipeline monenv default-token-z7n6z logstashsetup]: timed out waiting for the condition Warning FailedMount 19m (x43 over 4h57m) kubelet, controller-0 Unable to attach or mount volumes: unmounted volumes=[logstashsetup logstashconfig logstashpipeline], unattached volumes=[monenv default-token-z7n6z logstashsetup logstashconfig logstashpipeline]: timed out waiting for the condition Warning FailedMount 13m (x17 over 4h13m) kubelet, controller-0 Unable to attach or mount volumes: unmounted volumes=[logstashpipeline logstashsetup logstashconfig], unattached volumes=[logstashpipeline monenv default-token-z7n6z logstashsetup logstashconfig]: timed out waiting for the condition Warning FailedMount 8m53s (x40 over 4h55m) kubelet, controller-0 Unable to attach or mount volumes: unmounted volumes=[logstashsetup logstashconfig logstashpipeline], unattached volumes=[default-token-z7n6z logstashsetup logstashconfig logstashpipeline monenv]: timed out waiting for the condition Warning FailedMount 4m6s (x151 over 4h59m) kubelet, controller-0 MountVolume.SetUp failed for volume "logstashpipeline" : stat /var/lib/kubelet/pods/b0bb523c-7d9a-449e-86a0-01fc96a551c5/volumes/kubernetes.io~configmap/logstashpipeline: no such file or directory