# system looked healthy when this happened.
System was alarm free, all nodes were unlocked-enabled via system host-list, and all nodes were ready via kubectl get nodes -n deployment.
# rbd pod did not like controller-1, because it was tainted for some reason. App apply worked after removing the taint.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 36s (x21 over 26m) default-scheduler 0/4 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't satisfy existing pods anti-affinity rules, 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 2 node(s) didn't match node selector.
[sysadmin@controller-0 ~(keystone_admin)]$
# system looked healthy when this happened.
System was alarm free, all nodes were unlocked-enabled via system host-list, and all nodes were ready via kubectl get nodes -n deployment.
# rbd pod did not like controller-1, because it was tainted for some reason. App apply worked after removing the taint.
[sysadmin@ controller- 0 ~(keystone_admin)]$ kubectl get pods --all-namespaces -o wide | grep rbd -77bfb6dbb- k98vh 0/1 Pending 0 26m <none> <none> <none> <none> -77bfb6dbb- vm242 1/1 Running 1 26m dead:beef: :8e22:765f: 6121:eb5b controller-0 <none> <none>
kube-system rbd-provisioner
kube-system rbd-provisioner
Events: anti-affinity, 1 node(s) didn't satisfy existing pods anti-affinity rules, 1 node(s) had taint {node-role. kubernetes. io/master: }, that the pod didn't tolerate, 2 node(s) didn't match node selector. controller- 0 ~(keystone_admin)]$
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 36s (x21 over 26m) default-scheduler 0/4 nodes are available: 1 node(s) didn't match pod affinity/
[sysadmin@
[sysadmin@ controller- 0 ~(keystone_admin)]$ kubectl describe node controller-1 | grep Taints kubernetes. io/master: NoSchedule controller- 0 ~(keystone_admin)]$ kubectl describe node controller-0 | grep Taints
Taints: node-role.
[sysadmin@
Taints: <none>
# Workaround: kubernetes. io/master: NoSchedule-
kubectl taint node controller-1 node-role.
Following app apply already failed before the workaround: controller- 0 ~(keystone_admin)]$ ls -lrt /var/log/ armada/ platform- integ-apps- apply_2020- 06-2* armada/ platform- integ-apps- apply_2020- 06-20-15- 30-07.log armada/ platform- integ-apps- apply_2020- 06-20-16- 00-39.log armada/ platform- integ-apps- apply_2020- 06-20-16- 31-11.log armada/ platform- integ-apps- apply_2020- 06-20-17- 01-43.log armada/ platform- integ-apps- apply_2020- 06-20-17- 32-15.log armada/ platform- integ-apps- apply_2020- 06-21-19- 47-22.log armada/ platform- integ-apps- apply_2020- 06-21-20- 17-54.log armada/ platform- integ-apps- apply_2020- 06-21-20- 48-26.log armada/ platform- integ-apps- apply_2020- 06-21-21- 18-58.log
[sysadmin@
-rw-r--r-- 1 1000 users 21784 Jun 20 16:00 /var/log/
-rw-r--r-- 1 1000 users 21784 Jun 20 16:30 /var/log/
-rw-r--r-- 1 1000 users 21784 Jun 20 17:01 /var/log/
-rw-r--r-- 1 1000 users 21784 Jun 20 17:31 /var/log/
-rw-r--r-- 1 1000 users 21784 Jun 20 18:02 /var/log/
-rw-r--r-- 1 1000 users 21987 Jun 21 20:17 /var/log/
-rw-r--r-- 1 1000 users 22122 Jun 21 20:47 /var/log/
-rw-r--r-- 1 1000 users 22122 Jun 21 21:18 /var/log/
-rw-r--r-- 1 1000 users 9154 Jun 21 21:19 /var/log/
[sysadmin@ controller- 0 ~(keystone_admin)]$ ls -lrt /var/log/ armada/ oidc-auth- apps-apply_ 2020-* armada/ oidc-auth- apps-apply_ 2020-06- 20-20-52- 52.log
-rw-r--r-- 1 1000 users 19243 Jun 20 21:22 /var/log/