I believe we are having problems with pods not running like LP "https://bugs.launchpad.net/starlingx/+bug/1817723" but this time in 2+2+2 cluster. [wrsroot@controller-0 ~(keystone_admin)]$ kubectl get pods --all-namespaces | grep 0/1 kube-system calico-node-5kw4w 0/1 Init:0/2 0 6h4m kube-system calico-node-whrbc 0/1 Init:0/2 0 6h4m kube-system ceph-pools-audit-1558479000-zcmmr 0/1 Completed 0 11m kube-system ceph-pools-audit-1558479300-dxj65 0/1 Completed 0 6m10s kube-system ceph-pools-audit-1558479600-cm9rs 0/1 Completed 0 69s kube-system kube-multus-ds-amd64-rglr7 0/1 ContainerCreating 0 6h4m kube-system kube-multus-ds-amd64-wsqvv 0/1 ContainerCreating 0 6h4m kube-system kube-proxy-kh8rs 0/1 ContainerCreating 0 6h4m kube-system kube-proxy-pcxgc 0/1 ContainerCreating 0 6h4m openstack osh-openstack-garbd-garbd-85564795f6-9sj6v 0/1 Pending 0 5h56m [wrsroot@controller-0 ~(keystone_admin)]$ kubectl get pods -n openstack NAME READY STATUS RESTARTS AGE ingress-7bf7c8458f-dtr7t 1/1 Running 0 5h59m ingress-7bf7c8458f-grw8h 1/1 Running 0 5h59m ingress-error-pages-cf8cf7ccd-lw8sr 1/1 Running 0 5h59m ingress-error-pages-cf8cf7ccd-s6hfg 1/1 Running 0 5h59m mariadb-ingress-66c7f9964b-d4t9p 1/1 Running 0 5h59m mariadb-ingress-66c7f9964b-jnhws 1/1 Running 0 5h59m mariadb-ingress-error-pages-749cf64f44-dcr5h 1/1 Running 0 5h59m mariadb-server-0 1/1 Running 0 5h59m mariadb-server-1 1/1 Running 0 5h59m osh-openstack-garbd-garbd-85564795f6-9sj6v 0/1 Pending 0 5h57m [wrsroot@controller-0 ~(keystone_admin)]$ kubectl describe pod osh-openstack-garbd-garbd-85564795f6-9sj6v -n openstack Name: osh-openstack-garbd-garbd-85564795f6-9sj6v Namespace: openstack Priority: 0 PriorityClassName: Node: Labels: application=garbd component=server pod-template-hash=85564795f6 release_group=osh-openstack-garbd Annotations: configmap-bin-hash: e9eadebcdb0d47224b47dfa165edeca8ac773b0939b1672ee5d4a438c2341f5a Status: Pending IP: Controlled By: ReplicaSet/osh-openstack-garbd-garbd-85564795f6 Init Containers: init: Image: 10.10.58.2:9001/quay.io/stackanetes/kubernetes-entrypoint:v0.3.1 Port: Host Port: Command: kubernetes-entrypoint Environment: POD_NAME: osh-openstack-garbd-garbd-85564795f6-9sj6v (v1:metadata.name) NAMESPACE: openstack (v1:metadata.namespace) INTERFACE_NAME: eth0 PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/ DEPENDENCY_SERVICE: DEPENDENCY_DAEMONSET: DEPENDENCY_CONTAINER: DEPENDENCY_POD_JSON: COMMAND: echo done Mounts: /var/run/secrets/kubernetes.io/serviceaccount from osh-openstack-garbd-garbd-token-9mscp (ro) Containers: garbd: Image: 10.10.58.2:9001/docker.io/starlingx/stx-mariadb:master-centos-stable-20190520T233000Z.0 Port: Host Port: Command: /tmp/garbd.sh Environment: GROUP_NAME: mariadb-server_openstack GROUP_ADDRESS: gcomm://mariadb-server-0.mariadb-discovery.openstack.svc.cluster.local,mariadb-server-1.mariadb-discovery.openstack.svc.cluster.local Mounts: /tmp/garbd.sh from garbd-bin (ro) /var/run/secrets/kubernetes.io/serviceaccount from osh-openstack-garbd-garbd-token-9mscp (ro) Conditions: Type Status PodScheduled False Volumes: garbd-bin: Type: ConfigMap (a volume populated by a ConfigMap) Name: osh-openstack-garbd-garbd-bin Optional: false osh-openstack-garbd-garbd-token-9mscp: Type: Secret (a volume populated by a Secret) SecretName: osh-openstack-garbd-garbd-token-9mscp Optional: false QoS Class: BestEffort Node-Selectors: openstack-compute-node=enabled Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 29s (x367 over 6h) default-scheduler 0/4 nodes are available: 2 node(s) didn't match node selector, 2 node(s) had taints that the pod didn't tolerate. [wrsroot@controller-0 ~(keystone_admin)]$ kubectl describe pod osh-openstack-garbd-garbd-85564795f6-9sj6v -n openstack Using 20190521T132734Z in the BM 2+2+2. ### ### StarlingX ### Built from master ### OS="centos" SW_VERSION="19.01" BUILD_TARGET="Host Installer" BUILD_TYPE="Formal" BUILD_ID="20190521T132734Z" JOB="STX_build_master_master"