Waiting for 3 kube-system pods to start

[Marian Gasparovic]

All three kubernetes-master units are in waiting state, reporting `Waiting for 3 kube-system pods to start`

log shows
juju-log Checking system pods status: calico-kube-controllers-7b68f77f4c-gpxkk=Running, coredns-7bb4d77796-9c56s=Pending, kube-state-metrics-6f586bb967-mln76=Pending, metrics-server-v0.3.6-7d66499544-2d7mp=Pending

versions
kubernetes-master 1.20.2 waiting 3 kubernetes-master jujucharms 926 ubuntu

logs and configs

https://oil-jenkins.canonical.com/artifacts/4ef7365e-c958-4175-a0da-04ddcba1f826/index.html

[Marian Gasparovic (marosg)]

sorry, wrong link

https://oil-jenkins.canonical.com/artifacts/4608517c-820c-4611-9811-0db10bbcbb27/index.html

[George Kraft (cynerva)]

The calico CNI plugin is getting a 403 Forbidden response when trying to talk to kube-apiserver. From kubelet logs:

failed to setup network for sandbox "d879438af431291739566cd9d47f357b62fdc9c04dd49e7bc1cb7f9b4aa58477": Get https://10.246.64.82:6443/api/v1/namespaces/ingress-nginx-kubernetes-worker: Forbidden

10.246.64.82 is the kubernetes-master VIP. Looking in overlay_kubernetes_options.yaml, you have containerd configured with:

containerd:
  options:
    http_proxy: http://squid.internal:3128/
    https_proxy: http://squid.internal:3128/
    no_proxy: 10.244.0.0/15,192.168.0.0/16,172.16.0.0/12

10.246.64.82 does not fall within any of the no_proxy CIDRs, so the traffic is getting proxied, and the proxy is returning 403 Forbidden. You need to fix your containerd no_proxy or your kubernetes-master VIP, whichever is wrong.