Kubernetes-control-plane status stuck waiting

[David Gomez]

When deploying the charmed kubernetes bundle, kubernetes-control-plane app 1.24.3 stuck in waiting status for over 30+ minutes.

Deployed with "juju deploy charmed-kubernetes --overlay ./openstack-overlay.yaml --trust"

openstack-overlay.yaml:

description: Charmed Kubernetes overlay to add native OpenStack support.
applications:
  openstack-integrator:
    annotations:
      gui-x: "600"
      gui-y: "300"
    charm: openstack-integrator
    num_units: 1
    trust: true
relations:
  - ['openstack-integrator', 'kubernetes-control-plane:openstack']
  - ['openstack-integrator', 'kubernetes-worker:openstack']

See juju-crashdump tarball attached

Workaround:
Switching from calico to flannel in the deployment seems to get around this issue

[George Kraft (cynerva)]

From the crashdump, Calico CNI is getting a Forbidden response trying to talk to the Kubernetes API. From kubelet:

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e61f7be828eb2f62bfef34c9388cb
592311becf8a8eb84fe2fd6594ab1217228": Get "https://10.5.0.157:6443/api/v1/namespaces/ingress-nginx-kubernetes-worker": Forbidden

It looks like your containerd charm is configured with:

http_proxy: http://squid.internal:3128
https_proxy: http://squid.internal:3128
no_proxy: <blank>

The Calico CNI plugin inherits HTTP proxy variables from containerd. When it tries to reach the Kubernetes API at 10.5.0.157, the request gets sent to the HTTP proxy, which refuses to proxy the request and replies with 403 Forbidden.

I recommend setting the containerd charm's no_proxy config to:

127.0.0.1,localhost,::1,10.5.0.0/16

to ensure that traffic to the 10.5.0.0/16 network does not go through the HTTP proxy. Can you give that a try and let us know if it works?

[David Gomez (dvgomez)]

Hey George, that seemed to do the trick. Juju status now shows calico and kubernetes-control-plane are all green now(active).