juju debug log doesn't point to issue with kubelet service

Bug #1960402 reported by Natalia Nowakowska
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Kubernetes Worker Charm
New
Undecided
Unassigned

Bug Description

When snap.kubelet.daemon fails to start on the worker machines and thus don't map the nodes to charmed k8s, `juju debug-log` doesn't explicitly point to that issue. Instead, it says that the pods couldn't be scheduled due to nodes unavailability:
```
unit-kubernetes-worker-0: 10:42:44 WARNING unit.kubernetes-worker/0.update-status Error from server (NotFound): nodes "juju-401d1a-kubeflow-k8s-7" not found
unit-kubernetes-worker-2: 10:42:44 WARNING unit.kubernetes-worker/2.coordinator-relation-changed Error from server (NotFound): nodes "juju-401d1a-kubeflow-k8s-9" not found
unit-kubernetes-worker-0: 10:42:44 INFO unit.kubernetes-worker/0.juju-log Failed to apply label juju-application=kubernetes-worker. Will retry.
```

`juju status` shows the workload in ‘waiting’ state with message ‘Waiting for 4 kube-system pods to start’.
`journalctl -u snap.kubelet.daemon` on the worker nodes shows that the service couldn't be started:
```
Feb 08 10:03:36 juju-401d1a-kubeflow-k8s-9 kubelet.daemon[27797]: E0208 10:03:36.833393 27797 server.go:294] "Failed to run kubelet" err="failed to run Kubelet: failed to get docker version: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
Feb 08 10:03:36 juju-401d1a-kubeflow-k8s-9 systemd[1]: snap.kubelet.daemon.service: Main process exited, code=exited, status=1/FAILURE
```

It would be great to log at least the systemctl status output into the Juju log if it ends up needing to exit the hook with that status.

Revision history for this message
Natalia Nowakowska (natalia-nowakowska) wrote :

juju version: 2.9.22-ubuntu-amd64
charmed-k8s version: 1.21 (bundle 733)

Revision history for this message
Amanda Hager Lopes de Andrade Katz (amandahla) wrote :

I got the same message but in my case it was fixed after set no_proxy with:
"juju config containerd no_proxy=10.0.0.0/8,192.168.0.0/16,127.0.0.1,172.16.0.0/16"

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.