Comment 9 for bug 2064145

This bug, in my case, is caused by kube-proxy failing to start in kubernetes-control-plane and kubernetes-worker nodes. The ipset package is not installed and causes kube-proxy process to crash when kubernetes-control-plane and kubernetes-worker apps are configured with {mode: ipvs, ipvs: {strictARP: true}}

see https://bugs.launchpad.net/charm-calico/+bug/2045651

*****************

kubectl logs calico-node-2glrn -n kube-system -c install-cni:

2024-07-19 14:59:08.896 [INFO][1] cni-installer/<nil> <nil>: Running as a Kubernetes pod
2024-07-19 14:59:08.896 [INFO][1] cni-installer/<nil> <nil>: Installing any TLS assets
2024-07-19 14:59:08.973 [INFO][1] cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/bandwidth"
2024-07-19 14:59:08.973 [INFO][1] cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/bandwidth
2024-07-19 14:59:09.158 [INFO][1] cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/calico"
2024-07-19 14:59:09.158 [INFO][1] cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/calico
2024-07-19 14:59:09.362 [INFO][1] cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/calico-ipam"
2024-07-19 14:59:09.362 [INFO][1] cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/calico-ipam
2024-07-19 14:59:09.367 [INFO][1] cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/flannel"
2024-07-19 14:59:09.367 [INFO][1] cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/flannel
2024-07-19 14:59:09.372 [INFO][1] cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/host-local"
2024-07-19 14:59:09.372 [INFO][1] cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/host-local
2024-07-19 14:59:09.378 [INFO][1] cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/loopback"
2024-07-19 14:59:09.378 [INFO][1] cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/loopback
2024-07-19 14:59:09.385 [INFO][1] cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/portmap"
2024-07-19 14:59:09.385 [INFO][1] cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/portmap
2024-07-19 14:59:09.391 [INFO][1] cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/tuning"
2024-07-19 14:59:09.391 [INFO][1] cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/tuning
2024-07-19 14:59:09.391 [INFO][1] cni-installer/<nil> <nil>: Wrote Calico CNI binaries to /host/opt/cni/bin

2024-07-19 14:59:09.494 [INFO][1] cni-installer/<nil> <nil>: CNI plugin version: v3.27.3
2024-07-19 14:59:09.494 [INFO][1] cni-installer/<nil> <nil>: /host/secondary-bin-dir is not writeable, skipping
2024-07-19 14:59:09.494 [WARNING][1] cni-installer/<nil> <nil>: Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
2024-07-19 14:59:39.522 [ERROR][1] cni-installer/<nil> <nil>: Unable to create token for CNI kubeconfig error=Post "https://10.152.183.1:443/api/v1/namespaces/kube-system/serviceaccounts/calico-cni-plugin/token": dial tcp 10.152.183.1:443: i/o timeout
2024-07-19 14:59:39.522 [FATAL][1] cni-installer/<nil> <nil>: Unable to create token for CNI kubeconfig error=Post "https://10.152.183.1:443/api/v1/namespaces/kube-system/serviceaccounts/calico-cni-plugin/token": dial tcp 10.152.183.1:443: i/o timeout

*****************

journalctl:

Jul 20 06:50:06 game-ewe systemd[1]: Started Service for snap application kube-proxy.daemon.
Jul 20 06:50:06 game-ewe kube-apiserver.daemon[105450]: I0720 06:50:06.995084 105450 httplog.go:132] "HTTP" verb="GET" URI="/api/v1/nodes/game-ewe" latency="4.58224ms" userAgent="kube-proxy/v1.30.3 (linux/amd64) kubernetes/6fc0a69" audit-ID="601b0ed6-a8ac-4f4a-ba47-9da13b1a86ee" srcIP="127.0.0.1:54318" apf_pl="global-default" apf_fs="global-default" apf_iseats=1 apf_fseats=0 apf_additionalLatency="0s" apf_execution_time="3.85055ms" resp=200
Jul 20 06:50:06 game-ewe kube-proxy.daemon[156558]: I0720 06:50:06.996654 156558 server.go:1062] "Successfully retrieved node IP(s)" IPs=["192.168.2.49"]
Jul 20 06:50:07 game-ewe kube-proxy.daemon[156558]: I0720 06:50:07.000981 156558 conntrack.go:59] "Setting nf_conntrack_max" nfConntrackMax=131072
Jul 20 06:50:07 game-ewe kube-proxy.daemon[156558]: I0720 06:50:07.007944 156558 server.go:659] "kube-proxy running in dual-stack mode" primary ipFamily="IPv4"
Jul 20 06:50:07 game-ewe kube-proxy.daemon[156558]: E0720 06:50:07.011189 156558 server.go:558] "Error running ProxyServer" err="can't use the IPVS proxier: error getting ipset version, error: executable file not found in $PATH"
Jul 20 06:50:07 game-ewe kube-proxy.daemon[156558]: E0720 06:50:07.011230 156558 run.go:74] "command failed" err="can't use the IPVS proxier: error getting ipset version, error: executable file not found in $PATH"
Jul 20 06:50:07 game-ewe systemd[1]: snap.kube-proxy.daemon.service: Main process exited, code=exited, status=1/FAILURE
Jul 20 06:50:07 game-ewe systemd[1]: snap.kube-proxy.daemon.service: Failed with result 'exit-code'.
Jul 20 06:50:07 game-ewe kube-apiserver.daemon[105450]: I0720 06:50:07.910227 105450 httplog.go:132] "HTTP" verb="GET" URI="/api/v1/nodes/brief-skink" latency="7.769693ms" userAgent="kube-proxy/v1.30.3 (linux/amd64) kubernetes/6fc0a69" audit-ID="5abb9492-b891-49bf-8043-beaa3bd46696" srcIP="192.168.2.26:37758" apf_pl="global-default" apf_fs="global-default" apf_iseats=1 apf_fseats=0 apf_additionalLatency="0s" apf_execution_time="7.058536ms" resp=200