snap kube-proxy missing kernel modules on kubernetes control plane on lxd when on ipvs mode

[Gustavo Sanchez]

# ------- Info
Juju agents 2.9.42
Cloud substrate MAAS
Ubuntu Focal 20.04 LTS

kuberntetes-control-plane on LXD
kuberntetes-control-plane charm 1.26/stable 247
kuberntetes-control-plane snaps 1.26.8

# ------- Juju info
$ juju config kubernetes-control-plane proxy-extra-args
proxy-mode=ipvs ipvs-strict-arp=true

$ juju status | grep -vE 'active|started'
Model Controller Cloud/Region Version SLA Timestamp
kubernetes foundations-maas maas_cloud 2.9.42 unsupported 19:02:38Z

SAAS Status Store URL

App Version Status Scale Charm Channel Rev Exposed Message
kubernetes-control-plane 1.26.8 blocked 3 kubernetes-control-plane 1.26/stable 247 no Stopped services: kube-proxy

Unit Workload Agent Machine Public address Ports Message
kubernetes-control-plane/0* blocked idle 0/lxd/3 192.168.20.167 6443/tcp Stopped services: kube-proxy
kubernetes-control-plane/2 maintenance idle 2/lxd/2 192.168.20.168 6443/tcp Restarting snap.kube-apiserver.daemon service

Machine State Address Inst id Series AZ Message

Offer Application Charm Rev Connected Endpoint Interface Role
vault-certificates vault vault 68 2/2 certificates tls-certificates provider
vault-secrets vault vault 68 1/1 secrets vault-kv provider

# ------- kube-proxy snap info
root@juju-712203-2-lxd-2:~# systemctl status snap.kube-proxy.daemon.service
● snap.kube-proxy.daemon.service - Service for snap application kube-proxy.daemon
     Loaded: loaded (/etc/systemd/system/snap.kube-proxy.daemon.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/snap.kube-proxy.daemon.service.d
             └─always-restart.conf
     Active: activating (auto-restart) (Result: exit-code) since Thu 2023-08-31 18:57:54 UTC; 7s ago
    Process: 1761822 ExecStart=/usr/bin/snap run kube-proxy.daemon (code=exited, status=1/FAILURE)
   Main PID: 1761822 (code=exited, status=1/FAILURE)

root@juju-712203-2-lxd-2:~# snap list
Name Version Rev Tracking Publisher Notes
cdk-addons 1.26.5 17193 1.26/stable canonical✓ in-cohort
core 16-2.60.2 15925 latest/stable canonical✓ core
core18 20230530 2785 latest/stable canonical✓ base
core20 20230622 1974 latest/stable canonical✓ base
kube-apiserver 1.26.8 3442 1.26/stable canonical✓ in-cohort
kube-controller-manager 1.26.8 3220 1.26/stable canonical✓ in-cohort
kube-proxy 1.26.8 3136 1.26/stable canonical✓ classic,in-cohort
kube-scheduler 1.26.8 3103 1.26/stable canonical✓ in-cohort
kubectl 1.26.8 3063 1.26/stable canonical✓ classic,in-cohort
kubelet 1.26.8 3064 1.26/stable canonical✓ classic,in-cohort
lxd 4.0.9-a29c6f1 24061 4.0/stable/… canonical✓ -
snapd 2.59.5 19457 latest/stable canonical✓ snapd

# ------- Logs
$ grep kube-proxy /var/log/syslog
# [..]
Aug 31 18:39:47 juju-712203-2-lxd-2 kube-proxy.daemon[1707148]: I0831 18:39:47.433463 1707148 proxier.go:680] "Failed to load kernel module with modprobe, you can ignore this message when kube-proxy is running inside container without mounting /lib/modules" moduleName="ip_vs_sh"
Aug 31 18:39:47 juju-712203-2-lxd-2 kube-proxy.daemon[1707148]: I0831 18:39:47.435911 1707148 proxier.go:680] "Failed to load kernel module with modprobe, you can ignore this message when kube-proxy is running inside container without mounting /lib/modules" moduleName="nf_conntrack"
Aug 31 18:39:47 juju-712203-2-lxd-2 kube-proxy.daemon[1707148]: E0831 18:39:47.436160 1707148 server.go:492] "Error running ProxyServer" err="can't use the IPVS proxier: IPVS proxier will not be used because the following required kernel modules are not loaded: [ip_vs_wrr ip_vs_sh ip_vs ip_vs_rr]"
Aug 31 18:39:47 juju-712203-2-lxd-2 kube-proxy.daemon[1707148]: E0831 18:39:47.436220 1707148 run.go:74] "command failed" err="can't use the IPVS proxier: IPVS proxier will not be used because the following required kernel modules are not loaded: [ip_vs_wrr ip_vs_sh ip_vs ip_vs_rr]"
Aug 31 18:39:47 juju-712203-2-lxd-2 systemd[1]: snap.kube-proxy.daemon.service: Main process exited, code=exited, status=1/FAILURE
Aug 31 18:39:47 juju-712203-2-lxd-2 systemd[1]: snap.kube-proxy.daemon.service: Failed with result 'exit-code'.
Aug 31 18:39:57 juju-712203-2-lxd-2 systemd[1]: snap.kube-proxy.daemon.service: Scheduled restart job, restart counter is at 546.
Aug 31 18:39:57 juju-712203-2-lxd-2 systemd[1]: Stopped Service for snap application kube-proxy.daemon.
Aug 31 18:39:57 juju-712203-2-lxd-2 systemd[1]: Started Service for snap application kube-proxy.daemon.
Aug 31 18:39:57 juju-712203-2-lxd-2 kube-proxy.daemon[1707244]: I0831 18:39:57.646742 1707244 server.go:224] "Warning, all flags other than --config, --write-config-to, and --cleanup are deprecated, please begin using a config file ASAP"
Aug 31 18:39:57 juju-712203-2-lxd-2 kube-apiserver.daemon[1698248]: I0831 18:39:57.656789 1698248 httplog.go:132] "HTTP" verb="GET" URI="/api/v1/nodes/juju-712203-2-lxd-2" latency="3.095028ms" userAgent="kube-proxy/v1.26.8 (linux/amd64) kubernetes/395f0a2" audit-ID="f7ec624e-6545-4b0b-92c2-cc561591eefd" srcIP="192.168.20.168:43492" apf_pl="global-default" apf_fs="global-default" apf_iseats=1 apf_fseats=0 apf_additionalLatency="0s" apf_execution_time="2.73572ms" resp=200
Aug 31 18:39:57 juju-712203-2-lxd-2 kube-proxy.daemon[1707244]: I0831 18:39:57.657175 1707244 node.go:163] Successfully retrieved node IP: 192.168.20.168
Aug 31 18:39:57 juju-712203-2-lxd-2 kube-proxy.daemon[1707244]: I0831 18:39:57.657222 1707244 server_others.go:109] "Detected node IP" address="192.168.20.168"
Aug 31 18:39:57 juju-712203-2-lxd-2 kube-proxy.daemon[1707244]: E0831 18:39:57.667157 1707244 proxier.go:670] "Failed to read builtin modules file, you can ignore this message when kube-proxy is running inside container without mounting /lib/modules" err="open /lib/modules/5.4.0-156-generic/modules.builtin: no such file or directory" filePath="/lib/modules/5.4.0-156-generic/modules.builtin"
Aug 31 18:39:57 juju-712203-2-lxd-2 kube-proxy.daemon[1707244]: I0831 18:39:57.669690 1707244 proxier.go:680] "Failed to load kernel module with modprobe, you can ignore this message when kube-proxy is running inside container without mounting /lib/modules" moduleName="ip_vs"
Aug 31 18:39:57 juju-712203-2-lxd-2 kube-proxy.daemon[1707244]: I0831 18:39:57.672188 1707244 proxier.go:680] "Failed to load kernel module with modprobe, you can ignore this message when kube-proxy is running inside container without mounting /lib/modules" moduleName="ip_vs_rr"
Aug 31 18:39:57 juju-712203-2-lxd-2 kube-proxy.daemon[1707244]: I0831 18:39:57.674117 1707244 proxier.go:680] "Failed to load kernel module with modprobe, you can ignore this message when kube-proxy is running inside container without mounting /lib/modules" moduleName="ip_vs_wrr"
Aug 31 18:39:57 juju-712203-2-lxd-2 kube-proxy.daemon[1707244]: I0831 18:39:57.676527 1707244 proxier.go:680] "Failed to load kernel module with modprobe, you can ignore this message when kube-proxy is running inside container without mounting /lib/modules" moduleName="ip_vs_sh"
Aug 31 18:39:57 juju-712203-2-lxd-2 kube-proxy.daemon[1707244]: I0831 18:39:57.678400 1707244 proxier.go:680] "Failed to load kernel module with modprobe, you can ignore this message when kube-proxy is running inside container without mounting /lib/modules" moduleName="nf_conntrack"
Aug 31 18:39:57 juju-712203-2-lxd-2 kube-proxy.daemon[1707244]: E0831 18:39:57.678576 1707244 server.go:492] "Error running ProxyServer" err="can't use the IPVS proxier: IPVS proxier will not be used because the following required kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh]"
Aug 31 18:39:57 juju-712203-2-lxd-2 kube-proxy.daemon[1707244]: E0831 18:39:57.678609 1707244 run.go:74] "command failed" err="can't use the IPVS proxier: IPVS proxier will not be used because the following required kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh]"
Aug 31 18:39:57 juju-712203-2-lxd-2 systemd[1]: snap.kube-proxy.daemon.service: Main process exited, code=exited, status=1/FAILURE
Aug 31 18:39:57 juju-712203-2-lxd-2 systemd[1]: snap.kube-proxy.daemon.service: Failed with result 'exit-code'.
(END)

# ----- LXD profile
root@k8s-control-03:/home/ubuntu# lxc profile list
+----------------------------------------------+---------------------+---------+
| NAME | DESCRIPTION | USED BY |
+----------------------------------------------+---------------------+---------+
| default | Default LXD profile | 6 |
+----------------------------------------------+---------------------+---------+
| juju-kubernetes-kubernetes-control-plane-247 | | 1 |
+----------------------------------------------+---------------------+---------+
root@k8s-control-03:/home/ubuntu# lxc profile show juju-kubernetes-kubernetes-control-plane-247
config:
  linux.kernel_modules: ip_tables,ip6_tables,netlink_diag,nf_nat,overlay,rbd
  raw.lxc: |
    lxc.apparmor.profile=unconfined
    lxc.mount.auto=proc:rw sys:rw
    lxc.cgroup.devices.allow=a
    lxc.cap.drop=
  security.nesting: "true"
  security.privileged: "true"
description: ""
devices:
  aadisable:
    path: /dev/kmsg
    source: /dev/kmsg
    type: unix-char
name: juju-kubernetes-kubernetes-control-plane-247
used_by:
- /1.0/instances/juju-712203-2-lxd-2

[Gustavo Sanchez (gustavosr98)]

I manually added missing kernel modules to the lxc profile
linux.kernel_modules: ip_tables,ip6_tables,netlink_diag,nf_nat,overlay,rbd,ip_vs,ip_vs_rr,ip_vs_wrr,ip_vs_sh

And restarted the LXD machine

# -----
Now it is not complaining any more about the missing required modules
But I am getting a different error

-- The job identifier is 3100.
Aug 31 19:34:15 juju-712203-0-lxd-3 kube-proxy.daemon[20359]: I0831 19:34:15.291947 20359 server.go:224] "Warning, all flags other than --config, --write-config-to, and --cleanup are deprecated, please begin using a config file ASAP"
Aug 31 19:34:15 juju-712203-0-lxd-3 kube-proxy.daemon[20359]: I0831 19:34:15.317546 20359 node.go:163] Successfully retrieved node IP: 192.168.20.167
Aug 31 19:34:15 juju-712203-0-lxd-3 kube-proxy.daemon[20359]: I0831 19:34:15.317578 20359 server_others.go:109] "Detected node IP" address="192.168.20.167"
Aug 31 19:34:15 juju-712203-0-lxd-3 kube-proxy.daemon[20359]: E0831 19:34:15.324909 20359 proxier.go:670] "Failed to read builtin modules file, you can ignore this message when kube-proxy is running inside container without mounting /lib/modules" err="open /lib/modules/5.4.0-155-generic/modules.builtin: no such file or directory" filePath="/lib/modules/5.4.0-155-generic/modules.builtin"
Aug 31 19:34:15 juju-712203-0-lxd-3 kube-proxy.daemon[20359]: I0831 19:34:15.327412 20359 proxier.go:680] "Failed to load kernel module with modprobe, you can ignore this message when kube-proxy is running inside container without mounting /lib/modules" moduleName="ip_vs"
Aug 31 19:34:15 juju-712203-0-lxd-3 kube-proxy.daemon[20359]: I0831 19:34:15.329861 20359 proxier.go:680] "Failed to load kernel module with modprobe, you can ignore this message when kube-proxy is running inside container without mounting /lib/modules" moduleName="ip_vs_rr"
Aug 31 19:34:15 juju-712203-0-lxd-3 kube-proxy.daemon[20359]: I0831 19:34:15.332300 20359 proxier.go:680] "Failed to load kernel module with modprobe, you can ignore this message when kube-proxy is running inside container without mounting /lib/modules" moduleName="ip_vs_wrr"
Aug 31 19:34:15 juju-712203-0-lxd-3 kube-proxy.daemon[20359]: I0831 19:34:15.334656 20359 proxier.go:680] "Failed to load kernel module with modprobe, you can ignore this message when kube-proxy is running inside container without mounting /lib/modules" moduleName="ip_vs_sh"
Aug 31 19:34:15 juju-712203-0-lxd-3 kube-proxy.daemon[20359]: I0831 19:34:15.337175 20359 proxier.go:680] "Failed to load kernel module with modprobe, you can ignore this message when kube-proxy is running inside container without mounting /lib/modules" moduleName="nf_conntrack"
Aug 31 19:34:15 juju-712203-0-lxd-3 kube-proxy.daemon[20359]: E0831 19:34:15.337361 20359 server.go:492] "Error running ProxyServer" err="can't use the IPVS proxier: error getting ipset version, error: executable file not found in $PATH"
Aug 31 19:34:15 juju-712203-0-lxd-3 kube-proxy.daemon[20359]: E0831 19:34:15.337387 20359 run.go:74] "command failed" err="can't use the IPVS proxier: error getting ipset version, error: executable file not found in $PATH"
Aug 31 19:34:15 juju-712203-0-lxd-3 systemd[1]: snap.kube-pr...

[Gustavo Sanchez (gustavosr98)]

This was the other missing piece
https://bugs.launchpad.net/bugs/2020059

----- So,
Apt installing ipset and conntrack on the control-plane units
As well as adding this to the lxc profile
linux.kernel_modules: <existings>, ip_vs,ip_vs_rr,ip_vs_wrr,ip_vs_sh

Fixes the issue

[Gustavo Sanchez (gustavosr98)]

Added PR https://github.com/charmed-kubernetes/charm-kubernetes-control-plane/pull/300

[George Kraft (cynerva)]

Cherry-pick to release_1.28 branch:
https://github.com/charmed-kubernetes/charm-kubernetes-control-plane/commit/138dd0180c6194fb3b8adedea3479024eb44eb58

[Adam Dyess (addyess)]

Actually, the backport has already been applied -- charm just needs to be rebuilt