kubelet on lxd fails to start with Failed to start ContainerManager [invalid kernel flag: kernel/panic, expected value: 10, actual value: 0, invalid kernel flag: kernel/panic_on_oops, expected value: 1, actual value: 0, invalid kernel flag: vm/overcommit_memory, expected value: 1, actual value: 0]

[Jason Hobbs]

during a deploy on baremetal, a kubelet failed to start with this error and a lot of go tracebacks. This repeated many times:

Nov 06 10:28:17 juju-431bee-1-lxd-2 kubelet.daemon[253561]: F1106 16:28:17.063216 253561 kubelet.go:1296] Failed to start ContainerManager [invalid kernel flag: kernel/panic, expected value: 10, actual value: 0, invalid kernel flag: kernel/panic_on_oops, expected value: 1, actual value: 0, invalid kernel flag: vm/overcommit_memory, expected value: 1, actual value: 0]

example run https://solutions.qa.canonical.com/testruns/testRun/795ea282-ca6a-42bb-b5be-1d7ce15d24b3

[George Kraft (cynerva)]

This will happen any time kubernetes-worker is placed on LXD. The charm normally sets these kernel parameters as part of the charm's sysctl config[1], however on LXD, the sysctl config is simply ignored[2].

A workaround is to configure kubelet with protectKernelDefaults=false:

juju config kubernetes-worker kubelet-extra-config='{protectKernelDefaults: false}'

When you do this, Kubelet will stop protecting kernel defaults and will set the kernel.panic, kernel.panic_on_oops, and vm.overcommit_memory kernel parameters automatically.

[1]: https://github.com/charmed-kubernetes/layer-kubernetes-master-worker-base/blob/809f06c6f6521be59e21859eaebeccd13f4d8c28/config.yaml#L4
[2]: https://github.com/charmed-kubernetes/layer-kubernetes-master-worker-base/blob/809f06c6f6521be59e21859eaebeccd13f4d8c28/reactive/kubernetes_master_worker_base.py#L78

[Jason Hobbs (jason-hobbs)]

Thanks; we didn't intend to deploy k-w to lxd; that was due to a bug in fce
which we've fixed now. I guess this is still a bug but we don't expect to
hit it again.

On Mon, Nov 9, 2020 at 3:25 PM George Kraft <email address hidden>
wrote:

> This will happen any time kubernetes-worker is placed on LXD. The charm
> normally sets these kernel parameters as part of the charm's sysctl
> config[1], however on LXD, the sysctl config is simply ignored[2].
>
> A workaround is to configure kubelet with protectKernelDefaults=false:
>
> juju config kubernetes-worker kubelet-extra-
> config='{protectKernelDefaults: false}'
>
> When you do this, Kubelet will stop protecting kernel defaults and will
> set the kernel.panic, kernel.panic_on_oops, and vm.overcommit_memory
> kernel parameters automatically.
>
> [1]:
> https://github.com/charmed-kubernetes/layer-kubernetes-master-worker-base/blob/809f06c6f6521be59e21859eaebeccd13f4d8c28/config.yaml#L4
> [2]:
> https://github.com/charmed-kubernetes/layer-kubernetes-master-worker-base/blob/809f06c6f6521be59e21859eaebeccd13f4d8c28/reactive/kubernetes_master_worker_base.py#L78
>
> ** Also affects: charm-kubernetes-worker
> Importance: Undecided
> Status: New
>
> ** No longer affects: charm-kubernetes-master
>
> ** Changed in: charm-kubernetes-worker
> Importance: Undecided => High
>
> ** Changed in: charm-kubernetes-worker
> Status: New => Triaged
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1903566
>
> Title:
> kubelet on lxd fails to start with Failed to start ContainerManager
> [invalid kernel flag: kernel/panic, expected value: 10, actual value:
> 0, invalid kernel flag: kernel/panic_on_oops, expected value: 1,
> actual value: 0, invalid kernel flag: vm/overcommit_memory, expected
> value: 1, actual value: 0]
>
> Status in Kubernetes Worker Charm:
> Triaged
>
> Bug description:
> during a deploy on baremetal, a kubelet failed to start with this
> error and a lot of go tracebacks. This repeated many times:
>
> Nov 06 10:28:17 juju-431bee-1-lxd-2 kubelet.daemon[253561]: F1106
> 16:28:17.063216 253561 kubelet.go:1296] Failed to start
> ContainerManager [invalid kernel flag: kernel/panic, expected value:
> 10, actual value: 0, invalid kernel flag: kernel/panic_on_oops,
> expected value: 1, actual value: 0, invalid kernel flag:
> vm/overcommit_memory, expected value: 1, actual value: 0]
>
> example run
> https://solutions.qa.canonical.com/testruns/testRun/795ea282-ca6a-
> 42bb-b5be-1d7ce15d24b3
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/charm-kubernetes-worker/+bug/1903566/+subscriptions
>

[George Kraft (cynerva)]

PR: https://github.com/charmed-kubernetes/charm-kubernetes-worker/pull/93