Thanks; we didn't intend to deploy k-w to lxd; that was due to a bug in fce
which we've fixed now. I guess this is still a bug but we don't expect to
hit it again.
On Mon, Nov 9, 2020 at 3:25 PM George Kraft <email address hidden>
wrote:
> This will happen any time kubernetes-worker is placed on LXD. The charm
> normally sets these kernel parameters as part of the charm's sysctl
> config[1], however on LXD, the sysctl config is simply ignored[2].
>
> A workaround is to configure kubelet with protectKernelDefaults=false:
>
> juju config kubernetes-worker kubelet-extra-
> config='{protectKernelDefaults: false}'
>
> When you do this, Kubelet will stop protecting kernel defaults and will
> set the kernel.panic, kernel.panic_on_oops, and vm.overcommit_memory
> kernel parameters automatically.
>
> [1]:
> https://github.com/charmed-kubernetes/layer-kubernetes-master-worker-base/blob/809f06c6f6521be59e21859eaebeccd13f4d8c28/config.yaml#L4
> [2]:
> https://github.com/charmed-kubernetes/layer-kubernetes-master-worker-base/blob/809f06c6f6521be59e21859eaebeccd13f4d8c28/reactive/kubernetes_master_worker_base.py#L78
>
> ** Also affects: charm-kubernetes-worker
> Importance: Undecided
> Status: New
>
> ** No longer affects: charm-kubernetes-master
>
> ** Changed in: charm-kubernetes-worker
> Importance: Undecided => High
>
> ** Changed in: charm-kubernetes-worker
> Status: New => Triaged
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1903566
>
> Title:
> kubelet on lxd fails to start with Failed to start ContainerManager
> [invalid kernel flag: kernel/panic, expected value: 10, actual value:
> 0, invalid kernel flag: kernel/panic_on_oops, expected value: 1,
> actual value: 0, invalid kernel flag: vm/overcommit_memory, expected
> value: 1, actual value: 0]
>
> Status in Kubernetes Worker Charm:
> Triaged
>
> Bug description:
> during a deploy on baremetal, a kubelet failed to start with this
> error and a lot of go tracebacks. This repeated many times:
>
> Nov 06 10:28:17 juju-431bee-1-lxd-2 kubelet.daemon[253561]: F1106
> 16:28:17.063216 253561 kubelet.go:1296] Failed to start
> ContainerManager [invalid kernel flag: kernel/panic, expected value:
> 10, actual value: 0, invalid kernel flag: kernel/panic_on_oops,
> expected value: 1, actual value: 0, invalid kernel flag:
> vm/overcommit_memory, expected value: 1, actual value: 0]
>
> example run
> https://solutions.qa.canonical.com/testruns/testRun/795ea282-ca6a-
> 42bb-b5be-1d7ce15d24b3
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/charm-kubernetes-worker/+bug/1903566/+subscriptions
>
Thanks; we didn't intend to deploy k-w to lxd; that was due to a bug in fce
which we've fixed now. I guess this is still a bug but we don't expect to
hit it again.
On Mon, Nov 9, 2020 at 3:25 PM George Kraft <email address hidden>
wrote:
> This will happen any time kubernetes-worker is placed on LXD. The charm faults= false: '{protectKernel Defaults: false}' panic_on_ oops, and vm.overcommit_ memory /github. com/charmed- kubernetes/ layer-kubernete s-master- worker- base/blob/ 809f06c6f6521be 59e21859eaebecc d13f4d8c28/ config. yaml#L4 /github. com/charmed- kubernetes/ layer-kubernete s-master- worker- base/blob/ 809f06c6f6521be 59e21859eaebecc d13f4d8c28/ reactive/ kubernetes_ master_ worker_ base.py# L78 s-worker s-master s-worker s-worker /bugs.launchpad .net/bugs/ 1903566 panic_on_ oops, expected value: 1, memory, expected daemon[ 253561] : F1106 panic_on_ oops, memory, expected value: 1, actual value: 0] /solutions. qa.canonical. com/testruns/ testRun/ 795ea282- ca6a- 1d7ce15d24b3 /bugs.launchpad .net/charm- kubernetes- worker/ +bug/1903566/ +subscriptions
> normally sets these kernel parameters as part of the charm's sysctl
> config[1], however on LXD, the sysctl config is simply ignored[2].
>
> A workaround is to configure kubelet with protectKernelDe
>
> juju config kubernetes-worker kubelet-extra-
> config=
>
> When you do this, Kubelet will stop protecting kernel defaults and will
> set the kernel.panic, kernel.
> kernel parameters automatically.
>
> [1]:
> https:/
> [2]:
> https:/
>
> ** Also affects: charm-kubernete
> Importance: Undecided
> Status: New
>
> ** No longer affects: charm-kubernete
>
> ** Changed in: charm-kubernete
> Importance: Undecided => High
>
> ** Changed in: charm-kubernete
> Status: New => Triaged
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https:/
>
> Title:
> kubelet on lxd fails to start with Failed to start ContainerManager
> [invalid kernel flag: kernel/panic, expected value: 10, actual value:
> 0, invalid kernel flag: kernel/
> actual value: 0, invalid kernel flag: vm/overcommit_
> value: 1, actual value: 0]
>
> Status in Kubernetes Worker Charm:
> Triaged
>
> Bug description:
> during a deploy on baremetal, a kubelet failed to start with this
> error and a lot of go tracebacks. This repeated many times:
>
> Nov 06 10:28:17 juju-431bee-1-lxd-2 kubelet.
> 16:28:17.063216 253561 kubelet.go:1296] Failed to start
> ContainerManager [invalid kernel flag: kernel/panic, expected value:
> 10, actual value: 0, invalid kernel flag: kernel/
> expected value: 1, actual value: 0, invalid kernel flag:
> vm/overcommit_
>
> example run
> https:/
> 42bb-b5be-
>
> To manage notifications about this bug go to:
>
> https:/
>